Shrink ASM Diskgroup and Exadata Grid Disks
Here I will cover the shrink of ASM diskgroup in Exadata environment running VM’s. The process here is the opposite of what I wrote in the previous post, but have a tricky part that demands attention to avoid errors. The same points that you checked for extending are valid now: number the cells, disks per cell, ASM mirroring, and the VM that you want to change continue to be important, but we have more now. Besides that, the post shows how to verify if you have something in the ASM internal extent map that can block the shrink and “fix” this.
Here, in this scenario, I will reduce the size for grid disks linked with diskgroup DATAC8 (that run in a cluster for VM #08).
And before continuing, be aware of Exadata disk division:
ASM Extent Map and Moves
So, my actual usage space for DATAC8 is:
ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED NORMAL N 512 512 4096 4194304 29921280 29790140 498688 14645726 0 Y DATAC8/
MOUNTED NORMAL N 512 512 4096 4194304 6144000 6142020 102400 3019810 0 N RECOC8/
ASMCMD>
SQL> select name, total_mb, free_mb, total_mb - free_mb used_mb, round(100*free_mb/total_mb,2) pct_free
2 from v$asm_diskgroup
3 order by 1;
NAME TOTAL_MB FREE_MB USED_MB PCT_FREE
------------------------------ ---------- ---------- ---------- ----------
DATAC8 29921280 29790140 131140 99.56
RECOC8 6144000 6142020 1980 99.97
SQL>
As you can see above, the RAW size is around 29.2 TB (14.6 after the ASM NORMAL mirror) and the actual free space is around 14.3TB. So, plenty of space for the reduction in this case. Unfortunately, even if I have 99% of free space don’t mean that I can reduce instantly to 1TB.
This occurs because, probably, I have a fragmented diskgroup and something can be allocated at the end of some disk and will deny me do reduce the disk. I will not dig in this topic but will show you how to check this. If you want to check with more details you can read these two articles: http://asmsupportguy.blogspot.com/2011/06/asm-file-extent-map.html and http://asmsupportguy.blogspot.com/2012/10/where-is-my-data.html.
The idea is to use the ASM extend map from X$KFFXP to discover the object that has the MAX AU_KFFXP into some disk (whatever which one). So, I made:
SQL> select GROUP_NUMBER, name from v$asm_diskgroup;
GROUP_NUMBER NAME
------------ ------------------------------
1 DATAC8
2 RECOC8
SQL>
SQL> select VALUE from V$ASM_ATTRIBUTE where NAME='au_size' and GROUP_NUMBER=1;
VALUE
--------------------------------------------------------------------------------
4194304
SQL>
SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;
MAX(AU_KFFXP)
-------------
114195
SQL>
SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 114195;
NUMBER_KFFXP
------------
262
SQL>
SQL> select name from v$asm_alias where FILE_NUMBER = 262 and GROUP_NUMBER = 1;
NAME
----------------------------------------------------------------------
group_1.262.983716383
SQL>
Above I discovered that the file 252 (that it is a redo – will show after), is allocated in the allocation unit 114195 from diskgroup 1 (that have – and default fore Exadata – allocation unit of 4MB). Doing a little count (114195*4), this means that this file resides in somewhere around 446GB into some disk. And if I try to reduce the disk below this value, I will receive an error.
To solve this we need to move files, in this case, the file is related to MGMTDB:
ASMCMD> ls -l +DATAC8/_MGMTDB/ONLINELOG/
Type Redund Striped Time Sys Name
ONLINELOG MIRROR COARSE MAY 03 10:00:00 Y group_1.262.983716383
ONLINELOG MIRROR COARSE MAY 03 10:00:00 Y group_2.264.983716383
ONLINELOG MIRROR COARSE MAY 03 10:00:00 Y group_3.263.983716383
ASMCMD>
ASMCMD> ls -l +DATAC8
Type Redund Striped Time Sys Name
Y ASM/
Y _MGMTDB/
Y exa-cl8/
PASSWORD HIGH COARSE JAN 20 2017 N orapwasm => +DATAC8/ASM/PASSWORD/pwdasm.256.933784119
PASSWORD HIGH COARSE AUG 09 2018 N orapwasm_backup => +DATAC8/ASM/PASSWORD/pwdasm.1471.983713235
ASMCMD> ls -l +RECOC8
ASMCMD>
To move MGMTDB we can follow the steps from note 2065175.1 and use the script mdbutil.pl. In this case, I moved to RECOC8:
[grid@exa01vm08 +ASM1]$ /tmp/MGMTDB/mdbutil.pl --mvmgmtdb --target=+RECOC8 -debug
mdbutil.pl version : 1.98
2019-06-21 11:43:33: D Executing: /u01/app/18.0.0/grid/bin/srvctl status diskgroup -g RECOC8
2019-06-21 11:43:34: D Exit code: 0
2019-06-21 11:43:34: D Output of last command execution:
Disk Group RECOC8 is running on exa01vm08
2019-06-21 11:43:34: D Executing: /u01/app/18.0.0/grid/bin/srvctl status mgmtdb
2019-06-21 11:43:35: D Exit code: 0
2019-06-21 11:43:35: D Output of last command execution:
…
…
2019-06-21 11:53:20: D Executing: /u01/app/18.0.0/grid/bin/crsctl query crs activeversion
2019-06-21 11:53:20: D Exit code: 0
2019-06-21 11:53:20: D Output of last command execution:
Oracle Clusterware active version on the cluster is [18.0.0.0.0]
2019-06-21 11:53:20: I Starting the Cluster Health Analysis Resource
2019-06-21 11:53:20: D Executing: /u01/app/18.0.0/grid/bin/srvctl start cha
2019-06-21 11:53:22: D Exit code: 0
2019-06-21 11:53:22: D Output of last command execution:
2019-06-21 11:53:22: I MGMTDB Successfully moved to +RECOC8!
[grid@exa01vm08 +ASM1]$
The output above was cropped and you can see in raw here. And after that, doing the same query as above, I checked again the extent map:
SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;
MAX(AU_KFFXP)
-------------
114132
SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 114132;
NUMBER_KFFXP
------------
255
SQL> select name from v$asm_alias where FILE_NUMBER = 255 and GROUP_NUMBER = 1;
NAME
----------------------------------------------------------------------
REGISTRY.255.933784121
SQL>
So, more one file to move and not it is related to OCR. To move I made:
[root@exa01vm08 ~]# export ORACLE_HOME=/u01/app/18.0.0/grid
[root@exa01vm08 ~]# export PATH=$ORACLE_HOME/bin:$PATH
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# ocrconfig -add +RECOC8
[root@exa01vm08 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 491684
Used space (kbytes) : 88624
Available space (kbytes) : 403060
ID : 354072626
Device/File Name : +DATAC8
Device/File integrity check succeeded
Device/File Name : +RECOC8
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# ocrconfig -delete +DATAC8
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 491684
Used space (kbytes) : 88624
Available space (kbytes) : 403060
ID : 354072626
Device/File Name : +RECOC8
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@exa01vm08 ~]#
Above I: added a new place to OCR store the OCR files(RECOC8) and deleted the other (DATAC8). So, we still have more to move (votedisks, older OCR backups and ASM password file). To move votedisks I made:
[root@exa01vm08 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
ONLINE 077367b255804f0abf804a3a3ca8045d (o/200.200.10.11;200.200.10.12/DATAC8_CD_02_exaceladm04) [DATAC8]
ONLINE 59ef748a7d2a4f43bffb54a90fe1b1a9 (o/200.200.10.13;200.200.10.14/DATAC8_CD_02_exaceladm05) [DATAC8]
ONLINE cbaad30809f94fdcbfe5d60f6529ba63 (o/200.200.10.7;200.200.10.8/DATAC8_CD_02_exaceladm02) [DATAC8]
Located 3 voting disk(s).
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# crsctl replace votedisk +RECOC8
Successful addition of voting disk 081e9e767bc44ff2bff6067229378db5.
Successful addition of voting disk 82649e3309d34fa0bf4fd3c89c93e42f.
Successful addition of voting disk 57f3ec44b44b4fcdbf35e716e13011e9.
Successful deletion of voting disk 077367b255804f0abf804a3a3ca8045d.
Successful deletion of voting disk 59ef748a7d2a4f43bffb54a90fe1b1a9.
Successful deletion of voting disk cbaad30809f94fdcbfe5d60f6529ba63.
Successfully replaced voting disk group with +RECOC8.
CRS-4266: Voting file(s) successfully replaced
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
ONLINE 081e9e767bc44ff2bff6067229378db5 (o/200.200.10.11;200.200.10.12/RECOC8_CD_02_exaceladm04) [RECOC8]
ONLINE 82649e3309d34fa0bf4fd3c89c93e42f (o/200.200.10.9;200.200.10.10/RECOC8_CD_02_exaceladm03) [RECOC8]
ONLINE 57f3ec44b44b4fcdbf35e716e13011e9 (o/200.200.10.13;200.200.10.14/RECOC8_CD_03_exaceladm05) [RECOC8]
Located 3 voting disk(s).
[root@exa01vm08 ~]#
And to backups for OCR:
[root@exa01vm08 ~]# ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+DATAC8].
[root@exa01vm08 ~]# ocrconfig -backuploc +RECOC8
[root@exa01vm08 ~]#
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+RECOC8].
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# ocrconfig -manualbackup
exa01vm08 2019/06/21 12:08:05 +RECOC8:/exa-cl8/OCRBACKUP/backup_20190621_120805.ocr.4722.1011528485 671056737
exa01vm08 2018/08/16 12:38:11 +DATAC8:/exa-cl8/OCRBACKUP/backup_20180816_123811.ocr.1511.984314291 671056737
exa01vm08 2018/08/09 15:56:08 +DATAC8:/exa-cl8/OCRBACKUP/backup_20180809_155608.ocr.1508.983721369 2960767134
[root@exa01vm08 ~]#
After one more backup, I deleted manually the old OCR backups. And to move ASM password:
[grid@exa01vm08 +ASM1]$ asmcmd pwget --asm
+DATAC8/orapwASM
[grid@exa01vm08 +ASM1]$ srvctl config asm -detail
ASM home: <CRS home>
Password file: +DATAC8/orapwASM
Backup of Password file: +DATAC8/orapwASM_backup
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM
[grid@exa01vm08 +ASM1]$
[grid@exa01vm08 +ASM1]$
[grid@exa01vm08 +ASM1]$ asmcmd pwmove --asm +DATAC8/orapwASM +RECOC8/orapwASM -f
moving +DATAC8/orapwASM -> +RECOC8/orapwASM
[grid@exa01vm08 +ASM1]$
[grid@exa01vm08 +ASM1]$ srvctl config asm -detail
ASM home: <CRS home>
Password file: +RECOC8/orapwASM
Backup of Password file: +DATAC8/orapwASM_backup
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM
[grid@exa01vm08 +ASM1]$
[grid@exa01vm08 +ASM1]$ asmcmd pwmove --asm +DATAC8/orapwASM_backup +RECOC8/orapwASM_backup -f
moving +DATAC8/orapwASM_backup -> +RECOC8/orapwASM_backup
[grid@exa01vm08 +ASM1]$
[grid@exa01vm08 +ASM1]$
Maybe you don’t need to move everything, I recommend that after every move you check the extent map for ASM and verify if the value less the minimum size per disk that you want.
Since I changed a lot of things I made a restart for the cluster in both nodes:
[root@exa01vm08 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'exa01vm08'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'exa01vm08'
…
CRS-2673: Attempting to stop 'ora.diskmon' on 'exa01vm08'
CRS-2677: Stop of 'ora.diskmon' on 'exa01vm08' succeeded
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exa01vm08'
…
…
CRS-2672: Attempting to start 'ora.crsd' on 'exa01vm08'
CRS-2676: Start of 'ora.crsd' on 'exa01vm08' succeeded
[root@exa01vm08 ~]#
And after all of these moves I got the extent map with:
SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;
MAX(AU_KFFXP)
-------------
15149
SQL>
SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 15149;
NUMBER_KFFXP
------------
1
SQL>
Basically, this means that something is around 60Gb of some disk. But the link is for a file #1. So, to check what is this you can query inside X$KFFXP:
SQL> select NUMBER_KFFXP "ASM file number",
2 DECODE (NUMBER_KFFXP,
3 1, 'File directory'
4 , 2, 'Disk directory'
5 , 3, 'Active change directory'
6 , 4, 'Continuing operations directory'
7 , 5, 'Template directory'
8 , 6, 'Alias directory'
9 , 7, 'ADVM file directory'
10 , 8, 'Disk free space directory'
11 , 9, 'Attributes directory'
12 , 10, 'ASM User directory'
13 , 11, 'ASM user group directory'
14 , 12, 'Staleness directory'
15 , 253, 'spfile for ASM instance'
16 , 254, 'Stale bit map space registry '
17 , 255, 'Oracle Cluster Repository registry') "ASM metadata file name",
18 count(AU_KFFXP) "Allocation units"
19 from X$KFFXP
20 where GROUP_KFFXP=1
21 group by NUMBER_KFFXP
22 order by 1;
ASM file number ASM metadata file name Allocation units
--------------- ---------------------------------- ----------------
1 File directory 15
2 Disk directory 3
3 Active change directory 69
4 Continuing operations directory 6
5 Template directory 3
6 Alias directory 3
8 Disk free space directory 3
9 Attributes directory 3
12 Staleness directory 3
13 3
16 3
ASM file number ASM metadata file name Allocation units
--------------- ---------------------------------- ----------------
120 3
121 180
253 spfile for ASM instance 2
254 Stale bit map space registry 18
255 Oracle Cluster Repository registry 83
262 99
321 185
428 221
492 305
551 129
584 705
ASM file number ASM metadata file name Allocation units
--------------- ---------------------------------- ----------------
740 1865
1003 105
1417 2
4708 6
4710 6
4713 6
4716 6
4719 6
4722 6
4731 52
4733 52
ASM file number ASM metadata file name Allocation units
--------------- ---------------------------------- ----------------
4736 52
4739 379
4742 379
4744 52
4747 379
4750 24
4753 129
4756 5125
4759 7685
4762 1029
4765 517
ASM file number ASM metadata file name Allocation units
--------------- ---------------------------------- ----------------
4768 52
4771 4
4773 11819
47 rows selected.
SQL>
In this case, it is a file directory ASM, but I will not try to move because it is even impossible in my case because does not appears in ASMCMD command. Basically, can be internal file/directory that we can’t see. Since I know that the diskgroup is empty and the allocation is related to internal ASM, I continued.
Remember at the end of the procedure, after shrinking in grid disk (that I will show later) to move again these files to original diskgroup. I will not cover this new move here in the post but the steps are the same that I showed before.
Shrink in ASM
The shrinking part start in ASM it is simple and you just need to define the value that you want per disk and reduce them. The most critical part here is to define one size for the disk that it is aligned with 16 MB of the grid disk. Different from when you are increasing the space, here, for shrink, you start to define the size in ASM side and after go to grid disk. And since the grid disk in storage cell round down the value to the nearest 16MB value, you can misalign and have ASM disk with a different value than grid disk (and this will be REALLY BAD). A trick secret is to think directly in GB to avoid the error.
For 16 Mb explanation you can check in the Exadata docs https://docs.oracle.com/en/engineered-systems/exadata-database-machine/sagug/exadata-administering-asm.html#GUID-42DA2512-667D-443C-93C5-6E5110DFAE21:
Find the closest 16 MB boundary for the new grid disk size. If you do not perform this check, then the cell will round down the grid disk size to the nearest 16 MB boundary automatically, and you could end up with a mismatch in size between the Oracle ASM disks and the grid disks.
Since I defined to reduce the DATAC8 to around 1.8TB I defined the disks with 30GB:
SQL> select ((30*12)*5) as sizeGB FROM dual;
SIZEGB
----------
1800
SQL>
Remember in the beginning that you need to know your environment? Here I have 12 disks per cell and 5 cells:
SQL> set linesize 250
SQL> select dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status, count(1) num_disks
2 from v$asm_disk d, v$asm_diskgroup dg
3 where d.group_number = dg.group_number
4 and dg.name IN ('DATAC8')
5 group by dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status
6 order by 1,2,3;
NAME FAILGROUP STATE HEADER_STATU MOUNT_S MODE_ST NUM_DISKS
------ ------------------------------ -------- ------------ ------- ------- ----------
DATAC8 EXACELADM01 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELADM02 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELADM03 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELADM04 NORMAL MEMBER CACHED ONLINE 12
DATAC8 EXACELADM05 NORMAL MEMBER CACHED ONLINE 12
SQL>
After choosing the size that you want, you need to check if all the grid disk as online. It is not recommended to do the shrink with missing disks or even faulty failgroups. To check both at the same time you can do:
SQL> col path format a100
SQL> select dg.name, d.failgroup, d.path
2 from v$asm_disk d, v$asm_diskgroup dg
3 where d.group_number = dg.group_number
4 and dg.name IN ('DATAC8') and state = 'NORMAL'
5 order by 1,2,3;
NAME FAILGROUP PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_00_exaceladm01
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_01_exaceladm01
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_02_exaceladm01
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_03_exaceladm01
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_04_exaceladm01
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_05_exaceladm01
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_06_exaceladm01
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_07_exaceladm01
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_08_exaceladm01
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_09_exaceladm01
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_10_exaceladm01
NAME FAILGROUP PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8 EXACELADM01 o/200.200.10.5;200.200.10.6/DATAC8_CD_11_exaceladm01
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_00_exaceladm02
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_01_exaceladm02
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_02_exaceladm02
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_03_exaceladm02
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_04_exaceladm02
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_05_exaceladm02
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_06_exaceladm02
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_07_exaceladm02
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_08_exaceladm02
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_09_exaceladm02
NAME FAILGROUP PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_10_exaceladm02
DATAC8 EXACELADM02 o/200.200.10.7;200.200.10.8/DATAC8_CD_11_exaceladm02
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_00_exaceladm03
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_01_exaceladm03
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_02_exaceladm03
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_03_exaceladm03
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_04_exaceladm03
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_05_exaceladm03
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_06_exaceladm03
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_07_exaceladm03
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_08_exaceladm03
NAME FAILGROUP PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_09_exaceladm03
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_10_exaceladm03
DATAC8 EXACELADM03 o/200.200.10.9;200.200.10.10/DATAC8_CD_11_exaceladm03
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_00_exaceladm04
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_01_exaceladm04
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_02_exaceladm04
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_03_exaceladm04
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_04_exaceladm04
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_05_exaceladm04
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_06_exaceladm04
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_07_exaceladm04
NAME FAILGROUP PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_08_exaceladm04
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_09_exaceladm04
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_10_exaceladm04
DATAC8 EXACELADM04 o/200.200.10.11;200.200.10.12/DATAC8_CD_11_exaceladm04
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_00_exaceladm05
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_01_exaceladm05
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_02_exaceladm05
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_03_exaceladm05
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_04_exaceladm05
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_05_exaceladm05
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_06_exaceladm05
NAME FAILGROUP PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_07_exaceladm05
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_08_exaceladm05
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_09_exaceladm05
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_10_exaceladm05
DATAC8 EXACELADM05 o/200.200.10.13;200.200.10.14/DATAC8_CD_11_exaceladm05
60 rows selected.
SQL>
After all the checks 16MB align, missing disks, and online failgroups you can do the resize in ASM:
SQL> alter diskgroup DATAC8 resize all size 30720M rebalance power 1024;
Diskgroup altered.
As you can see above, I specified in MB and I made this to show you where the error can occurs. If I defined (as example) disk size as 30700M it will not be aligned to 16MB (30700/16 = 1918,75) and the griddisk will be 30688M and if ASM allocate something in the end of the disk you will corrupt something.
After executing the change in ASM you just continue after having nothing in v$asm_operation. And for ASM now you see the size for your diskgroup:
SQL> select name, total_mb, free_mb, total_mb - free_mb used_mb, round(100*free_mb/total_mb,2) pct_free
2 from v$asm_diskgroup
3 order by 1;
NAME TOTAL_MB FREE_MB USED_MB PCT_FREE
------------------------------ ---------- ---------- ---------- ----------
DATAC8 1843200 1837896 5304 99.71
RECOC8 6144000 6016048 127952 97.92
SQL>
Shrink for GRID DISK
After shrinking in ASM side, you need to reduce in storage side the grid disk to release the space to celldisk. The procedure is the same for the increase and you use ALTER GRIDDISK to specify the new value.
Just to show, that before the shrink in storage cell I have disk defined for grid disk (487GB) and celldisk (346GB free):
CellCLI> list griddisk where name = 'DATAC8_CD_00_exaceladm01' detail;
name: DATAC8_CD_00_exaceladm01
asmDiskGroupName: DATAC8
asmDiskName: DATAC8_CD_00_EXACELADM01
asmFailGroupName: EXACELADM01
availableTo:
cachedBy: FD_00_exaceladm01
cachingPolicy: default
cellDisk: CD_00_exaceladm01
comment: "Cluster exa-cl8 diskgroup DATAC8"
creationTime: 2017-01-20T16:19:29+01:00
diskType: HardDisk
errorCount: 0
id: 0ddfb7c0-1351-4df3-b5d6-82d3bbffa6e2
size: 487G
status: active
CellCLI>
CellCLI> list celldisk where name = 'CD_00_exaceladm01' detail;
name: CD_00_exaceladm01
comment:
creationTime: 2016-11-29T10:44:55+01:00
deviceName: /dev/sda
devicePartition: /dev/sda3
diskType: HardDisk
errorCount: 0
freeSpace: 346.0625G
id: f73cfdb7-aa40-47d9-99e0-e39e456b0b55
physicalDisk: PUUK3V
size: 7.1192474365234375T
status: normal
CellCLI>
As before, you have two option: execute manually disk by disk or using the script with dcli. Below you see that I created and called the script (I cropped the output but you can see the RAW execution here):
[DOM0 - root@exadbadm01 tmp]$ vi Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$ chmod +x Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$ cat Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh
dcli -l root -c exaceladm01 cellcli -e ALTER GRIDDISK DATAC8_CD_00_EXACELADM01 size=30720M;
dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC8_CD_00_EXACELADM02 size=30720M;
…
…
dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM02 size=30720M;
dcli -l root -c exaceladm03 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM03 size=30720M;
dcli -l root -c exaceladm04 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM04 size=30720M;
dcli -l root -c exaceladm05 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM05 size=30720M;
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$ ./Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh
exaceladm01: GridDisk DATAC8_CD_00_exaceladm01 successfully altered
exaceladm02: GridDisk DATAC8_CD_00_exaceladm02 successfully altered
…
…
exaceladm02: GridDisk DATAC8_CD_11_exaceladm02 successfully altered
exaceladm03: GridDisk DATAC8_CD_11_exaceladm03 successfully altered
exaceladm04: GridDisk DATAC8_CD_11_exaceladm04 successfully altered
exaceladm05: GridDisk DATAC8_CD_11_exaceladm05 successfully altered
[DOM0 - root@exadbadm01 tmp]$
Check again that I used the value defined in MB, 30720MB in this case. Again, be careful with 16MB align. After the change I have in storage cell:
CellCLI> list griddisk where name = 'DATAC8_CD_00_exaceladm01' detail;
name: DATAC8_CD_00_exaceladm01
asmDiskGroupName: DATAC8
asmDiskName: DATAC8_CD_00_EXACELADM01
asmFailGroupName: EXACELADM01
availableTo:
cachedBy: FD_00_exaceladm01
cachingPolicy: default
cellDisk: CD_00_exaceladm01
comment: "Cluster exa-cl8 diskgroup DATAC8"
creationTime: 2017-01-20T16:19:29+01:00
diskType: HardDisk
errorCount: 0
id: 0ddfb7c0-1351-4df3-b5d6-82d3bbffa6e2
size: 30G
status: active
CellCLI> list celldisk where name = 'CD_00_exaceladm01' detail;
name: CD_00_exaceladm01
comment:
creationTime: 2016-11-29T10:44:55+01:00
deviceName: /dev/sda
devicePartition: /dev/sda3
diskType: HardDisk
errorCount: 0
freeSpace: 803.0625G
id: f73cfdb7-aa40-47d9-99e0-e39e456b0b55
physicalDisk: PUUK3V
size: 7.1192474365234375T
status: normal
CellCLI>
Conclusion
Execute the shrink for Exadata is something that you don’t realize every day in your daily tasks. I suppose that not in a quarterly task because today is always “add, add, add”. Besides the increase of space, the shrink is trickier, and you really need to take care for more steps. Maybe you need to move some data, restart the cluster, move data again.
But, the most critical part is the 16MB alignment, more than when you add space. This is important because for shrink you define the value in two places: for ASM and for grid disk. If you choose a bad value, the size for grid disk can differ ASM and you will corrupt something. As I told, if you think directly in GB instead of MB for disk sizes, you are safer because GB is always compatible with 16MB.
This post was published in my personal blog too: http://www.fernandosimon.com/blog/shrink-asm-diskgroup-and-exadata-grid-disks/
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”
