DELETE ASMDISK from DISKGROUP with VOTEDISK

Recent Oracle community conversation on replacing disk from a diskgroup having votedisk triggered me to test the scenario

https://community.oracle.com/thread/3785252?sr=stream

Currently OCR/Votedisk is on External redundancy diskgroup

[oracle@VM2 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 0d67c8c5c1754fe5bfb3e4513cdad2f5 (ORCL:ASMDISK1) [DATA]

Created new Normal Redundancy diskgroup with 3 ASM Disks and Added OCR to this diskgroup, later removed the OCR from DATA diskgroup.

[root@VM1 ~]# ocrconfig -add +OCR_GROUP
[root@VM1 ~]# crsctl replace votedisk +OCR_GROUP
Successful addition of voting disk 8af044a05e624f15bfdd42383d299f79.
Successful addition of voting disk 4822dd1445194fffbf876bac9a8ac588.
Successful addition of voting disk 5c60703bc2d64f9abf5d65cb5157f4c1.
Successful deletion of voting disk 0d67c8c5c1754fe5bfb3e4513cdad2f5.
Successfully replaced voting disk group with +OCR_GROUP.
CRS-4266: Voting file(s) successfully replaced

[root@VM1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 8af044a05e624f15bfdd42383d299f79 (ORCL:ASMDISK5) [OCR_GROUP]
2. ONLINE 4822dd1445194fffbf876bac9a8ac588 (ORCL:ASMDISK6) [OCR_GROUP]
3. ONLINE 5c60703bc2d64f9abf5d65cb5157f4c1 (ORCL:ASMDISK7) [OCR_GROUP]
Located 3 voting disk(s).

Add 4th ASMDISK to normal redundancy diskgroup.

SQL> alter diskgroup OCR_GROUP add disk ‘ORCL:ASMDISK8’;

Diskgroup altered.

[root@VM1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 8af044a05e624f15bfdd42383d299f79 (ORCL:ASMDISK5) [OCR_GROUP]
2. ONLINE 4822dd1445194fffbf876bac9a8ac588 (ORCL:ASMDISK6) [OCR_GROUP]
3. ONLINE 5c60703bc2d64f9abf5d65cb5157f4c1 (ORCL:ASMDISK7) [OCR_GROUP]
Located 3 voting disk(s).

Drop one of ASMDISK which has votedisk.

SQL> alter diskgroup OCR_GROUP drop disk ASMDISK7;

Diskgroup altered.

ASM will choose the available Disk for placing the votedisk.

[root@VM1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 8af044a05e624f15bfdd42383d299f79 (ORCL:ASMDISK5) [OCR_GROUP]
2. ONLINE 4822dd1445194fffbf876bac9a8ac588 (ORCL:ASMDISK6) [OCR_GROUP]
3. ONLINE 1537fcd557584fd9bf349fb94dddc486 (ORCL:ASMDISK8) [OCR_GROUP]
Located 3 voting disk(s).

Conclusion: So as long as you have sufficient failgroups to maintain the required number of votedisk , ASM will move the votedisk to available disks after dropping any of the ASM DISK

Password File Maintenance ASM, RAC

Prior 11GR2:
ASM password file needs to be updated on both instances whenever changes are made.
11GR2:
Starting with oracle 11Gr2 ASM password file will be updated on both nodes

SQL> select * from v$version;

BANNER
——————————————————————————–
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – 64bit Production
PL/SQL Release 11.2.0.1.0 – Production
CORE 11.2.0.1.0 Production
TNS for Linux: Version 11.2.0.1.0 – Production
NLSRTL Version 11.2.0.1.0 – Production

SQL> select * from gv$pwfile_users order by inst_id;

INST_ID USERNAME SYSDB SYSOP SYSAS
———- —————————— —– —– —–
1 SYS TRUE TRUE TRUE
1 ASMSNMP TRUE FALSE FALSE
2 SYS TRUE TRUE TRUE
2 ASMSNMP TRUE FALSE FALSE
SQL> create user rchemik identified by rakesh;

User created.

SQL> grant sysasm to rchemik;

Grant succeeded.

SQL> select * from gv$pwfile_users order by inst_id;

INST_ID USERNAME SYSDB SYSOP SYSAS
———- —————————— —– —– —–
1 SYS TRUE TRUE TRUE
1 ASMSNMP TRUE FALSE FALSE
1 RCHEMIK FALSE FALSE TRUE
2 SYS TRUE TRUE TRUE
2 ASMSNMP TRUE FALSE FALSE
2 RCHEMIK FALSE FALSE TRUE

6 rows selected.
SQL> drop user rchemik;

User dropped.

SQL> select * from gv$pwfile_users order by inst_id;

INST_ID USERNAME SYSDB SYSOP SYSAS
———- —————————— —– —– —–
1 SYS TRUE TRUE TRUE
1 ASMSNMP TRUE FALSE FALSE
2 SYS TRUE TRUE TRUE
2 ASMSNMP TRUE FALSE FALSE

RAC file:

The change in password file update is only for ASM , rac database password still needs to updated on both instances whenever change is made.

12C:

Starting Oracle 12c, ASM supports password file in a diskgroup. You can store the oracle database password file directly in an ASM diskgroup. This support is available for password file of the ASM cluster as well.

Ref:https://prutser.wordpress.com/2011/01/14/clustered-asm-and-rac-password-file-maintenance/

CRS-2640: Required resource ‘ora.DEMO.dg’ is missing

What Happened ?

After creating diskgroup DEMO for testing, I have created few tablespaces under this diskgroup and later dropped both the tablespaces and diskgroup. During the next reboot the database wasn’t started by cluster.

Issue Troubleshooting:

[root@VM1 ~]# crsctl start resource ora.test.db
CRS-2640: Required resource ‘ora.DEMO.dg’ is missing.
CRS-4000: Command Start failed, or completed with errors.

$ srvctl start database -d TEST
PRCR-1079 : Failed to start resource ora.test.db
CRS-2640: Required resource ‘ora.DEMO.dg’ is missing.

Check the resource start or shutdown dependency 

[root@VM1 ~]# crsctl status resource ora.test.db -p |grep ora.DEMO.dg
START_DEPENDENCIES=hard(ora.DATA.dg,ora.DATA2.dg,ora.DEMO.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,uniform:ora.eons) pullup(ora.DATA.dg,ora.DATA2.dg)
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg,shutdown:ora.DATA2.dg,shutdown:ora.DEMO.dg)

Solution:

Remove the dependency of the diskgroup DEMO for the database

[root@VM1 ~]# srvctl modify database -d TEST -a “DATA,DATA2”

Diskgroup was already dropped manually, but try removing it using srvctl

[root@VM1 ~]# srvctl disable diskgroup -g DEMO
PRCR-1001 : Resource ora.DEMO.dg does not exist

[root@VM1 ~]# srvctl remove diskgroup -g DEMO -f
PRCR-1001 : Resource ora.DEMO.dg does not exist

Validate:

[root@VM1 ~]# crsctl status resource ora.test.db -p |grep ora.DEMO.dg

[root@VM1 ~]# srvctl start database -d TEST

[root@VM1 ~]# srvctl status database -d TEST

Instance TEST1 is running on node vm1
Instance TEST2 is running on node vm2

REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB

SQL> select label,os_mb from v$asm_disk order by label;

LABEL OS_MB
——————————- ———-
ASMDISK1 4777
ASMDISK2 4777
ASMDISK3 4777
ASMDISK4 6142
ASMDISK5 5114
ASMDISK6 5122
ASMDISK7 5122
ASMDISK8 5114

Create diskgroup with external redundancy

SQL> create diskgroup DATA3 external redundancy

disk ‘ORCL:ASMDISK5’
disk ‘ORCL:ASMDISK6’
disk ‘ORCL:ASMDISK7’
disk ‘ORCL:ASMDISK8’
attribute ‘compatible.asm’ = ‘11.2.0.0.0’; 2 3 4 5 6

Diskgroup created.

SQL>
select name, state, type, total_mb, free_mb,required_mirror_free_mb req_free, usable_file_mb
from v$asm_diskgroup where name = ‘DATA3’;SQL> 2

NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_FILE_MB
—————————— ———– —— ———- ———- ———- ———-
DATA3 MOUNTED EXTERN 20472 20414 0 20414

Required free space is 0 as the diskgroup is external redundancy . Entire space(USE_FILE_MB) is available for usage.

Create diskgroup with normal redundancy

SQL> Create diskgroup DATA4 normal redundancy
failgroup group1 disk
‘ORCL:ASMDISK5’
failgroup group2 disk
‘ORCL:ASMDISK6’
failgroup group3 disk
‘ORCL:ASMDISK7’
failgroup group4 disk
‘ORCL:ASMDISK8’
attribute ‘compatible.asm’ = ‘11.2.0.0.0’; 2 3 4 5 6 7 8 9 10

Diskgroup created.

Check size of each failgroup

SQL> select failgroup,sum(total_mb) from v$asm_disk
where group_number=2 group by failgroup order by failgroup; 2

FAILGROUP SUM(TOTAL_MB)
—————————— ————-
GROUP1 5114
GROUP2 5122
GROUP3 5122
GROUP4 5114

we require 41MB free space to restore redundancy incase of loss of a disk and we have 10135MB space for additional space

SQL> Select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
from v$asm_diskgroup where name = ‘DATA3′;
2
NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
—————————— ———– —— ———- ———- ———- ———-
DATA3 MOUNTED NORMAL 20472 20311 41 10135
select trunc((free_mb – required_mirror_free_mb) / 2) as useable
from v$asm_diskgroup where name=’DATA3’;
SQL> 2
USEABLE
———-
10135

Create some files on the diskgroup and check the required and usable space

SQL> create tablespace TEST datafile ‘+DEMO’ size 200m;
Tablespace created.

SQL> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
from v$asm_diskgroup where name = ‘DATA3’; 2

NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
—————————— ———– —— ———- ———- –
DATA3 MOUNTED NORMAL 20472 19885 150 9867

SQL> SQL> alter tablespace TEST add datafile ‘+DEMO’ size 200m;
Tablespace altered.

SQL> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
from v$asm_diskgroup where name = ‘DATA3’;
2
NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
—————————— ———– —— ———- ———- ———- ———-
DATA3 MOUNTED NORMAL 20472 19480 252 9614

Drop the tablespace created and check the required and usable space

SQL> SQL> drop tablespace TEST including contents and datafiles;

Tablespace dropped.

NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
—————————— ———– —— ———- ———- –
DATA3 MOUNTED NORMAL 20472 20290 47 10121

USABLE_FILE_MB = (FREE_MB – REQUIRED_MIRROR_FREE_MB) / [2|3]

ref: https://prutser.wordpress.com/2013/01/03/demystifying-asm-required_mirror_free_mb-and-usable_file_mb/

ASM Disk Header Repair

Below scenario is to test the ASM diskheader repair using kfed.

SQL> select path from v$asm_disk where group_number=3;

PATH
————————————————————————–
ORCL:ASMDISK5

[root@VM1 ~]# oracleasm querydisk -p ASMDISK5
 Disk "ASMDISK5" is a valid ASM disk
 /dev/sdd1: LABEL="ASMDISK5" TYPE="oracleasm"

[root@VM1 ~]# kfed read /dev/sdd1
 kfbh.endian: 1 ; 0x000: 0x01
 kfbh.hard: 130 ; 0x001: 0x82
 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
 kfbh.datfmt: 1 ; 0x003: 0x01
 kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
 kfbh.block.obj: 2147483648 ; 0x008: TYPE=0x8 NUMB=0x0
 kfbh.check: 2365485406 ; 0x00c: 0x8cfe715e
 kfbh.fcn.base: 0 ; 0x010: 0x00000000
 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
 kfbh.spare1: 0 ; 0x018: 0x00000000
 kfbh.spare2: 0 ; 0x01c: 0x00000000
 kfdhdb.driver.provstr: ORCLDISKASMDISK5 ; 0x000: length=16
 kfdhdb.driver.reserved[0]: 1145918273 ; 0x008: 0x444d5341
 kfdhdb.driver.reserved[1]: 894128969 ; 0x00c: 0x354b5349
 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
 kfdhdb.compat: 168820736 ; 0x020: 0x0a100000
 kfdhdb.dsknum: 0 ; 0x024: 0x0000
 kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
 kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
 kfdhdb.dskname: ASMDISK5 ; 0x028: length=8

ASM will also store copy of ASM Disk header under allocation unit 1 penultimate block. 

[root@VM1 ~]# kfed read /dev/sdd1 aun=1 blkn=254 |more
 kfbh.endian: 1 ; 0x000: 0x01
 kfbh.hard: 130 ; 0x001: 0x82
 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
 kfbh.datfmt: 1 ; 0x003: 0x01
 kfbh.block.blk: 254 ; 0x004: T=0 NUMB=0xfe
 kfbh.block.obj: 2147483648 ; 0x008: TYPE=0x8 NUMB=0x0
 kfbh.check: 2365485472 ; 0x00c: 0x8cfe71a0
 kfbh.fcn.base: 0 ; 0x010: 0x00000000
 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
 kfbh.spare1: 0 ; 0x018: 0x00000000
 kfbh.spare2: 0 ; 0x01c: 0x00000000
 kfdhdb.driver.provstr: ORCLDISKASMDISK5 ; 0x000: length=16
 kfdhdb.driver.reserved[0]: 1145918273 ; 0x008: 0x444d5341
 kfdhdb.driver.reserved[1]: 894128969 ; 0x00c: 0x354b5349
 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
 kfdhdb.compat: 168820736 ; 0x020: 0x0a100000
 kfdhdb.dsknum: 0 ; 0x024: 0x0000
 kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
 kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
 kfdhdb.dskname: ASMDISK5 ; 0x028: length=8
 kfdhdb.grpname: DATA3 ; 0x048: length=5
 kfdhdb.fgname: ASMDISK5 ; 0x068: length=8

Corrupt the header of the Disk for the diskgroup DATA3

[root@VM1 ~]# dd if=/dev/zero of=/dev/sdd1 bs=4096 count=1
 1+0 records in
 1+0 records out
 4096 bytes (4.1 kB) copied, 0.00111 seconds, 3.7 MB/s

Check the header status now, it will show as INVALID

[root@VM1 ~]# kfed read /dev/sdd1 |more
 kfbh.endian: 0 ; 0x000: 0x00
 kfbh.hard: 0 ; 0x001: 0x00
 kfbh.type: 0 ; 0x002: KFBTYP_INVALID
 kfbh.datfmt: 0 ; 0x003: 0x00
 kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
 kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0
 kfbh.check: 0 ; 0x00c: 0x00000000
 kfbh.fcn.base: 0 ; 0x010: 0x00000000
 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
 kfbh.spare1: 0 ; 0x018: 0x00000000
 kfbh.spare2: 0 ; 0x01c: 0x00000000
 ERROR!!!, failed to get the oracore error message
SQL> alter diskgroup DATA3 dismount;

Diskgroup altered.

SQL> alter diskgroup DATA3 mount;
 alter diskgroup DATA3 mount
 *
 ERROR at line 1:
 ORA-15032: not all alterations performed
 ORA-15017: diskgroup "DATA3" cannot be mounted
 ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA3"

Repair the ASM disk

[root@VM1 ~]# kfed repair /dev/sdd1

Check if the header status repaired using kfed

[root@VM1 ~]# kfed read /dev/sdd1 |more
 kfbh.endian: 1 ; 0x000: 0x01
 kfbh.hard: 130 ; 0x001: 0x82
 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
 kfbh.datfmt: 1 ; 0x003: 0x01
 kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
 kfbh.block.obj: 2147483648 ; 0x008: TYPE=0x8 NUMB=0x0
 kfbh.check: 2365485406 ; 0x00c: 0x8cfe715e
 kfbh.fcn.base: 0 ; 0x010: 0x00000000
 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
 kfbh.spare1: 0 ; 0x018: 0x00000000
 kfbh.spare2: 0 ; 0x01c: 0x00000000
 kfdhdb.driver.provstr: ORCLDISKASMDISK5 ; 0x000: length=16
 kfdhdb.driver.reserved[0]: 1145918273 ; 0x008: 0x444d5341
 kfdhdb.driver.reserved[1]: 894128969 ; 0x00c: 0x354b5349
 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
 kfdhdb.compat: 168820736 ; 0x020: 0x0a100000
 kfdhdb.dsknum: 0 ; 0x024: 0x0000
 kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
 kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
 kfdhdb.dskname: ASMDISK5 ; 0x028: length=8

Mount the diskgroup now and check the status

SQL> alter diskgroup DATA3 mount;

Diskgroup altered.

SQL> select group_number,name,type,state from v$asm_diskgroup ;

GROUP_NUMBER NAME TYPE STATE
 ------------ ------------------------------ ------ -----------
 1 DATA2 NORMAL MOUNTED
 2 DATA EXTERN MOUNTED
 3 DATA3 EXTERN MOUNTED

ASM Metadata files

ASM File Directory- file 1

http://asmsupportguy.blogspot.com.au/2012/01/asm-file-number-1.html

ASM Disk Directory- file2

http://asmsupportguy.blogspot.com.au/2012/01/asm-disk-directory.html

ASM Active Change Directory- File 3

http://asmsupportguy.blogspot.com.au/2012/01/asm-file-number-3.html

ASM Continuing Operations Directory- file 4

http://asmsupportguy.blogspot.com.au/2012/01/asm-file-number-4.html

ASM file number 5

http://asmsupportguy.blogspot.com.au/2012/01/asm-file-number-5.html

ASM file number 6

http://asmsupportguy.blogspot.com.au/2012/01/asm-file-number-6.html

ASM file number 7

http://asmsupportguy.blogspot.com.au/2012/01/asm-file-number-7.html

ASM file number 8

http://asmsupportguy.blogspot.com.au/2012/01/asm-file-number-8.html

ASM Attributes Directory- file 9

http://asmsupportguy.blogspot.com.au/2012/01/asm-file-number-9.html

ASM files number 10 and 11
http://asmsupportguy.blogspot.com.au/2012/01/asm-files-number-10-and-11.html

ASM files number 12 and 254
http://asmsupportguy.blogspot.com.au/2012/01/asm-files-number-12-and-254.html