介绍
有时候服务器或存储的突发情况，导致数据库服务器重启同时出现OCR磁盘丢失报错，导致oracle数据库集群宕机。这时，就需要我们来恢复OCR磁盘和voting disk，从而恢复集群！整个过程中，数据库是无法使用的！
本文数据库版本11.2.0.4 RAC，使用dd命令来模拟ocr磁盘损坏，然后通过ocr的自动备份来恢复ocr信息，然后重建voting disk！
检查当前可用资源
[grid@rac1 ~]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 1048576 20473 18230 0 18230 0 N DBFILE/
MOUNTED EXTERN N 512 4096 1048576 13178 12448 0 12448 0 N FRA/
MOUNTED NORMAL N 512 4096 1048576 6141 5215 2047 1584 0 Y OCR/
检查ocr备份
[grid@rac1 ~]$ ocrconfig -showbackup
注意：因为我使用的是虚拟机，且不常使用这个环境，所以这个自动备份时间有点早，但是依然可以使用！
检查vote disk
[grid@rac1 ~]$ crsctl query css votedisk
图中显示的是当前votedisk相关信息！
创建pfile文件
创建一个ASM实例的pfile文件，后面会用到！！！ ---- 或者可以手动创建一个
SQL> create pfile= '/tmp/pfile' from spfile;
模拟OCR磁盘损坏
由于OCR磁盘组使用normal冗余模式，所以损坏掉了两个才无法启动！！！
使用DD命令将OCR的磁盘损坏！！！
dd if=/dev/zero of=/dev/mapper/mpathfp1 bs=1M count=100 ----损坏一块照常工作！！！
dd if=/dev/zero of=/dev/mapper/mpathgp1 bs=1M count=100
dd if=/dev/zero of=/dev/mapper/mpathhp1 bs=1M count=100
这里使用dd命令将磁盘头的信息清空！！
等一段时间后，grid下的alter日志会出现如下信息：
[cssd(2912)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /dev/oracleasm/disks/OCR01 will be considered not functional in 99370 milliseconds
[cssd(2957)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /dev/oracleasm/disks/OCR01 will be considered not functional in 49600 milliseconds
.......
[crsd(3534)]CRS-1006:The OCR location +OCR is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.
2021-03-18 13:14:33.229:
[/u01/app/grid/bin/oraagent.bin(3693)]CRS-5822:Agent '/u01/app/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:3:6} in /u01/app/grid/log/rac1/agent/crsd/oraagent_grid/oraagent_grid.log.
如果需要更快出现报错，直接重启CRS！！！！
[root@rac1 bin]# ps -elf|grep smon
4 S root 2914 1 3 -40 - - 193678 hrtime 12:38 ? 00:01:21 /u01/app/grid/bin/osysmond.bin
0 S grid 3500 1 0 80 0 - 54961 semtim 12:39 ? 00:00:00 asm_smon_+ASM1
0 S root 9469 2759 0 80 0 - 25814 pipe_w 13:17 pts/0 00:00:00 grep smon
[root@rac1 bin]# ./crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
[root@rac1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
进程还在但是无法显示集群任何信息！！！！！
日志文件中会显示：
[crsd(9190)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.
2021-03-18 13:15:11.080:
[crsd(9190)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/grid/log/rac1/crsd/crsd.log.
2021-03-18 13:15:11.668:
[ohasd(2636)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.
2021-03-18 13:15:13.158:
[crsd(9203)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.
2021-03-18 13:15:13.170:
[crsd(9203)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/grid/log/rac1/crsd/crsd.log.
2021-03-18 13:15:13.785:
[ohasd(2636)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.
2021-03-18 13:15:15.274:
[crsd(9220)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.
2021-03-18 13:15:15.294:
[crsd(9220)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/grid/log/rac1/crsd/crsd.log.
2021-03-18 13:15:20.339:
[ohasd(2636)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.
2021-03-18 13:15:20.340:
[ohasd(2636)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
2021-03-18 13:15:20.420:
[ohasd(2636)]CRS-2769:Unable to failover resource 'ora.crsd'.
........
[/u01/app/grid/bin/oraagent.bin(9910)]CRS-5019:All OCR locations are on ASM disk groups [OCR], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/grid/log/rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".
[client(12353)]CRS-10001:18-Mar-21 13:31 ACFS-9290: Waiting for ASM to shutdown.
[client(12406)]CRS-10001:18-Mar-21 13:31 ACFS-9290: Waiting for ASM to shutdown.
[client(12424)]CRS-10001:18-Mar-21 13:31 ACFS-9290: Waiting for ASM to shutdown.
由于OCR磁盘损坏了，在正常停止crs时，会一直等待 ASM关闭，所以这里直接手动abort掉！！！
强制停止crs
[root@rac1 bin]# ./crsctl stop crs -f
注意：使用-f 强制停止crs，把2个节点的crs都停止掉！
以独占模式启动crs
在节点1上：以独占模式启动crs
[root@rac1 bin]# ./crsctl start crs -excl -nocrs
参数说明：
-excl 该参数指定使用独占模式
-nocrs 该参数指定忽略查找crs及voting
11.2.0.1:
# $CRS_HOME/bin/crsctl start crs -excl
11.2.0.2 and above:
# $CRS_HOME/bin/crsctl start crs -excl -nocrs
使用创建好的pfile文件重启启动asm实例
这里使用最开始已经准备好的pfile文件(grid用户执行)
SQL>startup pfile='/tmp/pfile';
查询信息可以看到ASM磁盘信息为空：
SQL>select group_number, disk_number, mount_status, header_status, path from v$asm_disk;
创建OCR磁盘组
我们还是使用原来asm磁盘来重新创建一个OCR磁盘组，并且取同样的名称！(grid用户)
sqlplus / as sysasm
SQL>create diskgroup OCR normal redundancy disk '/dev/oracleasm/disks/OCR01','/dev/oracleasm/disks/OCR02', '/dev/oracleasm/disks/OCR03' attribute 'compatible.asm'='11.2.0.0.0';
发现OCR已经上线，但是磁盘组中没有任何文件！！！其他两个磁盘组没有损坏，所以文件依旧存在。
[grid@rac1 ~]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 1048576 20473 18230 0 18230 0 N DBFILE/
MOUNTED EXTERN N 512 4096 1048576 13178 12448 0 12448 0 N FRA/
MOUNTED NORMAL N 512 4096 1048576 6141 5982 2047 1967 0 N OCR/
创建ASM实例的spfile文件
SQL>create spfile='+OCR' from pfile='/tmp/pfile';
还原OCR文件
OCR文件会有自动备份功能：
/u01/app/grid/cdata/raccluster
找一个最新的文件用来做还原！！！！
[root@rac1 ~]# cd /u01/app/grid/bin/
[root@rac1 bin]# ./ocrconfig -restore /u01/app/grid/cdata/raccluster/backup00.ocr
检查OCR：
恢复voting disk
如果遇到报错，需要将crs强制停止，然后重新以独占模式启动！！！！
[root@rac1 bin]# ./crsctl replace votedisk +OCR
[root@rac1 bin]# ./crsctl query css votedisk
检查集群环境
强制关闭crs,然后正常重启2个节点的crs！！！！ 查看日志， 启动需要一定时间！！！
[root@rac1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
通过下面的命令检查：
crsctl status resource -t
ocrcheck
crsctl query css votedisk
crsctl check cluster -all
检查完成后，简单测试下数据库能否正常使用！
注意：直接重启服务器后，,有可能丢失磁盘标签，导致找不到磁盘，这时通过查看alter日志可以发现，报错信息是找不到OCR磁盘，这是因为使用dd命令有可能将磁盘上得asm磁盘标签给清理了，需要重新将标签补充上！！！！！
oracleasm renamedisk -f /dev/mapper/mpathfp1 ocr01
oracleasm renamedisk -f /dev/mapper/mpathgp1 ocr02
oracleasm renamedisk -f /dev/mapper/mpathhp1 ocr03