Insights and Strategies for DBA: OCR/VOTING DISK RECOVERY WITHOUT ANY BACKUP OF OCR/OLR/VOTING DISK

Tested Demonstration Read Must Doc ID 1377349.1 (Repeat all steps on all the nodes locally and verify node by node)

Fully Deconfig & reconfig after Lost the OCR/VOTING DISK without any backup OCR/OLR/VOTING DISK

1^stMethod Normal with Same Name of Disk & Same Size of Disk.

Steps to deconfigure

Before deconfiguring, collect the following as grid user if possible to generate a list of user resources to be added back to the cluster after reconfigure finishes:

$GRID_HOME/bin/crsctl stat res -t
$GRID_HOME/bin/crsctl stat res -p
$GRID_HOME/bin/crsctl query css votedisk
$GRID_HOME/bin/ocrcheck
$GRID_HOME/bin/oifcfg getif
$GRID_HOME/bin/srvctl config nodeapps -a
$GRID_HOME/bin/srvctl config scan
$GRID_HOME/bin/srvctl config asm -a
$GRID_HOME/bin/srvctl config listener -l <listener-name> -a
$DB_HOME/bin/srvctl config database -d <dbname> -a
$DB_HOME/bin/srvctl config service -d <dbname> -s <service-name> -v

Note: In my case OCR & VOTING disk available are both on DISK1.

Note down the Total number of disks available.

[root@rac1 grid]# oracleasm listdisks

DISK1

DISK2

DISK3

DISK4

Corrupted the disk through dd command

Pagesize = 4096

Block count = 1000000 = 1MB

[root@rac1 disks]# dd if=/dev/zero of=DISK1 bs=4096 count=30000 ### You are using oracle oracleasm –deletedisk DISK1

dd: writing `DISK1': No space left on device

524113+0 records in

524112+0 records out

2146765824 bytes (2.1 GB) copied, 110.873 seconds, 19.4 MB/s

[root@rac1 disks]#

[root@rac1 disks]# crsctl stat res -t

--------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

ONLINE ONLINE rac1

ora.LISTENER.lsnr

ONLINE ONLINE rac1

ora.OCR.dg

ONLINE ONLINE rac1

ora.asm

ONLINE ONLINE rac1 Started

ora.eons

ONLINE ONLINE rac1

ora.gsd

ONLINE ONLINE rac1

ora.net1.network

ONLINE ONLINE rac1

ora.ons

ONLINE ONLINE rac1

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE rac1

ora.oc4j

1 ONLINE ONLINE rac1

ora.rac1.vip

1 ONLINE ONLINE rac1

ora.racdb.db

1 ONLINE ONLINE rac1 Open

ora.scan1.vip

1 ONLINE ONLINE rac1

[root@rac1 disks]#

[root@rac1 disks]# crsctl query css votedisk

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 7ce03eba16dd4f81bffaedbdbf91f613 (ORCL:DISK1) [OCR]

Located 1 voting disk(s).

[root@rac1 disks]# crsctl stop crs # [root@rac1 disks]# crsctl stop crs -f

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.oc4j' on 'rac1'

CRS-2673: Attempting to stop 'ora.OCR.dg' on 'rac1'

CRS-2673: Attempting to stop 'ora.racdb.db' on 'rac1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'

CRS-2673: Attempting to stop 'ora.gsd' on 'rac1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'

CRS-2677: Stop of 'ora.gsd' on 'rac1' succeeded

CRS-4549: Unexpected disconnect while executing shutdown request.

CRS-2675: Stop of 'ora.crsd' on 'rac1' failed

CRS-2679: Attempting to clean 'ora.crsd' on 'rac1'

CRS-4548: Unable to connect to CRSD

CRS-2678: 'ora.crsd' on 'rac1' has experienced an unrecoverable failure

CRS-0267: Human intervention required to resume its availability.

CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has failed

CRS-4687: Shutdown command has completed with error(s).

CRS-4000: Command Stop failed, or completed with errors.

[root@rac1 disks]# crsctl stat res -t

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Status failed, or completed with errors.

[root@rac1 disks]# crsctl check has

CRS-4638: Oracle High Availability Services is online

[root@rac1 disks]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

[root@rac1 disks]# ps -ef|grep smon

oracle 5068 1 0 18:32 ? 00:00:00 asm_smon_+ASM1

root 10735 4371 0 20:03 pts/1 00:00:00 grep smon

oracle 10895 1 0 19:42 ? 00:00:01 ora_smon_racdb1

[root@rac1 disks]# su - oracle

-bash-3.2$ db_env

-bash-3.2$ sqlplus sys as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Thu Mar 17 20:04:35 2016

Enter password:

Connected.

SQL> !

[oracle@rac1 ~]$ ps -ef|grep smon

oracle 5068 1 0 18:32 ? 00:00:00 asm_smon_+ASM1

oracle 10895 1 0 19:42 ? 00:00:02 ora_smon_racdb1

oracle 12337 12195 0 20:05 pts/1 00:00:00 grep smon

SQL> shutdown abort

ORACLE instance shut down.

SQL>

REBOOT NODE with the help of root user => reboot

[root@rac1 ~]# crsctl check has

CRS-4638: Oracle High Availability Services is online

[root@rac1 ~]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

CRS-4533: Event Manager is online

[root@rac1 ~]# crsctl start crs

CRS-4640: Oracle High Availability Services is already active

CRS-4000: Command Start failed, or completed with errors.

crsd.log

2016-03-17 20:00:39.260: [GIPCXCPT][2039349984] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)

2016-03-17 20:00:39.288: [GIPCXCPT][2039349984] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)

2016-03-17 20:00:48.928: [ CRSMAIN][1091852608] Policy Engine is not initialized yet!

2016-03-17 20:01:00.753: [ OCRRAW][2039349984]propriogid:1_2: INVALID FORMAT

2016-03-17 20:01:00.753: [ OCRRAW][2039349984]proprioini: all disks are not OCR/OLR formatted

2016-03-17 20:01:00.753: [ OCRRAW][2039349984]proprinit: Could not open raw device

2016-03-17 20:01:01.342: [ OCRAPI][2039349984]a_init:16!: Backend init unsuccessful : [26]

2016-03-17 20:01:01.351: [ CRSOCR][2039349984] OCR context init failure. Error: PROC-26: Error while accessing the physical storage

2016-03-17 20:01:01.352: [ CRSD][2039349984][PANIC] CRSD exiting: Could not init OCR, code: 26

2016-03-17 20:01:01.352: [ CRSD][2039349984] Done.

alertrac1.log

2016-03-17 19:54:19.722

[cssd(4650)]CRS-1605:CSSD voting file is online: ORCL:DISK1; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

2016-03-17 19:59:06.086

[crsd(5124)]CRS-1006:The OCR location +OCR is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

2016-03-17 19:59:35.606

[/u01/app/11.2.0/grid/bin/oraagent.bin(5289)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/crsd/oraagent_oracle/oraagent_oracle.log.

2016-03-17 19:59:38.604

[/u01/app/11.2.0/grid/bin/orarootagent.bin(5294)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/crsd/orarootagent_root/orarootagent_root.log.

2016-03-17 19:59:41.132

[/u01/app/11.2.0/grid/bin/scriptagent.bin(5382)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/scriptagent_oracle' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/crsd/ora_oc4j_type_oracle/ora_oc4j_type_oracle.log.

2016-03-17 20:00:18.260

[ohasd(4366)]CRS-2758:Resource 'ora.crsd' is in an unknown state.

2016-03-17 20:15:30.780

[cssd(4650)]CRS-1603:CSSD on node rac1 shutdown by user.

2016-03-17 20:15:31.308

[gpnpd(4582)]CRS-2341:GPNPD on node rac1 shutting down on signal 15.

2016-03-17 20:20:00.508

[ohasd(4250)]CRS-2112:The OLR service started on node rac1.

2016-03-17 20:20:06.874

[ohasd(4250)]CRS-8017:location: /etc/oracle/lastgasp has 44 reboot advisory log files, 0 were announced and 0 errors occurred

2016-03-17 20:20:41.581

[ohasd(4250)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.

2016-03-17 20:20:46.885

[ohasd(4250)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2016-03-17 20:20:47.172

[gpnpd(4440)]CRS-2328:GPNPD started on node rac1.

2016-03-17 20:20:58.451

[cssd(4513)]CRS-1713:CSSD daemon is started in clustered mode

2016-03-17 20:21:06.874

[cssd(4513)]CRS-1603:CSSD on node rac1 shutdown by user.

2016-03-17 20:21:07.088

[ohasd(4250)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac1'.

2016-03-17 20:21:20.679

[cssd(4623)]CRS-1713:CSSD daemon is started in clustered mode

2016-03-17 20:21:21.007

[cssd(4623)]CRS-1603:CSSD on node rac1 shutdown by user.

2016-03-17 20:21:23.573

[ohasd(4250)]CRS-2765:Resource 'ora.diskmon' has failed on server 'rac1'.

[client(4747)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.

[client(4761)]CRS-10001:ACFS-9322: done.

orarootagent_root.log

2016-03-17 19:59:35.294: [ora.rac1.vip][1555609920] [check] Checking if IP 192.168.88.10 is present on NIC eth0

2016-03-17 19:59:35.294: [ AGFW][1555609920] check for resource: ora.rac1.vip 1 1 completed with status: ONLINE

2016-03-17 19:59:35.625: [ AGFW][1354258752] CHECK initiated by timer for: ora.scan1.vip 1 1

2016-03-17 19:59:35.626: [ AGFW][1354258752] CHECK initiated by timer for: ora.net1.network rac1 1

2016-03-17 19:59:35.626: [ AGFW][1354258752] CHECK initiated by timer for: ora.rac1.vip 1 1

2016-03-17 19:59:35.636: [ AGFW][1488492864] Executing command: check for resource: ora.scan1.vip 1 1

2016-03-17 19:59:35.636: [ora.scan1.vip][1488492864] [check] Checking if IP 192.168.88.21 is present on NIC eth0

2016-03-17 19:59:35.636: [ AGFW][1488492864] check for resource: ora.scan1.vip 1 1 completed with status: ONLINE

2016-03-17 19:59:35.636: [ AGFW][1488492864] Executing command: check for resource: ora.net1.network rac1 1

2016-03-17 19:59:35.637: [ AGFW][1488492864] check for resource: ora.net1.network rac1 1 completed with status: ONLINE

2016-03-17 19:59:35.646: [ AGFW][1488492864] Executing command: check for resource: ora.rac1.vip 1 1

2016-03-17 19:59:35.646: [ora.rac1.vip][1488492864] [check] Checking if IP 192.168.88.10 is present on NIC eth0

2016-03-17 19:59:35.646: [ AGFW][1488492864] check for resource: ora.rac1.vip 1 1 completed with status: ONLINE

2016-03-17 19:59:35.674: [ CRSCOMM][1186466112][FFAIL] Couldnt clscreceive message, no message: 11

2016-03-17 19:59:35.674: [ CRSCOMM][1186466112] Client disconnected.

2016-03-17 19:59:35.674: [ CRSCOMM][1186466112] IPC client connection 0x1f8e9de0 to member 0 has been removed

2016-03-17 19:59:35.674: [CLSFRAME][1186466112] Removing IPC Member:{Relative|Node:0|Process:0|Type:1}

2016-03-17 19:59:35.744: [CLSFRAME][1186466112] Disconnected from CRSD:rac1 process: {Relative|Node:0|Process:0|Type:1}

2016-03-17 19:59:38.680: [ AGFW][1354258752]Created alert : (:CRSAGF00117:) : Disconnected from server. Commiting suicide.

2016-03-17 19:59:41.836: [ AGFW][1354258752] Agent is exiting with exit code: 1

[root@rac1]# cd $ORACLE_GRID/crs/install/

[root@rac1 install]# ./rootcrs.pl -deconfig -force -verbose

2016-03-17 20:34:34: Parsing the host name

2016-03-17 20:34:34: Checking for super user privileges

2016-03-17 20:34:34: User has super user privileges

Using configuration parameter file: ./crsconfig_params

PRCR-1035 : Failed to look up CRS resource ora.cluster_vip.type for 1

PRCR-1068 : Failed to query resources

Cannot communicate with crsd

PRCR-1070 : Failed to check if resource ora.gsd is registered

Cannot communicate with crsd

PRCR-1070 : Failed to check if resource ora.ons is registered

Cannot communicate with crsd

PRCR-1070 : Failed to check if resource ora.eons is registered

Cannot communicate with crsd

ACFS-9200: Supported

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Stop failed, or completed with errors.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'

CRS-4548: Unable to connect to CRSD

CRS-2675: Stop of 'ora.crsd' on 'rac1' failed

CRS-2679: Attempting to clean 'ora.crsd' on 'rac1'

CRS-4548: Unable to connect to CRSD

CRS-2678: 'ora.crsd' on 'rac1' has experienced an unrecoverable failure

CRS-0267: Human intervention required to resume its availability.

CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has failed

CRS-4687: Shutdown command has completed with error(s).

CRS-4000: Command Stop failed, or completed with errors.

You must kill crs processes or reboot the system to properly

cleanup the processes started by Oracle clusterware

error: package cvuqdisk is not installed

Successfully deconfigured Oracle clusterware stack on this node

[root@rac1 ~]# ocrcheck

PROT-601: Failed to initialize ocrcheck

PROC-33: Oracle Cluster Registry is not configured Storage layer error [Error opening ocr.loc file. No such file or directory] [2]

[root@rac1 ~]# crsctl check crs

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Check failed, or completed with errors.

[root@rac1 ~]# crsctl check has

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Check failed, or completed with errors.

[root@rac1 grid]# oracleasm listdisks

DISK2

DISK3

DISK4

[root@rac1 grid]# oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

[root@rac1 grid]# oracleasm listdisks

DISK2

DISK3

DISK4

-bash-3.2$ grid_env

sqlplu-bash-3.2$ sqlplus sys as sysasm

SQL*Plus: Release 11.2.0.1.0 Production on Thu Mar 17 21:03:48 2016

Enter password:

Connected to an idle instance.

SQL> startup mount

ORA-01078: failure in processing system parameters

ORA-29701: unable to connect to Cluster Synchronization Service

SQL> exit

[root@rac1 grid]# crsctl query css votedisk

Unable to communicate with the Cluster Synchronization Services daemon.

[root@rac1 grid]# ocrcheck

PROT-602: Failed to retrieve data from the cluster registry

PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfolclcpi1, dep=210, loc=kgfokge

AMDU-00210: No disks found in diskgroup OCR

AMDU-00210: No disks found in diskgroup OCR] [8]

[root@rac1 grid]# fdisk -l

[root@rac1 grid]# oracleasm querydisk /dev/sdb1

Device "/dev/sdb1" is not marked as an ASM disk

[root@rac1 grid]# oracleasm querydisk /dev/sdc1

Device "/dev/sdc1" is marked an ASM disk with the label "DISK2"

[root@rac1 grid]# oracleasm querydisk /dev/sdd1

Device "/dev/sdd1" is marked an ASM disk with the label "DISK3"

[root@rac1 grid]# oracleasm querydisk /dev/sde1

Device "/dev/sde1" is marked an ASM disk with the label "DISK4"

[root@rac1 grid]# fdisk /dev/sdb

Command (m for help): p

Disk /dev/sdb: 2147 MB, 2147483648 bytes

255 heads, 63 sectors/track, 261 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sdb1 1 261 2096451 83 Linux

Command (m for help): d

Selected partition 1

Command (m for help): p

Disk /dev/sdb: 2147 MB, 2147483648 bytes

255 heads, 63 sectors/track, 261 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

Command (m for help): n

Command action

e extended

p primary partition (1-4)

Partition number (1-4):

Value out of range.

Partition number (1-4): 1

First cylinder (1-261, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-261, default 261):

Using default value 261

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

[root@rac1 grid]# partprobe

NOTE:- Create same name of oracleasm disk and same size (Which name is belongs to OCR diskgroup to DISK)

[root@rac1 grid]# oracleasm createdisk DISK1 /dev/sdb1

Writing disk header: done

Instantiating disk: done

[root@rac1 grid]# oracleasm scandisks

[root@rac1 grid]# oracleasm listdisks

DISK1

DISK2

DISK3

DISK4

[root@rac1 grid]# olsnodes -s

PRCO-19: Failure retrieving list of nodes in the cluster

PRCO-2: Unable to communicate with the clusterware

[root@rac1 install]# ./rootcrs.pl -deconfig -force -verbose

2016-03-17 21:27:35: Parsing the host name

2016-03-17 21:27:35: Checking for super user privileges

2016-03-17 21:27:35: User has super user privileges

Using configuration parameter file: ./crsconfig_params

PRCR-1035 : Failed to look up CRS resource ora.cluster_vip.type for 1

PRCR-1068 : Failed to query resources

Cannot communicate with crsd

PRCR-1070 : Failed to check if resource ora.gsd is registered

Cannot communicate with crsd

PRCR-1070 : Failed to check if resource ora.ons is registered

Cannot communicate with crsd

PRCR-1070 : Failed to check if resource ora.eons is registered

Cannot communicate with crsd

ACFS-9200: Supported

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Stop failed, or completed with errors.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1'

CRS-2677: Stop of 'ora.drivers.acfs' on 'rac1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

error: package cvuqdisk is not installed

Successfully deconfigured Oracle clusterware stack on this node

In 11gR2 onword You are using as per below the manual steps use GUI and configure and creating the OCR file through $GRID_HOME/config/config.sh

Get GUI & configure it then execute root.sh after that all node and verify it it’s all working fine.

[root@rac1 grid]#

--------------------------------------------------------------------------------------

Grid user added back the resource cluster after reconfigure finish

--------------------------------------------------------------------------------------

Look those command output match before doing the activity

[root@rac1 grid]# ./root.sh

Running Oracle 11g root.sh script...

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying dbhome to /usr/local/bin ...

The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying oraenv to /usr/local/bin ...

The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root.sh script.

Now product-specific root actions will be performed.

2016-03-17 21:29:30: Parsing the host name

2016-03-17 21:29:30: Checking for super user privileges

2016-03-17 21:29:30: User has super user privileges

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

LOCAL ADD MODE

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

Adding daemon to inittab

CRS-4123: Oracle High Availability Services has been started.

ohasd is starting

CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'

CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'

CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded

CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'

CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'

CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'rac1'

CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'

CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded

CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'

CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded

ASM created and started successfully.

DiskGroup OCR created successfully.

clscfg: -install mode specified

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

CRS-2672: Attempting to start 'ora.crsd' on 'rac1'

CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded

Successful addition of voting disk 74ff31e9cb424fdbbfe9cb17227ad6a0.

Successfully replaced voting disk group with +OCR.

CRS-4266: Voting file(s) successfully replaced

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 74ff31e9cb424fdbbfe9cb17227ad6a0 (ORCL:DISK1) [OCR]

Located 1 voting disk(s).

CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'

CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'rac1'

CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'

CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'

CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'

CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'

CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'

CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'

CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'

CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'

CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'

CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'

CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'rac1'

CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'

CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded

CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'

CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'rac1'

CRS-2676: Start of 'ora.asm' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'rac1'

CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.evmd' on 'rac1'

CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'rac1'

CRS-2676: Start of 'ora.asm' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.OCR.dg' on 'rac1'

CRS-2676: Start of 'ora.OCR.dg' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac1'

CRS-2676: Start of 'ora.registry.acfs' on 'rac1' succeeded

rac1 2016/03/17 21:46:06 /u01/app/11.2.0/grid/cdata/rac1/backup_20160317_214606.olr

Preparing packages for installation...

cvuqdisk-1.0.7-1

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Updating inventory properties for clusterware

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 3999 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

'UpdateNodeList' was successful.

[root@rac1 grid]# crsctl check has

CRS-4638: Oracle High Availability Services is online

[root@rac1 grid]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

[root@rac1 grid]# crsctl stat res -t

--------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.OCR.dg

ONLINE ONLINE rac1

ora.asm

ONLINE ONLINE rac1 Started

ora.eons

ONLINE ONLINE rac1

ora.gsd

OFFLINE OFFLINE rac1

ora.net1.network

ONLINE ONLINE rac1

ora.ons

ONLINE ONLINE rac1

ora.registry.acfs

ONLINE ONLINE rac1

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE rac1

ora.oc4j

1 OFFLINE OFFLINE

ora.rac1.vip

1 ONLINE ONLINE rac1

ora.scan1.vip

1 ONLINE ONLINE rac1

[root@rac1 grid]#

-----------------------------------------------------------------------------------------------------------------------------------

Skip If you are using $GRID_HOME/config/config.sh if not using follow below the commands manually

Grid user added back the resource cluster after reconfigure finish

------------------------------------------------------------------------------------------------------------------------------------

Look those command output match before doing the activity

Syntax for 11g R2 RAC

srvctl add database -d db_unique_name -o ORACLE_HOME [-x node_name] [-m domain_name] [-p spfile] [-r {PRIMARY|PHYSICAL_STANDBY|LOGICAL_STANDBY|SNAPSHOT_STANDBY}] [-s start_options] [-t stop_options] [-n db_name] [-y {AUTOMATIC|MANUAL}] [-g server_pool_list] [-a "diskgroup_list"]

==>Add Resources

Verify Listener Owner => srvctl config listener -l listener -a

Listener enable => srvctl enable listener -l listener -n rac1

Add Database resource => srvctl add database -d racdb -o /u01/app/oracle/product/11.2.0/db_1 -n racdb

Instance Mannualy add => srvctl add instance -d racdb -i racdb1 -n rac1

Add Database name => srvctl modify database -d racdb -n racdb

Add Diskgroup => srvctl modify database -d racdb -a DATA # Below the follow

Spfile Name set of database => srvctl modify database -d racdb -p "+DATA/racdb/spfileracdb.ora"

# Incase any failure like “Unable to modify database because the resource for diskgroup DATA could not be found. PRCR-1001 : Resource ora.DATA.dg does not exist”

-bash-3.2$ srvctl status DISKGROUP -g DATA -a

PRCR-1001 : Resource ora.DATA.dg does not exist

-bash-3.2$ srvctl status DISKGROUP -g +DATA -a

PRCA-1000 : ASM Disk Group +DATA does not exist

PRCR-1070 : Failed to check if resource ora.+DATA.dg is registered

Invalid resource name

+asm_alert.log

NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1

SUCCESS: diskgroup OCR was mounted

ORACLE_BASE value has been saved for future startups

SUCCESS: ALTER DISKGROUP ALL MOUNT /* asm agent */

Sat Mar 19 21:59:15 2016

WARNING: failed to online diskgroup resource ora.OCR.dg (unable to communicate with CRSD/OHASD)

SQL> ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */

SUCCESS: ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */

Cause :

In contrast to previous version of CRS, 11gR2 ASM does not depend on CRSD anymore; rather if OCR is located on ASM, ASM will start before CRSD while clusterware is coming up, and stops after CRSD while clusterware is going down. Due to the new startup sequence, ASM can not update diskgroup resource status in CRSD either during clusterware startup or shutdown, thus above message is logged.

Solution :

The message can be ignored, the status of diskgroup and diskgroup resource will be synchronize by oraagent once clusterware is fully up.

To find out diskgroup resource status:

$GRID_HOME/bin/srvctl status diskgroup -g diskgroupname -a

To start or sync diskgroup resource:

$GRID_HOME/bin/srvctl start diskgroup -g diskgroupname

But Still above Solution did not worked .............

So I have dimounted and mounted the diskgroup again and then the dg resource were up.

-bash$ sqlplus sys as sysasm

SQL> alter diskgroup data dismount;

Diskgroup altered.

SQL> alter diskgroup data dismount;

alter diskgroup data dismount

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15001: diskgroup "DATA" does not exist or is not mounted

SQL> alter diskgroup data mount;

Diskgroup altered.

-bash-3.2$ srvctl modify database -d racdb -a DATA

Note: For more detail about resource then follow the link http://oracledbabay.blogspot.in/2012/11/srvctl-commands-10g-rac-11g-r2-rac.html

--------------------------------------------------------------------------------------------------------------------------------------

[root@rac1 grid]# crsctl stop cluster -all

CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.OCR.dg' on 'rac1'

CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac1'

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'

CRS-2796: The command may not proceed when Cluster Ready Services is not running

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'

CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'

CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded

CRS-2677: Stop of 'ora.registry.acfs' on 'rac1' succeeded

CRS-2677: Stop of 'ora.OCR.dg' on 'rac1' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'rac1'

CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'rac1'

CRS-2673: Attempting to stop 'ora.eons' on 'rac1'

CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'

CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded

CRS-2677: Stop of 'ora.eons' on 'rac1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed

CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'

CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'

CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'

CRS-2673: Attempting to stop 'ora.asm' on 'rac1'

CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'

CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.diskmon' on 'rac1'

CRS-2677: Stop of 'ora.diskmon' on 'rac1' succeeded

CRS-4639: Could not contact Oracle High Availability Services

CRS-4000: Command Stop failed, or completed with errors.

[root@rac1 grid]# crsclt start cluster –all

Check log files and verify it

….. Keep Enjoy J J J J J J J J J J J J J J J J J J

Reference:

2nd Method
3rd Method

Insights and Strategies for DBA

Labels

Friday, 17 June 2016

OCR/VOTING DISK RECOVERY WITHOUT ANY BACKUP OF OCR/OLR/VOTING DISK - PART I

Steps to deconfigure

No comments: