Friday 17 June 2016

OCR/VOTING DISK RECOVERY WITHOUT ANY BACKUP OF OCR/OLR/VOTING DISK - PART I

Tested Demonstration Read Must Doc ID 1377349.1 (Repeat all steps on all the nodes locally and verify node by node)

Fully Deconfig & reconfig after Lost the OCR/VOTING DISK without any backup OCR/OLR/VOTING DISK


1stMethod Normal with Same Name of Disk & Same Size of Disk.

Steps to deconfigure

Before deconfiguring, collect the following as grid user if possible to generate a list of user resources to be added back to the cluster after reconfigure finishes:
$GRID_HOME/bin/crsctl stat res -t
$GRID_HOME/bin/crsctl stat res -p
$GRID_HOME/bin/crsctl query css votedisk
$GRID_HOME/bin/ocrcheck
$GRID_HOME/bin/oifcfg getif
$GRID_HOME/bin/srvctl config nodeapps -a
$GRID_HOME/bin/srvctl config scan
$GRID_HOME/bin/srvctl config asm -a
$GRID_HOME/bin/srvctl config listener -l <listener-name> -a
$DB_HOME/bin/srvctl config database -d <dbname> -a
$DB_HOME/bin/srvctl config service -d <dbname> -s <service-name> -v

Note: In my case OCR & VOTING disk available are both on DISK1.

Note down the Total number of disks available.
[root@rac1 grid]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4

Corrupted the disk through dd command
Pagesize = 4096
Block count = 1000000 = 1MB
[root@rac1 disks]# dd if=/dev/zero of=DISK1 bs=4096 count=30000                ### You are using oracle oracleasm –deletedisk DISK1
dd: writing `DISK1': No space left on device
524113+0 records in
524112+0 records out
2146765824 bytes (2.1 GB) copied, 110.873 seconds, 19.4 MB/s

 [root@rac1 disks]#
[root@rac1 disks]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
ora.OCR.dg
               ONLINE  ONLINE       rac1
ora.asm
               ONLINE  ONLINE       rac1                     Started
ora.eons
               ONLINE  ONLINE       rac1
ora.gsd
               ONLINE  ONLINE       rac1
ora.net1.network
               ONLINE  ONLINE       rac1
ora.ons
               ONLINE  ONLINE       rac1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.racdb.db
      1        ONLINE  ONLINE       rac1                     Open
ora.scan1.vip
      1        ONLINE  ONLINE       rac1

[root@rac1 disks]#

[root@rac1 disks]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7ce03eba16dd4f81bffaedbdbf91f613 (ORCL:DISK1) [OCR]
Located 1 voting disk(s).


[root@rac1 disks]# crsctl stop crs                                                                     # [root@rac1 disks]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac1'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.racdb.db' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.gsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.gsd' on 'rac1' succeeded
CRS-4549: Unexpected disconnect while executing shutdown request.
CRS-2675: Stop of 'ora.crsd' on 'rac1' failed
CRS-2679: Attempting to clean 'ora.crsd' on 'rac1'
CRS-4548: Unable to connect to CRSD
CRS-2678: 'ora.crsd' on 'rac1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has failed
CRS-4687: Shutdown command has completed with error(s).
CRS-4000: Command Stop failed, or completed with errors.


[root@rac1 disks]# crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

[root@rac1 disks]# crsctl check has
CRS-4638: Oracle High Availability Services is online

[root@rac1 disks]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

 [root@rac1 disks]# ps -ef|grep smon
oracle    5068     1  0 18:32 ?        00:00:00 asm_smon_+ASM1
root     10735  4371  0 20:03 pts/1    00:00:00 grep smon
oracle   10895     1  0 19:42 ?        00:00:01 ora_smon_racdb1

[root@rac1 disks]# su - oracle

-bash-3.2$ db_env

-bash-3.2$ sqlplus sys as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Thu Mar 17 20:04:35 2016
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Enter password:
Connected.
SQL> !
[oracle@rac1 ~]$ ps -ef|grep smon
oracle    5068     1  0 18:32 ?        00:00:00 asm_smon_+ASM1
oracle   10895     1  0 19:42 ?        00:00:02 ora_smon_racdb1
oracle   12337 12195  0 20:05 pts/1    00:00:00 grep smon

SQL> shutdown abort
ORACLE instance shut down.
SQL>


REBOOT NODE with the help of root user => reboot    


[root@rac1 ~]# crsctl check has
CRS-4638: Oracle High Availability Services is online

[root@rac1 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online

[root@rac1 ~]# crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.


crsd.log
2016-03-17 20:00:39.260: [GIPCXCPT][2039349984] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)
2016-03-17 20:00:39.288: [GIPCXCPT][2039349984] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
2016-03-17 20:00:48.928: [ CRSMAIN][1091852608] Policy Engine is not initialized yet!
2016-03-17 20:01:00.753: [  OCRRAW][2039349984]propriogid:1_2: INVALID FORMAT
2016-03-17 20:01:00.753: [  OCRRAW][2039349984]proprioini: all disks are not OCR/OLR formatted
2016-03-17 20:01:00.753: [  OCRRAW][2039349984]proprinit: Could not open raw device
2016-03-17 20:01:01.342: [  OCRAPI][2039349984]a_init:16!: Backend init unsuccessful : [26]
2016-03-17 20:01:01.351: [  CRSOCR][2039349984] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage
2016-03-17 20:01:01.352: [    CRSD][2039349984][PANIC] CRSD exiting: Could not init OCR, code: 26
2016-03-17 20:01:01.352: [    CRSD][2039349984] Done.


alertrac1.log
2016-03-17 19:54:19.722
[cssd(4650)]CRS-1605:CSSD voting file is online: ORCL:DISK1; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.
2016-03-17 19:59:06.086
[crsd(5124)]CRS-1006:The OCR location +OCR is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.
2016-03-17 19:59:35.606
[/u01/app/11.2.0/grid/bin/oraagent.bin(5289)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/crsd/oraagent_oracle/oraagent_oracle.log.
2016-03-17 19:59:38.604
[/u01/app/11.2.0/grid/bin/orarootagent.bin(5294)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/crsd/orarootagent_root/orarootagent_root.log.
2016-03-17 19:59:41.132
[/u01/app/11.2.0/grid/bin/scriptagent.bin(5382)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/scriptagent_oracle' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/crsd/ora_oc4j_type_oracle/ora_oc4j_type_oracle.log.
2016-03-17 20:00:18.260
[ohasd(4366)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
2016-03-17 20:15:30.780
[cssd(4650)]CRS-1603:CSSD on node rac1 shutdown by user.
2016-03-17 20:15:31.308
[gpnpd(4582)]CRS-2341:GPNPD on node rac1 shutting down on signal 15.
2016-03-17 20:20:00.508
[ohasd(4250)]CRS-2112:The OLR service started on node rac1.
2016-03-17 20:20:06.874
[ohasd(4250)]CRS-8017:location: /etc/oracle/lastgasp has 44 reboot advisory log files, 0 were announced and 0 errors occurred
2016-03-17 20:20:41.581
[ohasd(4250)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2016-03-17 20:20:46.885
[ohasd(4250)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2016-03-17 20:20:47.172
[gpnpd(4440)]CRS-2328:GPNPD started on node rac1.
2016-03-17 20:20:58.451
[cssd(4513)]CRS-1713:CSSD daemon is started in clustered mode
2016-03-17 20:21:06.874
[cssd(4513)]CRS-1603:CSSD on node rac1 shutdown by user.
2016-03-17 20:21:07.088
[ohasd(4250)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac1'.
2016-03-17 20:21:20.679
[cssd(4623)]CRS-1713:CSSD daemon is started in clustered mode
2016-03-17 20:21:21.007
[cssd(4623)]CRS-1603:CSSD on node rac1 shutdown by user.
2016-03-17 20:21:23.573
[ohasd(4250)]CRS-2765:Resource 'ora.diskmon' has failed on server 'rac1'.
[client(4747)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4761)]CRS-10001:ACFS-9322: done.


orarootagent_root.log
2016-03-17 19:59:35.294: [ora.rac1.vip][1555609920] [check] Checking if IP 192.168.88.10 is present on NIC eth0
2016-03-17 19:59:35.294: [    AGFW][1555609920] check for resource: ora.rac1.vip 1 1 completed with status: ONLINE
2016-03-17 19:59:35.625: [    AGFW][1354258752] CHECK initiated by timer for: ora.scan1.vip 1 1
2016-03-17 19:59:35.626: [    AGFW][1354258752] CHECK initiated by timer for: ora.net1.network rac1 1
2016-03-17 19:59:35.626: [    AGFW][1354258752] CHECK initiated by timer for: ora.rac1.vip 1 1
2016-03-17 19:59:35.636: [    AGFW][1488492864] Executing command: check for resource: ora.scan1.vip 1 1
2016-03-17 19:59:35.636: [ora.scan1.vip][1488492864] [check] Checking if IP 192.168.88.21 is present on NIC eth0
2016-03-17 19:59:35.636: [    AGFW][1488492864] check for resource: ora.scan1.vip 1 1 completed with status: ONLINE
2016-03-17 19:59:35.636: [    AGFW][1488492864] Executing command: check for resource: ora.net1.network rac1 1
2016-03-17 19:59:35.637: [    AGFW][1488492864] check for resource: ora.net1.network rac1 1 completed with status: ONLINE
2016-03-17 19:59:35.646: [    AGFW][1488492864] Executing command: check for resource: ora.rac1.vip 1 1
2016-03-17 19:59:35.646: [ora.rac1.vip][1488492864] [check] Checking if IP 192.168.88.10 is present on NIC eth0
2016-03-17 19:59:35.646: [    AGFW][1488492864] check for resource: ora.rac1.vip 1 1 completed with status: ONLINE
2016-03-17 19:59:35.674: [ CRSCOMM][1186466112][FFAIL] Couldnt clscreceive message, no message: 11
2016-03-17 19:59:35.674: [ CRSCOMM][1186466112] Client disconnected.
2016-03-17 19:59:35.674: [ CRSCOMM][1186466112] IPC client connection 0x1f8e9de0 to member 0 has been removed
2016-03-17 19:59:35.674: [CLSFRAME][1186466112] Removing IPC Member:{Relative|Node:0|Process:0|Type:1}
2016-03-17 19:59:35.744: [CLSFRAME][1186466112] Disconnected from CRSD:rac1 process: {Relative|Node:0|Process:0|Type:1}
2016-03-17 19:59:38.680: [    AGFW][1354258752]Created alert : (:CRSAGF00117:) :  Disconnected from server. Commiting suicide.
2016-03-17 19:59:41.836: [    AGFW][1354258752] Agent is exiting with exit code: 1


[root@rac1]# cd $ORACLE_GRID/crs/install/
[root@rac1 install]# ./rootcrs.pl -deconfig -force -verbose
2016-03-17 20:34:34: Parsing the host name
2016-03-17 20:34:34: Checking for super user privileges
2016-03-17 20:34:34: User has super user privileges
Using configuration parameter file: ./crsconfig_params
PRCR-1035 : Failed to look up CRS resource ora.cluster_vip.type for 1
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.eons is registered
Cannot communicate with crsd

ACFS-9200: Supported
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-4548: Unable to connect to CRSD
CRS-2675: Stop of 'ora.crsd' on 'rac1' failed
CRS-2679: Attempting to clean 'ora.crsd' on 'rac1'
CRS-4548: Unable to connect to CRSD
CRS-2678: 'ora.crsd' on 'rac1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has failed
CRS-4687: Shutdown command has completed with error(s).
CRS-4000: Command Stop failed, or completed with errors.
You must kill crs processes or reboot the system to properly
cleanup the processes started by Oracle clusterware
error: package cvuqdisk is not installed
Successfully deconfigured Oracle clusterware stack on this node


[root@rac1 ~]# ocrcheck
PROT-601: Failed to initialize ocrcheck
PROC-33: Oracle Cluster Registry is not configured Storage layer error [Error opening ocr.loc file. No such file or directory] [2]


[root@rac1 ~]# crsctl check crs
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Check failed, or completed with errors.

[root@rac1 ~]# crsctl check has
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Check failed, or completed with errors.

[root@rac1 grid]# oracleasm listdisks
DISK2
DISK3
DISK4


[root@rac1 grid]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

[root@rac1 grid]# oracleasm listdisks
DISK2
DISK3
DISK4


-bash-3.2$ grid_env
sqlplu-bash-3.2$ sqlplus sys as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on Thu Mar 17 21:03:48 2016
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Enter password:
Connected to an idle instance.
SQL> startup mount
ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service
SQL> exit

[root@rac1 grid]# crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.

[root@rac1 grid]# ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfolclcpi1, dep=210, loc=kgfokge
AMDU-00210: No disks found in diskgroup OCR
AMDU-00210: No disks found in diskgroup OCR] [8]


[root@rac1 grid]# fdisk -l

[root@rac1 grid]# oracleasm querydisk /dev/sdb1
Device "/dev/sdb1" is not marked as an ASM disk

[root@rac1 grid]# oracleasm querydisk /dev/sdc1
Device "/dev/sdc1" is marked an ASM disk with the label "DISK2"

[root@rac1 grid]# oracleasm querydisk /dev/sdd1
Device "/dev/sdd1" is marked an ASM disk with the label "DISK3"

[root@rac1 grid]# oracleasm querydisk /dev/sde1
Device "/dev/sde1" is marked an ASM disk with the label "DISK4"


[root@rac1 grid]# fdisk /dev/sdb
Command (m for help): p
Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         261     2096451   83  Linux

Command (m for help): d
Selected partition 1

Command (m for help): p

Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4):
                Value out of range.
                                Partition number (1-4): 1
                                First cylinder (1-261, default 1):
                                Using default value 1
                                                Last cylinder or +size or +sizeM or +sizeK (1-261, default 261):
                                Using default value 261
Command (m for help): w

The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

[root@rac1 grid]# partprobe


NOTE:- Create same name of oracleasm disk and same size (Which name is belongs to OCR diskgroup  to DISK)
[root@rac1 grid]# oracleasm createdisk DISK1 /dev/sdb1
Writing disk header: done
Instantiating disk: done

[root@rac1 grid]# oracleasm scandisks

[root@rac1 grid]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4


[root@rac1 grid]# olsnodes -s
PRCO-19: Failure retrieving list of nodes in the cluster
PRCO-2: Unable to communicate with the clusterware

[root@rac1 install]# ./rootcrs.pl -deconfig -force -verbose
2016-03-17 21:27:35: Parsing the host name
2016-03-17 21:27:35: Checking for super user privileges
2016-03-17 21:27:35: User has super user privileges
Using configuration parameter file: ./crsconfig_params
PRCR-1035 : Failed to look up CRS resource ora.cluster_vip.type for 1
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.eons is registered
Cannot communicate with crsd

ACFS-9200: Supported
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
error: package cvuqdisk is not installed
Successfully deconfigured Oracle clusterware stack on this node

In 11gR2 onword You are using as per below the manual steps use GUI and configure and creating the OCR file through $GRID_HOME/config/config.sh
Get GUI & configure it then execute root.sh after that all node and verify it it’s all working fine.

[root@rac1 grid]#
--------------------------------------------------------------------------------------
Grid user added back the resource cluster after reconfigure finish
--------------------------------------------------------------------------------------
Look those command output match before doing the activity
$GRID_HOME/bin/crsctl stat res -t
$GRID_HOME/bin/crsctl stat res -p
$GRID_HOME/bin/crsctl query css votedisk
$GRID_HOME/bin/ocrcheck
$GRID_HOME/bin/oifcfg getif
$GRID_HOME/bin/srvctl config nodeapps -a
$GRID_HOME/bin/srvctl config scan
$GRID_HOME/bin/srvctl config asm -a
$GRID_HOME/bin/srvctl config listener -l <listener-name> -a
$DB_HOME/bin/srvctl config database -d <dbname> -a
$DB_HOME/bin/srvctl config service -d <dbname> -s <service-name> -v


[root@rac1 grid]# ./root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2016-03-17 21:29:30: Parsing the host name
2016-03-17 21:29:30: Checking for super user privileges
2016-03-17 21:29:30: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded

ASM created and started successfully.

DiskGroup OCR created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
Successful addition of voting disk 74ff31e9cb424fdbbfe9cb17227ad6a0.
Successfully replaced voting disk group with +OCR.
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   74ff31e9cb424fdbbfe9cb17227ad6a0 (ORCL:DISK1) [OCR]
Located 1 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.OCR.dg' on 'rac1'
CRS-2676: Start of 'ora.OCR.dg' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac1'
CRS-2676: Start of 'ora.registry.acfs' on 'rac1' succeeded

rac1     2016/03/17 21:46:06     /u01/app/11.2.0/grid/cdata/rac1/backup_20160317_214606.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3999 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.


[root@rac1 grid]# crsctl check has
CRS-4638: Oracle High Availability Services is online

[root@rac1 grid]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


[root@rac1 grid]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.OCR.dg
               ONLINE  ONLINE       rac1
ora.asm
               ONLINE  ONLINE       rac1                     Started
ora.eons
               ONLINE  ONLINE       rac1
ora.gsd
               OFFLINE OFFLINE      rac1
ora.net1.network
               ONLINE  ONLINE       rac1
ora.ons
               ONLINE  ONLINE       rac1
ora.registry.acfs
               ONLINE  ONLINE       rac1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        OFFLINE OFFLINE
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.scan1.vip
      1        ONLINE  ONLINE       rac1


[root@rac1 grid]#
-----------------------------------------------------------------------------------------------------------------------------------
Skip If you are using $GRID_HOME/config/config.sh if not using follow below the commands manually
Grid user added back the resource cluster after reconfigure finish
------------------------------------------------------------------------------------------------------------------------------------
Look those command output match before doing the activity
$GRID_HOME/bin/crsctl stat res -t
$GRID_HOME/bin/crsctl stat res -p
$GRID_HOME/bin/crsctl query css votedisk
$GRID_HOME/bin/ocrcheck
$GRID_HOME/bin/oifcfg getif
$GRID_HOME/bin/srvctl config nodeapps -a
$GRID_HOME/bin/srvctl config scan
$GRID_HOME/bin/srvctl config asm -a
$GRID_HOME/bin/srvctl config listener -l <listener-name> -a
$DB_HOME/bin/srvctl config database -d <dbname> -a
$DB_HOME/bin/srvctl config service -d <dbname> -s <service-name> -v

Syntax for 11g R2 RAC

srvctl add database -d db_unique_name -o ORACLE_HOME [-x node_name] [-m domain_name] [-p spfile] [-r {PRIMARY|PHYSICAL_STANDBY|LOGICAL_STANDBY|SNAPSHOT_STANDBY}] [-s start_options] [-t stop_options] [-n db_name] [-y {AUTOMATIC|MANUAL}] [-g server_pool_list] [-a "diskgroup_list"]


==>Add Resources
login as Grid User execute           => $GRID_HOME/bin/srvctl add listener
Verify Listener Owner                   => srvctl config listener -l listener -a
Listener enable                                 => srvctl enable listener -l listener -n rac1
Add Database resource                 => srvctl add database -d racdb -o /u01/app/oracle/product/11.2.0/db_1 -n racdb
Instance Mannualy add                 => srvctl add instance -d racdb -i racdb1 -n rac1
Add Database name                       => srvctl modify database -d racdb  -n racdb

Add Diskgroup                                  => srvctl modify database -d racdb -a DATA       # Below the follow
Spfile Name set of database       => srvctl modify database -d racdb -p "+DATA/racdb/spfileracdb.ora"

 # Incase any failure like “Unable to modify database because the resource for diskgroup DATA could not be found. PRCR-1001 : Resource ora.DATA.dg does not exist”
-bash-3.2$ srvctl status DISKGROUP -g DATA -a
PRCR-1001 : Resource ora.DATA.dg does not exist

-bash-3.2$ srvctl status DISKGROUP -g +DATA -a
PRCA-1000 : ASM Disk Group +DATA does not exist
PRCR-1070 : Failed to check if resource ora.+DATA.dg is registered
Invalid resource name

+asm_alert.log
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup OCR was mounted
ORACLE_BASE value has been saved for future startups
SUCCESS: ALTER DISKGROUP ALL MOUNT /* asm agent */
Sat Mar 19 21:59:15 2016
WARNING: failed to online diskgroup resource ora.OCR.dg (unable to communicate with CRSD/OHASD)
SQL> ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */
SUCCESS: ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */


Cause :

In contrast to previous version of CRS, 11gR2 ASM does not depend on CRSD anymore; rather if OCR is located on ASM, ASM will start before CRSD while clusterware is coming up, and stops after CRSD while clusterware is going down. Due to the new startup sequence, ASM can not update diskgroup resource status in CRSD either during clusterware startup or shutdown, thus above message is logged.

Solution :
The message can be ignored, the status of diskgroup and diskgroup resource will be synchronize by oraagent once clusterware is fully up.

To find out diskgroup resource status:

$GRID_HOME/bin/srvctl status diskgroup -g diskgroupname -a

To start or sync diskgroup resource:

$GRID_HOME/bin/srvctl start diskgroup -g diskgroupname



But Still above Solution did not worked  .............
So I have  dimounted and mounted the diskgroup again and then the dg resource were up.


-bash$ sqlplus sys as sysasm

SQL> alter diskgroup data dismount;
Diskgroup altered.

SQL> alter diskgroup data dismount;
alter diskgroup data dismount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15001: diskgroup "DATA" does not exist or is not mounted

SQL> alter diskgroup data mount;
Diskgroup altered.

-bash-3.2$ srvctl modify database -d racdb -a DATA


Note: For more detail about resource then follow the link http://oracledbabay.blogspot.in/2012/11/srvctl-commands-10g-rac-11g-r2-rac.html



--------------------------------------------------------------------------------------------------------------------------------------

[root@rac1 grid]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'rac1' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1'
CRS-2673: Attempting to stop 'ora.eons' on 'rac1'
CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
CRS-2677: Stop of 'ora.eons' on 'rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'rac1'
CRS-2677: Stop of 'ora.diskmon' on 'rac1' succeeded
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Stop failed, or completed with errors.


[root@rac1 grid]# crsclt start cluster –all

Check log files and verify it

….. Keep Enjoy J J J J J J J J J J J J J J J J J J


Reference:


No comments: