Friday, 17 June 2016

OCR/VOTING DISK RECOVERY WITHOUT ANY BACKUP OF OCR/OLR/VOTING DISK - PART III


3rd Method Different disk of OCR & VOTEDISK then doing this activity again.

Note: Same name of oracleasm disk possible Only . other asm Disk name this secenario Not Possible
Now getting start with 3r method in my case OCR (DISK1 : OCR) and Voting (DISK2: VOTE) Disk is Different mount point and
Both Disk are failure in my case.
Then 2 more disk add and create new diskgroup (DISK6: DEMO_OCR) (DISK7: DEMO_VOTE) and try to mounting.

https://oraclehandson.wordpress.com/2010/08/05/recovering-lost-ocr-voting-disk-in-11gr2/
-------------------------------------------------------------------------------------------------------------------------------------------------------------------

[root@rac1 ~]# crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DATA.dg    ora....up.type OFFLINE   OFFLINE
ora...._OCR.dg ora....up.type ONLINE    ONLINE    rac1
ora....VOTE.dg ora....up.type ONLINE    ONLINE    rac1
ora.FRA.dg     ora....up.type OFFLINE   OFFLINE
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac1
ora.OCR.dg     ora....up.type ONLINE    ONLINE    rac1
ora.OCR_1.dg   ora....up.type OFFLINE   OFFLINE
ora.VOTE.dg    ora....up.type ONLINE    ONLINE    rac1
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1
ora.eons       ora.eons.type  ONLINE    ONLINE    rac1
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    rac1
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1
ora....SM1.asm application    ONLINE    ONLINE    rac1
ora....C1.lsnr application    ONLINE    ONLINE    rac1
ora.rac1.gsd   application    OFFLINE   OFFLINE
ora.rac1.ons   application    ONLINE    ONLINE    rac1
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1
ora.racdb.db   ora....se.type OFFLINE   OFFLINE
ora....ry.acfs ora....fs.type ONLINE    ONLINE    rac1
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac1


[root@rac1 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   3de994e77be74f60bf4ad8cd59eb79ff (ORCL:DISK2) [VOTE]
Located 1 voting disk(s).


[root@rac1 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   3de994e77be74f60bf4ad8cd59eb79ff (ORCL:DISK2) [VOTE]
Located 1 voting disk(s).


[root@rac1 ~]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.DEMO_OCR.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.DEMO_VOTE.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.VOTE.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-4549: Unexpected disconnect while executing shutdown request.
CRS-2675: Stop of 'ora.crsd' on 'rac1' failed
CRS-2679: Attempting to clean 'ora.crsd' on 'rac1'
CRS-4548: Unable to connect to CRSD
CRS-2678: 'ora.crsd' on 'rac1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has failed
CRS-4687: Shutdown command has completed with error(s).
CRS-4000: Command Stop failed, or completed with errors.

[root@rac1 ~]#

[root@rac1 ~]# crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.


[root@rac1 ~]# crsctl check has
CRS-4638: Oracle High Availability Services is online


[root@rac1 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


[root@rac1 disks]#

[root@rac1 disks]# ps -ef|grep smon
oracle    5068     1  0 18:32 ?        00:00:00 asm_smon_+ASM1
root     10735  4371  0 20:03 pts/1    00:00:00 grep smon
oracle   10895     1  0 19:42 ?        00:00:01 ora_smon_racdb1

[root@rac1 disks]# su - oracle

-bash-3.2$ db_env

-bash-3.2$ sqlplus sys as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Thu Mar 17 20:04:35 2016
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Enter password:
Connected.
SQL> !
[oracle@rac1 ~]$ ps -ef|grep smon
oracle    5068     1  0 18:32 ?        00:00:00 asm_smon_+ASM1
oracle   10895     1  0 19:42 ?        00:00:02 ora_smon_racdb1
oracle   12337 12195  0 20:05 pts/1    00:00:00 grep smon

SQL> shutdown abort
ORACLE instance shut down.
SQL>

root# reboot



[root@rac1 install]# ./rootcrs.pl -deconfig -force -verbose
2016-03-22 21:44:01: Parsing the host name
2016-03-22 21:44:01: Checking for super user privileges
2016-03-22 21:44:01: User has super user privileges
Using configuration parameter file: ./crsconfig_params
PRCR-1035 : Failed to look up CRS resource ora.cluster_vip.type for 1
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.eons is registered
Cannot communicate with crsd

ACFS-9200: Supported
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-4548: Unable to connect to CRSD
CRS-2675: Stop of 'ora.crsd' on 'rac1' failed
CRS-2679: Attempting to clean 'ora.crsd' on 'rac1'
CRS-4548: Unable to connect to CRSD
CRS-2678: 'ora.crsd' on 'rac1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has failed
CRS-4687: Shutdown command has completed with error(s).
CRS-4000: Command Stop failed, or completed with errors.
You must kill crs processes or reboot the system to properly
cleanup the processes started by Oracle clusterware
Successfully deconfigured Oracle clusterware stack on this node

--------------------------------------------------------------------------------------------------------------
$GRID_HOME/crs/install/crsconfig_params.
Just in case, we want to change the location, we can edit following variables
--------------------------------------------------------------------------------------------------------------
OCR_LOCATIONS=NO_VAL
VOTING_DISKS=NO_VAL

## In my case set DISKGROUP.
OCR_LOCATIONS=DEMO_OCR
VOTING_DISKS=DEMO_VOTE

(NOTE: If ASM not Up and not getting any diskgroup then giving you these parameter after setting up all replace with asm)
As above the values of parameter which i'll show in my case if any error then follow below the method.

CRS_VOTING_DISKS=/tmp/vote/vote.dat
CRS_OCR_LOCATIONS=/tmp/ocr/ocr.dat



[root@rac1 grid]# ./root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2016-03-22 22:03:36: Parsing the host name
2016-03-22 22:03:36: Checking for super user privileges
2016-03-22 22:03:36: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded

DiskGroup OCR creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification 'ORCL:DISK1' matches no disks


Configuration of ASM failed, see logs for details
Did not succssfully configure and start ASM
CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
CRS-4000: Command Stop failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
Stop of resource "ora.crsd -init" failed
Failed to stop CRSD
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
Initial cluster configuration failed.  See /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_rac1.log for details



[root@rac1 install]# ./rootcrs.pl -deconfig -force -verbose

---------------------------------------------------------------------------------------------------------------------------------------------------------------
Edit crsconfig_params parameter because in my case ASM_DISK_GROUP=OCR that will change to DEMO_OCR
ASM_DISKS=ORCL:DISK1 to DISK6
---------------------------------------------------------------------------------------------------------------------------------------------------------------
ASM_DISK_GROUP=DEMO_OCRL
ASM_DISKS=ORCL:DISK6


--------- FAILED AGAIN

[root@rac1 grid]# ./root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2016-03-22 22:21:37: Parsing the host name
2016-03-22 22:21:37: Checking for super user privileges
2016-03-22 22:21:37: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded

Disk Group DEMO_OCR already exists. Cannot be created again

Configuration of ASM failed, see logs for details
Did not succssfully configure and start ASM
CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
CRS-4000: Command Stop failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
Stop of resource "ora.crsd -init" failed
Failed to stop CRSD
CRS-2500: Cannot stop resource 'ora.asm' as it is not running
CRS-4000: Command Stop failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.asm -init
Stop of resource "ora.asm -init" failed
Failed to stop ASM
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
Initial cluster configuration failed.  See /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_rac1.log for details


## Again change the parameter because ASM is not coming up and no any diskgroup find.
CRS_VOTING_DISKS=/tmp/vote/vote.dat
CRS_OCR_LOCATIONS=/tmp/ocr/ocr.dat


./root.sh
Failed again
-----------------------------------------------------------------------------------------------------------------------------------------------------------
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded

DiskGroup OCR creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification 'ORCL:DISK1' matches no disks


Configuration of ASM failed, see logs for details
Did not succssfully configure and start ASM
CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
CRS-4000: Command Stop failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
Stop of resource "ora.crsd -init" failed
Failed to stop CRSD
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
Initial cluster configuration failed.  See /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_rac1.log for details



./rootcrs.pl -deconfig -force -verbose
./roothas.pl -delete -force -verbose



2016-03-23 18:41:54: Configuring ASM via ASMCA
2016-03-23 18:41:55: Executing as oracle: /u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList ORCL:DISK1 -redunda
ncy EXTERNAL -configureLocalASM
2016-03-23 18:41:55: Running as user oracle: /u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList ORCL:DISK1 -redu
ndancy EXTERNAL -configureLocalASM
2016-03-23 18:41:55:   Invoking "/u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList ORCL:DISK1 -redundancy EXTER
NAL -configureLocalASM" as user "oracle"
2016-03-23 18:45:31: Configuration of ASM failed, see logs for details
2016-03-23 18:45:31: Did not succssfully configure and start ASM
2016-03-23 18:45:31: Exiting exclusive mode
2016-03-23 18:45:32: Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
2016-03-23 18:45:32: Stop of resource "ora.crsd -init" failed
2016-03-23 18:45:32: Failed to stop CRSD
2016-03-23 18:46:27: Initial cluster configuration failed.  See /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_rac1.log for details



Status: Total OCR Failure.


Reference: 

    1st Method
    2nd Method

No comments: