Friday 17 June 2016

OCR/VOTING DISK RECOVERY WITHOUT ANY BACKUP OF OCR/OLR/VOTING DISK - PART II


2nd Method Now testing is different size of disk in GB & different mount device disk (In my case /dev/sdb1 to DISK1 to /dev/sdf1 to DISK1 is up the OCR no backup (why same disk why another ASM disk is used.?)


[root@rac1 install]# ./rootcrs.pl -deconfig -force -verbose
2016-03-19 21:39:30: Parsing the host name
2016-03-19 21:39:30: Checking for super user privileges
2016-03-19 21:39:30: User has super user privileges
Using configuration parameter file: ./crsconfig_params
PRCR-1035 : Failed to look up CRS resource ora.cluster_vip.type for 1
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.eons is registered
Cannot communicate with crsd

ACFS-9200: Supported
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
2560+0 records in
2560+0 records out
10485760 bytes (10 MB) copied, 0.113001 seconds, 92.8 MB/s
Successfully deconfigured Oracle clusterware stack on this node


[root@rac1 install]# ocrcheck
PROT-601: Failed to initialize ocrcheck
PROC-33: Oracle Cluster Registry is not configured Storage layer error [Error opening ocr.loc file. No such file or directory] [2]


[root@rac1 install]# crsctl check crs
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Check failed, or completed with errors.

[root@rac1 install]# crsctl check has
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Check failed, or completed with errors.

[root@rac1 install]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4

[root@rac1 install]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

[root@rac1 install]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4

[root@rac1 install]# oracleasm deletedisk DISK1
Clearing disk header: done
Dropping disk: done

# New device of disk mount
[root@rac1 install]# oracleasm createdisk DISK1 /dev/sdf1
Writing disk header: done
Instantiating disk: done

[root@rac1 install]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4


 [root@rac1 install]# ./rootcrs.pl -deconfig -force -verbose
2016-03-19 21:48:15: Parsing the host name
2016-03-19 21:48:15: Checking for super user privileges
2016-03-19 21:48:15: User has super user privileges
Using configuration parameter file: ./crsconfig_params
Usage: srvctl <command> <object> [<options>]
    commands: enable|disable|start|stop|status|add|remove|modify|getenv|setenv|unsetenv|config
    objects: database|service|asm|diskgroup|listener|home|ons|eons
For detailed help on each command and object and its options use:
  srvctl <command> -h or
  srvctl <command> <object> -h
PRKO-2012 : nodeapps object is not supported in Oracle Restart
ACFS-9200: Supported
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
You must kill crs processes or reboot the system to properly
cleanup the processes started by Oracle clusterware
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
/bin/dd: opening `': No such file or directory
error: package cvuqdisk is not installed
Successfully deconfigured Oracle clusterware stack on this node
[root@rac1 install]# more /etc/oracle/olr.loc
/etc/oracle/olr.loc: No such file or directory



[root@rac1 grid]# ./root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2016-03-19 21:51:23: Parsing the host name
2016-03-19 21:51:23: Checking for super user privileges
2016-03-19 21:51:23: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded

ASM created and started successfully.

DiskGroup OCR created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
Successful addition of voting disk 6c9ced874d6a4f52bfcea75dc93baa63.
Successfully replaced voting disk group with +OCR.
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--     -----       -----------------                        --------- ---------
 1. ONLINE   6c9ced874d6a4f52bfcea75dc93baa63 (ORCL:DISK1) [OCR]
Located 1 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.OCR.dg' on 'rac1'
CRS-2676: Start of 'ora.OCR.dg' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac1'
CRS-2676: Start of 'ora.registry.acfs' on 'rac1' succeeded

rac1     2016/03/19 22:02:10     /u01/app/11.2.0/grid/cdata/rac1/backup_20160319_220210.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3999 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.



==>Add Resources
login as Grid User execute           => $GRID_HOME/bin/srvctl add listener
Verify Listener Owner                   => srvctl config listener -l listener -a
Listener enable                                 => srvctl enable listener -l listener -n rac1
Add Database resource                 => srvctl add database -d racdb -o /u01/app/oracle/product/11.2.0/db_1 -n racdb
Instance Mannualy add                 => s srvctl add instance -d racdb -i racdb1 -n rac1
Add Database name                       => srvctl modify database -d racdb  -n racdb

Add Diskgroup                                  => srvctl modify database -d racdb -a DATA       # Below the follow
Spfile Name set of database       => srvctl modify database -d racdb -p "+DATA/racdb/spfileracdb.ora"



---------------------------------------------------------------------------------------------------
Getting Error: At the time of DATA diskgroup add into srvctl database config
---------------------------------------------------------------------------------------------------

-bash-3.2$ srvctl modify database -d racdb -a DATA
Unable to modify database because the resource for diskgroup DATA could not be found. PRCR-1001 : Resource ora.DATA.dg does not exist

-bash-3.2$ srvctl status DISKGROUP -g DATA -a
PRCR-1001 : Resource ora.DATA.dg does not exist

-bash-3.2$ srvctl status DISKGROUP -g +DATA -a
PRCA-1000 : ASM Disk Group +DATA does not exist
PRCR-1070 : Failed to check if resource ora.+DATA.dg is registered
Invalid resource name

-----------------------
  +asm_alert.log
-----------------------
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup OCR was mounted
ORACLE_BASE value has been saved for future startups
SUCCESS: ALTER DISKGROUP ALL MOUNT /* asm agent */
Sat Mar 19 21:59:15 2016
WARNING: failed to online diskgroup resource ora.OCR.dg (unable to communicate with CRSD/OHASD)
SQL> ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */
SUCCESS: ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */
Sat Mar 19 21:59:17 2016
Starting background process ASMB

-------------
  Cause :
-------------
In contrast to previous version of CRS, 11gR2 ASM does not depend on CRSD anymore; rather if OCR is located on ASM, ASM will start before CRSD while clusterware is coming up, and stops after CRSD while clusterware is going down. Due to the new startup sequence, ASM can not update diskgroup resource status in CRSD either during clusterware startup or shutdown, thus above message is logged.

-----------------
Solution :
-----------------
The message can be ignored, the status of diskgroup and diskgroup resource will be synchronize by oraagent once clusterware is fully up.

To find out diskgroup resource status:

$GRID_HOME/bin/srvctl status diskgroup -g diskgroupname -a

To start or sync diskgroup resource:

$GRID_HOME/bin/srvctl start diskgroup -g diskgroupname


But Still above Solution did not worked  .............
So I have  dimounted and mounted the diskgroup again and then the dg resource were up.


-bash$ sqlplus sys as sysasm

SQL> alter diskgroup data dismount;
Diskgroup altered.

SQL> alter diskgroup data dismount;
alter diskgroup data dismount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15001: diskgroup "DATA" does not exist or is not mounted


SQL> alter diskgroup data mount;
Diskgroup altered.


-bash-3.2$ srvctl modify database -d racdb -a DATA



-bash-3.2$ srvctl add database -d racdb -o /u01/app/oracle/product/11.2.0/db_1 -n racdb
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Getting database start issue its srvctl configuration issue
Reference : sqlplus Reports "Connected to an idle instance" When Connecting to Database That's Started by srvctl/Clusterware (Doc ID 1566828.1)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Problem :
when i check the database status through
crs_stat -t  & ps -ef|grep smon : It show database online then i was login to database through "oracle" unix user an connect the database instance sqlplus.
                                                     It show's database instance is ideal and apply STARTUP -> database on exclusive mode, How's that possible.
                                                     Vice versa condition apply.
                                                     I'm stop this database through SRVCTL else killing SMON process after that  login to database through "oracle" unix user an connect the database instance sqlplus.
                                                     It show's database instance is ideal and apply STARTUP database, Now database is OPEN mode and "crs_stat -t " shows online. How's that possible.
                                                    Then stop the database through SRVCTL i'm getting below error. This error i'm getting after configure the database on SRVCTL manually. It's silly mistake by my side.
                                solution below here:
---------
Error:
---------
[root@rac1 grid]# srvctl stop database -d racdb
PRCD-1124 : Failed to stop database racdb and its services
PRCR-1065 : Failed to stop resource (((((NAME STARTS_WITH ora.racdb.) && (NAME ENDS_WITH .svc)) && (TYPE == ora.service.type)) && ((STATE != OFFLINE) || (TARGET != OFFLINE))) || (((NAME == ora.racdb.db) && (TYPE == ora.database.type)) && (STATE != OFFLINE)))
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0

ORA-01012: not logged on
CRS-2675: Stop of 'ora.racdb.db' on 'rac1' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0

ORA-01012: not logged on
CRS-2675: Stop of 'ora.racdb.db' on 'rac1' failed

-----------
CAUSE:
------------
Environment variable ORACLE_HOME is incorrectly set:
$ env | grep ORACLE_HOME
ORACLE_HOME=/app/oracle/product/11.2.0/dbhome_1/
    
---------------
SOLUTION:
---------------
Removing trailing "/" from environment variable ORACLE_HOME:
$ env | grep ORACLE_HOME
ORACLE_HOME=/app/oracle/product/11.2.0/dbhome_1


-bash-3.2$  srvctl modify database -d racdb -a DATA      
-bash-3.2$  srvctl modify database -d racdb -p "+DATA/racdb/spfileracdb.ora"
-bash-3.2$ srvctl config database –d racdb -a
-bash-3.2$ crs_stat -t
-bash-3.2$ sqlplus sys as sysdba              # if it shows again ideal instance kill the SMON process & start database again srvctl start database –d racdb



……. Now Enjoying With 2nd method J J J J J J J J J J J J  J J J J J



Reference:

    1st Method
     3rd Method

No comments: