clusterware startup sequence

Oracle 11gR2 

ohasd -> orarootagent -> ora.cssdmonitor : Monitors CSSD and node health (along with the cssdagent). Try to restart the node if the node is unhealthy.

                                    -> ora.ctssd : Cluster Time Synchronization Services Daemon

                                    -> ora.crsd   -> oraagent  -> ora.LISTENER.lsnr

                                                                             -> ora.LISTENER_SCAN.lsnr

                                                                             -> ora.ons

                                                                             -> ora.eons

                                                                             -> ora.asm

                                                                             -> ora.DB.db

                                                         ->orarootagent -> ora.nodename.vip

                                                                                 -> ora.net1.network

                                                                                 -> ora.gns.vip

                                                                                 -> ora.gnsd

                                                                                 -> ora.SCANn.vip

         -> cssdagent -> ora.cssd : Cluster Synchronization Services

         -> oraagent -> ora.mdnsd : Used for DNS lookup
                            -> ora.evmd

                            -> ora.asmd

                            -> ora.gpnpd : Grid Plug and Play = adding a node to the cluster is easier (we need less configuration for the new node)

If a resource is written using blue & bold font => resource owned by root. The other resources are owner by oracle. (all this on UNIX environment)

When a resource is managed by root, we need to run the command crsctl asroot or oracle.

DIAG TOOL

11gR2 RAC 

Cluster Diagnostic Collection Tool

$GRID_HOME/bin/diagcollection.pl --> Script

To use cluster diagnostic collection tool helps you collect the diagnostic information for all the require components
like HOST, OS, CLUSTER, AGENT etc....


------ OPTIONS AVAILABLE ------

[root@rac1 ~]# diagcollection.pl

Production Copyright 2004, 2008, Oracle.  All rights reserved
Cluster Ready Services (CRS) diagnostic collection tool
diagcollection
    --collect
             [--crs] For collecting crs diag information
             [--adr] For collecting diag information for ADR
             [--ipd] For collecting IPD-OS data
             [--all] Default.For collecting all diag information.
             [--core] UNIX only. Package core files with CRS data
             [--afterdate] UNIX only. Collects archives from the specified date. Specify in mm/dd/yyyy format
             [--aftertime] Supported with -adr option. Collects archives after the specified time. Specify in YYYYMMDDHHMISS24 format
             [--beforetime] Supported with -adr option. Collects archives before the specified date. Specify in YYYYMMDDHHMISS24 format
             [--crshome] Argument that specifies the CRS Home location
             [--incidenttime] Collects IPD data from the specified time.  Specify in MM/DD/YYY24HH:MM:SS format
                  If not specified, IPD data generated in the past 24 hours are collected
             [--incidentduration] Collects IPD data for the duration after the specified time.  Specify in HH:MM format.
                 If not specified, all IPD data after incidenttime are collected
             NOTE:
             1. You can also do the following
                ./diagcollection.pl --collect --crs --crshome <CRS Home>

     --clean        cleans up the diagnosability
                    information gathered by this script

     --coreanalyze  UNIX only. Extracts information from core files
                    and stores it in a text file




--nocore option undocumented is to use to ignore core datafiles while collection diagnostic information using Oracle Cluster Diagnostic Tool.




Below Example.:-

[root@rac1 ~]# diagcollection.pl --collect --crs --crshome /app/gridHome
Production Copyright 2004, 2008, Oracle.  All rights reserved
Cluster Ready Services (CRS) diagnostic collection tool
The following CRS diagnostic archives will be created in the local directory.
crsData_rac1_20150314_1940.tar.gz -> logs,traces and cores from CRS home. Note: core files will be packaged only with the --core option.
ocrData_rac1_20150314_1940.tar.gz -> ocrdump, ocrcheck etc
coreData_rac1_20150314_1940.tar.gz -> contents of CRS core files in text format

osData_rac1_20150314_1940.tar.gz -> logs from Operating System
Collecting crs data
/bin/tar: log/rac1/cssd/ocssd.log: file changed as we read it
/bin/tar: log/rac1/ohasd/ohasd.log: file changed as we read it
/bin/tar: log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log: file changed as we read it
/bin/tar: log/rac1/agent/ohasd/oraagent_grid/oraagent_grid.log: file changed as we read it
/bin/tar: log/rac1/agent/crsd/oraagent_oracle/oraagent_oracle.log: file changed as we read it
Collecting OCR data
Collecting information from core files
No corefiles found
Collecting OS logs.

[root@rac1 ~]#


Note:- Check log One by one.

tail -100 $GRID_HOME/log/rac1/cssd/ocssd.log

OHASD

11gR2 RAC

How ohasd will know which disk contains voting disk? Following is the ASM disk header metadata (dumped using kfed). ohasd use the markers between vfstart & vfend.

If the markers between vfstart & vfend are 0 then disk does *NOT* contain voting disk

[root@rac1] kfed read ‘/dev/sdi’ | grep vf
kfdhdb.vfstart: 130; 0x0ec: 0×00000060
kfdhdb.vfend: 168; 0x0f0: 0x000000d0

[root@rac1] kfed read ‘/dev/sdg’ | grep vf
kfdhdb.vfstart: 0 ; 0x0ec: 0×00000000
kfdhdb.vfend: 0 ; 0x0f0: 0×00000000

[root@rac1] kfed read ‘/dev/sdh’ | grep vf
kfdhdb.vfstart: 0 ; 0x0ec: 0×00000000
kfdhdb.vfend: 0 ; 0x0f0: 0×00000000

So from above output, disk ‘/dev/sdi’ contains voting disk.

Conclusion: The way to handle Voting Disk and OCR has changed significantly – they can be kept inside of an ASM Diskgroup especially.

oifcfg

11gR2 RAC - oifcfg (Oracle Interface Configuration tool)


Utility is usefull while managing the hostname,IP address changes during datacenter migrations, or network
reconfiguration that require chaing IP address to cluster member. (Return value Global Public and global cluster_interconnect)

[root@rac1 peer]# oifcfg

Name:
        oifcfg - Oracle Interface Configuration Tool.

Usage:  oifcfg iflist [-p [-n]]
        oifcfg setif {-node <nodename> | -global} {<if_name>/<subnet>:<if_type>}...
        oifcfg getif [-node <nodename> | -global] [ -if <if_name>[/<subnet>] [-type <if_type>] ]
        oifcfg delif [{-node <nodename> | -global} [<if_name>[/<subnet>]]]
        oifcfg [-help]

        <nodename> - name of the host, as known to a communications network
        <if_name>  - name by which the interface is configured in the system
        <subnet>   - subnet address of the interface
        <if_type>  - type of the interface { cluster_interconnect | public }


[root@rac1 peer]# oifcfg iflist
eth0  192.168.0.0
eth1  192.168.1.0
eth2  192.168.2.0

[root@rac1 peer]# oifcfg getif
eth0  192.168.0.0  global  public
eth1  192.168.1.0  global  cluster_interconnect


[root@rac1 peer]# oifcfg delif -global 

[root@rac1 peer]# oifcfg setif -global <interface name>/<subnet>:public

[root@rac1 peer]# oifcfg setif -global <interface name>/<subnet>:cluster_interconnect

ORA-07445: exception encountered: core dump SIGSEGV

May 29, 2014 10:07:02 AM	~ORA-07445: exception encountered: core dump [_intel_fast_memcmp()+72] [SIGSEGV] [Address not mapped to object] [0x0] [] []
Mar 17, 2015 2:56:05 PM	~ORA-07445: exception encountered: core dump [_intel_fast_memcmp()+38] [SIGSEGV] [Address not mapped to object] [0x0] [] []

*ORA-600/ORA-7445 Error Look-up Tool 

ID 153788.1

ID 362953.1

Problem Symptoms

Whenever I check my alert log file I got the text as
Errors in file /var/opt/oracle/admin/udump/orastdby_ora_31795.trc
ORA-07445: exception encountered: core dump [_intel_fast_memcmp()+38] [SIGSEGV] [Address not mapped to object] [0x0] [] []
After checking the trace file I got,
Call Stack Trace shows: 
ksedst ksedmp ssexhd intel_fast_memcmp

Recent Changes of The Database
We have changes the CURSOR_SHARING parameter to SIMILAR from EXACT.

Cause of The Problem
This is oracle bug. Bug number 4456646. When the cursor sharing parameter is not set to EXACT this bug may fire. This occur while literal replacement when there are empty string literals in use. This bug happened in oracle version 10.2.0.1. In our environment it was RHL linux 32 bit production server.

Solution of The Problem 
Way 1: 
As a workaround disable literal replacement. You can do it by,
ALTER SYSTEM SET cursor_sharing='EXACT' SCOPE=both;
if you use spfile.

Way 2: 
This bug is fixed in 10.2.0.2 patchset. So, apply 10.2.0.2 patchset in order to fix the prolem.

Way 3: 
Apply One-off patch. Download one-off patch 4456646 from metalink and apply if that is avialable for your OS. For Linux x86 and linux x86-64 only available currently.

ABOUT THE CURSOR & CURSOR SHARING 

Cursor is a type of memory where store sql info,sql text,execution plans at all on shared pool -> library cache. And Bind variables usage the shared pool.

RAC BG PROCESS

RAC Background Processes

                Lock Monitor Processes (LMON)

                 Lock Monitor Services (LMS)

                 Lock Monitor Daemon Process ( LMD)

                 LCKn ( Lock Process)

                 DIAG (Diagnostic Daemon)

1. Lock Monitor Processes (LMON)

     LMON Maintains GCS memory structures.

     LMON handles the abnormal termination of processes and instances.

     LMON deals with Reconfiguration of locks & resources when an instance joins or leaves the cluster (During reconfiguration LMON generate the trace files)

     LMON is responsible for executing dynamic lock remastering every 10 mins ( Only in 10g R2 & later versions).

     LMON Processes manages the global locks & resources.

     It monitors all instances in cluster, primary for dictionary cache locks, library cache locks & deadlocks on deadlock sensitive on enqueue & resources.

     LMON also provides cluster group services.

     It is also called as  GES [Global Enqueue Service] monitor.

2. Lock Monitor Services (LMS)

Of all Background processes, LMS is the most active

It consumes significant amount of CPU time. (10g R2: Ensure that LMS process does not encounter the CPU starvation).

Its primary job is to transport blocks across the nodes for cache-fusion requests.

When there is a consistent-read request, the LMS process rolls back the block, makes a Consistent-Read image of the block and then ships this block across the HSI (High Speed Interconnect) to the process requesting from a remote node.

LMS also constantly checks with the LMD background process (or our GES process) to get the lock requests placed by the LMD process.

Each node will have 2 or more LMS processes.

GCS_SERVER_PROCESSES parameter denotes the number of LMS processes specified in init. ora file.

The above parameter value is set based on number of cpu’s (MIN(CPU_COUNT/2,2))

In10gR2, for a single CPU instance, only one LMS processes started.

It is advised to Increase the parameter value, if global cache activity is very high.

Also called the GCS (Global Cache Services) processes.

Internal View: X$KJMSDP

3. Lock Monitor Daemon Process ( LMDn)

LMD process performs lock and deadlock detection globally.

It also monitors for lock conversion timeouts.

Sometimes, it is also referred to as the GES (Global Enqueue Service) daemon since its job is to manage the global enqueue and global resource access.

LMD process also handles deadlock detection and remote enqueue requests.

Remote resource requests are the requests originating from another instance.

Internal View: X$KJMDDP

4. LCKn (Lock Process)

It manages instance resource requests & cross instance calls for shared resources.

During instance recovery, it builds a list of invalid lock elements and validates lock elements.

5. DIAG (Diagnostic Daemon)

A new background process introduced in Oracle 10g featuring new enhanced diagnosability framework

Regularly monitors the health of the instance.

Also checks instance hangs & deadlocks.

It captures the vital diagnostics data for instance & process failures.

QUICK REFERENCE:

RAC – Real Application Cluster

GCS – Global Cache Services

GES - Global Enqueue Service

LMD – Lock Monitor Deamon

LMON – Lock Monitor Processes

LMS –  Lock Monitor Services

Reference :- BG PROCESS LINK

Cache fusion Recovery

RAC 11gR2  

Internal Cache Fusion Recovery.

1. Instance crash

2. GRD Lock Freez

3. GES reconfigure

4. DLM recovery Perform

5. GCS (PCM remastered)

6. Pending Writes and Notification are processed

7. SINGLE PASS RECOVERY FOLLOW
Instance recovery lock is acquired by SMON.
Recovery set prepaired and built.SMON PGA allocate.
SMON acquires locks on buffer that need recovery.

8. TWO PASS RECOVERY
DB is partially available.
Blocks are made available as they are recovered.
Lock Release by SMON.Recovery is complete.
System is available.

Cluster Re-mastering and Fast re-configuring

RAC 11g R2

_gcs_file_to_lock ------ old lazy remastering (re-configure time locks/delete (Join/leave the node) the DLM in all instances then configure but in new Oracle version 11g below parameter.


_gcs_fast_reconfig, _lm_master_weight  Main Parameter to maintain.(GRD TC in 11g and JOIN/LEAVE node to re-distribute the resources.

The most common reasons for Re-mastering any Re-configuration.

1. when a node joins or leaves a cluster.

2. when an instance death is detected.Every instance updates the controlfile with a 

     heartbeat through its Checkpoint (CKPT) process. If heartbeat information is not 

     present for x amount of time (normally 30 seconds in UNIX variants), the instance is 

     considered to be dead and the Instance Membership Recovery (IMR) 

     process initiates reconfiguration.

3. Reconfiguration event is due to a communication failure.Communication channels are

     established between the Oracle processes across the nodes.

(GRD / CLUSTER Re-mastering and Re-configuring) 

GRD / CLUSTER re-mastering and re-configuring dynamically by LMON

Reconfigure process triggers IMR (Instance Membership Recovery), and Seven Steps process to ensures complete reconfiguration: -

1. The Name Service is Frozen.
2. IDLM is frozen.Lock Database is Frozen.
3. Determination of membership and validation and IMR occur.
4. Bitmap rebuild take place (GRD), including instance name.
5. Dead all instance
6. Unfreeze and release the name of services for use.
7. Hand over reconfiguration to GES/GCS.


IMR  starts and node eviction take place, let's look at the corresponding Messgaes in the ALERT.LOG and LMON trace files to get a better Picture.


(IMR is a Deep Level of Node Evection Process) Read IMR (instance Membership recovery) click : here 

Past Image

11gR2 RAC

PI - Past image  is a copy of a globally dirty block and is maintained in the database buffer cache.It can be createdand saved when dirty block is shipped across to another instance after setting the resource role and global(if it was not already set).

PI must be maintainde by an instance until it or a later version of block is written to disk.
GCS is responsible for informing an instance that its PI is no longer needed after another instance writes a newer version of the same block.

Please are discarded when GCS posts all the holding instances that a new and consistent version of that particular block is now on disk.


check logs and session logs into shows this type

st: PI md: NULL tch: 1 le: 0x7bfa6000                        -- Past Image

Note:- In simple Words It's  Eleminates Write/Write contentions

OMOTION

RAC 11g R2 11.2.0.2 -- OMOTION ---- Let’s try to get the online relocation done by ourselves and see what happens to the existing and on to the target instance.

After installation of the Oracle 11gr2 Grid Infrastructure and a "software only installation" of 11gr2 RAC, I installed patch 9004119 .

If you want to read about this article with all explanation read below the links:

Installing One Rac Node

Installing & RAC One Node handling of Planned Downtime

Below the steps follow After patching on 11.2.0.1.0 :

[oracle@host01 ~]$ srvctl config database -d orcl

Database unique name: orcl

Database name: orcl

Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1

Oracle user: oracle

Spfile: +FRA/orcl/spfileorcl.ora

Domain:

Start options: open

Stop options: immediate

Database role: PRIMARY

Management policy: AUTOMATIC

Server pools: orcl

Database instances:

Disk Groups: FRA

Mount point paths:

Services: srvc1

Type: RAC One Node

Online relocation timeout: 30

Instance name prefix: orcl

Candidate servers: host01,host02,host03

Database is administrator managed

Before we start the migration, let’s check the status of the database and it’s instance right now.

[oracle@host01 ~]$ srvctl status database -d orcl

 Instance orcl_1 is running on node host01

 Online relocation: INACTIVE

So what we have here is a RAC One Node database with the SID orcl and it is running on the node HOST01with the instance orcl_1.  So now, we shall try to relocate the instance from this node to the target nodeHOST02.  Also it’s shown that the online relocation is not active at the moment.

It’s important to mention that with the version 11201, this task was done by a utility OMOTION but from 11202 onwards, the use of this utility is not required. The release of the software used for this demo was 11203 so obviously, the utility wasn’t required.

The conversion is done using the command SRVCTL RELOCATE DATABASE in which we are going to pass the name of the target node and the option to be in verbose mode for the output. Below is the output of this command:

[oracle@host01 ~]$ srvctl relocate database -d orcl -n host02 -w 30 -v

Configuration updated to two instances

Instance orcl_2 started

Services relocated

Waiting for up to 30 minutes for instance orcl_1 to stop ...

Instance orcl_1 stopped

Configuration updated to one instance

And from another session, we can see that the migration is going on.

[oracle@host01 ~]$ srvctl status database -d orcl

Instance orcl_2 is running on node host02

Online relocation: ACTIVE

Source instance: orcl_1 on host03

Destination instance: orcl_2 on host02

We can see that the second instance has come up and the relocation status is also shown as ACTIVE which means that the relocation is going on. We would need to run the command couple of times as it may take longer for the instance to crash.

[oracle@host01 ~]$ srvctl status database -d orcl

Instance orcl_2 is running on node host02

Online relocation: ACTIVE

Source instance: orcl_1 on host03

Destination instance: orcl_2 on host02

[oracle@host01 ~]$ srvctl status database -d orcl

Instance orcl_2 is running on node host02

Online relocation: ACTIVE

Source instance: orcl_1 on host03

Destination instance: orcl_2 on host02

Finally when the relocation would be over, this would be shown as the output,

[oracle@host01 ~]$ srvctl status database -d orcl

Instance orcl_2 is running on node host02

Online relocation: INACTIVE [oracle@host01 ~]$

As we can see, one the relocation is complete only the instance “orcl_2” is going to be working and the status of ONLINE RELOCATION is going to be completed.

Application Context

RAC 11gR2 

Application Context for more usefull and simple Its new Feature in ORACLE RAC 11G

DBMS_APPLICATION_INFO to use and diagnose where the application creating a problem.And why query hang else what ever follow steps .



exec dbms_application_info.set_module(module_name => 'UPDATE RECORD', action_name => 'UPDATE TAB1');
exec DBMS_APPLICATION_INFO.SET_CLIENT_INFO(CLIENT_INFO => 'SUNRAY');
UPDATE TAB1 SET COL1='UPDATED';





create table TAB1 (COL1 varchar2(10)); --- Session 1

insert into tab1 values ('FIRST'); --- Session 1

update tab1 set col1='SECOND'; --- Session 2




Now the Session 2 is hang and going to waiting forever because the same row is beaing updated by another session i.e. Session2 and not commit but session 1
Because we set application context using the DBMS_APPLICATION_INFO package, we can easily identify the problematic code location via the simple query


select sid,event,module,action,client_info
from v$session where state='WAITING' and
WAIT_CLASS='Application';



-------------------

Else old method ----  v$session, v$process, v$sesison_wait, v$system_wait, dba_objects.

FLASHBACK Enable

11gR2 RAC

1. Verify Archive log and Flashback is enable.

archive log list

select FLASHBACK_ON from v$database;

2. cluster_database parameter false

alter system set cluster_database=false scope=false sid='racdb1';


3. Setting up FLASHBACK Area.

alter system set DB_RECOVERY_FILE_DEST_SIZE=200M scope=spfile;
alter system set DB_RECOVERY_FILE_DEST='+FRA/flashback/' scope=spfile;


4. shutdown all instance.

srvctl stop database -d RACDB

5. Mount database.

startup mlount

6. Enable Flashback.

alter database flashback on;


7. set parameter cluster_database true.

alter system set cluster_database=true scope=spfile side='racdb1';

8. shutdown instance.

shutdown ;

9. start all instances.

srvctl start database -d racdb

Archive Logs Enable

11gR2 RAC

1. Set clusetr_database to false for instance:

alter system set cluster_database=false scope=spfile sid='racdb1';

2. Shutdown all instance

srvctl stop database -d racdb

3. Mount Database

startup mount

4. Enable Archiving 

alter database archivelog;

5. Set the parameter cluster_database to true for the instance "racdb1".

alter system set cluster_database=true scope=spfile sid='racdb1';

6. Shutdown the local instance

shutdown

7. Bring up all instance

srvctl start database -d racdb

Once in archive log mode,each instance can archive redo logs automatically;