ORA-12537: TNS:connection closed - Oracle RAC 11g and above

 The client was getting Below error while connecting to the RAC database

sqlplus test@RACDB_1

SQL*Plus: Release 11.2.0.4.0 - Production on Tue Sep 22 11:34:07 2020

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

ERROR:

ORA-12537: TNS:connection closed

Issue : 

The Oracle Binary permissions got changed and were not allowing the connections

Current Permissions under Oracle Home

[oracle@Node1 ~]$ ls -lrt $ORACLE_HOME/bin/oracle

-rwxr-sr-x 1 oracle asmadmin 243043788 Jun  2 01:12 oracle

It should be set to 6751 and should look like "-rwsr-s--x"

But doing chmod 6751 on oracle binary was not setting the correct permissions

[oracle@Node1 bin]$ chmod 6751 oracle

[oracle@Node1 bin]$ ls -lrt $ORACLE_HOME/bin/oracle

-rwsr-x--x 1 oracle asmadmin 243043788 Jun  2 01:12 oracle

Running below as RDBMS database Owner user helped, in this case, "oracle" user

1) Stop the database instance where the permissions got changed

 srvctl stop instance -d RACDB -i RACDB1

 2) Run as Oracle Database owner, in this case its Oracle OS user.  

[oracle@Node1 ~]$ $GRID_HOME/bin/setasmgidwrap o=$ORACLE_HOME/bin/oracle

3) The permissions got changed and resolved the connection issues

[oracle@Node1 ~]$ ls -lrt $ORACLE_HOME/bin/oracle

-rwsr-s--x 1 oracle asmadmin 243043788 Jun  2 01:12 oracle

Hope this resolves your issue.. 

exadbcpatchmulti : Conflicts and ERROR: apply_async blocked on node

 Now a days i'm facing multiple issues on ExaCM boxes for patching on database nodes.

Sharing one scenario, Get request from customer apply APRIL_PSU2021 on ExaCM db boxes

ENV:
DB: 11204 version
GRID have 19c with APRIL2021 PSU.


*: Issue comes while running : exadbcpatchmulti -precheck_async 32537095 , failed with conflicts.
 Get details from exadbcpatch log / conflict log to identify more details.

*: How to resolve conflicts ?
Share the all details and conflict with oracle, They provide me the solution below
Edit /var/opt/oracle/exapatch/exadbcpatch.cfg - #For this file mostly root ownership, need to take help from GRID owner team or any other script available to update cfg file.

add the Cloud patch which needs to be rolled back as in below.
rollback_needed_before_apply="32328629::normal"

Refer Doc:
Exadbcpatchmulti patch conflict on cloud MLR "ERROR: precheck failed in conflict check" (Doc ID 2492936.1)

*: Now Pre-checks successfully done without any issue and move forward to apply the patch on next step.

*: Now facing the issue again : Node_1 successfully applied the patch but another issue occur on Node_2 below the details here.

++++++Error getting when apply patch: On Node_1 successfully done but having issue on Node_2
INFO: running on the node Xxx3a02c02vm01-adm
INFO: check for this action apply_async
INFO: check for this action rollback_async
INFO: Successfully released ohome lock. Proceeding to release local provisioning lock
INFO: Successfully released local provisioning lock
ERROR: apply_async blocked on node Xxx3a02c02vm01-adm: as Exadbcpatch operation already running on requested home
ERROR: apply_async blocked on node Xxx3a02c02vm01-adm: as Exadbcpatch operation already running on requested home

++++++Solution
execute on all the nodes then run exadbcpatchmulti again.
mv /var/opt/oracle/cstate.xml /var/opt/oracle/cstate.xml_old

Refer Doc:
Exadbcpatchmulti fails with Apply_async Blocked On Node X As Another Instance Of Exadbcpatch Running (Doc ID 2530468.1)

*: Now run exadbcpatchmulti  again and successfully applied the patch on ExaCM dbs.

ORA-20001: Latest xml inventory is not loaded into table

Symptoms: As i faced during execute datapatch verbose 19c.
During execution of datapatch or DBMS_QOPATCH following error seen

ERROR:
ORA-20001: Latest xml inventory is not loaded into table
ORA-06512: at "SYS.DBMS_QOPATCH", line 1448
ORA-06512: at "SYS.DBMS_QOPATCH", line 122

Solution: 

1. oraInventory issue
2. Recreate oraceinventory sometime works
3. 19c not using softlink so need to set correct entry into oratabs and run datapatch -verbose

-Because ORACLE_HOME using softlink that's why need to set this parameter for temporary purpose once datapatch done remove this parameter from pfile.

ORA-20001: Latest xml inventory is not loaded into table --->
1.
Startup upgrade
create or replace directory OPATCH_LOG_DIR as '$OH/QOpatch'; ## don't using ORACLE_HOME softlink
create or replace directory OPATCH_SCRIPT_DIR as '$OH/QOpatch';
 
ALTER SYSTEM SET "_disable_directory_link_check" = TRUE SCOPE=SPFILE;
shutdown immediate
startup upgrade
 
 
cd $ORACLE_HOME/OPatch
 
./datapatch -verbose
 
SELECT a.ksppinm "Parameter",b.ksppstvl "Session Value",c.ksppstvl "Instance Value" FROM x$ksppi a, x$ksppcv b, x$ksppsv c WHERE a.indx = b.indx AND a.indx = c.indx AND a.ksppinm LIKE '/_disable_direc%' escape '/';

Ref Doc:
datapatch fails with "ORA-20009:" “ORA-20013: DBMS_QOPATCH ran mostly in non install area” ORACLE_HOME is a SYMBOLIC LINK (Doc ID 2033620.1)
Queryable Patch Inventory - Issues/Solutions for ORA-20001: Latest xml inventory is not loaded into table (Doc ID 1602089.1)

GoldenGate 12.1.2 not firing insert trigger

Getting report of goldengate Replicate process which is shows after upgrade from 11g to 12c.

Even though checkpoint table also upgraded. But Replicate process unable to start. 

------------------------------------------------------------------------------------------------------------------------

2021-02-11 04:13:31  INFO    OGG-06451  Triggers will be suppressed by default.

--More--(36%)
2021-02-11 04:13:31  INFO    OGG-01815  Virtual Memory Facilities for: COM
    anon alloc: mmap(MAP_ANON)  anon free: munmap
    file alloc: mmap(MAP_SHARED)  file free: munmap
    target directories:
    /opt/app/p0bfo3d1/ggs/12.1.2.1.0/dirtmp.

CACHEMGR virtual memory values (may have been adjusted)
CACHESIZE:                                2G
CACHEPAGEOUTSIZE (default):               8M
PROCESS VM AVAIL FROM OS (min):           4G
CACHESIZEMAX (strict force to disk):   3.41G

Database Version:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
PL/SQL Release 12.1.0.2.0 - Production
CORE    12.1.0.2.0    Production
TNS for Linux: Version 12.1.0.2.0 - Production
NLSRTL Version 12.1.0.2.0 - Production

Database Language and Character Set:
NLS_LANGUAGE     = "AMERICAN"
NLS_TERRITORY    = "AMERICA"
NLS_CHARACTERSET = "WE8ISO8859P1"

***********************************************************************
**                     Run Time Messages                             **
***********************************************************************
 
Opened trail file /opt/app/ORCL/ggs/11.2.1.0.3/trails/y1000331 at 2021-02-11 05:01:20
 
2021-02-11 05:01:20  INFO    OGG-03506  The source database character set, as determined from the trail file, is we8iso8859p1.
 
2021-02-11 05:01:20  INFO    OGG-06505  MAP resolved (entry ggsuser.ggs_heartbeat): MAP "GGSUSER"."GGS_HEARTBEAT", TARGET ggsuser.ggs_lagtime, KEY
COLS (ID), INSERTALLRECORDS, COLMAP (USEDEFAULTS, id = 0, committime = @GETENV ('GGHEADER', 'COMMITTIMESTAMP'), groupname  = @GETENV ('GGENVIRONME
NT', 'GROUPNAME'), ).
Using the following default columns with matching names:
  SOURCETIME=SOURCETIME, TARGETTIME=TARGETTIME, LAGTIME=LAGTIME
 
2021-02-11 05:01:21  INFO    OGG-06510  Using the following key columns for target table GGSUSER.GGS_LAGTIME: ID.
 
 
2021-02-11 05:01:21  WARNING OGG-00869  OCI Error ORA-00001: unique constraint (GGSUSER.GGS_LAGTIME_PK) violated (status = 1), SQL <INSERT INTO "G
GSUSER"."GGS_LAGTIME" ("ID","COMMITTIME","GROUPNAME") VALUES (:a0,:a1,:a2)>.
 
2021-02-11 05:01:21  WARNING OGG-01004  Aborted grouped transaction on 'GGSUSER.GGS_LAGTIME', Database error 1 (OCI Error ORA-00001: unique constr
aint (GGSUSER.GGS_LAGTIME_PK) violated (status = 1), SQL <INSERT INTO "GGSUSER"."GGS_LAGTIME" ("ID","COMMITTIME","GROUPNAME") VALUES (:a0,:a1,:a2)
>).
 
2021-02-11 05:01:21  WARNING OGG-01003  Repositioning to rba 16083229 in seqno 331.
 
2021-02-11 05:01:21  WARNING OGG-01154  SQL error 1 mapping GGSUSER.GGS_HEARTBEAT to GGSUSER.GGS_LAGTIME OCI Error ORA-00001: unique constraint (G
GSUSER.GGS_LAGTIME_PK) violated (status = 1), SQL <INSERT INTO "GGSUSER"."GGS_LAGTIME" ("ID","COMMITTIME","GROUPNAME") VALUES (:a0,:a1,:a2)>.
 
2021-02-11 05:01:21  WARNING OGG-01003  Repositioning to rba 16083229 in seqno 331.
Source Context :
  SourceModule            : [er.errors]
  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/app/er/errors.cpp]
  SourceFunction          : [take_rep_err_action]
  SourceLine              : [701]
  ThreadBacktrace         : [13] elements
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f69e005bdae]]
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x340
) [0x7f69e0056ac0]]
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/libgglog.so(_MSG_ERR_MAP_TO_TANDEM_FAILED(CSourceContext*, ggs::gglib::ggapp::CQualD
BObjName<(DBObjType)1> const&, ggs::gglib::ggapp::CQualDBObjName<(DBObjType)1> const&, CMessageFactory::MessageDisposition)+0x53) [0x7f69e004fcff]
]
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/replicat(take_rep_err_action(short, int, char const*, extr_ptr_def*, __std_rec_hdr*,
 char*, file_def*, bool)+0xdda) [0x580b82]]
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/replicat(process_record(file_def*&, short&, extr_ptr_def*&, extr_ptr_def*&, int&, bo
ol&, int&, int&, bool)+0xcaa) [0x5d0f5a]]
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/replicat(process_replicat_loop()+0xe11) [0x5ad7f1]]
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/replicat(replicat_main(int, char**)+0x5ca) [0x5c6b7a]]
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/replicat(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x4f) [0x699c0f]]
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/replicat(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::T
hread::ThreadArgs*)+0x104) [0x699f44]]
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/replicat(ggs::gglib::MultiThreading::MainThread::Run(int, char**)+0x8b) [0x69a04b]]
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/replicat(main+0x3f) [0x5c580f]]
                          : [/lib64/libc.so.6(__libc_start_main+0x100) [0x3b5481ed20]]
                          : [/opt/app/ORCL/ggs/12.1.2.1.0/replicat() [0x517099]]
 
2021-02-11 05:01:21  ERROR   OGG-01296  Error mapping from GGSUSER.GGS_HEARTBEAT to GGSUSER.GGS_LAGTIME.

------------------------------------------------------------------------------------------------------------------------

 Solution:

I had to troubleshoot a situation, after GoldenGate capture some DML and replicate that, Oracle database needs to run insert trigger making some business integration.

After to upgrade this enviroment from GG 11.1.1.1 to 12.1.2 and DB 11.2.0.3 to 12.1.0.2, was identified that GoldenGate wasn’t running this triggers

So, found interesting resolution on Oracle Docs:

SUPPRESSTRIGGERS | NOSUPPRESSTRIGGERS

Valid for nonintegrated Replicat for Oracle. Controls whether or not triggers are fired during the Replicat session. Provides an alternative to manually disabling triggers. (Integrated Replicat does not require disabling of triggers on the target system.)

SUPPRESSTRIGGERS is the default and prevents triggers from firing on target objects that are configured for replication with Oracle GoldenGate. SUPPRESSTRIGGERS is valid for Oracle 11.2.0.2 and later 11gR2 versions. SUPPRESSTRIGGERS is not valid for 11gR1.

So, added 'DBOPTIONS NOSUPPRESSTRIGGERS' in the replicat parameter file.

OR

However, altough you probably want most of the triggers not to be executed on the target, it can be possible to override this setting, for example when you implement a heartbeat table.

The way to exclude the trigger from the SUPRRESSTRIGGERS setting is to grant the trigger an exception. This can be done as follows on the target:

SQL> EXEC dbms_ddl.set_trigger_firing_property('HR','trg_tbl_heartbeat', FALSE);'

The result of the statement above is that trigger hr.trg_tbl_heartbeat will go off in the target database.

=================================================================================

2nd.

If above steps are not working and still Replicate process Abended then go for below the steps.

 upgrade Goldengate from 11g to 12c ?

If yes, you have two options :

1) Change the double quotes to single quotes

2) Add NOUSEANSISQLQUOTES into your GLOBALS file and restart the processes
/
NOUSEANSISQLQUOTES is intended for backward compatibility with the parameter files of Oracle GoldenGate versions that predate version 12c, where strings in double quotes are intended to be literals and case-sensitive column names are not supported (whether or not they are within quotes).

Reference : https://docs.oracle.com/goldengate/1212/gg-winux/GWURF/gg_parameters183.htm#GWURF728

Exadata : Disk Replacement

Operational Activities:

*.Diagnostics & SR Creation

   CellCLI> list celldisk where name='<Cell_Disk>' details

       Disk Status -> Online/Offline  #Both Cases

       Error Count -> more than 0.

   SR Creation

         1. ASR Configure

         2. Manual SR - read Doc ID - 761868.1

Reference :

Sundiag & Other Diagnostics: Oracle Exadata Diagnostic Information required for Disk Failures and some other Hardware issues (Doc ID 761868.1)

*.Drop Faulted Disk

   1.Normal -> Confined offline

          *.Amber LED

          *.Rebalance Initiated

   2.Confined offline -> offline

         *.Blue LED

         *.Ready for Replacement

Reference:

Drop Online Disk: Things to Check in ASM When Replacing an ONLINE disk from Exadata Storage Cell (Doc ID 1326611.1)

*.Disk replacement

    FSE visit and Replacement

Reference:

Replace Procedure: How to Replace a Hard Drive in an Exadata Storage Server (Predictive Failure) (Doc ID 1390836.1)

CRS-5016 CRS-2674 ACFS-00591 ACFS-02037 CRS resource for ACFS

Once Add/Register ACFS filesystem as a Cluster 

srvctl add filesystem -d /dev/asm/sharedvol-64 -g GG_APP -v SHAREDVOL -m /gg_acfs/ggs -u oracle

[root@rac1 bin]# srvctl status filesystem -d /dev/asm/sharedvol-64

ACFS file system is not running

[root@rac1 bin]# srvctl start filesystem -d /dev/asm/sharedvol-64
PRCR-1079 : Failed to start resource ora.gg_app.sharedvol.acfs
CRS-5016: Process "/u01/app/11204/grid/bin/acfssinglefsmount" spawned by agent "/u01/app/11204/grid/bin/orarootagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11204/grid/log/rac2/agent/crsd/orarootagent_root//orarootagent_root.log"
CRS-5016: Process "/u01/app/11204/grid/bin/acfssinglefsmount" spawned by agent "/u01/app/11204/grid/bin/orarootagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11204/grid/log/rac1/agent/crsd/orarootagent_root//orarootagent_root.log"
CRS-5016: Process "/sbin/acfsutil" spawned by agent "/u01/app/11204/grid/bin/orarootagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11204/grid/log/rac2/agent/crsd/orarootagent_root//orarootagent_root.log"
CRS-5016: Process "/sbin/acfsutil" spawned by agent "/u01/app/11204/grid/bin/orarootagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11204/grid/log/rac1/agent/crsd/orarootagent_root//orarootagent_root.log"
CRS-2674: Start of 'ora.gg_app.sharedvol.acfs' on 'rac2' failed
CRS-2674: Start of 'ora.gg_app.sharedvol.acfs' on 'rac1' failed


# mount.acfs -o all
mount.acfs: ACFS-00591: error found in volume disk header
mount.acfs: ACFS-02037: File system not created on a Linux system. Cannot mount.
found error... so(make acfs)

[root@rac1 bin]# mkfs.acfs -f /dev/asm/sharedvol-64
mkfs.acfs: version                   = 11.2.0.4.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/sharedvol-64
mkfs.acfs: volume size               = 9663676416
mkfs.acfs: Format complete.

[root@rac1 bin]# mount.acfs -o all

[root@rac1 bin]#  acfsutil registry -a -f /dev/asm/sharedvol-64 /gg_acfs/ggs/

[root@rac1 bin]# srvctl start filesystem -d /dev/asm/sharedvol-64

[root@rac1 bin]# srvctl status filesystem -d /dev/asm/sharedvol-64
ACFS file system is running on rac1,rac2

[root@rac1 bin]# df -h /gg_acfs/ggs
Filesystem            Size  Used Avail Use% Mounted on
/dev/asm/sharedvol-64
                      9.0G   95M  9.0G   2% /gg_acfs/ggs

12c GoldenGate Silent Installation

Most of the customer and Env. not using GUI based installation so here i'm giving you small demonstration to installing Oracle goldengate 12c silent installation.

For GUI based. run simply ./runInstaller and pass require parameters according to you env.

Below the steps for silent installations.

=====================
-Software Location of GG
=====================
/u01/fbo_ggs_Linux_x64_shiphome/Disk1/response

oggcore.rsp ----> response file

less oggcore.rsp | grep oracle.install.responseFileVersion

==================================
-Below the parameters update in response file
==================================

oracle.install.responseFileVersion=/oracle/install/rspfmt_ogginstall_response_schema_v12_1_2

INSTALL_OPTION=ORA12c
SOFTWARE_LOCATION=/oracle/gg
START_MANAGER=true
MANAGER_PORT=7809
DATABASE_LOCATION=/u01/app/oracle/product/12.1.0/dbhome_1
INVENTORY_LOCATION=/u01/app/oraInventory
UNIX_GROUP_NAME=oinstall

$ ./runInstaller -silent -responseFile /home/oracle/ggresp.rsp

==================
-Inventory Verification
==================

You can confirm the installation of GoldenGate by simply looking at the Oracle inventory.
The inventory file is located at /u01/app/oraInventory/ContentsXML

-bash-3.2$ less inventory.xml | grep -i gg
<HOME NAME="OraHome15" LOC="/u01/app/ogg" TYPE="O" IDX="28"/>

Reference :
Oracle 11g GoldenGate Installation
https://oraclegurukul.blogspot.com/2015/01/oracle-11g-goldengate-installation.html

OGG-01033 There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using ENCRYPT) or an unknown error.

Initial Load issue in  12c.

-- Report error.

2018-10-28 10:08:02  ERROR   OGG-01033  There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using ENCRYPT) or an unknown error. (Remote file used is /u01/app/oracle/product/ogg/dirdat/initload.dat, reply received is Failed resolving output file /u01/app/oracle/product/ogg/dirdat/initload.dat (error: 2, No such file or directory)).
2018-10-28 10:08:02  

ERROR   OGG-01668  PROCESS ABENDING.

-- Solution.

This error will come in goldengate version 12.2, as new parameter ALLOWOUTPUTDIR  has been introduced , which need to be added in ./GLOBALS file on target .

syntax – ALLOWOUTPUTDIR   [ path of the remote trail location]

1. stop the manager and the replicates

ggsci> stop mgr

ggsci> stop *

ggsci> edit params ./GLOBAL     <-------------------- Put the parameter ALLOWOUTPUTDIR

2.  edit the ./GLOBAL

ALLOWOUTPUTDIR /u01/ogg/dirdat/

3. stat mgr and replicates
ggsci> start mgr
ggsci> start *

-- Source 

./extract <param_file> <report_file> for checking any issue before deploying automate.

rtld: 0712-001 Symbol CreateIoCompletionPort was referenced

In My Case Cause of below the issue at storage level.
failing currently due to LVM inconsistency

For more Detail follow below the reference links.

Ref: 
http://oracle-help.com/ora-errors/rtld-0712-001-symbol-createiocompletionport-was-referenced/
AIX 12.1.0.2 Installation Fails with "rtld: 0712-001 Symbol CreateIoCompletionPort was referenced " (Doc ID 1949184.1)

Highly Available VIP (HAVIP)

Highly Available IP (HAIP) comes from 11.2.0.2 on-words and maximum 4 define for Private network.

Now in 12c version introduces Highly Available VIP (HAVIP) a Clusterware-monitored VIP that can be used for non-Databaseapplications.

Typically, VIPs are created for VIP or SCAN listeners for Database connections. In some cases, the need arises to create a VIP monitored by Clusterware designed to support other applications.

For more Detail : Click Here

New feature in Oracle 11g database to kill session

Now we can kill session after finishing existing transaction to be finished. Means in previous release there was no scope to finish on going transaction. Instead of "alter system kill session" here we can use "alter system disconnect session". Using syntax "POST_TRANSACTION" we can kill session for active transaction has to be completed first and after that it will be automatically killed. Example of same syntax is given below.

SQL> alter system disconnect session '10,251' post_transaction;
System altered.

LREG New BG Process In 12c

In 11.2 Pmon process propogates the service metrics to the listeners regitered in local_listener adn remote_listener initialization parameter.

As the remote_listener specifies specifies the address of scan listener and the local_listener parameter specifies the address of the VIP listener, PMON process propogates the service metrics to both SCAN and VIP listeners.

You can trace listener registration using the following command.

alter system set event='immediate trace name listener_registration level 15';

Now in 12c version listener registration is permed by a new mandatory background process named LREG.

And if you want to trace then LREG trace file getting to all info about listener registration.

Advantage : Not recommneding that you create 50+ of unnecessary services,you should create

as many services as you need to split the application into manageable and disjointed workload.

In 11gR2 if there are 50+/100+ services and listeners then there is a possiblilyt that the PMON process

might spend more time on service registration to listener.

But in 12c this possibility is eliminated as the LREG backgroud process which is totally dedicated for registering the services to Listener.

and PMON is freed from listener registration.

PRCD-1027 PRCD-1229

PRCD-1027 : Failed to retrieve database racdb
PRCD-1229 : An attempt to access configuration of database racdb was rejected because its version 11.2.0.1.0 differs from the program version 11.2.0.4.0. Instead run the program from /oracle/product/11.2.0/db_1.



[root@rac1 ~]# srvctl status database -d racdb
PRCD-1027 : Failed to retrieve database racdb
PRCD-1229 : An attempt to access configuration of database racdb was rejected because its version 11.2.0.1.0 differs from the program version 11.2.0.4.0. Instead run the program from /oracle/product/11.2.0/db_1.

[root@rac1 ~]# srvctl config database -d racdb

PRCD-1027 : Failed to retrieve database racdb
PRCD-1229 : An attempt to access configuration of database racdb was rejected because its version 11.2.0.1.0 differs from the program version 11.2.0.4.0. Instead run the program from /oracle/product/11.2.0/db_1.

[root@rac1 ~]# srvctl config database
racdb


-- First Attempt
[root@rac1 ~]# srvctl modify database -d racdb -o /oracle/product/11204/db_1

PRCD-1027 : Failed to retrieve database racdb
PRCD-1229 : An attempt to access configuration of database racdb was rejected because its version 11.2.0.1.0 differs from the program version 11.2.0.4.0. Instead run the program from /oracle/product/11.2.0/db_1.

-- Second Attempt
[root@rac1 ~]# srvctl upgrade database -d  racdb -o /oracle/product/11204/db_1

PRCD-1231 : Failed to upgrade configuration of database racdb to version 11.2.0.4.0 in new Oracle home /oracle/product/11204/db_1
PRKH-1014 : Current user "root" is not the oracle owner user "oracle" of oracle home "/oracle/product/11204/db_1"

[root@rac1 ~]# su - oracle

-- Final Attempt
[oracle@rac1 ~]$ srvctl upgrade database -d  racdb -o /oracle/product/11204/db_1

[oracle@rac1 ~]$ exit
logout

[root@rac1 ~]# srvctl status database -d racdb
Instance racdb1 is running on node rac1
Instance racdb2 is running on node rac2

mv: cannot move `OPatch' to `OPatch_old': Permission denied

Applying Latest Opatch in GRID_HOME  Opatch (6880880) 


[grid@dbapath.com patch]$ mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch_old
mv: cannot move `/u01/app/11.2.0/grid/OPatch/’ to `/u01/app/11.2.0/grid/OPatch_old’: Permission denied

You must either unlock GRID_HOME or you can move OPatch directory as a superuser.

[oracle@grid_server1 ]$ cd /u01/app/11.2.0/grid/crs/install
[oracle@grid_server1 ]$ sudo perl rootcrs.pl -unlock -crshome /u01/app/11.2.0/grid

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Undefined subroutine &main::read_file called at /u01/app/11.2.0/grid/crs/install/crspatch.pm line 86.

If you hit an error like above, here is the workaround (according to this paper: roothas.pl -patch or rootcrs.pl -patch Fails with ‘Undefined subroutine’ [ID 1268390.1]):

# Take a backup of the file /crs/install/crsconfig_lib.pm

grid$ cd /crs/install

grid$ cp crsconfig_lib.pm crsconfig_lib.pm.bak

# Make the following change in that file crsconfig_lib.pm
#From
# my @exp_func = qw(check_CRSConfig validate_olrconfig validateOCR
#To
# my @exp_func = qw(check_CRSConfig validate_olrconfig validateOCR read_file

#Execute relock:

[grid@dbapath.com ]$ cd /u01/app/11.2.0/grid/crs/install

[grid@dbapath.com ]$ sudo perl rootcrs.pl -patch -crshome /u01/app/11.2.0/grid
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
ACFS-9200: Supported
CRS-4123: Oracle High Availability Services has been started.

Quiescing RAC Databases

To quiesce a RAC database, use the ALTER SYSTEM QUIESCE RESTRICTED statement from one instance.

It is not possible to open the database from any instance while the database is in the process of being quiesced from another instance.

After all non-DBA sessions become inactive, the ALTER SYSTEM QUIESCE RESTRICTED statement executes and the database is considered to be quiesced.
In a RAC environment, this statement affects all instances.

Cold backups cannot be taken when the database is in a quiesced state because the Oracle background processes may  still perform updates for internal purposes even when the database is in a quiesced state.
Also, the file headers of online data filescontinue to appear as if they are being accessed. They do not look the same as if a clean shutdownwere done.

Mongo ReplicaSet

Hi Guys this is my first post in Mongodb and today i give you a small demonstration to configure replica set.

• But first thing is know about the replica set? 
• what is replicaset? 
• How they work ? go through with google all this type of questions.

Now we are going to configure replicaset.

=============
My Envirenment
=============

3 Servers -> 1 Primary -> 2 Secondary
OS Useing RHEL7
MongoDB version 3.4

=========
# Host File
=========

- Entry in /etc/hosts in all members

[root@db1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.219.130         mongodb1.com    db1
192.168.219.140         mongodb2.com    db2
192.168.219.141         mongodb3.com    db3

==============
# Disable Selinux
==============

Check the SELinux status with the command.
[root@db1 ~]# getenforce
enabled

[root@db1 ~]# cat /etc/selinux/config
SELINUX=disabled

[root@db1 ~]# getenforce
disabled

==================
# Configure Firewalld
==================
In the first step, we already disabled SELinux. For security reasons, we will now enable firewalld on all nodes and open only the ports that are used by MongoDB and SSH.

- Install Firewalld with the yum command.

[root@db1 ~]# yum -y install firewalld

- Start firewalld and enable it to start at boot time.

[root@db1 ~]# systemctl start firewalld
[root@db1 ~]# systemctl enable firewalld

- Next, open your ssh port and the MongoDB default port 27017.

[root@db1 ~]# firewall-cmd --permanent --add-port=22/tcp
[root@db1 ~]# firewall-cmd --permanent --add-port=27017/tcp

- Reload firewalld to apply the changes.

[root@db1 ~]# firewall-cmd --reload



=====================
# Mongo Configuration File
=====================

- Below the configuration file parameter which is same in all the members

/etc/mongod.conf
# network interfaces
net:
  port: 27017
  #bindIp: 0.0.0.0 # Listen to local interface only, comment to listen on all interfaces.

replication:
  replSetName: rs0


=========================
# MongoDB Replica Set initiate
=========================
In this step, we will create the replica set. We will use the 'mongo1' server as 'PRIMARY' node, and 'db2' and 'db3' as 'SECONDARY' nodes.

Initiate the replica set from the mongo1 server with the query below.

[root@db1 ~]# rs.initiate()
Make sure 'ok' value is 1.

-Now add the 'db2' and 'db3' nodes to the replica sets.

[root@db1 ~]# rs.add("db2:27017")
[root@db1 ~]# rs.add("db3:27017")

You will see the results below and make sure there is no error.


- Next, check the replica sets status with the rs query below.

[root@db1 ~]# rs.status()

- Another query to check the status is:

[root@db1 ~]# rs.isMaster()



==================
# Test the Replication
==================

Test the data set replication from the 'PRIMARY' instance 'mongo1' to 'SECONDARY' nodes 'db2' and 'db3'.

In this step, we will try to write or create a new database on the 'PRIMARY' node 'db1',
then check if the replication is working by checking the database on 'SECONDARY' nodes 'db2' and 'db3'.

- Login to the 'db1' server and open mongo shell.

[root@db1 ~]# mongo

- Now create a new database 'lemp' and new 'stack' collection for the database.

> use lemp
> db.stack.save(
{
    "desc": "LEMP Stack",
    "apps":  ["Linux", "Nginx", "MySQL", "PHP"],
})




- Next, go to the 'SECONDARY' node 'db2' and open the mongo shell.

[root@db1 ~]# mongo

Enable reading from the 'SECONDARY' node with the query 'rs.slaveOk()', and then check if the 'lemp' database exists on the 'SECONDARY' nodes.

> rs.slaveOk()
> show dbs
> use lemp
> show collections
> db.stack.find()


===================
# Check Replication Lag
===================

> rs.printReplicationinfo()
> rs.printSlaveReplicationinfo() # show all the members with replication lag.

rs0:PRIMARY> rs.printSlaveReplicationInfo();
source: db2:27017
        syncedTo: Mon Jan 15 2018 21:49:07 GMT+0530 (IST)
        0 secs (0 hrs) behind the primary
source: db3:27017
        syncedTo: Thu Jan 01 1970 05:30:00 GMT+0530 (IST)
        1516033147 secs (421120.32 hrs) behind the primary


rs0:PRIMARY> rs.printSlaveReplicationInfo();
source: db2:27017
        syncedTo: Mon Jan 15 2018 21:49:07 GMT+0530 (IST)
        0 secs (0 hrs) behind the primary
source: db3:27017
        syncedTo: Thu Jan 01 1970 05:30:00 GMT+0530 (IST)
        1516033147 secs (421120.32 hrs) behind the primary

rs0:PRIMARY> rs.status();
Test Demo: Down the db3 member instanace throught db.serverShutdown(); and execute above command and see the status.

Udev Rule For ASM Disks

In Case of RAC configure on VM - After Adding ASM disk on Node 1 adding asm disk in UDEV rules for persistent.

------------------
IN RHEL/OEL-5/6/7
------------------Set Asm disk using UDEV
/sbin/scsi_id -g -u -d /dev/sdb
/sbin/scsi_id -g -u -d /dev/sdc
/sbin/scsi_id -g -u -d /dev/sdd
/sbin/scsi_id -g -u -d /dev/sde
/sbin/scsi_id -g -u -d /dev/sdf
/sbin/scsi_id -g -u -d /dev/sdg

-Above the command execute not showing any scsi_id for udev rule. I'm not testing much more but find some doc and fix this issue on above linux versions.

- Adding below rules into "/etc/udev/rules.d/50-udev.rules" in my case my OS version is 5. For more detail follow oracle-base.
# ASM DISK RULES
KERNEL=="sdb1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdc1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdd1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sde1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdf1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk5", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdg1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk6", OWNER="oracle", GROUP="dba", MODE="0660"

- No execute again below the command and wondering got the output of scsi_id.
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sdb
36000c299cae37d62af51ab7c55768959
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sdc
36000c2989736710c1b2bd8efea742e61
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sdd
36000c295eae87b8a4e919b5b3b077827
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sde
36000c293bc54c062ff70f20ae8c3078e
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sdf
36000c29759c3a92862de21e908f8516e
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sdg
36000c292f86ba892199aa8fd6bb7ffe8

- Creating a file and insert below commands into this file "vim /etc/udev/rules.d/99-oracle-asmdevices.rules"
KERNEL=="sdb1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c299cae37d62af51ab7c55768959",NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdc1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c2989736710c1b2bd8efea742e61",NAME="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdd1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c295eae87b8a4e919b5b3b077827",NAME="asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sde1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c293bc54c062ff70f20ae8c3078e",NAME="asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdf1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29759c3a92862de21e908f8516e",NAME="asm-disk5", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdg1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c292f86ba892199aa8fd6bb7ffe8",NAME="asm-disk6", OWNER="oracle", GROUP="dba", MODE="0660"

- Add the following to the "/etc/scsi_id.config" file to configure SCSI devices as trusted. Create the file if it doesn't already exist.
options=-g

- Execute below commands
/sbin/partprobe /dev/sdb1
/sbin/partprobe /dev/sdc1
/sbin/partprobe /dev/sdd1
/sbin/partprobe /dev/sde1
/sbin/partprobe /dev/sdf1
/sbin/partprobe /dev/sde1

# #OL5 and OL6
# /sbin/start_udev
ls -lrt /dev/asm*
Now proceed your next action plan for node 2.

DNS SERVER

DNS configure without CHROOT on RHEL/OEL For Oracle RAC


[root@dns1 ~]# rpm -qa | grep -i bind
ypbind-1.19-12.el5_6.1
kdebindings-3.5.4-6.el5
bind-utils-9.3.6-20.P1.el5_8.6
system-config-bind-4.0.3-6.0.1.el5
bind-libs-9.3.6-20.P1.el5_8.6
bind-9.3.6-20.P1.el5_8.6

[root@dns1 ~]# hostname
dns1.testenv.com



----- Create/Edit /etc/named.conf File ------

[root@dns1 ~]# cat /etc/named.conf
options {
        listen-on port 53 { 192.168.2.200; };
        directory "/var/named";
        dump-file "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        /*
         * If there is a firewall between you and nameservers you want
         * to talk to, you might need to uncomment the query-source
         * directive below.  Previous versions of BIND always asked
         * questions using port 53, but BIND 8.1 uses an unprivileged
         * port by default.
         */
         // query-source address * port 53;

        // My Additions
        // Forwarder: Anything this DNS can't resolve gets forwarded to my ISPs DNS.
        #forwarders { 194.168.4.100; 194.168.8.100; };
        // End My Additions
};

zone "testenv.com." IN {
                 type master;
                 file "testenv.com.zone";
                 allow-update { none; };
};

zone "2.168.192.in-addr.arpa." IN {
        type master;
        file "2.168.192.in-addr.arpa";
        allow-update { none; };
};

include "/etc/rndc.key";




----------- Create/Edit /var/named/2.168.192.in-addr.arpa File ----------------
[root@dns1 named]# pwd
/var/named
[root@dns1 named]# cat 2.168.192.in-addr.arpa
$ORIGIN 2.168.192.in-addr.arpa.
$TTL 1H
@          IN     SOA    testenv.com.  root.testenv.com. (
                         42 ; serial (d. adams)
                         3H ; refresh
                        15M ; retry
                         1W ; expiry
                         1D ) ; minimum

2.168.192.in-addr.arpa.   IN   NS     testenv.com.
151 IN PTR rac1.testenv.com.
152 IN PTR rac2.testenv.com.
161 IN PTR rac1-vip.testenv.com.
162 IN PTR rac2-vip.testenv.com.
51  IN PTR rac-scan.testenv.com.
52  IN PTR rac-scan.testenv.com.
53  IN PTR rac-scan.testenv.com.


------------------ Create/Edit /var/named/testenv.com.zone File --------

[root@dns1 named]# cat testenv.com.zone
$TTL    86400
@               IN SOA  testenv.com root.testenv.com (
                                        42              ; serial (d. adams)
                                        3H              ; refresh
                                        15M             ; retry
                                        1W              ; expiry
                                        1D )            ; minimum
                IN NS   testenv.com
rac1            IN A    192.168.2.151
rac2            IN A    192.168.2.152
rac1-priv       IN A    192.168.3.151
rac2-priv       IN A    192.168.3.152
rac1-vip        IN A    192.168.2.161
rac2-vip        IN A    192.168.2.162
rac-scan        IN A    192.168.2.51
rac-scan        IN A    192.168.2.52
rac-scan        IN A    192.168.2.53



-------- Give the Ownership --------

cd /var/named
chown named:named 2.168.192.in-addr.arpa
chown named:named testenv.com.zone

chkconfig named on

[root@dns1 named]# cat /etc/resolv.conf
nameserver 192.168.2.200



--------- Check the its working or not ----------

[root@dns1 named]# nslookup rac-scan
Server:         192.168.2.200
Address:        192.168.2.200#53

Name:   rac-scan.testenv.com
Address: 192.168.2.53
Name:   rac-scan.testenv.com
Address: 192.168.2.51
Name:   rac-scan.testenv.com
Address: 192.168.2.52


[root@dns1 named]# dig testenv.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-20.P1.el5_8.6 <<>> testenv.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50541
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;testenv.com.                   IN      A

;; AUTHORITY SECTION:
testenv.com.            86400   IN      SOA     testenv.com.testenv.com. root.testenv.com.testenv.com. 42 10800 900 604800 86400

;; Query time: 3 msec
;; SERVER: 192.168.2.200#53(192.168.2.200)
;; WHEN: Wed Jul  5 11:31:57 2017
;; MSG SIZE  rcvd: 82


---------------------------------------------------------------------------------------------------------------------------
=============
Now RAC Part
=============


Configure one node if working well athen follow all steps on all nodes.

[root@rac1 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               rac1.testenv.com rac1 localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
#Public-IP
192.168.2.151  rac1  rac1.testenv.com
192.168.2.152  rac2  rac2.testenv.com

#Virtual-IP
192.168.2.161  rac1-vip      rac1-vip.testenv.com
192.168.2.162 rac2-vip      rac2-vip.testenv.com

#Scan-IP
192.168.198.51  rac-scan        rac-scan.testenv.com
192.168.2.52    rac-scan        rac-scan.testenv.com
192.168.2.53    rac-scan        rac-scan.testenv.com

192.168.2.200   dns1    dns1.testenv.com


[root@rac1 ~]# hostname
rac1.testenv.com


[root@rac1 ~]# cat /etc/resolv.conf
search testenv.com      dns1.testenv.com
nameserver 192.168.2.200


[root@rac1 ~]# dig rac1

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> rac1
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 63859
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;rac1.                          IN      A

;; Query time: 4 msec
;; SERVER: 192.168.2.200#53(192.168.2.200)
;; WHEN: Wed Jul  5 11:36:32 2017
;; MSG SIZE  rcvd: 22



[root@rac1 ~]# dig rac-scan

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> rac-scan
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 359
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;rac-scan.                      IN      A

;; Query time: 13 msec
;; SERVER: 192.168.2.200#53(192.168.2.200)
;; WHEN: Wed Jul  5 11:36:05 2017
;; MSG SIZE  rcvd: 26



Now its working fine ... above the setting replicate on all nodes.

Data Guard Flow

PHYSICAL STANDBY DATA FLOW

Let us understand how the data flows in data guard setup as described above by Points 1 to 8 :

Point 1) On Primary Database, Transactions starts. All the buffer cache locks (exclusive locks) that are required for the transaction are acquired.

Point 2) On Primary Database, the redo blocks that describes the changes (or change vectors) are generated and stored in the processes’ Program Global Area (PGA). After successfully acquiring the redo allocation latch, space is then allocated in the redo log buffer. The redo generated then gets copied from the processes’ PGA into the redo log buffer.

Point 3) On Primary Database, The oracle foreground process tells the LGWR to flush the redo log buffers to disk. Remember that the database blocks in the database have not yet been updated with DML changes. The LGWR flushes the redo buffers to the ORL and acknowledges the completion to the session. At this point, the transaction is persistent on disk. No commit has occurred thus far.

---

At some future time, the database buffers that were previously changed will be written to disk by the database writer process (DBWR) at checkpoint time. This point is not marked in above diagram.

Note that before the DBWR process has flushed the database buffers to disks, the LGWR process must have already written the redo buffers to disk. This explicit sequence is enforced by the write-ahead logging protocol.

Also The ARCH process on the primary database archives the ORLs into archive log files. This point is also not marked in the above diagram.

---

Point 4) On Primary Database, the LNS process reads the recently flushed redo from the redo log buffer and sends the redo data to the standby database using the redo transport destination (LOG_ARCHIVE_DEST_n) that we defined during standby database creation. We are using ASYNC transport method, so the LGWR does not wait for any acknowledgment from the LNS for this network send operation. It does not communicate with the LNS except to start it up at the database start stage and after a failure of a standby connection.

Point 5) On Standby Database , the RFS reads the redo stream from the network socket into the network buffers, and then it writes this redo stream to the SRL.

Point 6) On Standby Database, The ARCH process archives the SRLs into archive log files when a log switch occurs at the primary database. The generated archive log file is then registered with the standby control file.
flow involves three distinct phases, as follows:

Point 7) On standby database, the actual recovery process starts from this step. The managed recovery process (MRP) will asynchronously read ahead the redo from the SRLs or the archived redo logs (when recovery falls behind or is not in real-time apply mode). The blocks that require redo apply are parsed out and placed into appropriate in-memory map segments.

Point 8) On standby database, the MRP process ships redo to the recovery slaves using the parallel query (PQ) interprocess communication framework. Parallel media
recovery (PMR) causes the required data blocks to be read into the buffer cache, and subsequently redo will be applied to these buffer cache buffers.

At checkpoint phase, the recently modified buffers (modified by the parallel recovery slaves) will be flushed to disk and also the update of datafile headers to record checkpoint completion.

PHYSICAL STANDBY DATABASE  RELATED PROCESSES

All the important processes are created in CIRCLE above diagram

On the Primary Database:

LGWR : The log writer process flushes log buffers from the SGA to Online Redo Log files.

LNS : The LogWriter Network Service (LNS below 12c version) reads the redo being flushed from the redo buffers by the LGWR and sends the redo over network to the standby database. The
main purpose of the LNS process is to free up the LGWR process from performing the redo transport role.

ARCH  : The archiver processes archives the ORL files to archive log files. Up to 30 ARCH processes can exist, and these ARCH processes are also used to fulfill gap resolution requests. Note that one ARCH process has a special role in that it is dedicated to local redo log archiving only and never communicates with a standby database.

On the Standby Database:

RFS : The main objective of the Remote File Server process is to perform a network receive of redo transmitted from the primary site and then writes the network buffer (redo data) to the standby redo log (SRL) files.

ARCH : The archive processes on the standby site perform the same functions performed on the primary site, except that on the standby site, an ARCH process generates archived log files from the SRLs.

MRP : The managed recovery process coordinates media recovery management. Remember that a physical standby is in perpetual recovery mode.

Basically we can categorize physical standby database  into three major components:

1) Data Guard Redo Transport Services

– To transfer the redo that is generated by the primary database to the standby database.

Point 4 and 5 in the above diagram are where Redo Transport works.

2) Data Guard Apply Services  

– To receive and apply the redo sent by Redo Transport Services to the standby database.

Point 7 and 8 in the above diagram are where Redo Apply works.

3) Data Guard Role Management Services

– To assist in the database role changes in switchover and failover scenarios.

This service works in the background and takes care of switchover/failover scenarios

GAP RESOLUTION FLOW

How to resolve the gap in between primary and standby.

When MRP finds that an archive log is missing during media recovery, it sends the fal_client information to the server identified by fal_server and requests fal_server to resend the file again.  The fal_client and fal_server init.ora (spfile) parameters are set on the standby instance.

The Oracle docs note that fal_server specifies the FAL (fetch archive log) server for a standby database. The value for fal_server is an Oracle*Net service name, which is assumed to be configured properly on the standby database system to point to the desired FAL server.

1.       The fetch archive log (FAL) client is the MRP process.  The fetch archive log (FAL) server is a foreground process that runs on the primary database and services the fetch archive log requests coming from the FAL client.  A separate FAL server is created for each incoming FAL client. 

2.       Fal_server and fal_client foreground process.

3.       Its resolve the gap in automatic mode when you define these parameter fal_Client and fal_server.

4.       Above diagram background process work to resolve the gap in automatic mode ARCH to RFS.

5.       If you not setup this parameter then manually transfer the log from primary to standby and register the archive logs to standby and perform recovery.

Role of Standby Redo Logs Why and How?

For more info Refer : https://docs.oracle.com/cd/B19306_01/server.102/b14239/log_transport.htm#i1268294

CURRENT SESSION

# script to find redo generated by current sessions

set lines 2000

set pages 1000

col sid for 99999

col name for a09

col username for a14

col PROGRAM for a21

col MODULE for a25

select s.sid,sn.SERIAL#,n.name, round(value/1024/1024,2) redo_mb, sn.username,sn.status,substr (sn.program,1,21) "program", sn.type, sn.module,sn.sql_id

from v$sesstat s join v$statname n on n.statistic# = s.statistic#

join v$session sn on sn.sid = s.sid where n.name like 'redo size' and s.value!=0 order by

redo_mb desc;

# script to check INACTIVE sessions with HIGH DISK IO

select p.spid,s.username, s.sid,s.status,t.disk_reads, s.last_call_et/3600 last_call_et_Hrs,

s.action,s.program,s.machine cli_mach,s.process cli_process,lpad(t.sql_text,30) "Last SQL"

from gv$session s, gv$sqlarea t,v$process p

where s.sql_address =t.address and

s.sql_hash_value =t.hash_value and

p.addr=s.paddr and

t.disk_reads > 5000

and s.status='INACTIVE'

and s.process='1234'

order by S.PROGRAM;

# script to analyze Disk IO’s

prompt SESSIONS PERFORMING HIGH I/O > 50000

select p.spid, s.sid,s.process cli_process, s.status,t.disk_reads, s.last_call_et/3600 last_call_et_Hrs,

s.action,s.program,lpad(t.sql_text,30) "Last SQL"

from v$session s, v$sqlarea t,v$process p

where s.sql_address =t.address and

s.sql_hash_value =t.hash_value and

p.addr=s.paddr and

t.disk_reads > 10000

order by t.disk_reads desc;