Thursday, 29 January 2015

Resolving Gaps in Data Guard Apply Using Incremental RMAN BAckup


Recently, we had a glitch on a Data Guard (physical standby database) on infrastructure.
This is not a critical database; so the monitoring was relatively lax. And that being done by an outsourcer does not help it either.
In any case, the laxness resulted in a failure remaining undetected for quite some time and it was eventually discovered only when the
customer complained. This standby database is usually opened for read only access from time to time.
This time, however, the customer saw that the data was significantly out of sync with primary and raised a red flag. Unfortunately,
at this time it had become a rather political issue.

Since the DBA in charge couldn’t resolve the problem, I was called in. In this post, I will describe the issue and how it was resolved. In summary,
there are two parts of the problem:

(1) What happened
(2) How to fix it

What Happened

Let’s look at the first question – what caused the standby to lag behind. First, I looked for the current SCN numbers of the primary and standby databases.


On the primary:
SQL> col current_scn format  999999999999999
SQL> select current_scn from v$database;

CURRENT_SCN
-----------
1447102


On the standby:
SQL> col current_scn format  999999999999999
SQL> select current_scn from v$database;

CURRENT_SCN
-----------
1301571

Clearly there is a difference. But this by itself does not indicate a problem; since the standby is expected to lag behind the primary (this is an asynchronous non-real time apply setup). The real question is how much it is lagging in the terms of wall clock. To know that I used the scn_to_timestamp function to translate the SCN to a timestamp:

SQL> select scn_to_timestamp(1447102) from dual;

SCN_TO_TIMESTAMP(1447102)
-------------------------------
18-DEC-09 08.54.28.000000000 AM

I ran the same query to know the timestamp associated with the SCN of the standby database as well (note, I ran it on the primary database, though; since it will fail in the standby in a mounted mode):

SQL> select scn_to_timestamp(1301571) from dual;

SCN_TO_TIMESTAMP(1301571)
-------------------------------
15-DEC-09 07.19.27.000000000 PM

This shows that the standby is two and half days lagging! The data at this point is not just stale; it must be rotten.

The next question is why it would be lagging so far back in the past. This is a 10.2 database where FAL server should automatically resolved any gaps in archived logs. Something must have happened that caused the FAL (fetch archived log) process to fail. To get that answer, first, I checked the alert log of the standby instance. I found these lines that showed the issue clearly:


Fri Dec 18 06:12:26 2009
Waiting for all non-current ORLs to be archived...
Media Recovery Waiting for thread 1 sequence 700
Fetching gap sequence in thread 1, gap sequence 700-700
… …
Fri Dec 18 06:13:27 2009
FAL[client]: Failed to request gap sequence
GAP - thread 1 sequence 700-700
DBID 846390698 branch 697108460
FAL[client]: All defined FAL servers have been attempted.


Going back in the alert log, I found these lines:

Tue Dec 15 17:16:15 2009
Fetching gap sequence in thread 1, gap sequence 700-700
Error 12514 received logging on to the standby
FAL[client, MRP0]: Error 12514 connecting to DEL1 for fetching gap sequence
Tue Dec 15 17:16:15 2009
Errors in file /opt/oracle/admin/DEL2/bdump/del2_mrp0_18308.trc:
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
Tue Dec 15 17:16:45 2009
Error 12514 received logging on to the standby
FAL[client, MRP0]: Error 12514 connecting to DEL1 for fetching gap sequence
Tue Dec 15 17:16:45 2009
Errors in file /opt/oracle/admin/DEL2/bdump/del2_mrp0_18308.trc:
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

This clearly showed the issue. On December 15th at 17:16:15, the Managed Recovery Process encountered an error while receiving the log information from the primary. The error was ORA-12514 “TNS:listener does not currently know of service requested in connect descriptor”.
This is usually the case when the TNS connect string is incorrectly specified. The primary is called DEL1 and there is a connect string called DEL1 in the standby server.

The connect string works well. Actually, right now there is no issue with the standby getting the archived logs; so there connect string is fine - now.
The standby is receiving log information from the primary. There must have been some temporary hiccups causing that specific archived log not to
travel to the standby. If that log was somehow skipped (could be an intermittent problem), then it should have been picked by the FAL process
later on; but that never happened. Since the sequence# 700 was not applied, none of the logs received later – 701, 702 and so on – were applied either.
This has caused the standby to lag behind since that time.

So, the fundamental question was why FAL did not fetch the archived log sequence# 700 from the primary. To get to that, I looked into the alert
log of the primary instance. The following lines were of interest:


Tue Dec 15 19:19:58 2009
Thread 1 advanced to log sequence 701 (LGWR switch)
Current log# 2 seq# 701 mem# 0: /u01/oradata/DEL1/onlinelog/o1_mf_2_5bhbkg92_.log
Tue Dec 15 19:20:29 2009Errors in file /opt/oracle/product/10gR2/db1/admin/DEL1/bdump/del1_arc1_14469.trc:
ORA-00308: cannot open archived log '/u01/oraback/1_700_697108460.dbf'
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
Tue Dec 15 19:20:29 2009
FAL[server, ARC1]: FAL archive failed, see trace file.
Tue Dec 15 19:20:29 2009
Errors in file /opt/oracle/product/10gR2/db1/admin/DEL1/bdump/del1_arc1_14469.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed.
Archiver continuing
Tue Dec 15 19:20:29 2009
ORACLE Instance DEL1 - Archival Error. Archiver continuing.


These lines showed everything clearly. The issue was:

ORA-00308: cannot open archived log '/u01/oraback/1_700_697108460.dbf'
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory

The archived log simply was not available. The process could not see the file and couldn’t get it across to the standby site.

Upon further investigation I found that the DBA actually removed the archived logs to make some room in the filesystem without
realizing that his action has removed the most current one which was yet to be transmitted to the remote site. The mystery surrounding why the
FAL did not get that log was finally cleared.


==========
Solution
==========
Now that I know the cause, the focus was now on the resolution. If the archived log sequence# 700 was available on the primary, I could have easily copied it over to the standby, registered the log file and let the managed recovery process pick it up. But unfortunately, the file was gone and I couldn’t just recreate the file. Until that logfile was applied, the recovery will not move forward. So, what are my options?

One option is of course to recreate the standby - possible one but not technically feasible considering the time required. The other option is to apply the incremental backup of primary from that SCN number. That’s the key – the backup must be from a specific SCN number. I have described the process since it is not very obvious. The following shows the step by step approach for resolving this problem. I have shown where the actions must be performed – [Standby] or [Primary].

1. [Standby] Stop the managed standby apply process:

SQL> alter database recover managed standby database cancel;

Database altered.

2. [Standby] Shutdown the standby database

3. [Primary] On the primary, take an incremental backup from the SCN number where the standby has been stuck:

------------------------------------------- Rman Backup Status ------------------------------------------

SELECT OPERATION, STATUS, MBYTES_PROCESSED, START_TIME, END_TIME from V$RMAN_STATUS;

select
  sid,
  start_time,
  totalwork
  sofar,
 (sofar/totalwork) * 100 pct_done
from
   v$session_longops
where totalwork > sofar
AND opname NOT LIKE '%aggregate%'
AND opname like 'RMAN%';


select sid, spid, client_info,
   event,  seconds_in_wait,
   p1, p2, p3
 from  v$process p, v$session s
 where p.addr = s.paddr
 and client_info like 'rman channel=%';
-----------------------------------------------------------------------------------------------------------

RMAN> run {
2> allocate channel c1 type disk format '/u01/oraback/%U.rmb';
3> backup incremental from scn 1301571 database;
4> }

using target database control file instead of recovery catalog
allocated channel: c1
channel c1: sid=139 devtype=DISK

Starting backup at 18-DEC-09
channel c1: starting full datafile backupset
channel c1: specifying datafile(s) in backupset
input datafile fno=00001 name=/u01/oradata/DEL1/datafile/o1_mf_system_5bhbh59c_.dbf
… …
piece handle=/u01/oraback/06l16u1q_1_1.rmb tag=TAG20091218T083619 comment=NONE
channel c1: backup set complete, elapsed time: 00:00:06
Finished backup at 18-DEC-09
released channel: c1

4. [Primary] On the primary, create a new standby controlfile:

SQL> alter database create standby controlfile as '/u01/oraback/DEL1_standby.ctl';

Database altered.

5. [Primary] Copy these files to standby host:

oracle@oradba1 /u01/oraback# scp *.rmb *.ctl oracle@oradba2:/u01/oraback
oracle@oradba2's password:
06l16u1q_1_1.rmb 100% 43MB 10.7MB/s 00:04
DEL1_standby.ctl 100% 43MB 10.7MB/s 00:04

6. [Standby] Bring up the instance in nomount mode:

SQL> startup nomount


7. [Standby] Check the location of the controlfile:

SQL> show parameter control_files

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_files string /u01/oradata/standby_cntfile.ctl


8. [Standby] Replace the controlfile with the one you just created in primary.

9. $ cp /u01/oraback/DEL1_standby.ctl /u01/oradata/standby_cntfile.ctl

10.[Standby] Mount the standby database:

SQL> alter database mount standby database;

11.[Standby] RMAN does not know about these files yet; so you must let it know – by a process called cataloging. Catalog these files:

$ rman target=/

Recovery Manager: Release 10.2.0.4.0 - Production on Fri Dec 18 06:44:25 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: DEL1 (DBID=846390698, not open)
RMAN> catalog start with '/u01/oraback';

using target database control file instead of recovery catalog
searching for all files that match the pattern /u01/oraback

List of Files Unknown to the Database
=====================================
File Name: /u01/oraback/DEL1_standby.ctl
File Name: /u01/oraback/06l16u1q_1_1.rmb

Do you really want to catalog the above files (enter YES or NO)? yes
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /u01/oraback/DEL1_standby.ctl
File Name: /u01/oraback/06l16u1q_1_1.rmb


12.Recover these files:

RMAN> recover database;

Starting recover at 18-DEC-09
using channel ORA_DISK_1
channel ORA_DISK_1: starting incremental datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: /u01/oradata/DEL2/datafile/o1_mf_system_5lptww3f_.dbf
...…
channel ORA_DISK_1: reading from backup piece /u01/oraback/05l16u03_1_1.rmb
channel ORA_DISK_1: restored backup piece 1
piece handle=/u01/oraback/05l16u03_1_1.rmb tag=TAG20091218T083619
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07

starting media recovery

archive log thread 1 sequence 8012 is already on disk as file /u01/oradata/1_8012_697108460.dbf
archive log thread 1 sequence 8013 is already on disk as file /u01/oradata/1_8013_697108460.dbf
… …

13. After some time, the recovery fails with the message:

archive log filename=/u01/oradata/1_8008_697108460.dbf thread=1 sequence=8009
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 12/18/2009 06:53:02
RMAN-11003: failure during parse/execution of SQL statement: alter database recover logfile '/u01/oradata/1_8008_697108460.dbf'
ORA-00310: archived log contains sequence 8008; sequence 8009 required
ORA-00334: archived log: '/u01/oradata/1_8008_697108460.dbf'

 ELSE

----I'm Getting this error
starting media recovery
unable to find archive log
archive log thread=1 sequence=37391
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 01/05/2016 10:33:44
RMAN-06054: media recovery requesting unknown log: thread 1 seq 8008 lowscn 2224474915633

RMAN> shutdown immediate



This happens because we have come to the last of the archived logs. The expected archived log with sequence# 8008 has not been generated yet.


14.At this point exit RMAN and start managed recovery process:

SQL> alter database recover managed standby database disconnect from session;

Database altered.

15.Check the SCN’s in primary and standby:

[Standby] SQL> select current_scn from v$database;

CURRENT_SCN
-----------
1447474

[Primary] SQL> select current_scn from v$database;

CURRENT_SCN
-----------
1447478

Now they are very close to each other. The standby has now caught up.


Use Roll Forward Feature12c R2 onwards:
Roll Forward Physical Standby Using RMAN Incremental Backup in Single Command (Doc ID 2431311.1) 

Sunday, 25 January 2015

adsplice

USE PROD instance

------- Pre-install Tasks -----------
You must shut down all Application tier services before performing the tasks in this section.



--------AD Splicer---------
• AD Splicer registers off-cycle product as a valid Oracle Applications product for the given release
• The splicing process enables AutoPatch and AD Administration to recognize the off-cycle product
• AD Splicer uses control files to manage the splicing process

– These are not related to the database control files
AD Splicer Control Files
Off-cycle products are generally released as patches. All product files and control files necessary to splice in the product are included in the patch.
There is one product configuration file, called newprods.txt. The product configuration file must be edited for your specific system before the off-cycle product can be properly spliced.
Note: The product definition files must not be altered.

There are two product definition files per off-cycle product. They are
• <prod>prod.txt: which contains language-independent information for product <prod>
• <prod>terr.txt: which contains language-dependent information for product <prod> <prod> represents the product abbreviation for an Applications product. For example GL for General Ledger and PO for Purchasing.
The AD Splicer control files must copied from the patch to the APPL_TOP/admin directory.



-------Editing newprod.txt (Controlfile)-------------

-bash-3.2$ more newprods.txt

%%% Splicer control file format 12.0.A
#
# $Header: newprods.txt 120.1 2006/09/20 12:57:45 jaobrien noship $
#
#
# Section for Oracle Support Diagnostic Tools

product=izu
base_product_top=*APPL_TOP*
oracle_schema=izu
sizing_factor=100
main_tspace=APPS_TS_TX_DATA
index_tspace=APPS_TS_TX_IDX
temp_tspace=TEMP1
default_tspace=APPS_TS_TX_DATA

# end of Oracle Support Diagnostic Tools



------Post-Splice Steps--------
After the new product is spliced in, the files and objects for this product can be installed. To do this you need to:
• Log out and log back in so that you are using the new environment file (or registry entries) to set up your environment
• Verify that <PROD>_TOP environment variables are set for your newly-spliced product
• Run AutoPatch to install files and database objects for your new product




Example : Patch - ADSPLICE PATCH FOR IZU - Release 12 - 6280873
IZU Splice Preparation Tasks for R12
Customers with multi-node installations of eBusiness Suite should apply this
adsplice patch on all nodes.

Before you run adpatch you must complete the steps below:

1. Unzip this patch and you will get directory 6280873.

2. Copy the following text files from 6280873/izu/admin to the admin directory under APPL_TOP.
     izuprod.txt
     izuterr.txt
     newprods.txt

3. Change directory to the admin directory under APPL_TOP.
   Review newprods.txt and verify that the tablespaces listed exist and are the ones planned for use by this product. If changes are required, edit the file as appropriate and save it.

   cd $APPL_TOP/admin
   vi newprod.txt

4. You must run "adsplice" from the admin directory under your APPL_TOP.
   When you are asked to enter the name of your AD Splicer control file, enter newprods.txt
   adsplice
   Filename [adsplice.log]

**********************************************************************************************************************

-bash-3.2$ cd $APPL_TOP/admin

-bash-3.2$ adsplice

                     Copyright (c) 2002 Oracle Corporation
                        Redwood Shores, California, USA

                                   AD Splicer

                                 Version 12.0.0

NOTE: You may not use this utility for custom development
      unless you have written permission from Oracle Corporation.

Your default directory is '/u01/oracle/VIS/apps/apps_st/appl'.
Is this the correct APPL_TOP [Yes] ?

AD Splicer records your AD Splicer session in a text file
you specify.  Enter your AD Splicer log file name or press [Return]
to accept the default file name shown in brackets.

Filename [adsplice.log] : izu_adsplice.log

************* Start of AD Splicer session *************
AD Splicer version: 12.0.0
AD Splicer started at: Sun Jan 25 2015 12:39:45

APPL_TOP is set to /u01/oracle/VIS/apps/apps_st/appl

Change DB = YES.


Please enter the name of the Oracle Applications System that this
APPL_TOP belongs to.

The Applications System name must be unique across all Oracle
Applications Systems at your site, must be from 1 to 30 characters
long, may only contain alphanumeric and underscore characters,
and must start with a letter.

Sample Applications System names are: "prod", "test", "demo" and
"Development_2".

Applications System Name [VIS] : VIS *


NOTE: If you do not currently have certain types of files installed
in this APPL_TOP, you may not be able to perform certain tasks.

Example 1: If you don't have files used for installing or upgrading
the database installed in this area, you cannot install or upgrade
the database from this APPL_TOP.

Example 2: If you don't have forms files installed in this area, you cannot
generate them or run them from this APPL_TOP.

Example 3: If you don't have concurrent program files installed in this area,
you cannot relink concurrent programs or generate reports from this APPL_TOP.


Do you currently have or want to install files  used for installing or upgrading
the database in this APPL_TOP [YES] ? YES *


Do you currently have or want to install Java and HTML files for HTML-based
functionality in this APPL_TOP [YES] ? YES *


Do you currently have or want to install Oracle Applications forms files
in this APPL_TOP [YES] ? YES *


Do you currently have or want to install concurrent program files
in this APPL_TOP [YES] ? YES *


Please enter the name Oracle Applications will use to identify this APPL_TOP.

The APPL_TOP name you select must be unique within an Oracle Applications
System, must be from 1 to 30 characters long, may only contain
alphanumeric and underscore characters, and must start with a letter.

Sample APPL_TOP Names are: "prod_all", "demo3_forms2", and "forms1".

APPL_TOP Name [oracle] : oracle *



You are about to install or upgrade Oracle Applications product tables
in your ORACLE database 'VIS'
using ORACLE executables in '/u01/oracle/VIS/apps/tech_st/10.1.2'.

Is this the correct database [Yes] ?

AD Splicer needs the password for your 'SYSTEM' ORACLE schema
in order to determine your installation configuration.

Enter the password for your 'SYSTEM' ORACLE schema:


The ORACLE username specified below for Application Object Library
uniquely identifies your existing product group: APPLSYS

Enter the ORACLE password of Application Object Library [APPS] :

AD Splicer is verifying your username/password.
The status of various features in this run of AD Splicer is:

                                           <-Feature version in->
Feature                          Active?   APPLTOP    Data model    Flags
------------------------------   -------   --------   -----------   -----------
CHECKFILE                        Yes       1          1             Y N N Y N Y
PREREQ                           Yes       6          6             Y N N Y N Y
CONCURRENT_SESSIONS              No        2          2             Y Y N Y Y N
PATCH_TIMING                     Yes       2          2             Y N N Y N Y
PATCH_HIST_IN_DB                 Yes       6          6             Y N N Y N Y
SCHEMA_SWAP                      Yes       1          1             Y N N Y Y Y
JAVA_WORKER                      Yes       1          1             Y N N Y N Y
CODELEVEL                        Yes       1          1             Y N N Y N Y



Reading product information from file...

Reading language and territory information from file...

Reading language information from applUS.txt ...

AD Splicer warning:
 Product Data File
 /u01/oracle/VIS/apps/apps_st/appl/admin/zfaprod.txt
 does not exist for product "zfa".
This product is registered in the database but the
above file does not exist in APPL_TOP.  The product
will be ignored without error.


AD Splicer warning:
 Product Data File
 /u01/oracle/VIS/apps/apps_st/appl/admin/zsaprod.txt
 does not exist for product "zsa".
This product is registered in the database but the
above file does not exist in APPL_TOP.  The product
will be ignored without error.


AD Splicer warning:
 Product Data File
 /u01/oracle/VIS/apps/apps_st/appl/admin/jtsprod.txt
 does not exist for product "jts".
This product is registered in the database but the
above file does not exist in APPL_TOP.  The product
will be ignored without error.


Reading database to see what industry is currently installed.

Reading FND_LANGUAGES to see what is currently installed.
Currently, the following language is installed:

Code   Language                                Status
----   --------------------------------------- ---------
US     American English                        Base

Your base language will be AMERICAN.

Setting up module information.
Reading database for information about the modules.
Saving module information.
Reading database for information about the products.
Reading database for information about how products depend on each other.
Reading topfile.txt ...

Saving product information.

Please enter the directory where your AD Splicer control file is located.

The default directory is [/u01/oracle/VIS/apps/apps_st/appl/admin] :

Please enter the name of your AD Splicer control file [newprods.txt] :


Processing file /u01/oracle/VIS/apps/apps_st/appl/admin/newprods.txt...
  Loading information for product 'izu'...

Warning: Because the database is in new tablespace mode, AD Splicer is ignoring the tablespace settings in your control file!

Successfully read file /u01/oracle/VIS/apps/apps_st/appl/admin/newprods.txt.


Verifying localizations...

Validating inter-product dependency information...

Re-organizing product information lists...


Adding new languages into FND_LANGUAGES...

Saving module actions...done.

Saving product actions...done.

Product top directory for product "izu" exists.
Directory is /u01/oracle/VIS/apps/apps_st/appl/izu/12.0.0

  Directory "log" for product "izu" exists.
  Directory "out" for product "izu" exists.
  Directory "mesg" for product "izu" exists.
  Directory "sql" for product "izu" exists.
    File "sql/IZUNLINS.sql" for product "izu" exists.
  Directory "admin/sql" for product "izu" exists.
    File "admin/sql/IZUNLADD.sql" for product "izu" exists.
  Directory "admin/driver" for product "izu" exists.


Do you wish to regenerate your environment file [Yes] ?

Checking if AutoConfig is enabled....

Running AutoConfig on : All products ...

The log file is located at:
/u01/oracle/VIS/apps/apps_st/appl/admin/VIS/log/autoconfig_1.log

Please check the log file for more details about the run of AutoConfig.

AD Splicer is complete.

Errors and warnings are listed in the log file
/u01/oracle/VIS/apps/apps_st/appl/admin/VIS/log/izu_adsplice.log

and in other log files in the same directory.

You have new mail in /var/spool/mail/oraapp

-bash-3.2$

**********************************************************************************************************************



5. AD Splicer will regenerate the environment file or registry. If you have made manual modifications to your environment file or your registry, you will need to apply those changes again.



APPLY THE PATCH
----------------------------
1. cd $ADMIN_SCRIPTS_HOME
sh adstpall.sh apps/apps

2. Enable Maintenance Mode

3. adpatch




Verifying splicing of IZU
---------------------------------
A. Log out and log back in so that you are using the new environment
  file (or registry entries ) to set up your environment
  exit
  su - oraapp
  . ./APPSVIS_oracle.env

B. Verify that IZU_TOP environment variable is set for your newly-spliced product IZU


C. Verify that the product has been correctly created in the database by connection as "APPS" user.
select fa.APPLICATION_ID, STATUS
from fnd_application fa, fnd_product_installations fpi
where fa.APPLICATION_ID=fpi.APPLICATION_ID and APPLICATION_SHORT_NAME = 'IZU' ;

This statement should return one row:
APPLICATION_ID S
-------------- -
278 I


D. Verify you can connect to the database as izu/izu using SQLPLUS.


Tuesday, 20 January 2015

Oracle 11g GoldenGate Installation. Part 1

 MY ENV.

2 DB 10g on two different
Machine 192.168.2.29    (10g)
Machine 192.168.2.30    (10g)
Each & every node install GG  because of the purpose to replicate a data. And tns / Listener configure.

Oracle 11g GoldenGate Installation.
Download the GoldenGate software from Oracle Edelivery.

Select a Product Pack: Oracle Fusion Middleware
Platform: Linux X86


Download the Oracle GoldenGate media pack.

Create the directory for GoldenGate.


1.
$ mkdir -p /u01/app/gg


2.
  Make changes in your bash profile.
  Add path variable.Like this.
  # PATH=/usr/sbin:/u01/app/gg:$PATH; export PATH
  # PATH=$ORACLE_HOME/bin:/u01/app/gg:$PATH; export PATH

  PATH=/u01/app/gg:$PATH; export PATH

  LD library entry should look something like this.
  # LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/u01/app/gg; export LD_LIBRARY_PATH

  LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/u01/app/gg; export LD_LIBRARY_PATH

3.
  Reload bash profile
  $ . ./.bash_profile

export PATH
# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
GG_HOME=/u01/app/gg; export GG_HOME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1; export ORACLE_HOME
ORACLE_SID=ORCLDB; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=/usr/sbin:/u01/app/gg:$PATH; export PATH
PATH=$ORACLE_HOME/bin:/u01/app/gg:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/u01/app/gg; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

4.
   copy the tar file to the "gg" directory and untar

   tar -xvof fbo_ggs_Linux_x86_ora11g_32bit.tar 

   Invoke the GoldenGate Software Command Interface.

-------------------------------------------------------------------------------------------------------------------------------------
5.
Using DB10g and GG11g
./ggsci      #Below Getting Error
./ggsci
./ggsci: error while loading shared libraries: libnnz11.so:
cannot open shared object file: No such file or directory

10g
[oracle@localhost gg]$ ln -s /u01/app/oracle/product/10.2.0/db_1/lib/libnnz10.so libnnz11.so
[oracle@localhost gg]$ ln -s /u01/app/oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 libclntsh.so.11.1

11g
ln -s /u01/app/oracle/product/11.2.0/db_1/lib/libnnz11.so libnnz11.so
ln -s /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1 libclntsh.so.11.1


[oracle@localhost gg]$ pwd
/u01/app/gg
[oracle@localhost gg]$ ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.1.1.1.2 OGGCORE_11.1.1.1.2_PLATFORMS_111004.2100 
Linux, x86, 32bit (optimized), Oracle 11g on Oct 4 2011 23:53:33

Copyright (C) 1995, 2011, Oracle and/or its affiliates. All rights reserved.



GGSCI (localhost) 1> create subdirs
Code:
Creating subdirectories under current directory /u01/app/gg

Parameter files                /u01/app/gg/dirprm: created
Report files                   /u01/app/gg/dirrpt: created
Checkpoint files               /u01/app/gg/dirchk: created
Process status files           /u01/app/gg/dirpcs: created
SQL script files               /u01/app/gg/dirsql: created
Database definitions files     /u01/app/gg/dirdef: created
Extract data files             /u01/app/gg/dirdat: created
Temporary files                /u01/app/gg/dirtmp: created
Veridata files                 /u01/app/gg/dirver: created
Veridata Lock files            /u01/app/gg/dirver/lock: created
Veridata Out-Of-Sync files     /u01/app/gg/dirver/oos: created
Veridata Out-Of-Sync XML files /u01/app/gg/dirver/oosxml: created
Veridata Parameter files       /u01/app/gg/dirver/params: created
Veridata Report files          /u01/app/gg/dirver/report: created
Veridata Status files          /u01/app/gg/dirver/status: created
Veridata Trace files           /u01/app/gg/dirver/trace: created
Stdout files                   /u01/app/gg/dirout: created

GGSCI (localhost) 2> exit

Before going any further make sure your database is in
archivelog mode.

Prepare your database for replication.

In order to extract committed transactions from the source database
the database must be configured with supplemental logging on primary
key columns.

SQL> alter database add supplemental log data (primary key) columns;

Database altered.

Do a logswitch to start supplemental logging.

SQL> alter system switch logfile;

System altered.

SQL> select supplemental_log_data_min, supplemental_log_data_pk,
supplemental_log_data_ui from v$database;


SUPPLEME SUP SUP
-------- --- ---
IMPLICIT YES NO


SUPPLEMENTAL_LOG_DATA_MIN: Ensures that logminer will have sufficient information.
SUPPLEMENTAL_LOG_DATA_PK: Indicates whether all columns of the primary key are placed into
the redo log whenever an update is performed.
SUPPLEMENTAL_LOG_DATA_UI: Indicated whether all columns belonging to the unique key are placed
into the redo log if any of the unique columns are modified.

Create a Tablespace and user for GoldenGate.

SQL> create tablespace golden datafile'/u01/app/oracle/oradata/orcl/golden01.dbf' size 100m
2 autoextend on
3 extent management local
4 segment space management auto;

Tablespace created.

SQL> create user ggs_admin identified by ggs_admin temporary tablespace temp default tablespace golden quota unlimited on golden;

User created.


SQL> grant dba to ggs_admin;

Grant succeeded.

SQL> alter system set recyclebin=off scope=spfile;

System altered.

--Set recyclebin to off, Necessary for DDL Replication from Oracle 10g onwards.


SQL> shutdown immediate
SQL> startup




Installing GoldenGate for Oracle RAC (Doc ID 966181.1)

How do I install GoldenGate on an Oracle RAC?

There are multiple variations of installing GoldenGate for Oracle RAC.


Installing GoldenGate on Shared Drives

This approach works very well. The software is installed on shared drives that all the RAC nodes have access to. The manager and GoldenGate processes can be started from one of the nodes. If the node that the GoldenGate processes were started from fails, the checkpoints are maintained. The manager and GoldenGate processes then can be started from another node without making any modifications to any parameter files. If the database alone crashes on the node that GoldenGate was installed, then the manager process will need to be stopped on that node before it be restarted on another node.

Installing GoldenGate on local disk

This approach will not work in a fail over situation. If there was a database failure alone on this node, the software could be installed on a different node. For the replicat , all the trails would need to be moved to that node. The extract and replicat could be added and positioned to that last known good checkpoint. If there were a complete node failure, this would be very difficult to recover from. It would be impossible to know what the last good checkpoint was and we would have no access to any trails that were located on those local disks.

Connecting to the RAC instance from a application server using SQL*Net

The software is installed on a separate machine. GoldenGate is setup to connect to the RAC Instance via the Global Database name which is setup in TNSNAME.ORA. By connecting to the Global Database using SQL*Net, Oracle will handle the fail over. For extract, the oracle logs will need to be located on shared disk that this server has access to.

When adding your online Extract groups, be sure to specify the THREADS option on the ADD command.



Part 2

Wednesday, 14 January 2015

Upgrading ASM from Oracle 10g to Oracle 11g Release 2

This note describes the steps for upgrading ASM running on Oracle 10.2.0.4 to Oracle 11.2.0.1 on a HP UX platform.

To upgrade the ASM, we need to install the 11g R2 Grid Infrastructure software as ASM is now no longer part of the database software as was the case in Oracle 10g.

We launch the installer from Grid Infrastructure software directory - unzip the file hpia64_11gR2_grid.zip which we have downloaded from OTN.



Optional -------------------- When you are using HP UX platform normally ignore this step
This error “Unable to get the current group” when we launch the installer is a known error for HP UX which is resolved by applying the OS patch OS patch PHCO_40381



----------------------------------------------------------------------------------------------

Ensure the “Upgrade Grid Infrastructure” is chosen

 
The OUI detects the running 10g ASM instance and automatically the Upgrade Grid Infrastructure option is selected.



Specify the password for ASMSNMP user



Select the appropriate group – we use dba group for all three – ignore the warning message which will follow. 




One of the new 11g OUI features is that it generates some fix-up scripts which can be run to change or modify certain kernel parameter settings.

Click on the “Fix and Check Again” button



A fix-up script runfixup.sh has been generated which needs to be run as root – note some parameters cannot be changed on the fly and require a system reboot.

After running the script, we click on OK and then on Check Again.



Even after running the fix-up script, some kernel parameters cannot be modified without a reboot.

Since this is a test, we do not do a machine reboot at this stage and we check the Ignore All box and click on Next


Note that Migrate ASM is showing the value true








root@inpsidt1# ./rootupgrade.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /data/oracle/product/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2011-05-17 12:24:49: Checking for super user privileges
2011-05-17 12:24:49: User has super user privileges
2011-05-17 12:24:49: Parsing the host name
Using configuration parameter file: /data/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
stty: : Not a typewriter
stty: : Not a typewriter
stty: : Not a typewriter
mesg: cannot change mode
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
CSS appears healthy
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
CRS-4664: Node inpsidt1 successfully pinned.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting

inpsidt1     2011/05/17 12:26:18     /data/oracle/product/11.2.0/grid/cdata/inpsidt1/backup_20110517_122618.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 7591 MB    Passed
The inventory pointer is located at /var/opt/oracle/oraInst.loc
The inventory is located at /home/oracle/oraInventory
'UpdateNodeList' was successful.





Automatic Storage Management Configuration Assistant ASMCA is a new tool in Oracle 11g for performing ASM administration.

Previously we used to do the same via DBCA in Oracle 10g.

From the installation log files we can see that ASM is being upgraded by the ASMCA which has been invoked by the OUI.


INFO: Starting 'Automatic Storage Management Configuration Assistant'
INFO: Starting 'Automatic Storage Management Configuration Assistant'
INFO: Executing ASMCA
INFO: Command /data/oracle/product/11.2.0/grid/bin/asmca -silent -oui_internal -upgradeASM
INFO: ... GenericInternalPlugIn.handleProcess() entered.
INFO: ... GenericInternalPlugIn: getting configAssistantParmas.
INFO: ... GenericInternalPlugIn: checking secretArguments.
INFO: ... GenericInternalPlugIn: starting read loop.
INFO: Read: ASMSNMP_PASSWORD_PROMPT
INFO: Processing: ASMSNMP_PASSWORD_PROMPT for argument tag –asmMonitorPassword
INFO: End of argument passing to stdin
INFO: Read:
INFO: Read: ASM upgraded and started successfully.
INFO: Read:
INFO: Completed Plugin named: Automatic Storage Management Configuration Assistant




We can see that the oratab file has also been updated and the ORACLE_HOME for the ASM instance is now showing the 11g Oracle Home
.
root@inpsidt1# cat /etc/oratab |grep ASM

+ASM:/data/oracle/product/11.2.0/grid:N


Let us now add a new disk to the previously existing disk group DATA which was created in 10g ASM.

We launch ASMCA via asmca from Grid Infrastructure software home









Note there are two types of compatibility for the diskgroup. One is the ASM compatibility and the other is Database compatibility.
  • ASM Compatibility: The minimum version of the ASM software that can access the disk group. In 11g, the default setting is 10.1.
  • Database Compatibility:            The minimum COMPATIBLE database initialization parameter setting for any database instance that uses the disk group. In 11g, the default setting is 10.1.


Note – before performing the ASM upgrade, shut down any Oracle databases on the same machine or they will be abnormally shutdown when communication to the ASM instance is lost as the existing 10g ASM instance is shutdown as part of the ASM upgrade process.

We can see lines like this in the database alert log:


Errors in file /data/oracle/product/10.2.0.4.5/admin/test11g/bdump/test11g_asmb_15099.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Tue May 17 12:25:08 2011
ASMB: terminating instance due to error 15064

Tuesday, 13 January 2015

TWO DIFFR EBS USER AT SAME TIME

Whenever we login with two different EBS users at the same time in the same machine,for the second user everything is working fine,but for the first user if we select any responsibility it showing the error message :

Two Method 

1.running on two different explorer.


2. Below the steps.







Errors and resolution in OAF

You have insufficient privileges for the current operation. Please contact your System Administrator.


You cannot run a page which is not SelfSecured when the MAC fails.



Solution:
=========


Set the profiles of the below three to 'None'


FND Validation Level
FND Function Validation Level
Framework Validation Level



=======
Error:
=======

when I am running, the fwktutorial.jsp, i am getting the links for hello world, etc. However when i click on the links, i am getting the follwoing error;

error:

Unexpected URL parameters have been detected and will be ignored. 



Solution:
=========

Go to Tools-->Embedded OC4J Preferences-->Global-->Startup-->Select Default Local IP Address.