Showing posts with label RAC / ASM. Show all posts
Showing posts with label RAC / ASM. Show all posts

Monday, 21 August 2023

Multiplexing Controlfile in ASM

 Manual Method:

================

1.

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

control_files                        string      +DATA/racdb/controlfile/current.260.979677429


[oracle@rac1 ~]$ srvctl stop database -d racdb


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

2.

ASMCMD> lsdg

State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  NORMAL  N         512   4096  1048576     30717    27208             1171           13018              0             N  DATA/

MOUNTED  NORMAL  N         512   4096  1048576      3069     2143              309             917              0             N  OCR/


cd OCR

mkdir racdb

cd racdb

mkdir controlfile


ASMCMD> pwd

+DATA/RACDB/CONTROLFILE


ASMCMD> cp +DATA/RACDB/CONTROLFILE/Current.260.979677429 +OCR/racdb/controlfile/current

copying +DATA/RACDB/CONTROLFILE/Current.260.979677429 -> +OCR/racdb/controlfile/current


ASMCMD> cd +OCR/racdb/controlfile/


ASMCMD> ls -l

Type         Redund  Striped  Time             Sys  Name

                                               N    current => +OCR/ASM/CONTROLFILE/current.256.1144794611


--asm_alert.log

Sun Aug 13 22:27:42 2023

SQL> /* ASMCMD */alter diskgroup /*ASMCMD*/ "OCR" add directory '+OCR/racdb'

SUCCESS: /* ASMCMD */alter diskgroup /*ASMCMD*/ "OCR" add directory '+OCR/racdb'

Sun Aug 13 22:27:56 2023

SQL> /* ASMCMD */alter diskgroup /*ASMCMD*/ "OCR" add directory '+OCR/racdb/controlfile'

SUCCESS: /* ASMCMD */alter diskgroup /*ASMCMD*/ "OCR" add directory '+OCR/racdb/controlfile'


--Output

ASMCMD> pwd

+OCR/racdb/controlfile

ASMCMD> ls -l

Type         Redund  Striped  Time             Sys  Name

                                               N    current => +OCR/ASM/CONTROLFILE/current.256.1144794611


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


3.

[oracle@rac1 ~]$ srvctl start database -d racdb -o nomount

show controlfile

alter system set control_files='+DATA/racdb/controlfile/current.260.979677429','+OCR/racdb/controlfile/current' scope=spfile sid='*';


srvctl stop database -d racdb


[oracle@rac1 ~]$ srvctl start database -d racdb

[oracle@rac1 ~]$ srvctl status database -d racdb

Instance racdb1 is running on node rac1

Instance racdb2 is running on node rac2


NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

control_files                        string      +DATA/racdb/controlfile/current.260.979677429, +OCR/racdb/controlfile/current


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

4. clean controlfile from old DG if you do moving.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


===============================

RMAN CONTROLFILE Multiplexing:

===============================


1. Identify the location of the current controlfile:

SQL> select name from v$controlfile;


NAME

--------------------------------------------------------------------------------

+DATA/racdb/controlfile/current.260.979677429



2. Shutdown the database and start the instance:

    SQL> shutdown normal

    SQL> startup nomount


3. Use RMAN to duplicate the controlfile:

    $ rman nocatalog

    RMAN>connect target

    RMAN>restore controlfile to '<DISKGROUP_NAME>' from '<OLD_PATH>';

    RMAN> restore controlfile to '+OCR' from '+DATA/racdb/controlfile/current.260.979677429';


4. On the ASM instance, identify the name of the controlfile:

    Using ASMCMD:

    $ asmcmd

    ASMCMD> cd OCR

    ASMCMD> find -t controlfile . *


 Changing the current directory to the diskgroup where the controlfile was created will speed the search.

    Output:

ASMCMD> find -t controlfile . *

WARNING:option 't' is deprecated for 'find'

please use 'type'

+OCR/RACDB/CONTROLFILE/current.256.1144796825


5. On the database side:

##OPTIONAL STEP spfile and init file manual changes.

            * Modify init.ora or spfile, adding the new path to parameter control_files.

            * if using init<SID>.ora, just modify the control_files parameter and restart the database.

            * If using spfile,


            1) startup nomount the database instance

            2) alter system set control_files='+DG1/P10R2/CONTROLFILE/backup.308.577785757','/oradata2/102b/oradata/P10R2/control01.ctl' scope=spfile;


            For RAC instance:

            alter system set control_files='+DATA/racdb/controlfile/current.260.979677429','+OCR/RACDB/CONTROLFILE/current.256.1144796825' scope=spfile sid='*';


            3) shutdown immediate



        * start the instance.


            Verify that new control file has been recognized. If the new controlfile was not used, the complete procedure needs to be repeated.


6. 

[oracle@rac1 ~]$ srvctl start database -d racdb

[oracle@rac1 ~]$ srvctl status database -d racdb

Instance racdb1 is running on node rac1

Instance racdb2 is running on node rac2


Saturday, 16 July 2022

How To setup Oracle RDA and Use

RDA captures data and provides Oracle support with a comprehensive picture of the customer's environment which aids in problem diagnosis. The goal of this article is to detail the steps on how to gather basic configuration, log and platform information from Oracle OpenSSO using the Remote Diagnostic Agent.


Instructions for UNIX/ZLinux Type Operating Systems

Choose or create a directory or area on your UNIX server. Make sure you have sufficient space for the RDA output (150MB). It does not matter where you create this directory or what it is named, but the same user that runs RDA must own it. Do not use a directory that contains an older version of RDA unless you have deleted the previous version of RDA first. If necessary, you can reuse prior setup files.

Note: The rda.zip creates a directory named "rda" containing all the required files when you extract it.


Installation Instruction

Do not extract the contents of the RDA archive on a Windows client first or you will have to remove the ^M characters from the end of each line in all of the shell scripts in order for them to run.

downloaded rda.zip file from oracle support as compitable your OS, I'm using UNIX server.

step 1.

Extract the .zip archive contents into a new directory, preserving the directory structure of the archive. Do not extract into a directory that contains an older RDA version. For example:

   unzip rda.zip


step 2.

Change directory to the 'rda' directory created by the above command, for example:

   cd rda


setp 3.

Make sure the RDA command 'rda.sh' is executable. To verify, enter the following command:

    ls -l rda.sh


If the script is not executable, update it using the command:

    chmod +x rda.sh


step 4.

The following command checks that you have a viable installation of RDA by checking that all file versions are as expected:

  ./rda.sh -cv

output like below
===================
Loading the file list ...
Checking the directory [D_RDA] . ...
Checking the directory [D_RDA] engine ...
Checking the directory [D_RDA_ADM] model ...
Checking the directory [D_RDA_CHK] APPS ...
Checking the directory [D_RDA_CHK] BI ...
Checking the directory [D_RDA_CHK] CGBU ...
Checking the directory [D_RDA_CHK] DB ...
Checking the directory [D_RDA_CHK] EM ...
Checking the directory [D_RDA_CHK] OFM ...
Checking the directory [D_RDA_CHK] TEST ...
Checking the directory [D_RDA_COL] APPS ...
Checking the directory [D_RDA_COL] BI ...
Checking the directory [D_RDA_COL] CGBU ...
Checking the directory [D_RDA_COL] CLOUD ...
Checking the directory [D_RDA_COL] DA ...
Checking the directory [D_RDA_COL] DB ...
Checking the directory [D_RDA_COL] EM ...
Checking the directory [D_RDA_COL] EXPLORER ...
Checking the directory [D_RDA_COL] OFM ...
Checking the directory [D_RDA_COL] OS ...
Checking the directory [D_RDA_COL] PGBU ...
Checking the directory [D_RDA_COL] RDA ...
Checking the directory [D_RDA_COL] SAMPLE ...
Checking the directory [D_RDA_COL] TOOL ...
Checking the directory [D_RDA_CSS] . ...
Checking the directory [D_RDA_DAT] . ...
Checking the directory [D_RDA_DFW] cv0200 ...
Checking the directory [D_RDA_INC] Convert/Common ...
Checking the directory [D_RDA_INC] Convert/DB/LOG ...
Checking the directory [D_RDA_INC] Convert/OFM/OIM ...
Checking the directory [D_RDA_INC] Convert/OS/INST ...
Checking the directory [D_RDA_INC] Convert/OS/OS ...
Checking the directory [D_RDA_INC] Convert/RDA/CONFIG ...
Checking the directory [D_RDA_INC] Convert/TOOL/ALERT ...
Checking the directory [D_RDA_INC] Convert/TOOL/COMPLY ...
Checking the directory [D_RDA_INC] IRDA ...
Checking the directory [D_RDA_INC] IRDA/CV0200 ...
Checking the directory [D_RDA_INC] RDA ...
Checking the directory [D_RDA_INC] RDA/Agent ...
Checking the directory [D_RDA_INC] RDA/Driver ...
Checking the directory [D_RDA_INC] RDA/Handle ...
Checking the directory [D_RDA_INC] RDA/Library ...
Checking the directory [D_RDA_INC] RDA/Limit ...
Checking the directory [D_RDA_INC] RDA/Local ...
Checking the directory [D_RDA_INC] RDA/Object ...
Checking the directory [D_RDA_INC] RDA/Operator ...
Checking the directory [D_RDA_INC] RDA/Request ...
Checking the directory [D_RDA_INC] RDA/SDCL ...
Checking the directory [D_RDA_INC] RDA/SDSL ...
Checking the directory [D_RDA_INC] RDA/Target ...
Checking the directory [D_RDA_INC] RDA/Token ...
Checking the directory [D_RDA_INC] RDA/UI ...
Checking the directory [D_RDA_INC] RDA/Value ...
Checking the directory [D_RDA_INC] RDA/Web ...
Checking the directory [D_RDA_MSG] charset ...
Checking the directory [D_RDA_MSG] desc ...
Checking the directory [D_RDA_MSG] en ...
Checking the directory [D_RDA_MSG] fr ...
Checking the directory [D_RDA_POD] en ...
No issues found

step 5.

Now run RDA here for Preinstallation checks and co-ordinate with SA for failed checks/fixes.

./rda.sh -T hcve

Output like below
==============
Processing HCVE tests ...
Available Pre-Installation Rule Sets:
   1.  Oracle Database 10g R1 (10.1.0) Preinstall (Linux)
   2.  Oracle Database 10g R2 (10.2.0) Preinstall (Linux)
   3.  Oracle Database 11g R1 (11.1) Preinstall (Linux)
   4.  Oracle Database 11g R2 (11.2.0) Preinstall (Linux)
   5.  Oracle Database 12c R1 (12.1.0) Preinstallation (Linux)
   6.  Oracle Database 12c R2 (12.2.0) Preinstallation (Linux)
   7.  Oracle Database 18c Preinstallation (Linux)
   8.  Oracle Database 19c Preinstallation (Linux)
   9.  Oracle Identity and Access Management PreInstall Check: Oracle Identity
       and Access Management 11g Release 2 (11.1.2) Linux
  10.  Oracle JDeveloper PreInstall Check: Oracle JDeveloper 11g Release 2
       (11.1.2.4) Linux
  11.  Oracle JDeveloper PreInstall Check: Oracle JDeveloper 12c (12.1.3)
       Linux
  12.  OAS PreInstall Check: Application Server 10g R2 (10.1.2) Linux
  13.  OAS PreInstall Check: Application Server 10g R3 (10.1.3) Linux
  14.  OFM PreInstall Check: Oracle Fusion Middleware 11g R1 (11.1.1) Linux
  15.  OFM PreInstall Check: Oracle Fusion Middleware 12c (12.1.3) Linux
  16.  OFM PreInstall Check: Oracle Fusion Middleware 12c (12.2.1.3.0) Linux
  17.  Oracle Forms and Reports PreInstall Check: Oracle Forms and Reports 11g
       Release 2 (11.1.2) Linux
  18.  Portal PreInstall Check: Oracle Portal Generic
  19.  IDM PreInstall Check: Identity Management 10g (10.1.4) Linux
  20.  BIEE PreInstall Check: Business Intelligence Enterprise Edition 11g
       (11.1.1) Linux
  21.  EPM PreInstall Check: Enterprise Performance Management Server (11.1.2)
       Generic
  22.  Oracle Enterprise Manager Cloud Control PreInstall Check: Oracle
       Enterprise Manager Cloud Control 12c Release 4 (12.1.0.4) Linux
  23.  Oracle E-Business Suite Release 11i (11.5.10) Preinstall (Linux x86 and
       x86_64)
  24.  Oracle E-Business Suite Release 12 (12.1.1) Preinstall (Linux x86 and
       x86_64)
  25.  Oracle E-Business Suite Release 12 (12.2.0) Preinstall (Linux x86_64)
Available Post-Installation Rule Sets:
  26.  RAC 10G DB and OS Best Practices (Linux)
  27.  Data Guard Postinstall (Generic)
  28.  WLS PostInstall Check: WebLogic Server 11g (10.3.x) Generic
  29.  WLS PostInstall Check: WebLogic Server 12c (12.x) Generic
  30.  Portal PostInstall Check: Oracle Portal Generic
  31.  OC4J PostInstall Check: Oracle Containers for J2EE 10g (10.1.x) Generic
  32.  SOA PostInstall Check: Service-Oriented Architecture 11g and Later
       Generic
  33.  OSB PostInstall Check: Service Bus 11g and Later Generic
  34.  Oracle Forms 11g Post Installation (Generic)
  35.  Oracle Enterprise Manager Agent 12c Post Installation (Generic)
  36.  Oracle Management Server 12c Post Installation (Generic)
  37.  Network Charging and Control Database Post Installation (Generic)
Enter the HCVE rule set number or 0 to cancel the test
Press Return to accept the default (0)
> 8
Performing HCVE checks ...
Enter value for < Planned ORACLE_HOME location >
Press Return to accept the default (/u01/app/oracle/product/12.1.0/db_1)
>

Test "Oracle Database 19c Preinstallation (Linux)" executed at 16-Jul-2022 21:36:39
Test Results
~~~~~~~~~~~~
(Note : Current Health Check rulesets do not have logic to support checks against Oracle Cloud or Exadata environments. If your product is installed in such an environment some test results will be erroneous. Refer instead to the product documentation rather than rely on the data below.)
ID     NAME                 RESULT  VALUE
====== ==================== ======= ==========================================
A00100 OS Certified?        FAILED  Not certified [Oracle Linux version]
A01020 User in /etc/passwd? PASSED  userOK
A01040 Group in /etc/group? PASSED  GroupOK
A01050 Enter ORACLE_HOME    RECORD  /u01/app/oracle/product/12.1.0/db_1
A01060 ORACLE_HOME Valid?   PASSED  OHexists
A01070 O_H Permissions OK?  PASSED  CorrectPerms
A01410 oraInventory Permiss PASSED  oraInventoryOK
A01420 Other OUI Up?        PASSED  NoOtherOUI
A01430 Got Software Tools?  PASSED  ld_nm_ar_make_found
A01440 Other O_Hs in PATH?  FAILED  OratabEntryInPath
A02010 Umask Set to 022?    PASSED  UmaskOK
A02030 Limits Processes     PASSED  Adequate
A02040 Limits Stacksize     PASSED  Adequate
A02050 Limits Descriptors   PASSED  Adequate
A02100 LDLIBRARYPATH Unset? FAILED  IsSet
A02170 JAVA_HOME Unset?     PASSED  UnSet
A03100 RAM (in MB)          PASSED  5982
A02210 Kernel Parameters OK PASSED  KernelOK
A02300 Tainted Kernel?      PASSED  NotVerifiable
A03010 Temp Adequate?       PASSED  TempSpaceOK
A03020 Disk Space OK?       PASSED  DiskSpaceOK
A03050 Swap (in MB)         RECORD  3999
A03100 RAM (in MB)          PASSED  5982
A03150 SwapToRam OK?        FAILED  SwapToRamTooLow
A03500 Network              PASSED  Connected
A03510 IP Address           RECORD  192.168.219.2
A03530 Domain Name          RECORD  NotFound
A03540 /etc/hosts Format    FAILED  No entry found
A03550 DNS Lookup           FAILED  nslookup host.domain
A03600 ip_local_port_range  PASSED  RangeOK
A04305 OL7 Server RPMs OK?  SKIPPED NotOL7
A04315 RHEL7 Server RPMs OK SKIPPED NotRedHat
A04321 SLES12 Server RPMs O SKIPPED NotSuSE
A04325 SLES15 Server RPMs O SKIPPED NotSuSE
Result file: output/collect/DB_HCVE_A_DB19c_lin_res.htm


In my case below the file location you just ftp on local machine and check the htm ext. file. 

/home/oracle/software/rda/output/collect
-rw-r----- 1 oracle oinstall 19312 Jul 16 21:36 DB_HCVE_A_DB19c_lin_res.txt
-rw-r----- 1 oracle oinstall 29553 Jul 16 21:36 DB_HCVE_A_DB19c_lin_res.htm


Reference:
Remote Diagnostic Agent (RDA) - Getting Started (Doc ID 314422.1)

Sunday, 12 September 2021

ORACLE RAC : TERMINATING THE INSTANCE DUE TO ERROR 304

 After refreshing my QA database using RMAN DUPLICATE, my instance startup was failing with the below error


USER (ospid: 60897): terminating the instance due to error 304

Instance terminated by USER, pid = 60897

Wed Mar 06 02:14:34 2019

Starting ORACLE instance (normal)


Looking into the spfile, I noticed the database was pulling wrong instance_number and thread numbers even though the DB configuration was correct


$ srvctl config database -d oradb

Database unique name: oradb

Database name: oradb

Oracle home: /opt/app/oradb/oracle/product/11.2.0.4

Oracle user: oradb

Spfile: +oradb_DATA/oradb/spfileoradb.ora

Domain: db.abc.com

Start options: open

Stop options: immediate

Database role: PRIMARY

Management policy: AUTOMATIC

Server pools: oradb

Database instances: oradb1,oradb2

Disk Groups: oradb_DATA,oradb_FRA,oradb_REDO1,oradb_REDO2

Mount point paths: 

Services: oradb_1_2.db.abc.com,oradb_2_1.db.abc.com

Type: RAC

Database is administrator managed


From the pfile I created from the current spfile I could see 

*.instance_number=2

*.thread=2


To resolve this, bring down the complete database and just start the failing instance, in our case instance 1


srvctl start instance -d oradb -i oradb1


Once the instance is started, login to SQL and run below


SQL> alter system set instance_number=1 scope=spfile sid='oradb1';

System altered.


SQL> alter system set thread=1 scope=spfile sid='oradb1';

System altered.


SQL> alter system set undo_tablespace='UNDO01' sid='oradb1';

System altered.


shutdown the instance and start the complete database

SQL> shutdown immediate


srvctl start database -d oradb



Hope this resolves your issue. 

ORA-12537: TNS:connection closed - Oracle RAC 11g and above

 The client was getting Below error while connecting to the RAC database


sqlplus test@RACDB_1

SQL*Plus: Release 11.2.0.4.0 - Production on Tue Sep 22 11:34:07 2020

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

ERROR:

ORA-12537: TNS:connection closed



Issue : 

The Oracle Binary permissions got changed and were not allowing the connections

Current Permissions under Oracle Home


[oracle@Node1 ~]$ ls -lrt $ORACLE_HOME/bin/oracle

-rwxr-sr-x 1 oracle asmadmin 243043788 Jun  2 01:12 oracle


It should be set to 6751 and should look like "-rwsr-s--x"

But doing chmod 6751 on oracle binary was not setting the correct permissions


[oracle@Node1 bin]$ chmod 6751 oracle


[oracle@Node1 bin]$ ls -lrt $ORACLE_HOME/bin/oracle

-rwsr-x--x 1 oracle asmadmin 243043788 Jun  2 01:12 oracle


Running below as RDBMS database Owner user helped, in this case, "oracle" user


1) Stop the database instance where the permissions got changed

 srvctl stop instance -d RACDB -i RACDB1


 2) Run as Oracle Database owner, in this case its Oracle OS user.  

[oracle@Node1 ~]$ $GRID_HOME/bin/setasmgidwrap o=$ORACLE_HOME/bin/oracle


3) The permissions got changed and resolved the connection issues

[oracle@Node1 ~]$ ls -lrt $ORACLE_HOME/bin/oracle

-rwsr-s--x 1 oracle asmadmin 243043788 Jun  2 01:12 oracle



Hope this resolves your issue.. 

Tuesday, 14 July 2020

CRS-5016 CRS-2674 ACFS-00591 ACFS-02037 CRS resource for ACFS

Once Add/Register ACFS filesystem as a Cluster 

srvctl add filesystem -d /dev/asm/sharedvol-64 -g GG_APP -v SHAREDVOL -m /gg_acfs/ggs -u oracle

[root@rac1 bin]# srvctl status filesystem -d /dev/asm/sharedvol-64

ACFS file system is not running

[root@rac1 bin]# srvctl start filesystem -d /dev/asm/sharedvol-64
PRCR-1079 : Failed to start resource ora.gg_app.sharedvol.acfs
CRS-5016: Process "/u01/app/11204/grid/bin/acfssinglefsmount" spawned by agent "/u01/app/11204/grid/bin/orarootagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11204/grid/log/rac2/agent/crsd/orarootagent_root//orarootagent_root.log"
CRS-5016: Process "/u01/app/11204/grid/bin/acfssinglefsmount" spawned by agent "/u01/app/11204/grid/bin/orarootagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11204/grid/log/rac1/agent/crsd/orarootagent_root//orarootagent_root.log"
CRS-5016: Process "/sbin/acfsutil" spawned by agent "/u01/app/11204/grid/bin/orarootagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11204/grid/log/rac2/agent/crsd/orarootagent_root//orarootagent_root.log"
CRS-5016: Process "/sbin/acfsutil" spawned by agent "/u01/app/11204/grid/bin/orarootagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11204/grid/log/rac1/agent/crsd/orarootagent_root//orarootagent_root.log"
CRS-2674: Start of 'ora.gg_app.sharedvol.acfs' on 'rac2' failed
CRS-2674: Start of 'ora.gg_app.sharedvol.acfs' on 'rac1' failed


# mount.acfs -o all
mount.acfs: ACFS-00591: error found in volume disk header
mount.acfs: ACFS-02037: File system not created on a Linux system. Cannot mount.
found error... so(make acfs)

[root@rac1 bin]# mkfs.acfs -f /dev/asm/sharedvol-64
mkfs.acfs: version                   = 11.2.0.4.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/sharedvol-64
mkfs.acfs: volume size               = 9663676416
mkfs.acfs: Format complete.

[root@rac1 bin]# mount.acfs -o all

[root@rac1 bin]#  acfsutil registry -a -f /dev/asm/sharedvol-64 /gg_acfs/ggs/

[root@rac1 bin]# srvctl start filesystem -d /dev/asm/sharedvol-64

[root@rac1 bin]# srvctl status filesystem -d /dev/asm/sharedvol-64
ACFS file system is running on rac1,rac2

[root@rac1 bin]# df -h /gg_acfs/ggs
Filesystem            Size  Used Avail Use% Mounted on
/dev/asm/sharedvol-64
                      9.0G   95M  9.0G   2% /gg_acfs/ggs

Monday, 16 July 2018

rtld: 0712-001 Symbol CreateIoCompletionPort was referenced

In My Case Cause of below the issue at storage level.
failing currently due to LVM inconsistency

For more Detail follow below the reference links.

Ref: 
http://oracle-help.com/ora-errors/rtld-0712-001-symbol-createiocompletionport-was-referenced/
AIX 12.1.0.2 Installation Fails with "rtld: 0712-001 Symbol CreateIoCompletionPort was referenced " (Doc ID 1949184.1)

Highly Available VIP (HAVIP)

Highly Available IP (HAIP) comes from 11.2.0.2 on-words and maximum 4 define for Private network.

Now in 12c version introduces Highly Available VIP (HAVIP) a Clusterware-monitored VIP that can be used for non-Databaseapplications.

Typically, VIPs are created for VIP or SCAN listeners for Database connections. In some cases, the need arises to create a VIP monitored by Clusterware designed to support other applications.

For more Detail : Click Here

Thursday, 12 July 2018

LREG New BG Process In 12c

In 11.2 Pmon process propogates the service metrics to the listeners regitered in local_listener adn remote_listener initialization parameter.

As the remote_listener specifies specifies the address of scan listener and the local_listener parameter specifies the address of the VIP listener, PMON process propogates the service metrics to both SCAN and VIP listeners.

You can trace listener registration using the following command.

alter system set event='immediate trace name listener_registration level 15';

Now in 12c version listener registration is permed by a new mandatory background process named LREG.
And if you want to trace then LREG trace file getting to all info about listener registration.

Advantage : Not recommneding that you create 50+ of unnecessary services,you should create
as many services as you need to split the application into manageable and disjointed workload.

In 11gR2 if there are 50+/100+ services and listeners then there is a possiblilyt that the PMON process
might spend more time on service registration to listener.

But in 12c this possibility is eliminated as the LREG backgroud process which is totally dedicated for registering the services to Listener.

and PMON is freed from listener registration.

Monday, 2 July 2018

PRCD-1027 PRCD-1229

PRCD-1027 : Failed to retrieve database racdb
PRCD-1229 : An attempt to access configuration of database racdb was rejected because its version 11.2.0.1.0 differs from the program version 11.2.0.4.0. Instead run the program from /oracle/product/11.2.0/db_1.



[root@rac1 ~]# srvctl status database -d racdb
PRCD-1027 : Failed to retrieve database racdb
PRCD-1229 : An attempt to access configuration of database racdb was rejected because its version 11.2.0.1.0 differs from the program version 11.2.0.4.0. Instead run the program from /oracle/product/11.2.0/db_1.

[root@rac1 ~]# srvctl config database -d racdb

PRCD-1027 : Failed to retrieve database racdb
PRCD-1229 : An attempt to access configuration of database racdb was rejected because its version 11.2.0.1.0 differs from the program version 11.2.0.4.0. Instead run the program from /oracle/product/11.2.0/db_1.

[root@rac1 ~]# srvctl config database
racdb


-- First Attempt
[root@rac1 ~]# srvctl modify database -d racdb -o /oracle/product/11204/db_1

PRCD-1027 : Failed to retrieve database racdb
PRCD-1229 : An attempt to access configuration of database racdb was rejected because its version 11.2.0.1.0 differs from the program version 11.2.0.4.0. Instead run the program from /oracle/product/11.2.0/db_1.

-- Second Attempt
[root@rac1 ~]# srvctl upgrade database -d  racdb -o /oracle/product/11204/db_1

PRCD-1231 : Failed to upgrade configuration of database racdb to version 11.2.0.4.0 in new Oracle home /oracle/product/11204/db_1
PRKH-1014 : Current user "root" is not the oracle owner user "oracle" of oracle home "/oracle/product/11204/db_1"

[root@rac1 ~]# su - oracle

-- Final Attempt
[oracle@rac1 ~]$ srvctl upgrade database -d  racdb -o /oracle/product/11204/db_1

[oracle@rac1 ~]$ exit
logout

[root@rac1 ~]# srvctl status database -d racdb
Instance racdb1 is running on node rac1
Instance racdb2 is running on node rac2

Saturday, 30 June 2018

mv: cannot move `OPatch' to `OPatch_old': Permission denied

Applying Latest Opatch in GRID_HOME  Opatch (6880880) 


[grid@dbapath.com patch]$ mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch_old
mv: cannot move `/u01/app/11.2.0/grid/OPatch/’ to `/u01/app/11.2.0/grid/OPatch_old’: Permission denied

You must either unlock GRID_HOME or you can move OPatch directory as a superuser.

[oracle@grid_server1 ]$ cd /u01/app/11.2.0/grid/crs/install
[oracle@grid_server1 ]$ sudo perl rootcrs.pl -unlock -crshome /u01/app/11.2.0/grid

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Undefined subroutine &main::read_file called at /u01/app/11.2.0/grid/crs/install/crspatch.pm line 86.

If you hit an error like above, here is the workaround (according to this paper: roothas.pl -patch or rootcrs.pl -patch Fails with ‘Undefined subroutine’ [ID 1268390.1]):

# Take a backup of the file /crs/install/crsconfig_lib.pm

grid$ cd /crs/install

grid$ cp crsconfig_lib.pm crsconfig_lib.pm.bak

# Make the following change in that file crsconfig_lib.pm
#From
# my @exp_func = qw(check_CRSConfig validate_olrconfig validateOCR
#To
# my @exp_func = qw(check_CRSConfig validate_olrconfig validateOCR read_file

#Execute relock:

[grid@dbapath.com ]$ cd /u01/app/11.2.0/grid/crs/install

[grid@dbapath.com ]$ sudo perl rootcrs.pl -patch -crshome /u01/app/11.2.0/grid
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
ACFS-9200: Supported
CRS-4123: Oracle High Availability Services has been started.

Thursday, 28 June 2018

Quiescing RAC Databases


To quiesce a RAC database, use the ALTER SYSTEM QUIESCE RESTRICTED statement from one instance.

It is not possible to open the database from any instance while the database is in the process of being quiesced from another instance.

After all non-DBA sessions become inactive, the ALTER SYSTEM QUIESCE RESTRICTED statement executes and the database is considered to be quiesced.
In a RAC environment, this statement affects all instances.

Cold backups cannot be taken when the database is in a quiesced state because the Oracle background processes may  still perform updates for internal purposes even when the database is in a quiesced state.
Also, the file headers of online data filescontinue to appear as if they are being accessed. They do not look the same as if a clean shutdownwere done.

Monday, 18 June 2018

Udev Rule For ASM Disks

In Case of RAC configure on VM - After Adding ASM disk on Node 1 adding asm disk in UDEV rules for persistent.

------------------
IN RHEL/OEL-5/6/7
------------------
Set Asm disk using UDEV
/sbin/scsi_id -g -u -d /dev/sdb
/sbin/scsi_id -g -u -d /dev/sdc
/sbin/scsi_id -g -u -d /dev/sdd
/sbin/scsi_id -g -u -d /dev/sde
/sbin/scsi_id -g -u -d /dev/sdf
/sbin/scsi_id -g -u -d /dev/sdg

-Above the command execute not showing any scsi_id for udev rule. I'm not testing much more but find some doc and fix this issue on above linux versions.

- Adding below rules into "/etc/udev/rules.d/50-udev.rules" in my case my OS version is 5. For more detail follow oracle-base.

# ASM DISK RULES
KERNEL=="sdb1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdc1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdd1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sde1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdf1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk5", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdg1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk6", OWNER="oracle", GROUP="dba", MODE="0660"

- No execute again below the command and wondering got the output of scsi_id.
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sdb
36000c299cae37d62af51ab7c55768959
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sdc
36000c2989736710c1b2bd8efea742e61
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sdd
36000c295eae87b8a4e919b5b3b077827
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sde
36000c293bc54c062ff70f20ae8c3078e
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sdf
36000c29759c3a92862de21e908f8516e
[root@rac1 rules.d]# /sbin/scsi_id -g -u -s /block/sdg
36000c292f86ba892199aa8fd6bb7ffe8

- Creating a file and insert below commands into this file "vim /etc/udev/rules.d/99-oracle-asmdevices.rules"
KERNEL=="sdb1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c299cae37d62af51ab7c55768959",NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdc1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c2989736710c1b2bd8efea742e61",NAME="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdd1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c295eae87b8a4e919b5b3b077827",NAME="asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sde1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c293bc54c062ff70f20ae8c3078e",NAME="asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdf1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29759c3a92862de21e908f8516e",NAME="asm-disk5", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdg1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c292f86ba892199aa8fd6bb7ffe8",NAME="asm-disk6", OWNER="oracle", GROUP="dba", MODE="0660"

- Add the following to the "/etc/scsi_id.config" file to configure SCSI devices as trusted. Create the file if it doesn't already exist.

options=-g

- Execute below commands
/sbin/partprobe /dev/sdb1
/sbin/partprobe /dev/sdc1
/sbin/partprobe /dev/sdd1
/sbin/partprobe /dev/sde1
/sbin/partprobe /dev/sdf1
/sbin/partprobe /dev/sde1

# #OL5 and OL6
# /sbin/start_udev
ls -lrt /dev/asm*
Now proceed your next action plan for node 2.

Monday, 14 August 2017

DNS SERVER

DNS configure without CHROOT on RHEL/OEL For Oracle RAC


[root@dns1 ~]# rpm -qa | grep -i bind
ypbind-1.19-12.el5_6.1
kdebindings-3.5.4-6.el5
bind-utils-9.3.6-20.P1.el5_8.6
system-config-bind-4.0.3-6.0.1.el5
bind-libs-9.3.6-20.P1.el5_8.6
bind-9.3.6-20.P1.el5_8.6

[root@dns1 ~]# hostname
dns1.testenv.com



----- Create/Edit /etc/named.conf File ------

[root@dns1 ~]# cat /etc/named.conf
options {
        listen-on port 53 { 192.168.2.200; };
        directory "/var/named";
        dump-file "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        /*
         * If there is a firewall between you and nameservers you want
         * to talk to, you might need to uncomment the query-source
         * directive below.  Previous versions of BIND always asked
         * questions using port 53, but BIND 8.1 uses an unprivileged
         * port by default.
         */
         // query-source address * port 53;

        // My Additions
        // Forwarder: Anything this DNS can't resolve gets forwarded to my ISPs DNS.
        #forwarders { 194.168.4.100; 194.168.8.100; };
        // End My Additions
};

zone "testenv.com." IN {
                 type master;
                 file "testenv.com.zone";
                 allow-update { none; };
};

zone "2.168.192.in-addr.arpa." IN {
        type master;
        file "2.168.192.in-addr.arpa";
        allow-update { none; };
};

include "/etc/rndc.key";




----------- Create/Edit /var/named/2.168.192.in-addr.arpa File ----------------
[root@dns1 named]# pwd
/var/named
[root@dns1 named]# cat 2.168.192.in-addr.arpa
$ORIGIN 2.168.192.in-addr.arpa.
$TTL 1H
@          IN     SOA    testenv.com.  root.testenv.com. (
                         42 ; serial (d. adams)
                         3H ; refresh
                        15M ; retry
                         1W ; expiry
                         1D ) ; minimum

2.168.192.in-addr.arpa.   IN   NS     testenv.com.
151 IN PTR rac1.testenv.com.
152 IN PTR rac2.testenv.com.
161 IN PTR rac1-vip.testenv.com.
162 IN PTR rac2-vip.testenv.com.
51  IN PTR rac-scan.testenv.com.
52  IN PTR rac-scan.testenv.com.
53  IN PTR rac-scan.testenv.com.


------------------ Create/Edit /var/named/testenv.com.zone File --------

[root@dns1 named]# cat testenv.com.zone
$TTL    86400
@               IN SOA  testenv.com root.testenv.com (
                                        42              ; serial (d. adams)
                                        3H              ; refresh
                                        15M             ; retry
                                        1W              ; expiry
                                        1D )            ; minimum
                IN NS   testenv.com
rac1            IN A    192.168.2.151
rac2            IN A    192.168.2.152
rac1-priv       IN A    192.168.3.151
rac2-priv       IN A    192.168.3.152
rac1-vip        IN A    192.168.2.161
rac2-vip        IN A    192.168.2.162
rac-scan        IN A    192.168.2.51
rac-scan        IN A    192.168.2.52
rac-scan        IN A    192.168.2.53



-------- Give the Ownership --------

cd /var/named
chown named:named 2.168.192.in-addr.arpa
chown named:named testenv.com.zone

chkconfig named on

[root@dns1 named]# cat /etc/resolv.conf
nameserver 192.168.2.200



--------- Check the its working or not ----------

[root@dns1 named]# nslookup rac-scan
Server:         192.168.2.200
Address:        192.168.2.200#53

Name:   rac-scan.testenv.com
Address: 192.168.2.53
Name:   rac-scan.testenv.com
Address: 192.168.2.51
Name:   rac-scan.testenv.com
Address: 192.168.2.52


[root@dns1 named]# dig testenv.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-20.P1.el5_8.6 <<>> testenv.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50541
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;testenv.com.                   IN      A

;; AUTHORITY SECTION:
testenv.com.            86400   IN      SOA     testenv.com.testenv.com. root.testenv.com.testenv.com. 42 10800 900 604800 86400

;; Query time: 3 msec
;; SERVER: 192.168.2.200#53(192.168.2.200)
;; WHEN: Wed Jul  5 11:31:57 2017
;; MSG SIZE  rcvd: 82


---------------------------------------------------------------------------------------------------------------------------
=============
Now RAC Part
=============


Configure one node if working well athen follow all steps on all nodes.

[root@rac1 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               rac1.testenv.com rac1 localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
#Public-IP
192.168.2.151  rac1  rac1.testenv.com
192.168.2.152  rac2  rac2.testenv.com

#Virtual-IP
192.168.2.161  rac1-vip      rac1-vip.testenv.com
192.168.2.162 rac2-vip      rac2-vip.testenv.com

#Scan-IP
192.168.198.51  rac-scan        rac-scan.testenv.com
192.168.2.52    rac-scan        rac-scan.testenv.com
192.168.2.53    rac-scan        rac-scan.testenv.com

192.168.2.200   dns1    dns1.testenv.com


[root@rac1 ~]# hostname
rac1.testenv.com


[root@rac1 ~]# cat /etc/resolv.conf
search testenv.com      dns1.testenv.com
nameserver 192.168.2.200


[root@rac1 ~]# dig rac1

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> rac1
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 63859
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;rac1.                          IN      A

;; Query time: 4 msec
;; SERVER: 192.168.2.200#53(192.168.2.200)
;; WHEN: Wed Jul  5 11:36:32 2017
;; MSG SIZE  rcvd: 22



[root@rac1 ~]# dig rac-scan

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> rac-scan
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 359
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;rac-scan.                      IN      A

;; Query time: 13 msec
;; SERVER: 192.168.2.200#53(192.168.2.200)
;; WHEN: Wed Jul  5 11:36:05 2017
;; MSG SIZE  rcvd: 26



Now its working fine ... above the setting replicate on all nodes.




Tuesday, 1 November 2016

ORACLE EXA-DATA SIMULATION SETUP


Use Virtual box / vmware  for Exa-Simulation
11.2.3.2.1 : 64-bit > V36290-01.zip
GI + DB = 11.2.0.4      
OS: First tested (below protocol is working or not)
rds, rds_tcp, rds_rdma
I’m tested on RHEL/OEL ( > 5.10 and 6.x )

Discussion and configure below the points.:
1.     Install an Oracle Linux / RHEL which is support rds, rds_tcp, rds_rdma protocols (because we are not using infiniband switch H/w)
2.     Add storage cell software
3.     Prepare some virtual disks
Installation Oracle Linux

1.       Minimum 2 GB RAM Required and 40 GB HDD only for illustration. (Exa-Simulation on laptop).
2.       1 NIC only (Host only Adapter) for storage cell. If you want Internet on this machine then one more NIC add with bridge connection.
3.       After OS installation completion.
4.       Selinux and firewall disabled.
Add storage cell software

[root@storagecell ]# vi /etc/hosts

[root@storagecell ]# more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       storagecell     storagecell.localdomain localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6

Put the software cell image zip (V36290-01.zip) in the path of host system choosed as shared folder so it can be directly used inside the VM. In fact there is a special file system mounted

[root@storagecell ~]# mount
OS on /home type vboxsf (gid=160,rw)
[root@storagecell ~]# cd /home/
[root@storagecell ~]# ll
total 1506576
-rwxrwx— 1 root vboxsf 1542729472 Oct 10 22:25 V36290-01.zip

Uncompress the file

[root@storagecell ]# unzip V36290-01.zip
Archive:  V36290-01.zip
inflating: README.txt
inflating: cellImageMaker_11.2.3.2.1_LINUX.X64_130109-1.x86_64.tar

And then untar

[root@storagecell ]# tar pxvf cellImageMaker_11.2.3.2.1_LINUX.X64_130109-1.x86_64.tar

(there are some errors due permissions on host folder, but there is no problem because the only file needed i cell.bin)

[root@storagecell cellbits]# ll dl180/boot/cellbits/
total 1446844
-rwxrwx— 1 root vboxsf       729 Jan  9  2013 c7rpms.tbz
-rwxrwx— 1 root vboxsf 245231205 Jan  9  2013 cell.bin
-rwxrwx— 1 root vboxsf  12705374 Jan  9  2013 cellboot.tbz
-rwxrwx— 1 root vboxsf 141444416 Jan  9  2013 cellfw.tbz
-rwxrwx— 1 root vboxsf 142434203 Jan  9  2013 cellrpms.tbz
-rwxrwx— 1 root vboxsf 208612489 Jan  9  2013 commonos.tbz
-rwxrwx— 1 root vboxsf 375683818 Jan  9  2013 debugos.tbz
-rwxrwx— 1 root vboxsf  55704927 Jan  9  2013 doclib.zip
-rwxrwx— 1 root vboxsf 199485158 Jan  9  2013 exaos.tbz
-rwxrwx— 1 root vboxsf  18186084 Jan  9  2013 hputils.tbz
-rwxrwx— 1 root vboxsf  53387742 Jan  9  2013 kernel.tbz
-rwxrwx— 1 root vboxsf  16165382 Jan  9  2013 ofed.tbz
-rwxrwx— 1 root vboxsf  12485584 Jan  9  2013 sunutils.tbz

Then copy cell.bin and remove all

[root@storagecell ]# cp dl180/boot/cellbits/cell.bin .
[root@storagecell ]# ll cell.bin
total 1746060
-rwxrwx— 1 root vboxsf  245231205 Dec 12 10:43 cell.bin
-rwxrwx— 1 root vboxsf 1542729472 Oct 10 22:25 V36290-01.zip

Now unzip cell.bin

[root@storagecell ]# unzip cell.bin
Archive:  cell.bin
warning [cell.bin]:  6408 extra bytes at beginning or within zipfile
(attempting to process anyway)
inflating: cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm
inflating: jdk-1_5_0_15-linux-amd64.rpm

Install jdk
[root@storagecell sf_OS]# rpm -ivh jdk-1_5_0_15-linux-amd64.rpm
Preparing…                ########################################### [100%]
1:jdk                    ########################################### [100%]

Preparing to install cell rpm
[root@storagecell ]# mkdir /var/log/oracle
[root@storagecell ]# chmod 775 /var/log/oracle

(It will be used also by celladmin user …)
Install cell sw
[root@storagecell ]# rpm -ivh cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm
Preparing…                ########################################### [100%]
Pre Installation steps in progress …
1:cell                   ########################################### [100%]
Post Installation steps in progress …
Set cellusers group for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory
Set 775 permissions for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory
/
/
Installation SUCCESSFUL.
Starting RS and MS… as user celladmin
Done. Please Login as user celladmin and create cell to startup CELLSRV to complete cell configuration.
WARNING: Using the current shell as root to restart cell services.
Restart the cell services using a new shell.


Prepare some virtual disks


In 12c use at least 1GB file size. esp

1.      The storage cell software installed and some disks must be created.
2.      Shutdown the machine and with VirtualBox Settings create all the disks: 12 with size 500M (data) and 6 with size 400M (flash)
3.      After storage cell software install and Adding a disks. Some changes in gnome environment produce this strange behaviour.
4.      Switch to failsafe session > clieck on session then see the failsafe option.
5.      Log again in an xterm
6.      Edit /etc/bashrc
[root@storagecell]# vi /etc/bashrc
export DISPLAY=:0
7.      Save, exit and login again.
Now identify disks
[root@storagecell ~]# fdisk -l 2>/dev/null |grep “B,”
Disk /dev/sda: 26.8 GB, 26843545600 bytes
Disk /dev/sdb: 524 MB, 524288000 bytes
Disk /dev/sdc: 524 MB, 524288000 bytes
Disk /dev/sdd: 524 MB, 524288000 bytes
Disk /dev/sde: 524 MB, 524288000 bytes
Disk /dev/sdf: 524 MB, 524288000 bytes
Disk /dev/sdg: 524 MB, 524288000 bytes
Disk /dev/sdh: 524 MB, 524288000 bytes
Disk /dev/sdi: 524 MB, 524288000 bytes
Disk /dev/sdj: 524 MB, 524288000 bytes
Disk /dev/sdk: 524 MB, 524288000 bytes
Disk /dev/sdl: 524 MB, 524288000 bytes
Disk /dev/sdm: 524 MB, 524288000 bytes
Disk /dev/sdn: 419 MB, 419430400 bytes
Disk /dev/sdo: 419 MB, 419430400 bytes
Disk /dev/sdp: 419 MB, 419430400 bytes
Disk /dev/sdq: 419 MB, 419430400 bytes
Disk /dev/sdr: 419 MB, 419430400 bytes
Disk /dev/sds: 419 MB, 419430400 bytes

Looking at environment of the new celladmin user, it could be found an interesting variable that define where the cellsrv process will search for disks:
[root@storagecell unix]# echo $T_WORK
/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks

Under that path a new folder is needed (raw) and symbolic link to “real” disks should be created
cd /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks

mkdir disks

mkdir disks/raw

cd disks/raw
ln -s /dev/sdb storagecell_DISK01
ln -s /dev/sdc storagecell_DISK02
ln -s /dev/sdd storagecell_DISK03
ln -s /dev/sde storagecell_DISK04
ln -s /dev/sdf storagecell_DISK05
ln -s /dev/sdg storagecell_DISK06
ln -s /dev/sdh storagecell_DISK07
ln -s /dev/sdi storagecell_DISK08
ln -s /dev/sdj storagecell_DISK09
ln -s /dev/sdk storagecell_DISK10
ln -s /dev/sdl storagecell_DISK11
ln -s /dev/sdm storagecell_DISK12
ln -s /dev/sdn storagecell_FLASH01
ln -s /dev/sdo storagecell_FLASH02
ln -s /dev/sdp storagecell_FLASH03
ln -s /dev/sdq storagecell_FLASH04
ln -s /dev/sdr storagecell_FLASH05
ln -s /dev/sds storagecell_FLASH06

Some Changes and Errors:
Add/Set  in /etc/sysctl.conf
fs.file-max = 65536

Edit the /etc/security/limit.conf files and add/set

* soft nofile 65536
* hard nofile 65536

To communicate over InfiniBand Oracle uses rds protocol. All the modules of rds must be loaded (and configured to be loaded over machine restarts)

[root@storagecell ~]# lsmod |grep rds*
rdma_cm                73429  2 rds_rdma,ib_iser
ib_cm                  72041  1 rdma_cm
iw_cm                  43593  1 rdma_cm
ib_sa                  76489  2 rdma_cm,ib_cm
ib_core               108097  7 rds_rdma,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
ib_addr                42697  1 rdma_cm
ipv6                  438369  3 rdma_cm,ib_addr,cnic

[root@storagecell ~]# modprobe rds
[root@storagecell ~]# modprobe rds_tcp
[root@storagecell ~]# modprobe rds_rdma
[root@storagecell ~]# su - celladmin

[celladmin@storagecell ~]$ cellcli -e alter cell restart services all
CELL-01509: Restart Server (RS) not responding.
Starting the RS, CELLSRV, and MS services…
Getting the state of RS services… running
Starting CELLSRV services…
The STARTUP of CELLSRV services was not successful.
CELL-01547: CELLSRV startup failed due to unknown reasons.
Starting MS services…
The STARTUP of MS services was successful.

The error in not unknown (as stated) but well known and expected : [Required IP parameters missing]
So, set the interconnect (= InfiniBand connection)
[celladmin@storagecell ~]$ cellcli -e create cell storagecell interconnect1=eth0
Cell stocell1 successfully created
Starting CELLSRV services…
The STARTUP of CELLSRV services was successful.
Flash cell disks, FlashCache, and FlashLog will be created…
CellDisk FD_00_stocell1 successfully created
CellDisk FD_01_stocell1 successfully created
CellDisk FD_02_stocell1 successfully created
CellDisk FD_03_stocell1 successfully created
CellDisk FD_04_stocell1 successfully created
CellDisk FD_05_stocell1 successfully created
Flash log stocell1_FLASHLOG successfully created
Flash cache stocell1_FLASHCACHE successfully created

(I’m not sure why Flash components are auto configured, but it can be modified later if needed)
Configure cell disks
[celladmin@storagecell ~]$ cellcli -e create celldisk all
CellDisk CD_DISK01_stocell1 successfully created
CellDisk CD_DISK02_stocell1 successfully created
CellDisk CD_DISK03_stocell1 successfully created
CellDisk CD_DISK04_stocell1 successfully created
CellDisk CD_DISK05_stocell1 successfully created
CellDisk CD_DISK06_stocell1 successfully created
CellDisk CD_DISK07_stocell1 successfully created
CellDisk CD_DISK08_stocell1 successfully created
CellDisk CD_DISK09_stocell1 successfully created
CellDisk CD_DISK10_stocell1 successfully created
CellDisk CD_DISK11_stocell1 successfully created
CellDisk CD_DISK12_stocell1 successfully created

and grid disks
[celladmin@storagecell ~]$ cellcli -e create griddisk all harddisk prefix=DATA
GridDisk DATA_CD_DISK01_stocell1 successfully created
GridDisk DATA_CD_DISK02_stocell1 successfully created
GridDisk DATA_CD_DISK03_stocell1 successfully created
GridDisk DATA_CD_DISK04_stocell1 successfully created
GridDisk DATA_CD_DISK05_stocell1 successfully created
GridDisk DATA_CD_DISK06_stocell1 successfully created
GridDisk DATA_CD_DISK07_stocell1 successfully created
GridDisk DATA_CD_DISK08_stocell1 successfully created
GridDisk DATA_CD_DISK09_stocell1 successfully created
GridDisk DATA_CD_DISK10_stocell1 successfully created
GridDisk DATA_CD_DISK11_stocell1 successfully created
GridDisk DATA_CD_DISK12_stocell1 successfully created

Ø  Now configure another Machine for GI setup.

Note:- Some command & scripts not working on Fake H/W module of Exa-data. Like (imageinfo, imagehistory, patching etc.. doesn’t work on it).


And special thanks to my friends/Bro..
Skant Gupta, Vj sharma for completing this doc.