GPNP PROFILE 11G R2 RAC

In this post, I will explain what is GPnP profile, what does it contain and how is it used by clusterware.

WHAT IS GPNP PROFILE?


GPNP - is a Grid plug & Play profile management tool its new feature in oracle 11g

The GPnP profile is a small XML file located in GRID_HOME/gpnp/<hostname>/profiles/peer under the name profile.xml. It is used to establish the correct global personality of a node. Each node maintains a local copy of the GPnP Profile and is maintanied by the GPnP Deamon (GPnPD) .

WHAT DOES GPNP PROFILE CONTAIN?

GPnP Profile  is used to store necessary information required for the startup of Oracle Clusterware like  SPFILE location,ASM DiskString  etc.

It contains various attributes defining node personality.

- Cluster name

- Network classifications (Public/Private)

- Storage to be used for CSS

- Storage to be used for ASM : SPFILE location,ASM DiskString  etc

- Digital signature information : The profile is security sensitive. It might identify the storage to be used as the root partition of a machine.  Hence, it contains digital signature information of the provisioning authority.

Here is the GPnP profile of my RAC setup.

gpnptool can be  used  for reading/editing the gpnp profile.

[root@host01 peer]# gpnptool get

<?xml version=”1.0″ encoding=”UTF-8″?><gpnp:GPnP-Profile Version=”1.0″ xmlns=”http://www.grid-pnp.org/2005/11/gpnp-profile” xmlns:gpnp=”http://www.grid-pnp.org/2005/11/gpnp-profile” xmlns:orcl=”http://www.oracle.com/gpnp/2005/11/gpnp-profile” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd” ProfileSequence=”7″ ClusterUId=”14cddaccc0464f92bfc703ec1004a386″ ClusterName=”cluster01″ PALocation=””><gpnp:Network-Profile><gpnp:HostNetwork id=”gen” HostName=”*”><gpnp:Network id=”net1″ IP=”192.9.201.0″ Adapter=”eth0″ Use=”public”/><gpnp:Network id=”net2″ IP=”10.0.0.0″ Adapter=”eth1″ Use=”cluster_interconnect”/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id=”css” DiscoveryString=”+asm” LeaseDuration=”400″/><orcl:ASM-Profile id=”asm” DiscoveryString=”” SPFile=”+DATA/cluster01/asmparameterfile/registry.253.783619911″/><ds:Signature xmlns:ds=”http://www.w3.org/2000/09/xmldsig#”><ds:SignedInfo><ds:CanonicalizationMethod Algorithm=”http://www.w3.org/2001/10/xml-exc-c14n#”/><ds:SignatureMethodAlgorithm=”http://www.w3.org/2000/09/xmldsig#rsa-sha1″/><ds:Reference URI=””><ds:Transforms><ds:Transform Algorithm=”http://www.w3.org/2000/09/xmldsig#enveloped-signature”/><ds:Transform Algorithm=”http://www.w3.org/2001/10/xml-exc-c14n#”> <InclusiveNamespaces xmlns=”http://www.w3.org/2001/10/xml-exc-c14n#” PrefixList=”gpnp orcl xsi”/></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm=”http://www.w3.org/2000/09/xmldsig#sha1″/><ds:DigestValue>4VMorzxVNa+FeOx2SCk1unVBpfU=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>bbzV04n2zSGTtUEvqqB+pjw1vH7i8MOEUqkhXAyloX0a41T2FkDEA++ksc0BafndAk7tR+6LGdppE1aOsaJUtYxQqaHJdpVsJF+sj2jN7LPJlT5NBt+K7b08TLjDID92Se6vEiDAeeKlEbpVWKMUIvQvp6LrYK8cDB/YjUnXuGU=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile>

WHO UPDATES GPNP PROFILE?

GPnPd daemon replicates changes to the profile during

  - installation,

  - system boot or

  - when updated

Profile is updated Whenever changes are made to a cluster with configuration tools like

  . oifcfg (Change network),

  . crsctl (change location of voting disk),

  . asmcmd (change ASM_DISKSTRING, SPfile location) etc.

HOW IS GPNP PROFILE USED BY CLUSTERWARE?

To start clusterware, voting disk needs to be accessed. If voting disk is on ASM, this information (that voting disk is on ASM) is read from GPnP profile (<orcl:CSS-Profile id=”css” DiscoveryString=”+asm” LeaseDuration=”400″/>).   The voting disk is read using kfed utility  even if ASM is not up.

Next,  the clusterware checks if all the nodes have the updated GPnP profile and the node joins the cluster based on the GPnP configuration . Whenver a node is started/added to the cluster, the clusterware software on the starting node starts a GPnP agent.

§  - If the node is already part of the cluster, the GPnP agent reads the existing profile on that node.

§  - If the node is being added to the cluster, GPnP agent locates agent on another existing node using multicast protocol (provided by mDNS) and gets the profile from that agent.

Next CRSD needs to read OCR to startup various resources on the node and hence update it as status of resources changes. Since OCR is also on ASM, location of ASM SPfile should be known.

The order of searching the ASM SPfile is

§  - GPnP profile

§  - ORACLE_HOME/dbs/spfile<sid.ora>

§  - ORACLE_HOME/dbs/init<sid.ora>

In cluster environment, the location of  SPfile for ASMread from GPnP profile.

[grid@host01 peer]$ gpnptool getpval -asm_spf

Warning: some command line parameters were defaulted. Resulting command line:

         /u01/app/11.2.0/grid/bin/gpnptool.bin getpval -asm_spf -p=profile.xml -o-

+DATA/cluster01/asmparameterfile/registry.253.793721441

The oputput of the query shows that SPfile is on ASM in DATA diskgroup. To find out the

 location of ASM disks, following query is issued :

[root@host01 peer]# gpnptool getpval -asm_dis

ASM-Profile id=”asm” DiscoveryString=””

The  device headers of every device in the disk string returned by the above query are scanned  (if configured by you at ASM initial setup time). Here Discovery String is blank is as ASMDISKSTRINGS parameter has not been set. Hence, headers of all the ASM disks are scanned .

Here, I have shown the output of the query only on the disk which contains SPfile.(spfflg is not null)

[root@host01 ~]#  kfed read /dev/sdb3 | grep -E ‘spf|ausize’

kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000

kfdhdb.spfile:                       16 ; 0x0f4: 0x00000010

kfdhdb.spfflg:                        1 ; 0x0f8: 0x00000001

In the output above, we see that

     the device /dev/sdb3 contains a copy of the ASM spfile (spfflg=1).

     The ASM spfile location starts at the disk offset of 16 (spfile=16)

Considering the allocation unit size (kfdhdb.ausize = 1M), let’s dump the ASM spfile from the device:

[root@host01 ~]#  dd if=/dev/sdb3 of=spfileASM_Copy2.ora skip=16  bs=1M count=1

1+0 records in

1+0 records out

1048576 bytes (1.0 MB) copied, 0.170611 seconds, 6.1 MB/s

[root@host01 ~]# strings spfileASM_Copy2.ora

+ASM1.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value

+ASM2.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value

+ASM3.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value

+ASM3.asm_diskgroups=’FRA’#Manual Mount

+ASM2.asm_diskgroups=’FRA’#Manual Mount

+ASM1.asm_diskgroups=’FRA’#Manual Mount

*.asm_power_limit=1

*.diagnostic_dest=’/u01/app/grid’

*.instance_type=’asm’

*.large_pool_size=12M

*.remote_login_passwordfile=’EXCLUSIVE’

Using the parameters in SPfile, ASM is started.

Once ASM is up, OCR is read by CRSD and various resources on the node are started.

Each node reads network information in GPnP profile and using GNS,  negotiates appropriate network identity for itself . Hence, nodes can be dynamically added/deleted.

What happens if GPnP profile is lost?

To know please click  here.

————————————————————

GPNPTOOL COMMAND REFERENCE:

How to get GPNP profile info. into local node RAC1
[root@rac1 ~]# gpnptool get      

- How to read the profile

[root@inssc3 bin]# ./gpnptool get

- How to find GPnP Deamons are running on the local node

[root@host01 peer]# gpnptool lfind

Success. Local gpnpd found.

- How to find the location of ASM spfile if the ASM is down

[root@host01 peer]# gpnptool getpval -asm_spf

+DATA/cluster01/asmparameterfile/registry.253.783619911

- How to find all RD-discoverable resources of given type

[root@host01 peer]# gpnptool find

Found 3 instances of service ‘gpnp’.

        mdns:service:gpnp._tcp.local.://host03:18015/agent=gpnpd,cname=cluster01,host=host03,pid=5066/gpnpd h:host03 c:cluster01

        mdns:service:gpnp._tcp.local.://host02:17637/agent=gpnpd,cname=cluster01,host=host02,pid=5236/gpnpd h:host02 c:cluster01

        mdns:service:gpnp._tcp.local.://host01:16633/agent=gpnpd,cname=cluster01,host=host01,pid=5206/gpnpd h:host01 c:cluster01

Related Topics:

EDIT GPNP PROFILE

LOSE GPNP PROFILE

EDIT GPNP PROFILE 11G R2 RAC

In 11g R2 RAC, Voting disk is stored on an ASM diskgroup. . But CSSD needs the voting files before ASM is online.  At its startup, CSSD scans the device headers of all devices specified in the gpnp profile xml file, tag “DiscoveryString” which contains the same value as specified in asm_diskstring parameter for ASM instance.

If it can find at least more than half the number of total no. of voting files, the party takes place,  otherwise,  CSSD will cycle with appropriate error messages in $GRID_HOME/log/hostname/log/cssd/ocssd.log for each loop.

To verify it, I modified ASM_DISKSTRING to such a value that the ASM disk containing voting disk was not included and tried to restart crs on a node.. Looking up the CSSD logfile, I saw that the CSSD had trouble identifying its voting files.

Now, I had to change back the ASM disk_string parameter without having ASM running, and with no CSSD available, which is necessary to start it? How do we tell the CSSD, that’s running fairly in advance of ASM, to scan the right devices?  I used gpnptool to edit the gpnp profile and restored the discovery string to the appropriate value. Now I could get crs running on my machine.

Overview:

– Move voting disk to a new diskgroup VOTE.

– Set ASM_DISKTRING to all the disks which are members of all the other diskgroups (DATA/FRA) using SQL

— stop and restart crs on the node –

– check that HAS services have started but rest of the services are not up

–Check the ocssd.log — Scans all the disks which are part of the discovery string  but does not find voting disk

– Edit the gpnp profile to modify discovery string  for asm

– Try to restart crs

– check that all the daemons have started and cluster servicves are up

Implementation:

– cretae a diskgroup with external redundancy to be used to store VD

– Move voting disk to the diskgroup vote – fails because

    — diskgroup not mounted on all the nodes – mount it

   — diskgroup compatibility < 11.2.0.0 – Modify it

[root@host01 ~]# crsctl replace votedisk +vote

Failed to create voting files on disk group vote.

Change to configuration failed, but was successfully rolled back.

CRS-4000: Command Replace failed, or completed with errors.

  – Mount vote diskgroup on all the nodes

  —  Modify diskgroup compatibility to 11.2.0.0

  — Move voting disk to the diskgroup vote – succeeds

[root@host01 ~]#  crsctl replace votedisk +vote

Successful addition of voting disk 443f1c60e16f4fa5bfbfeaae6b2f919d.

Successful deletion of voting disk 3d9da0d16baa4f10bf4e4b9b4aa688c6.

Successful deletion of voting disk 369621097c034f6dbf29a8cc97dc4bbc.

Successful deletion of voting disk 407b5d6588ea4f6fbf1503f7f6cc2951.

Successfully replaced voting disk group with +vote.

CRS-4266: Voting file(s) successfully replaced

– check that voting disk has been moved to vote diskgroup

[root@host01 ~]#  crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

–  —–    —————–                ——— ———

 1. ONLINE   443f1c60e16f4fa5bfbfeaae6b2f919d (ORCL:ASMDISK014) [VOTE]

Located 1 voting disk(s).

– Find out the disks in the other diskgroups – DATA and FRA

[grid@host01 ~]$ asmcmd lsdsk -G data

Path

ORCL:ASMDISK01

ORCL:ASMDISK010

ORCL:ASMDISK011

ORCL:ASMDISK012

[grid@host01 ~]$ asmcmd lsdsk -G fra

Path

ORCL:ASMDISK02

ORCL:ASMDISK03

– Set ASM_DISKTRING to all the disks which are members of DATA/FRA diskgroups

 SQL> sho parameter asm_diskstring

NAME                                 TYPE        VALUE

———————————— ———– ————

asm_diskstring                       string

SQL> alter system set asm_diskstring='ORCL:ASMDISK01','ORCL:ASMDISK010','ORCL:ASMDISK011','ORCL:ASMDISK012','ORCL:ASMDISK02','ORCL:ASMDISK03';

– check that discovery string in gpnp profile  points to the disks as specified 

[root@host01 ~]# cd /u01/app/11.2.0/grid/gpnp/host01/profiles/peer/

                         vi profile.xml

DiscoveryString=”ORCL:ASMDISK01,ORCL:ASMDISK010,ORCL:ASMDISK011,ORCL:ASMDISK012,ORCL:ASMDISK02,ORCL:ASMDISK03“

— stop and restart crs on the node –

]

[root@host01 ~]# crsctl stop crs

  [root@host01 ~]# crsctl start  crs

– check that has has started but rest of the services are not up

[root@host01 ~]# crsctl check has

CRS-4638: Oracle High Availability Services is online

[root@host01 ~]# crsctl check crs

]CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

CRS-4534: Cannot communicate with Event Manager

[root@host01 ~]# crsctl check css

CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

[root@host01 ~]# crsctl stat res -t

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Status failed, or completed with errors.

–Check the ocssd.log

– Scans all the disks which are part of the discovery string but does not find voting disk

 [root@host01 ~]# vi $ORACLE_HOME/log/host01/cssd/ocssd.log

2013-01-21 12:55:14.759: [    CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK01:0:0

2013-01-21 12:55:14.759: [   SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d0410 for disk :ORCL:ASMDISK01:

2013-01-21 12:55:14.759: [    CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK010:0:0

2013-01-21 12:55:14.759: [   SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d0c10 for disk :ORCL:ASMDISK010:

2013-01-21 12:55:14.759: [    CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK011:0:0

2013-01-21 12:55:14.759: [   SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d1508 for disk :ORCL:ASMDISK011:

2013-01-21 12:55:14.759: [    CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK012:0:0

2013-01-21 12:55:14.759: [   SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d1e00 for disk :ORCL:ASMDISK012:

2013-01-21 12:55:14.760: [    CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK02:0:0

2013-01-21 12:55:14.760: [   SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d26f8 for disk :ORCL:ASMDISK02:

2013-01-21 12:55:14.760: [    CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK03:0:0

@

2013-01-21 12:55:14.760: [    CLSF][2985733008]Read ASM header off dev:ORCL:ASMDISK03:0:0

2013-01-21 12:55:14.760: [   SKGFD][2985733008]Lib :ASM:/opt/oracle/extapi/32/asm/orcl/1/libasm.so: closing handle 0x91d2ff0 for disk :ORCL:ASMDISK03:

2013-01-21 12:55:14.760: [    CSSD][2985733008]clssnmvDiskVerify: file is not a voting file, cannot recognize on-disk signature for a voting

.

.

.

2013-01-21 12:55:14.760: [    CSSD][2985733008]clssnmvDiskVerify: Successful discovery of 0 disks

2013-01-21 12:55:14.760: [    CSSD][2985733008]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery

2013-01-21 12:55:14.760: [    CSSD][2985733008]clssnmvFindInitialConfigs: No voting files found

– Copy the existing gpnp profile to profile.bak  and edit the copy to modify discovery string  for asm

[root@host01 ~]# cd /u01/app/11.2.0/grid/gpnp/host01/profiles/peer/

[root@host01 peer]# cp profile.xml profile.bak

– remove the oracle signature from the file –

[root@host01 peer]# gpnptool unsign -p=profile.bak

Warning: some command line parameters were defaulted. Resulting command line:

         /u01/app/11.2.0/grid/bin/gpnptool.bin unsign -p=profile.bak -o-

– change the DiscoveryString itself –

[root@host01 peer]# gpnptool edit -asm_dis='ORCL:*' -p=profile.bak -o=profile.bak -ovr

Resulting profile written to “profile.bak”.

Success.

–  sign the profile xml file with the wallet (Notice: the path is only the directory to the wallet, NOT the wallet file itself) –

[root@host01 peer]# gpnptool sign -p=profile.bak -w=file:/u01/app/11.2.0/grid/gpnp/host01/wallets/peer/ -o=profile.new

Resulting profile written to “profile.new”.

Success.

–  move the original profile.xml out of the way 

[root@host01 peer]# mv profile.new profile.xml

– check that discovery string has been modified

[root@host01 peer]# vi profile.xml

DiscoveryString=”ORCL:*”

– Try to restart crs 

[root@host01 peer]# ps -ef |grep d.bin

kill all the processes

[root@host01 peer]# crsctl start crs

– check that all the daemons have started and cluster servicves are up

[root@host01 peer]# ps -ef |grep d.bin

[root@host01 peer]#crsctl stat res -t

Conclusion:

To access voting disk, gpnp profile is read to find the location of voting disk : found as asm

The asm disks as per the discovery string in gpnp profile are scanned to search for voting disk.

– Note : GPNP profile on other nodes also contains erroneous discovery string.

                  Hence copy the profile from the current node to the other nodes of