ORACLE EXA-DATA SIMULATION SETUP

Use Virtual box / vmware  for Exa-Simulation

Download Software from Oracle Edelivery : https://edelivery.oracle.com/osdc/faces/Home.jspx;jsessionid=BRsZ6x_AFUhA9vhujM0Ot-pPtD92a3S-qlM4qLABO9NC1fwNqhJA!1493615362

11.2.3.2.1 : 64-bit > V36290-01.zip

Install an Oracle Linux

GI + DB = 11.2.0.4      

OS: First tested (below protocol is working or not)

rds, rds_tcp, rds_rdma

I’m tested on RHEL/OEL ( > 5.10 and 6.x )

Discussion and configure below the points.:

1.     Install an Oracle Linux / RHEL which is support rds, rds_tcp, rds_rdma protocols (because we are not using infiniband switch H/w)

2.     Add storage cell software

3.     Prepare some virtual disks

Installation Oracle Linux

1.       Minimum 2 GB RAM Required and 40 GB HDD only for illustration. (Exa-Simulation on laptop).

2.       1 NIC only (Host only Adapter) for storage cell. If you want Internet on this machine then one more NIC add with bridge connection.

3.       After OS installation completion.

4.       Selinux and firewall disabled.

Add storage cell software

[root@storagecell ]# vi /etc/hosts
…
[root@storagecell ]# more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       storagecell     storagecell.localdomain localhost.localdomain localhost

::1             localhost6.localdomain6 localhost6

Put the software cell image zip (V36290-01.zip) in the path of host system choosed as shared folder so it can be directly used inside the VM. In fact there is a special file system mounted

[root@storagecell ~]# mount OS on /home type vboxsf (gid=160,rw) [root@storagecell ~]# cd /home/ [root@storagecell ~]# ll total 1506576 -rwxrwx— 1 root vboxsf 1542729472 Oct 10 22:25 V36290-01.zip

Uncompress the file

[root@storagecell ]# unzip V36290-01.zip Archive:  V36290-01.zip inflating: README.txt inflating: cellImageMaker_11.2.3.2.1_LINUX.X64_130109-1.x86_64.tar

And then untar

[root@storagecell ]# tar pxvf cellImageMaker_11.2.3.2.1_LINUX.X64_130109-1.x86_64.tar

(there are some errors due permissions on host folder, but there is no problem because the only file needed i cell.bin)

[root@storagecell cellbits]# ll dl180/boot/cellbits/ total 1446844 -rwxrwx— 1 root vboxsf       729 Jan  9  2013 c7rpms.tbz -rwxrwx— 1 root vboxsf 245231205 Jan  9  2013 cell.bin -rwxrwx— 1 root vboxsf  12705374 Jan  9  2013 cellboot.tbz -rwxrwx— 1 root vboxsf 141444416 Jan  9  2013 cellfw.tbz -rwxrwx— 1 root vboxsf 142434203 Jan  9  2013 cellrpms.tbz -rwxrwx— 1 root vboxsf 208612489 Jan  9  2013 commonos.tbz -rwxrwx— 1 root vboxsf 375683818 Jan  9  2013 debugos.tbz -rwxrwx— 1 root vboxsf  55704927 Jan  9  2013 doclib.zip -rwxrwx— 1 root vboxsf 199485158 Jan  9  2013 exaos.tbz -rwxrwx— 1 root vboxsf  18186084 Jan  9  2013 hputils.tbz -rwxrwx— 1 root vboxsf  53387742 Jan  9  2013 kernel.tbz -rwxrwx— 1 root vboxsf  16165382 Jan  9  2013 ofed.tbz -rwxrwx— 1 root vboxsf  12485584 Jan  9  2013 sunutils.tbz

Then copy cell.bin and remove all

[root@storagecell ]# cp dl180/boot/cellbits/cell.bin . [root@storagecell ]# ll cell.bin total 1746060 -rwxrwx— 1 root vboxsf  245231205 Dec 12 10:43 cell.bin -rwxrwx— 1 root vboxsf 1542729472 Oct 10 22:25 V36290-01.zip

Now unzip cell.bin

[root@storagecell ]# unzip cell.bin Archive:  cell.bin warning [cell.bin]:  6408 extra bytes at beginning or within zipfile (attempting to process anyway) inflating: cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm inflating: jdk-1_5_0_15-linux-amd64.rpm

Install jdk

[root@storagecell sf_OS]# rpm -ivh jdk-1_5_0_15-linux-amd64.rpm Preparing…                ########################################### [100%] 1:jdk                    ########################################### [100%]

Preparing to install cell rpm

[root@storagecell ]# mkdir /var/log/oracle [root@storagecell ]# chmod 775 /var/log/oracle

(It will be used also by celladmin user …)

Install cell sw

[root@storagecell ]# rpm -ivh cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm Preparing…                ########################################### [100%] Pre Installation steps in progress … 1:cell                   ########################################### [100%] Post Installation steps in progress … Set cellusers group for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory Set 775 permissions for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory / / Installation SUCCESSFUL. Starting RS and MS… as user celladmin Done. Please Login as user celladmin and create cell to startup CELLSRV to complete cell configuration. WARNING: Using the current shell as root to restart cell services. Restart the cell services using a new shell.

Prepare some virtual disks

In 12c use at least 1GB file size. esp

1.      The storage cell software installed and some disks must be created.

2.      Shutdown the machine and with VirtualBox Settings create all the disks: 12 with size 500M (data) and 6 with size 400M (flash)

3.      After storage cell software install and Adding a disks. Some changes in gnome environment produce this strange behaviour.

4.      Switch to failsafe session > clieck on session then see the failsafe option.

5.      Log again in an xterm

6.      Edit /etc/bashrc

[root@storagecell]# vi /etc/bashrc

export DISPLAY=:0

7.      Save, exit and login again.

Now identify disks

[root@storagecell ~]# fdisk -l 2>/dev/null |grep “B,”
Disk /dev/sda: 26.8 GB, 26843545600 bytes
Disk /dev/sdb: 524 MB, 524288000 bytes
Disk /dev/sdc: 524 MB, 524288000 bytes
Disk /dev/sdd: 524 MB, 524288000 bytes
Disk /dev/sde: 524 MB, 524288000 bytes
Disk /dev/sdf: 524 MB, 524288000 bytes
Disk /dev/sdg: 524 MB, 524288000 bytes
Disk /dev/sdh: 524 MB, 524288000 bytes
Disk /dev/sdi: 524 MB, 524288000 bytes
Disk /dev/sdj: 524 MB, 524288000 bytes
Disk /dev/sdk: 524 MB, 524288000 bytes
Disk /dev/sdl: 524 MB, 524288000 bytes
Disk /dev/sdm: 524 MB, 524288000 bytes
Disk /dev/sdn: 419 MB, 419430400 bytes
Disk /dev/sdo: 419 MB, 419430400 bytes
Disk /dev/sdp: 419 MB, 419430400 bytes
Disk /dev/sdq: 419 MB, 419430400 bytes
Disk /dev/sdr: 419 MB, 419430400 bytes
Disk /dev/sds: 419 MB, 419430400 bytes

Looking at environment of the new celladmin user, it could be found an interesting variable that define where the cellsrv process will search for disks:

[root@storagecell unix]# echo $T_WORK /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks

Under that path a new folder is needed (raw) and symbolic link to “real” disks should be created

cd /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks mkdir disks mkdir disks/raw cd disks/raw ln -s /dev/sdb storagecell_DISK01 ln -s /dev/sdc storagecell_DISK02 ln -s /dev/sdd storagecell_DISK03 ln -s /dev/sde storagecell_DISK04 ln -s /dev/sdf storagecell_DISK05 ln -s /dev/sdg storagecell_DISK06 ln -s /dev/sdh storagecell_DISK07 ln -s /dev/sdi storagecell_DISK08 ln -s /dev/sdj storagecell_DISK09 ln -s /dev/sdk storagecell_DISK10 ln -s /dev/sdl storagecell_DISK11 ln -s /dev/sdm storagecell_DISK12 ln -s /dev/sdn storagecell_FLASH01 ln -s /dev/sdo storagecell_FLASH02 ln -s /dev/sdp storagecell_FLASH03 ln -s /dev/sdq storagecell_FLASH04 ln -s /dev/sdr storagecell_FLASH05 ln -s /dev/sds storagecell_FLASH06

Some Changes and Errors:

Add/Set  in /etc/sysctl.conf

fs.file-max = 65536

Edit the /etc/security/limit.conf files and add/set

* soft nofile 65536 * hard nofile 65536

To communicate over InfiniBand Oracle uses rds protocol. All the modules of rds must be loaded (and configured to be loaded over machine restarts)

[root@storagecell ~]# lsmod |grep rds*

rdma_cm                73429  2 rds_rdma,ib_iser

ib_cm                  72041  1 rdma_cm

iw_cm                  43593  1 rdma_cm

ib_sa                  76489  2 rdma_cm,ib_cm

ib_core               108097  7 rds_rdma,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad

ib_addr                42697  1 rdma_cm

ipv6                  438369  3 rdma_cm,ib_addr,cnic

[root@storagecell ~]# modprobe rds
[root@storagecell ~]# modprobe rds_tcp
[root@storagecell ~]# modprobe rds_rdma

[root@storagecell ~]# su - celladmin

[celladmin@storagecell ~]$ cellcli -e alter cell restart services all

CELL-01509: Restart Server (RS) not responding.
Starting the RS, CELLSRV, and MS services…
Getting the state of RS services… running
Starting CELLSRV services…
The STARTUP of CELLSRV services was not successful.
CELL-01547: CELLSRV startup failed due to unknown reasons.
Starting MS services…
The STARTUP of MS services was successful.

The error in not unknown (as stated) but well known and expected : [Required IP parameters missing]

So, set the interconnect (= InfiniBand connection)

[celladmin@storagecell ~]$ cellcli -e create cell storagecell interconnect1=eth0
Cell stocell1 successfully created
Starting CELLSRV services…
The STARTUP of CELLSRV services was successful.
Flash cell disks, FlashCache, and FlashLog will be created…
CellDisk FD_00_stocell1 successfully created
CellDisk FD_01_stocell1 successfully created
CellDisk FD_02_stocell1 successfully created
CellDisk FD_03_stocell1 successfully created
CellDisk FD_04_stocell1 successfully created
CellDisk FD_05_stocell1 successfully created
Flash log stocell1_FLASHLOG successfully created
Flash cache stocell1_FLASHCACHE successfully created

(I’m not sure why Flash components are auto configured, but it can be modified later if needed)
Configure cell disks

[celladmin@storagecell ~]$ cellcli -e create celldisk all
CellDisk CD_DISK01_stocell1 successfully created
CellDisk CD_DISK02_stocell1 successfully created
CellDisk CD_DISK03_stocell1 successfully created
CellDisk CD_DISK04_stocell1 successfully created
CellDisk CD_DISK05_stocell1 successfully created
CellDisk CD_DISK06_stocell1 successfully created
CellDisk CD_DISK07_stocell1 successfully created
CellDisk CD_DISK08_stocell1 successfully created
CellDisk CD_DISK09_stocell1 successfully created
CellDisk CD_DISK10_stocell1 successfully created
CellDisk CD_DISK11_stocell1 successfully created
CellDisk CD_DISK12_stocell1 successfully created

and grid disks

[celladmin@storagecell ~]$ cellcli -e create griddisk all harddisk prefix=DATA
GridDisk DATA_CD_DISK01_stocell1 successfully created
GridDisk DATA_CD_DISK02_stocell1 successfully created
GridDisk DATA_CD_DISK03_stocell1 successfully created
GridDisk DATA_CD_DISK04_stocell1 successfully created
GridDisk DATA_CD_DISK05_stocell1 successfully created
GridDisk DATA_CD_DISK06_stocell1 successfully created
GridDisk DATA_CD_DISK07_stocell1 successfully created
GridDisk DATA_CD_DISK08_stocell1 successfully created
GridDisk DATA_CD_DISK09_stocell1 successfully created
GridDisk DATA_CD_DISK10_stocell1 successfully created
GridDisk DATA_CD_DISK11_stocell1 successfully created
GridDisk DATA_CD_DISK12_stocell1 successfully created

Ø  Now configure another Machine for GI setup.

Note:- Some command & scripts not working on Fake H/W module of Exa-data. Like (imageinfo, imagehistory, patching etc.. doesn’t work on it).

And special thanks to my friends/Bro..
Skant Gupta, Vj sharma for completing this doc.