exadbcpatchmulti : Conflicts and ERROR: apply_async blocked on node

 Now a days i'm facing multiple issues on ExaCM boxes for patching on database nodes.

Sharing one scenario, Get request from customer apply APRIL_PSU2021 on ExaCM db boxes

ENV:
DB: 11204 version
GRID have 19c with APRIL2021 PSU.


*: Issue comes while running : exadbcpatchmulti -precheck_async 32537095 , failed with conflicts.
 Get details from exadbcpatch log / conflict log to identify more details.

*: How to resolve conflicts ?
Share the all details and conflict with oracle, They provide me the solution below
Edit /var/opt/oracle/exapatch/exadbcpatch.cfg - #For this file mostly root ownership, need to take help from GRID owner team or any other script available to update cfg file.

add the Cloud patch which needs to be rolled back as in below.
rollback_needed_before_apply="32328629::normal"

Refer Doc:
Exadbcpatchmulti patch conflict on cloud MLR "ERROR: precheck failed in conflict check" (Doc ID 2492936.1)

*: Now Pre-checks successfully done without any issue and move forward to apply the patch on next step.

*: Now facing the issue again : Node_1 successfully applied the patch but another issue occur on Node_2 below the details here.

++++++Error getting when apply patch: On Node_1 successfully done but having issue on Node_2
INFO: running on the node Xxx3a02c02vm01-adm
INFO: check for this action apply_async
INFO: check for this action rollback_async
INFO: Successfully released ohome lock. Proceeding to release local provisioning lock
INFO: Successfully released local provisioning lock
ERROR: apply_async blocked on node Xxx3a02c02vm01-adm: as Exadbcpatch operation already running on requested home
ERROR: apply_async blocked on node Xxx3a02c02vm01-adm: as Exadbcpatch operation already running on requested home

++++++Solution
execute on all the nodes then run exadbcpatchmulti again.
mv /var/opt/oracle/cstate.xml /var/opt/oracle/cstate.xml_old

Refer Doc:
Exadbcpatchmulti fails with Apply_async Blocked On Node X As Another Instance Of Exadbcpatch Running (Doc ID 2530468.1)

*: Now run exadbcpatchmulti  again and successfully applied the patch on ExaCM dbs.

Exadata : Disk Replacement

Operational Activities:

*.Diagnostics & SR Creation

   CellCLI> list celldisk where name='<Cell_Disk>' details

       Disk Status -> Online/Offline  #Both Cases

       Error Count -> more than 0.

   SR Creation

         1. ASR Configure

         2. Manual SR - read Doc ID - 761868.1

Reference :

Sundiag & Other Diagnostics: Oracle Exadata Diagnostic Information required for Disk Failures and some other Hardware issues (Doc ID 761868.1)

*.Drop Faulted Disk

   1.Normal -> Confined offline

          *.Amber LED

          *.Rebalance Initiated

   2.Confined offline -> offline

         *.Blue LED

         *.Ready for Replacement

Reference:

Drop Online Disk: Things to Check in ASM When Replacing an ONLINE disk from Exadata Storage Cell (Doc ID 1326611.1)

*.Disk replacement

    FSE visit and Replacement

Reference:

Replace Procedure: How to Replace a Hard Drive in an Exadata Storage Server (Predictive Failure) (Doc ID 1390836.1)

ORACLE EXA-DATA SIMULATION SETUP

Use Virtual box / vmware  for Exa-Simulation

Download Software from Oracle Edelivery : https://edelivery.oracle.com/osdc/faces/Home.jspx;jsessionid=BRsZ6x_AFUhA9vhujM0Ot-pPtD92a3S-qlM4qLABO9NC1fwNqhJA!1493615362

11.2.3.2.1 : 64-bit > V36290-01.zip

Install an Oracle Linux

GI + DB = 11.2.0.4      

OS: First tested (below protocol is working or not)

rds, rds_tcp, rds_rdma

I’m tested on RHEL/OEL ( > 5.10 and 6.x )

Discussion and configure below the points.:

1.     Install an Oracle Linux / RHEL which is support rds, rds_tcp, rds_rdma protocols (because we are not using infiniband switch H/w)

2.     Add storage cell software

3.     Prepare some virtual disks

Installation Oracle Linux

1.       Minimum 2 GB RAM Required and 40 GB HDD only for illustration. (Exa-Simulation on laptop).

2.       1 NIC only (Host only Adapter) for storage cell. If you want Internet on this machine then one more NIC add with bridge connection.

3.       After OS installation completion.

4.       Selinux and firewall disabled.

Add storage cell software

[root@storagecell ]# vi /etc/hosts
…
[root@storagecell ]# more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       storagecell     storagecell.localdomain localhost.localdomain localhost

::1             localhost6.localdomain6 localhost6

Put the software cell image zip (V36290-01.zip) in the path of host system choosed as shared folder so it can be directly used inside the VM. In fact there is a special file system mounted

[root@storagecell ~]# mount OS on /home type vboxsf (gid=160,rw) [root@storagecell ~]# cd /home/ [root@storagecell ~]# ll total 1506576 -rwxrwx— 1 root vboxsf 1542729472 Oct 10 22:25 V36290-01.zip

Uncompress the file

[root@storagecell ]# unzip V36290-01.zip Archive:  V36290-01.zip inflating: README.txt inflating: cellImageMaker_11.2.3.2.1_LINUX.X64_130109-1.x86_64.tar

And then untar

[root@storagecell ]# tar pxvf cellImageMaker_11.2.3.2.1_LINUX.X64_130109-1.x86_64.tar

(there are some errors due permissions on host folder, but there is no problem because the only file needed i cell.bin)

[root@storagecell cellbits]# ll dl180/boot/cellbits/ total 1446844 -rwxrwx— 1 root vboxsf       729 Jan  9  2013 c7rpms.tbz -rwxrwx— 1 root vboxsf 245231205 Jan  9  2013 cell.bin -rwxrwx— 1 root vboxsf  12705374 Jan  9  2013 cellboot.tbz -rwxrwx— 1 root vboxsf 141444416 Jan  9  2013 cellfw.tbz -rwxrwx— 1 root vboxsf 142434203 Jan  9  2013 cellrpms.tbz -rwxrwx— 1 root vboxsf 208612489 Jan  9  2013 commonos.tbz -rwxrwx— 1 root vboxsf 375683818 Jan  9  2013 debugos.tbz -rwxrwx— 1 root vboxsf  55704927 Jan  9  2013 doclib.zip -rwxrwx— 1 root vboxsf 199485158 Jan  9  2013 exaos.tbz -rwxrwx— 1 root vboxsf  18186084 Jan  9  2013 hputils.tbz -rwxrwx— 1 root vboxsf  53387742 Jan  9  2013 kernel.tbz -rwxrwx— 1 root vboxsf  16165382 Jan  9  2013 ofed.tbz -rwxrwx— 1 root vboxsf  12485584 Jan  9  2013 sunutils.tbz

Then copy cell.bin and remove all

[root@storagecell ]# cp dl180/boot/cellbits/cell.bin . [root@storagecell ]# ll cell.bin total 1746060 -rwxrwx— 1 root vboxsf  245231205 Dec 12 10:43 cell.bin -rwxrwx— 1 root vboxsf 1542729472 Oct 10 22:25 V36290-01.zip

Now unzip cell.bin

[root@storagecell ]# unzip cell.bin Archive:  cell.bin warning [cell.bin]:  6408 extra bytes at beginning or within zipfile (attempting to process anyway) inflating: cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm inflating: jdk-1_5_0_15-linux-amd64.rpm

Install jdk

[root@storagecell sf_OS]# rpm -ivh jdk-1_5_0_15-linux-amd64.rpm Preparing…                ########################################### [100%] 1:jdk                    ########################################### [100%]

Preparing to install cell rpm

[root@storagecell ]# mkdir /var/log/oracle [root@storagecell ]# chmod 775 /var/log/oracle

(It will be used also by celladmin user …)

Install cell sw

[root@storagecell ]# rpm -ivh cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm Preparing…                ########################################### [100%] Pre Installation steps in progress … 1:cell                   ########################################### [100%] Post Installation steps in progress … Set cellusers group for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory Set 775 permissions for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory / / Installation SUCCESSFUL. Starting RS and MS… as user celladmin Done. Please Login as user celladmin and create cell to startup CELLSRV to complete cell configuration. WARNING: Using the current shell as root to restart cell services. Restart the cell services using a new shell.

Prepare some virtual disks

In 12c use at least 1GB file size. esp

1.      The storage cell software installed and some disks must be created.

2.      Shutdown the machine and with VirtualBox Settings create all the disks: 12 with size 500M (data) and 6 with size 400M (flash)

3.      After storage cell software install and Adding a disks. Some changes in gnome environment produce this strange behaviour.

4.      Switch to failsafe session > clieck on session then see the failsafe option.

5.      Log again in an xterm

6.      Edit /etc/bashrc

[root@storagecell]# vi /etc/bashrc

export DISPLAY=:0

7.      Save, exit and login again.

Now identify disks

[root@storagecell ~]# fdisk -l 2>/dev/null |grep “B,”
Disk /dev/sda: 26.8 GB, 26843545600 bytes
Disk /dev/sdb: 524 MB, 524288000 bytes
Disk /dev/sdc: 524 MB, 524288000 bytes
Disk /dev/sdd: 524 MB, 524288000 bytes
Disk /dev/sde: 524 MB, 524288000 bytes
Disk /dev/sdf: 524 MB, 524288000 bytes
Disk /dev/sdg: 524 MB, 524288000 bytes
Disk /dev/sdh: 524 MB, 524288000 bytes
Disk /dev/sdi: 524 MB, 524288000 bytes
Disk /dev/sdj: 524 MB, 524288000 bytes
Disk /dev/sdk: 524 MB, 524288000 bytes
Disk /dev/sdl: 524 MB, 524288000 bytes
Disk /dev/sdm: 524 MB, 524288000 bytes
Disk /dev/sdn: 419 MB, 419430400 bytes
Disk /dev/sdo: 419 MB, 419430400 bytes
Disk /dev/sdp: 419 MB, 419430400 bytes
Disk /dev/sdq: 419 MB, 419430400 bytes
Disk /dev/sdr: 419 MB, 419430400 bytes
Disk /dev/sds: 419 MB, 419430400 bytes

Looking at environment of the new celladmin user, it could be found an interesting variable that define where the cellsrv process will search for disks:

[root@storagecell unix]# echo $T_WORK /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks

Under that path a new folder is needed (raw) and symbolic link to “real” disks should be created

cd /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks mkdir disks mkdir disks/raw cd disks/raw ln -s /dev/sdb storagecell_DISK01 ln -s /dev/sdc storagecell_DISK02 ln -s /dev/sdd storagecell_DISK03 ln -s /dev/sde storagecell_DISK04 ln -s /dev/sdf storagecell_DISK05 ln -s /dev/sdg storagecell_DISK06 ln -s /dev/sdh storagecell_DISK07 ln -s /dev/sdi storagecell_DISK08 ln -s /dev/sdj storagecell_DISK09 ln -s /dev/sdk storagecell_DISK10 ln -s /dev/sdl storagecell_DISK11 ln -s /dev/sdm storagecell_DISK12 ln -s /dev/sdn storagecell_FLASH01 ln -s /dev/sdo storagecell_FLASH02 ln -s /dev/sdp storagecell_FLASH03 ln -s /dev/sdq storagecell_FLASH04 ln -s /dev/sdr storagecell_FLASH05 ln -s /dev/sds storagecell_FLASH06

Some Changes and Errors:

Add/Set  in /etc/sysctl.conf

fs.file-max = 65536

Edit the /etc/security/limit.conf files and add/set

* soft nofile 65536 * hard nofile 65536

To communicate over InfiniBand Oracle uses rds protocol. All the modules of rds must be loaded (and configured to be loaded over machine restarts)

[root@storagecell ~]# lsmod |grep rds*

rdma_cm                73429  2 rds_rdma,ib_iser

ib_cm                  72041  1 rdma_cm

iw_cm                  43593  1 rdma_cm

ib_sa                  76489  2 rdma_cm,ib_cm

ib_core               108097  7 rds_rdma,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad

ib_addr                42697  1 rdma_cm

ipv6                  438369  3 rdma_cm,ib_addr,cnic

[root@storagecell ~]# modprobe rds
[root@storagecell ~]# modprobe rds_tcp
[root@storagecell ~]# modprobe rds_rdma

[root@storagecell ~]# su - celladmin

[celladmin@storagecell ~]$ cellcli -e alter cell restart services all

CELL-01509: Restart Server (RS) not responding.
Starting the RS, CELLSRV, and MS services…
Getting the state of RS services… running
Starting CELLSRV services…
The STARTUP of CELLSRV services was not successful.
CELL-01547: CELLSRV startup failed due to unknown reasons.
Starting MS services…
The STARTUP of MS services was successful.

The error in not unknown (as stated) but well known and expected : [Required IP parameters missing]

So, set the interconnect (= InfiniBand connection)

[celladmin@storagecell ~]$ cellcli -e create cell storagecell interconnect1=eth0
Cell stocell1 successfully created
Starting CELLSRV services…
The STARTUP of CELLSRV services was successful.
Flash cell disks, FlashCache, and FlashLog will be created…
CellDisk FD_00_stocell1 successfully created
CellDisk FD_01_stocell1 successfully created
CellDisk FD_02_stocell1 successfully created
CellDisk FD_03_stocell1 successfully created
CellDisk FD_04_stocell1 successfully created
CellDisk FD_05_stocell1 successfully created
Flash log stocell1_FLASHLOG successfully created
Flash cache stocell1_FLASHCACHE successfully created

(I’m not sure why Flash components are auto configured, but it can be modified later if needed)
Configure cell disks

[celladmin@storagecell ~]$ cellcli -e create celldisk all
CellDisk CD_DISK01_stocell1 successfully created
CellDisk CD_DISK02_stocell1 successfully created
CellDisk CD_DISK03_stocell1 successfully created
CellDisk CD_DISK04_stocell1 successfully created
CellDisk CD_DISK05_stocell1 successfully created
CellDisk CD_DISK06_stocell1 successfully created
CellDisk CD_DISK07_stocell1 successfully created
CellDisk CD_DISK08_stocell1 successfully created
CellDisk CD_DISK09_stocell1 successfully created
CellDisk CD_DISK10_stocell1 successfully created
CellDisk CD_DISK11_stocell1 successfully created
CellDisk CD_DISK12_stocell1 successfully created

and grid disks

[celladmin@storagecell ~]$ cellcli -e create griddisk all harddisk prefix=DATA
GridDisk DATA_CD_DISK01_stocell1 successfully created
GridDisk DATA_CD_DISK02_stocell1 successfully created
GridDisk DATA_CD_DISK03_stocell1 successfully created
GridDisk DATA_CD_DISK04_stocell1 successfully created
GridDisk DATA_CD_DISK05_stocell1 successfully created
GridDisk DATA_CD_DISK06_stocell1 successfully created
GridDisk DATA_CD_DISK07_stocell1 successfully created
GridDisk DATA_CD_DISK08_stocell1 successfully created
GridDisk DATA_CD_DISK09_stocell1 successfully created
GridDisk DATA_CD_DISK10_stocell1 successfully created
GridDisk DATA_CD_DISK11_stocell1 successfully created
GridDisk DATA_CD_DISK12_stocell1 successfully created

Ø  Now configure another Machine for GI setup.

Note:- Some command & scripts not working on Fake H/W module of Exa-data. Like (imageinfo, imagehistory, patching etc.. doesn’t work on it).

And special thanks to my friends/Bro..
Skant Gupta, Vj sharma for completing this doc.   

You are trying to install the 64-bit Cell rpm but a 32-bit JDK is installed. You should remove the 32-bit JDK rpm and use the cell binary to install the 64-bit JDK and the 64 bit CELL

Some Work Arround and fix this issue.

Below the error When I'm installing the cell rpm > After extracting cell.bin > install jdk > install cell but getting below error.


***************************************************************************************

[root@storagecell mnt]# rpm -ivh cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm
Preparing...                ########################################### [100%]
You are trying to install the 64-bit Cell rpm but a 32-bit JDK is installed. You should remove the 32-bit JDK rpm and use the cell binary to install the 64-bit JDK and the 64 bit CELL
error: %pre(cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64) scriptlet failed, exit status 4
error:   install: %pre scriptlet failed (2), skipping cell-11.2.3.2.1_LINUX.X64_130109-1

***************************************************************************************

[root@storagecell ~]# rpm -qa | grep jdk
jdk-1.5.0_15-fcs.x86_64
java-1.6.0-openjdk-devel-1.6.0.0-1.39.1.9.7.el6
java-1.6.0-openjdk-1.6.0.0-1.39.1.9.7.el6

***************************************************************************************
-----------
Solution:
-----------

RH6 “rpm -q” gives output with a different format from RH5.
That produces an error when pre script checks the installed rpm.

.rpmmacros in your home should fix the problem.

Creat .rpmmacro @ root home and change the query format.

[root@storagecell ~]# vim .rpmmacros

[root@storagecell ~]# rpm -qa | grep jdk
jdk-1.5.0_15-fcs
java-1.6.0-openjdk-devel-1.6.0.0-1.39.1.9.7.el6
java-1.6.0-openjdk-1.6.0.0-1.39.1.9.7.el6


[root@stocell1 ~]# cat .rpmmacros
%_query_all_fmt %%{name}-%%{version}-%%{release}



[root@storagecell mnt]# rpm -ivh cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm
Preparing...                ########################################### [100%]
Pre Installation steps in progress ...
ping: unknown host storagecell.localdomain
Check hostname setting. Cannot install cell
error: %pre(cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64) scriptlet failed, exit status 5
error:   install: %pre scriptlet failed (2), skipping cell-11.2.3.2.1_LINUX.X64_130109-1

[root@storagecell mnt]# vim /etc/hosts

[root@storagecell mnt]# cat /etc/hosts
127.0.0.1   storagecell storagecell.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

[root@storagecell mnt]# rpm -ivh cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm
Preparing...                ########################################### [100%]
Pre Installation steps in progress ...
   1:cell                   ########################################### [100%]
Post Installation steps in progress ...
Set cellusers group for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory
Set 775 permissions for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory
/
/
Installation SUCCESSFUL.
Starting RS and MS... as user celladmin
Done. Please Login as user celladmin and create cell to startup CELLSRV to complete cell configuration.
WARNING: Using the current shell as root to restart cell services.
Restart the cell services using a new shell.

[root@storagecell mnt]#

Exadata

SMART_FLASH_CACHE

Smart flash_cache in 11gR2 and > ( 12c ) Version  supports on OEL or Solaris only DB Smart Flash Cache in Oracle 11g

In case you don’t have budget to buy Exadata you can still buy huge number of flash disks and put on them part of your database. But what should be stored on flash disks(very fast) and what on magnetic disks(very slow) ?

It’s not your businesses to know let decide database.

Introduction

DB Smart Flash Cache is new extension for buffer cache area. This extra area should be defined on solid state disks (SSD) and has following features:

•          make performance improvements at moderate cost(cheaper than DRAM)

•          low latency compared to magnetic disks

•          higher throughput compared to magnetic disks

•          easy to setup

•          easy to control

•          can be used for RAC cache fusion keeps consistency

•          direct I/O bypasses buffer cache so as well bypasses DB smart flash cache

•          can cache only clean blocks from buffer cache

•          flash cache is not auto-tuned

•          only blocks from standard buffer pool are cached in DB smart flash cache

Oracle recommends:

•          flash disks should have comparable read IOPs and IOPs write

•          this new layer should be at least 2-10 times bigger than buffer cache in DRAM

•          mainly for OLTP systems

Architecture

if a oracle server process needs to read a block from database at first it must read it from magnetic disk(physical read). Then the block is stored in buffer cache memory and added to standard “LRU chain” list.

When “in memory buffers area” is getting full Oracle must decide which blocks needs to be removed from cache. If you have DB Smart Flash Cache enabled “clean” blocks are written to “Flash cache” by DBWR process so next time they can be read into memory from Flash Cache and improve your performance.

NOTE: “Dirty” blocks are never stored in Flash Cache

List of blocks cached in DB smart flash cache are stored in buffer cache area on two dedicated flash “LRU lists” depending on object attribute FLASH_CACHE:

•          DEFAULT – standard last recently used algorithm decides how long such blocks are cached in flash cache. It’s default value assigned to each object in database.

•          KEEP – such blocks are not removed from flash cache as long as the flash cache is large enough

alter|create table|index objectname

storage

(  

 buffer_pool { keep | recycle | default }

   flash_cache { keep | none    | default }

);

NONE value for FLASH_CACHE is blocking flash caching for a given object.

Statistics

All I/O operations from DB smart flash cache are counted as physical I/O however Oracle also collects such informations in new columns.

V$SQL - OPTIMIZED_PHY_READ_REQUESTS
V$SQLAREA - OPTIMIZED_PHY_READ_REQUESTS
V$FILESTAT - OPTIMIZED_PHYBLKRD

select name from v$statname where name like 'physical%optimized%';

NAME                                                           
----------------------------------------------------------------
physical read requests optimized                                 
physical read total bytes optimized

You can see such stats in V$SESSTAT and V$SYSSTAT

Setup

Two parameters must be set on database level to turn on DB smart flash cache:

• DB_FLASH_CACHE_FILE – defines (OS disk path or ASM disk group) and file name to store this data

• DB_FLASH_CACHE_SIZE – defines size of the flash cache

DB_FLASH_CACHE_FILE='/os path/flash_cache_file.dbf'
DB_FLASH_CACHE_FILE='+FLASH_DISK_GROUP/flash_cache_file.dbf'
DB_FLASH_CACHE_SIZE=200m

After setting both parameters you need to restart database.

DB_FLASH_CACHE_FILE

•          can’t be shared between many databases or instances DB_FLASH_CACHE_SIZE

•          can’t be dynamically resized

•           can be set to 0 to disable DB smart flash cache

•          can be set to original size to re-enable DB smart flash cache

Performance improvements

Oracle conducted interesting test for a OLTP database 70GB size with 8GB SGA. From below picture you can see improvements for Transactions versus size of DB smart cache size.

Following picture shows improvement in transaction response time versus DB smart cache size

Example

I simulate SSD disk by creation ramdisk – disk based in memory using following steps:

1. create directory to mount ramdisk and change owner to oracle and group dba

[root@oel5 /]mkdir /ramdisk
[root@oel5 /]chown oracle:dba -R /ramdisk

2. mount ramdisk and check it

[root@oel5 /]# mount -t tmpfs none /ramdisk -o size=256m

[root@oel5 /]# mount | grep ramdisk

none on /ramdisk type tmpfs (rw,size=256m)

3. set parameters for database and restart it as user oracle

SQL> alter system set db_flash_cache_file='/ramdisk/ram.dbf'
SQL> scope=spfile;
System altered.

SQL> alter system set db_flash_cache_size=200M scope=sp;
System altered. 

SQL> startup force
ORACLE instance started.
Total System Global Area  835104768 bytes
Fixed Size                  2232960 bytes
Variable Size             507514240 bytes
Database Buffers          322961408 bytes
Redo Buffers                2396160 bytes
Database mounted.
Database opened.

SQL> show parameter flash_cache

NAME                    TYPE        VALUE
----------------------- ----------- ------------------------------
db_flash_cache_file     string      /ramdisk/ram.dbf
db_flash_cache_size     big integer 200M

4. Check new file exists in /ramdisk

[root@oel5 ramdisk]# ll
total 8
-rw-r----- 1 oracle asmadmin 209715200 Feb 24 22:54 ram.dbf

5. Let’s create tables with flash_cache keep reference in storage clause so Oracle will try to keep the blocks in DB smart cache as long as possible.

create table test_tbl1(id number,id1 number,id2 number)storage(flash_cache keep);

begin

  for i in 1..1000000

  loop

    insert into test_tbl1 values(i, i, i);

  end loop;

  commit;

end;

/

6. Eventually after some time you can see some data in flash cache – v$bh view.

select status, count(*) from v$bh
group by status;

STATUS       COUNT(*)
---------- ----------
xcur            36915
flashcur        25583
cr                 13

7. If you clean buffer cache as well db smart flash cache is purged

alter system flush buffer_cache;
system FLUSH altered. 

STATUS       COUNT(*)
---------- ----------
xcur              520
free            36411

ERROR:-

I do all steps of your manual, but after ‘startup force’ I have an error:

SQL> startup force
ORA-00439: feature not enabled: Server Flash Cache
What am I doing wrong?
My configuration:
Oracle Linux Server release 6.4
2.6.39-400.24.1.el6uek.x86_64
Oracle Database 11g 11.2.0.3.0

Patch 12949806: FLASH CACHE CHECK IS AGAINST ENTERPRISE-RELEASE
Now it works!

------FOR TESTING PURPOSE------

    # fdisk -l /dev/sdc

    Disk /dev/sdc: 24.5 GB, 24575868928 bytes

    255 heads, 63 sectors/track, 2987 cylinders

    Units = cylinders of 16065 * 512 = 8225280 bytes

    Disk /dev/sdc doesn't contain a valid partition table

    # chmod 777 /dev/sdc

set Oracle initialization parameters:

    $ sqlplus / as sysdba

    SQL> alter system set db_flash_cache_file='/dev/sdc' scope=spfile;

    System altered.

    SQL> alter system set db_flash_cache_size=10G scope=spfile;

    System altered.

Stop/Start database

    SQL> shutdown immediate

Reference:

12c Below one more my favrt. link to configure and understand smart flash cache.

  https://oracle-base.com/articles/12c/database-smart-flash-cache-12cr1