exadbcpatchmulti : Conflicts and ERROR: apply_async blocked on node

 Now a days i'm facing multiple issues on ExaCM boxes for patching on database nodes.

Sharing one scenario, Get request from customer apply APRIL_PSU2021 on ExaCM db boxes

ENV:
DB: 11204 version
GRID have 19c with APRIL2021 PSU.


*: Issue comes while running : exadbcpatchmulti -precheck_async 32537095 , failed with conflicts.
 Get details from exadbcpatch log / conflict log to identify more details.

*: How to resolve conflicts ?
Share the all details and conflict with oracle, They provide me the solution below
Edit /var/opt/oracle/exapatch/exadbcpatch.cfg - #For this file mostly root ownership, need to take help from GRID owner team or any other script available to update cfg file.

add the Cloud patch which needs to be rolled back as in below.
rollback_needed_before_apply="32328629::normal"

Refer Doc:
Exadbcpatchmulti patch conflict on cloud MLR "ERROR: precheck failed in conflict check" (Doc ID 2492936.1)

*: Now Pre-checks successfully done without any issue and move forward to apply the patch on next step.

*: Now facing the issue again : Node_1 successfully applied the patch but another issue occur on Node_2 below the details here.

++++++Error getting when apply patch: On Node_1 successfully done but having issue on Node_2
INFO: running on the node Xxx3a02c02vm01-adm
INFO: check for this action apply_async
INFO: check for this action rollback_async
INFO: Successfully released ohome lock. Proceeding to release local provisioning lock
INFO: Successfully released local provisioning lock
ERROR: apply_async blocked on node Xxx3a02c02vm01-adm: as Exadbcpatch operation already running on requested home
ERROR: apply_async blocked on node Xxx3a02c02vm01-adm: as Exadbcpatch operation already running on requested home

++++++Solution
execute on all the nodes then run exadbcpatchmulti again.
mv /var/opt/oracle/cstate.xml /var/opt/oracle/cstate.xml_old

Refer Doc:
Exadbcpatchmulti fails with Apply_async Blocked On Node X As Another Instance Of Exadbcpatch Running (Doc ID 2530468.1)

*: Now run exadbcpatchmulti  again and successfully applied the patch on ExaCM dbs.

ORA-20001: Latest xml inventory is not loaded into table

Symptoms: As i faced during execute datapatch verbose 19c.
During execution of datapatch or DBMS_QOPATCH following error seen

ERROR:
ORA-20001: Latest xml inventory is not loaded into table
ORA-06512: at "SYS.DBMS_QOPATCH", line 1448
ORA-06512: at "SYS.DBMS_QOPATCH", line 122

Solution: 

1. oraInventory issue
2. Recreate oraceinventory sometime works
3. 19c not using softlink so need to set correct entry into oratabs and run datapatch -verbose

-Because ORACLE_HOME using softlink that's why need to set this parameter for temporary purpose once datapatch done remove this parameter from pfile.

ORA-20001: Latest xml inventory is not loaded into table --->
1.
Startup upgrade
create or replace directory OPATCH_LOG_DIR as '$OH/QOpatch'; ## don't using ORACLE_HOME softlink
create or replace directory OPATCH_SCRIPT_DIR as '$OH/QOpatch';
 
ALTER SYSTEM SET "_disable_directory_link_check" = TRUE SCOPE=SPFILE;
shutdown immediate
startup upgrade
 
 
cd $ORACLE_HOME/OPatch
 
./datapatch -verbose
 
SELECT a.ksppinm "Parameter",b.ksppstvl "Session Value",c.ksppstvl "Instance Value" FROM x$ksppi a, x$ksppcv b, x$ksppsv c WHERE a.indx = b.indx AND a.indx = c.indx AND a.ksppinm LIKE '/_disable_direc%' escape '/';

Ref Doc:
datapatch fails with "ORA-20009:" “ORA-20013: DBMS_QOPATCH ran mostly in non install area” ORACLE_HOME is a SYMBOLIC LINK (Doc ID 2033620.1)
Queryable Patch Inventory - Issues/Solutions for ORA-20001: Latest xml inventory is not loaded into table (Doc ID 1602089.1)