Blog
Fast-Start Failover, Observe-Only Mode and Health Conditions
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

Fast-Start Failover, Observe-Only Mode and Health Conditions

Oracle Data Guard Broker allows the database administrators to automate some tasks and an easy way to configure properly a lot of features and details for data guard environments. The Fast-Start FailOver (FSFO) allows the broker to automatically failover to standby database in case of failure of the primary. But until 19c the only option is always to trigger the failover. This changed at 19c with a nice new feature that allows us to put FSFO in Observe-Only Mode.
In this post, I will focus just on new features for FSFO like Observe-Only Mode and Health Conditions for it. Lag and other details will not be covered here.

 

Observe-Only Mode

The Observe-Only Mode is a simple change that allows putting the FSFO to just observing/monitoring the DG environment, but in case of failure, it does not change the roles between primary and standby. Simple like that. As the Broker documentation for Observe-Only Mode says:
The observe-only mode enables you to test the impact of using fast-start failover in your configuration, without making any actual changes to the configuration.
Mode details can be checked in this link at documentation too. But FSFO is that:

 

 

Enable Observe-Only

So, to enable it is very simple, just need to call “ENABLE FAST_START FAILOVER OBSERVE ONLY”:

 

DGMGRL> ENABLE FAST_START FAILOVER OBSERVE ONLY;

Enabled in Observe-Only Mode.

DGMGRL>

 

And at drc* trace file at primary side we can see:

 

2020-06-11T23:45:19.329+02:00

ENABLE FAST_START FAILOVER OBSERVE ONLY

FSFO SetState(st=47 "ENABLE OBONLY", fl=0x0 "", ob=0x2b621d39, tgt=2, v=0)

Setup log_archive_dest_n of GROUP=0 PRIORITY=0 with 'golds19c' as FSFO target

Fast-Start Failover (FSFO) has been enabled under observe-only mode between:

  Primary = "gold19c"

  Standby = "golds19c"

2020-06-11T23:45:20.527+02:00

ENABLE FAST_START FAILOVER OBSERVE ONLY completed successfully

 

And the result it is FSFO at Observe-Only Mode

 

DGMGRL> show fast_start failover;




Fast-Start Failover: Enabled in Observe-Only Mode




  Protection Mode:    MaxAvailability

  Lag Limit:          0 seconds




  Threshold:          30 seconds

  Active Target:      golds19c

  Potential Targets:  "golds19c"

    golds19c   valid

  Observer:           goldsn1.oralocal

  Shutdown Primary:   TRUE

  Auto-reinstate:     TRUE

  Observer Reconnect: (none)

  Observer Override:  FALSE




Configurable Failover Conditions

  Health Conditions:

    Corrupted Controlfile          YES

    Corrupted Dictionary           YES

    Inaccessible Logfile            NO

    Stuck Archiver                  NO

    Datafile Write Errors          YES




  Oracle Error Conditions:

    (none)




DGMGRL>

 

And after we force the shutdown of the database, we can see that the roles not changed.

 

[oracle@goldpn1 ~]$ srvctl stop database -d gold19c -o abort

[oracle@goldpn1 ~]$

 

At Observer log file we can see some information that the error with primary was detected but nothing is done since it is in observe mode:

 

…

Unable to connect to database using gold19c

[W000 2020-06-12T00:13:22.248+02:00] Primary database cannot be reached.

[W000 2020-06-12T00:13:22.248+02:00] Fast-Start Failover threshold has expired.

[W000 2020-06-12T00:13:22.248+02:00] Try to connect to the standby.

[W000 2020-06-12T00:13:22.248+02:00] Making a last connection attempt to primary database before proceeding with Fast-Start Failover.

[W000 2020-06-12T00:13:22.248+02:00] Check if the standby is ready for failover.

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor




Unable to connect to database using gold19c

[W000 2020-06-12T00:13:22.261+02:00] A fast-start failover would have been initiated...

[W000 2020-06-12T00:13:22.261+02:00] Unable to failover since this observer is in observe-only mode

[W000 2020-06-12T00:13:22.261+02:00] Fast-Start Failover is not possible because observe-only mode.

[W000 2020-06-12T00:13:22.261+02:00] Try to connect to the primary.

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor




Unable to connect to database using gold19c

[W000 2020-06-12T00:13:22.269+02:00] Primary database cannot be reached.

[W000 2020-06-12T00:13:22.269+02:00] Fast-Start Failover observe-only mode enabled.

[W000 2020-06-12T00:13:22.269+02:00] Will not attempt a Fast-Start Failover.

[W000 2020-06-12T00:13:22.269+02:00] Retry connecting to primary.

[W000 2020-06-12T00:13:23.270+02:00] Try to connect to the primary.

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor




Unable to connect to database using gold19c

[W000 2020-06-12T00:13:23.277+02:00] Primary database cannot be reached.

[W000 2020-06-12T00:13:24.278+02:00] Try to connect to the primary.

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

…

 

And at drc* trace file at standby side we can see:

 

2020-06-12T00:13:21.103+02:00

Fast-Start Failover cannot proceed because: "observe-only mode"

 

Until now, this means that error with primary was detected, logged at logs, but no action was taken. The roles continue the same. The show report confirm this too:

 

DGMGRL> show configuration verbose;




Configuration - gold19c




  Protection Mode: MaxAvailability

  Members:

  gold19c  - Primary database

    golds19c - (*) Physical standby database




  (*) Fast-Start Failover target




  Properties:

    FastStartFailoverThreshold      = '30'

    OperationTimeout                = '30'

    TraceLevel                      = 'USER'

    FastStartFailoverLagLimit       = '0'

    CommunicationTimeout            = '180'

    ObserverReconnect               = '0'

    FastStartFailoverAutoReinstate  = 'TRUE'

    FastStartFailoverPmyShutdown    = 'TRUE'

    BystandersFollowRoleChange      = 'ALL'

    ObserverOverride                = 'FALSE'

    ExternalDestination1            = ''

    ExternalDestination2            = ''

    PrimaryLostWriteAction          = 'CONTINUE'

    ConfigurationWideServiceName    = 'gold19c_CFG'




Fast-Start Failover: Enabled in Observe-Only Mode

  Lag Limit:          0 seconds

  Threshold:          30 seconds

  Active Target:      golds19c

  Potential Targets:  "golds19c"

    golds19c   valid

  Observer:           goldsn1.oralocal

  Shutdown Primary:   TRUE

  Auto-reinstate:     TRUE

  Observer Reconnect: (none)

  Observer Override:  FALSE




Configuration Status:

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

ORA-16625: cannot reach member "gold19c"

DGM-17017: unable to determine configuration status




DGMGRL> show fast_start failover;




Fast-Start Failover: Enabled in Observe-Only Mode




  Protection Mode:    MaxAvailability

  Lag Limit:          0 seconds




  Threshold:          30 seconds

  Active Target:      golds19c

  Potential Targets:  "golds19c"

    golds19c   valid

  Observer:           goldsn1.oralocal

  Shutdown Primary:   TRUE

  Auto-reinstate:     TRUE

  Observer Reconnect: (none)

  Observer Override:  FALSE




Configurable Failover Conditions

  Health Conditions:

    Corrupted Controlfile          YES

    Corrupted Dictionary           YES

    Inaccessible Logfile            NO

    Stuck Archiver                  NO

    Datafile Write Errors          YES




  Oracle Error Conditions:

    (none)




DGMGRL>

 

In some scenarios, this can be good because it allows us to fix the problem and not pass from a failover event (manual reinstate and so on in some cases). Another option is to use the Observe-Only mode to do what the name says, just observer. Think in one environment that you want to test some conditions and the health of the environment (network and others) before you really enable the FSFO.
 
So, if the primary database returns, the FSFO returns normally:

 

[oracle@goldpn1 ~]$ srvctl start database -d gold19c

[oracle@goldpn1 ~]$

 

At drc* file for standby:

 

2020-06-12T00:16:52.837+02:00

Primary connected to this instance.

2020-06-12T00:17:00.186+02:00

FSFO SetState(st=2 "UNSYNC", fl=0x1 "AVAIL", ob=0x0, tgt=2, v=11)

2020-06-12T00:17:06.951+02:00

FSFO SetState(st=1 "SYNC", fl=0x1 "AVAIL", ob=0x0, tgt=2, v=12)

 

At Broker:

 

DGMGRL> show configuration;




Configuration - gold19c




  Protection Mode: MaxAvailability

  Members:

  gold19c  - Primary database

    golds19c - (*) Physical standby database




Fast-Start Failover: Enabled in Observe-Only Mode




Configuration Status:

SUCCESS   (status updated 51 seconds ago)




DGMGRL> show fast_start failover;




Fast-Start Failover: Enabled in Observe-Only Mode




  Protection Mode:    MaxAvailability

  Lag Limit:          0 seconds




  Threshold:          30 seconds

  Active Target:      golds19c

  Potential Targets:  "golds19c"

    golds19c   valid

  Observer:           goldsn1.oralocal

  Shutdown Primary:   TRUE

  Auto-reinstate:     TRUE

  Observer Reconnect: (none)

  Observer Override:  FALSE




Configurable Failover Conditions

  Health Conditions:

    Corrupted Controlfile          YES

    Corrupted Dictionary           YES

    Inaccessible Logfile            NO

    Stuck Archiver                  NO

    Datafile Write Errors          YES




  Oracle Error Conditions:

    (none)




DGMGRL>

 

Upgrade and Downgrade modes

If the FSFO is operating  in Observer-Only ode it is impossible to “upgrade” it to normal mode:

 

DGMGRL>  ENABLE FAST_START FAILOVER

Error: ORA-16889: Fast-start failover mode cannot be changed between normal and observe-only modes.




Failed.

DGMGRL>

 

To do that, we need to disable the FSFO and enable it in normal mode:

 

DGMGRL>  ENABLE FAST_START FAILOVER

Error: ORA-16889: Fast-start failover mode cannot be changed between normal and observe-only modes.




Failed.

DGMGRL>

DGMGRL> DISABLE FAST_START FAILOVER ;

Disabled.

DGMGRL> ENABLE FAST_START FAILOVER ;

Enabled in Zero Data Loss Mode.

DGMGRL>

 

To downgrade is the same, we can’t downgrade directly, need to disable and change to Observer-Only mode:

 

DGMGRL> ENABLE FAST_START FAILOVER OBSERVE ONLY;

Error: ORA-16889: Fast-start failover mode cannot be changed between normal and observe-only modes.




Failed.

DGMGRL>

 

Health Conditions

This is not a new feature for 19c, but help to reduce the scenarios where FSFO is triggered. It is possible to control the Health Conditions and disable/enable some errors like corrupted controlfiles or stuck archive. All options can be checked here at the documentation.
Look below at “Configurable Failover Conditions”, everything there can be set:

 

DGMGRL>  show fast_start failover;




Fast-Start Failover: Enabled in Observe-Only Mode




  Protection Mode:    MaxAvailability

  Lag Limit:          0 seconds




  Threshold:          30 seconds

  Active Target:      golds19c

  Potential Targets:  "golds19c"

    golds19c   valid

  Observer:           goldsn1.oralocal

  Shutdown Primary:   TRUE

  Auto-reinstate:     TRUE

  Observer Reconnect: (none)

  Observer Override:  FALSE




Configurable Failover Conditions

  Health Conditions:

    Corrupted Controlfile          YES

    Corrupted Dictionary           YES

    Inaccessible Logfile            NO

    Stuck Archiver                  NO

    Datafile Write Errors          YES




  Oracle Error Conditions:

    (none)




DGMGRL>

 

Some examples

 

DGMGRL> ENABLE FAST_START FAILOVER CONDITION "Inaccessible Logfile";

Succeeded.

DGMGRL> show fast_start failover;




Fast-Start Failover: Enabled in Observe-Only Mode




  Protection Mode:    MaxAvailability

  Lag Limit:          0 seconds




  Threshold:          30 seconds

  Active Target:      golds19c

  Potential Targets:  "golds19c"

    golds19c   valid

  Observer:           goldsn1.oralocal

  Shutdown Primary:   TRUE

  Auto-reinstate:     TRUE

  Observer Reconnect: (none)

  Observer Override:  FALSE




Configurable Failover Conditions

  Health Conditions:

    Corrupted Controlfile          YES

    Corrupted Dictionary           YES

    Inaccessible Logfile           YES

    Stuck Archiver                  NO

    Datafile Write Errors          YES




  Oracle Error Conditions:

    (none)




DGMGRL>




DGMGRL> ENABLE FAST_START FAILOVER CONDITION "Corrupted Dictionary";

Succeeded.

DGMGRL> DISABLE FAST_START FAILOVER CONDITION "Inaccessible Logfile";

Succeeded.

DGMGRL> show fast_start failover;




Fast-Start Failover: Enabled in Observe-Only Mode




  Protection Mode:    MaxAvailability

  Lag Limit:          0 seconds




  Threshold:          30 seconds

  Active Target:      golds19c

  Potential Targets:  "golds19c"

    golds19c   valid

  Observer:           goldsn1.oralocal

  Shutdown Primary:   TRUE

  Auto-reinstate:     TRUE

  Observer Reconnect: (none)

  Observer Override:  FALSE




Configurable Failover Conditions

  Health Conditions:

    Corrupted Controlfile          YES

    Corrupted Dictionary           YES

    Inaccessible Logfile            NO

    Stuck Archiver                  NO

    Datafile Write Errors          YES




  Oracle Error Conditions:

    (none)




DGMGRL>

 

Another option is to enable (or disable) for special condition/error from controlfile. The error ORA-240 can be set at trigger option:

 

DGMGRL> ENABLE FAST_START FAILOVER CONDITION 240;

Succeeded.

DGMGRL> show fast_start failover;




Fast-Start Failover: Enabled in Observe-Only Mode




  Protection Mode:    MaxAvailability

  Lag Limit:          0 seconds




  Threshold:          30 seconds

  Active Target:      golds19c

  Potential Targets:  "golds19c"

    golds19c   valid

  Observer:           goldsn1.oralocal

  Shutdown Primary:   TRUE

  Auto-reinstate:     TRUE

  Observer Reconnect: (none)

  Observer Override:  FALSE




Configurable Failover Conditions

  Health Conditions:

    Corrupted Controlfile          YES

    Corrupted Dictionary           YES

    Inaccessible Logfile            NO

    Stuck Archiver                  NO

    Datafile Write Errors          YES




  Oracle Error Conditions:

    ORA-240: control file enqueue held for more than %s seconds




DGMGRL> DISABLE FAST_START FAILOVER CONDITION 240;

Succeeded.

DGMGRL>

 

But just for ORA-240, other errors are not yet enabled:

 

DGMGRL> ENABLE FAST_START FAILOVER CONDITION 600;

Error: ORA-16524: unsupported command, option, or argument




Failed.

DGMGRL>

 

Observe-Only and Conditions

The new feature Observe-Only mode for 19c is a good feature because it allows more control where and when the FSFO is triggered. Until this, the only option was ON or OFF. And in scenarios that you want to test, or even validate the environment before enable (for real) was impossible.
And if we combine this with Heath Condition check, it is a powerful control for the DG environment. It allows a better tune.
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”

 

 


ZDLRA, ORDERING_WAIT task state
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

ZDLRA, ORDERING_WAIT task state

Tasks for ZDLRA are the pillar of how the backups are processed, everything is one task. So, when you ingest incremental backup one task is created but can occur that it get a freeze at ORDERING_WAIT state. These tasks are hard to identify and can create a big problem for your virtual full backup and backup strategy. Below I will show how they occur and how to solve the problem.

 

Incremental backups

To understand how they appear I need to show a little how the incremental backups work. Basically, look at the example below and one datafile with some blocks. When you do backup level 0, all the blocks are copied, and if I do backup incremental level 1 just the “new blocks” are copied (just blocks that changed).

 

 

As you can see above, the first backup of datafile (level 0), have SCN 3333. So, the consequent backup level 1 will copy everything that changed from that (and in this example have SCN 4444). And, the next incremental backup will pick up everything since the last backup, in this example, every block change since SCN 4444. So, it will generate a new backup with/until SCN 5555.
As you know, this is the definition of incremental backup. As you can see in the definition at “About RMAN Incremental Backups” from docs it is:
“An incremental backup copies only those data blocks that have changed since a previous backup. You can use RMAN to create incremental backups of data files, tablespaces, or the whole database.”
But the point it is that database/rman knows the scn from the last backup, and when does incremental backup it copy everything since from the last scn. Each incremental backup “knows” (internally, with the database blocks that are inside) the start scn and endpoint scn. So, to “reconstruct” the datafile, the database/rman uses the full backup and all subsequent incremental backups.

 

Incremental backup and ZDLRA

 

As I already wrote about incremental backups and ZDLRA, they are used to construct the “Virtual Full Backup”. In a very resumed way, ZDLRA merge the stored backups and create the virtual full backup (as I explained here too).

 

 

But even with this virtual backup, the way how the incremental backups work not change. The procedure is the same, check the scn from the last backup and copy all block change since that. As you can see in the image above, the full backups (blue in the image) are created merging the previous full with the ingested backup and are used as the base for the subsequent incremental.

 

ORDERING_WAIT

The ORDERING_WAIT occurs when the task INDEX_BACKUP that creates the index (and the virtual full backup) can’t finish because it doesn’t have all the required data. And this occurs because (by some reason) one backup is created and not stored at ZDLRA. And can be even a duplicate to create the standby (remember that basically the duplicate is rman backup copied from one side to another).

 

 

Look the image above, when after the SCN 4444 (that was the last backup stored at ZDLRA), another backup was taken and it is not inside of ZDLRA. So, when the new incremental backup is taken, it will copy all blocks changed from the last backup, but this last backup was the one that is not at ZDLRA (for rman side, it don’t care where it is the backup. By definition, incremental backup it is from the last backup, whatever or wherever it is).

 

And when this incremental backup it is ingested at ZDLRA, it will try to create the virtual full. But since the last stored backup have the SCN 4444, and the new incremental pickup blocks changed since SCN 5555 and go until the SCN 6666, ZDLRA knows that it is a gap when opens this ingested backup. ZDLRA doesn’t have the blocks that are between SCN 4444 and 5555 (look the yellow block, backup exists just outside of ZDLRA).

 

So, it is impossible to create the virtual full backup for SCN 6666, and the INDEX_BACKUP task will be at hold in state ORDERING_WAIT. To solve, there is two option, you can take a new level 0 backup or use BACKUP [CUMULATIVE] INCREMENTAL LEVEL 1 … FOR RECOVER OF TAG ‘<TAG>’ command. I will show you below how to do that.

 

How this occurs and how to solve

Bellow, I will show how you can identify and solve the problem. I will use the solution wrote in the last paragraph. But you can check the internal details of how occurs, to identify, and how to solve the issue.
In this scenario, I will use one database (number 12 – tablespace users for one PDB). So, first, check the backups for datafile 12:

 

RMAN> list backup of datafile 12 completed after "sysdate - 20/1440";







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24283   Incr 1  40.00K     SBT_TAPE    00:00:04     20/04/2020 23:49:26

        BP Key: 24284   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T234921

        Handle: VB$_1891149551_24282I   Media:

  List of Datafiles in backup set 24283

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2013573    20/04/2020 23:49:22              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24287   Incr 0  40.00K     SBT_TAPE    00:00:04     20/04/2020 23:49:26

        BP Key: 24288   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T234921

        Handle: VB$_1891149551_24282_12   Media:

  List of Datafiles in backup set 24287

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   0  Incr 2013573    20/04/2020 23:49:22              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




RMAN>

 

As you can see, we have Incr 1 and Incr 0 levels with the same scn. And the name of the handle starts with VB$. This means that virtual full backup it is OK and created by ZDLRA.
And if I do a new incremental backup, ZDLRA generates a new virtual full backup:

 



RMAN> BACKUP INCREMENTAL LEVEL 1 DEVICE TYPE SBT FILESPERSET 1 DATAFILE 12;




Starting backup at 20/04/2020 23:51:58

using channel ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 20/04/2020 23:51:59

channel ORA_SBT_TAPE_1: finished piece 1 at 20/04/2020 23:52:02

piece handle=ORCL18C_a1uu5dsv_1_1 tag=TAG20200420T235158 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03

Finished backup at 20/04/2020 23:52:02




Starting Control File and SPFILE Autobackup at 20/04/2020 23:52:02

piece handle=c-558466555-20200420-0b comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 20/04/2020 23:52:10




RMAN> list backup of datafile 12 completed after "sysdate - 20/1440";







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24283   Incr 1  40.00K     SBT_TAPE    00:00:04     20/04/2020 23:49:26

        BP Key: 24284   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T234921

        Handle: VB$_1891149551_24282I   Media:

  List of Datafiles in backup set 24283

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2013573    20/04/2020 23:49:22              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24287   Incr 0  40.00K     SBT_TAPE    00:00:04     20/04/2020 23:49:26

        BP Key: 24288   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T234921

        Handle: VB$_1891149551_24282_12   Media:

  List of Datafiles in backup set 24287

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   0  Incr 2013573    20/04/2020 23:49:22              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24355   Incr 1  40.00K     SBT_TAPE    00:00:03     20/04/2020 23:52:02

        BP Key: 24356   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T235158

        Handle: VB$_1891149551_24354I   Media:

  List of Datafiles in backup set 24355

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2013810    20/04/2020 23:51:59              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24359   Incr 0  40.00K     SBT_TAPE    00:00:03     20/04/2020 23:52:02

        BP Key: 24360   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T235158

        Handle: VB$_1891149551_24354_12   Media:

  List of Datafiles in backup set 24359

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   0  Incr 2013810    20/04/2020 23:51:59              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




RMAN>

 

Simulating the error

Just to remember that this new incremental have all scn from 2013573 until 2013810.
But by some reason one full backup it takes to another place (like disk, or duplicate for standby). Look that the channel type it is not ZDLRA:

 

RMAN> BACKUP INCREMENTAL LEVEL 0 DEVICE TYPE disk format '/tmp/%U' DATAFILE 12 TAG 'BKP-DBF-TO-DISK';




Starting backup at 20/04/2020 23:54:01

released channel: ORA_SBT_TAPE_1

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=65 device type=DISK

channel ORA_DISK_1: starting incremental level 0 datafile backup set

channel ORA_DISK_1: specifying datafile(s) in backup set

input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf

channel ORA_DISK_1: starting piece 1 at 20/04/2020 23:54:01

channel ORA_DISK_1: finished piece 1 at 20/04/2020 23:54:02

piece handle=/tmp/a3uu5e0p_1_1 tag=BKP-DBF-TO-DISK comment=NONE

channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01

Finished backup at 20/04/2020 23:54:02




Starting Control File and SPFILE Autobackup at 20/04/2020 23:54:02

piece handle=/u01/app/oracle/oradata/ORCL18C/autobackup/2020_04_20/o1_mf_s_1038268443_h9w6hwht_.bkp comment=NONE

Finished Control File and SPFILE Autobackup at 20/04/2020 23:54:10




RMAN> list backup tag = 'BKP-DBF-TO-DISK';







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24391   Incr 0  1.18M      DISK        00:00:00     20/04/2020 23:54:01

        BP Key: 24394   Status: AVAILABLE  Compressed: NO  Tag: BKP-DBF-TO-DISK

        Piece Name: /tmp/a3uu5e0p_1_1

  List of Datafiles in backup set 24391

  Container ID: 3, PDB Name: ORCL18P

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   0  Incr 2013972    20/04/2020 23:54:01              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




RMAN>

 

Now, if I try to execute the incremental level 1, the ingested backup does not generate a new virtual backup:

 

RMAN> BACKUP INCREMENTAL LEVEL 1 DEVICE TYPE SBT FILESPERSET 1 DATAFILE 12;




Starting backup at 20/04/2020 23:55:23

released channel: ORA_DISK_1

allocated channel: ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: SID=65 device type=SBT_TAPE

channel ORA_SBT_TAPE_1: RA Library (ZDLRAS1) SID=A3C0F4C16DAA11FAE053010310ACC1C4

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 20/04/2020 23:55:24

channel ORA_SBT_TAPE_1: finished piece 1 at 20/04/2020 23:55:27

piece handle=ORCL18C_a5uu5e3c_1_1 tag=TAG20200420T235524 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03

Finished backup at 20/04/2020 23:55:27




Starting Control File and SPFILE Autobackup at 20/04/2020 23:55:27

piece handle=c-558466555-20200420-0d comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 20/04/2020 23:55:35




RMAN> list backup of datafile 12 completed after "sysdate - 20/1440";







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24283   Incr 1  40.00K     SBT_TAPE    00:00:04     20/04/2020 23:49:26

        BP Key: 24284   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T234921

        Handle: VB$_1891149551_24282I   Media:

  List of Datafiles in backup set 24283

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2013573    20/04/2020 23:49:22              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24287   Incr 0  40.00K     SBT_TAPE    00:00:04     20/04/2020 23:49:26

        BP Key: 24288   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T234921

        Handle: VB$_1891149551_24282_12   Media:

  List of Datafiles in backup set 24287

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   0  Incr 2013573    20/04/2020 23:49:22              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24355   Incr 1  40.00K     SBT_TAPE    00:00:03     20/04/2020 23:52:02

        BP Key: 24356   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T235158

        Handle: VB$_1891149551_24354I   Media:

  List of Datafiles in backup set 24355

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2013810    20/04/2020 23:51:59              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24359   Incr 0  40.00K     SBT_TAPE    00:00:03     20/04/2020 23:52:02

        BP Key: 24360   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T235158

        Handle: VB$_1891149551_24354_12   Media:

  List of Datafiles in backup set 24359

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   0  Incr 2013810    20/04/2020 23:51:59              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24391   Incr 0  1.18M      DISK        00:00:00     20/04/2020 23:54:01

        BP Key: 24394   Status: AVAILABLE  Compressed: NO  Tag: BKP-DBF-TO-DISK

        Piece Name: /tmp/a3uu5e0p_1_1

  List of Datafiles in backup set 24391

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   0  Incr 2013972    20/04/2020 23:54:01              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24430   Incr 1  256.00K    SBT_TAPE    00:00:00     20/04/2020 23:55:24

        BP Key: 24431   Status: AVAILABLE  Compressed: NO  Tag: TAG20200420T235524

        Handle: ORCL18C_a5uu5e3c_1_1   Media: Recovery Appliance (ZDLRAS1)

  List of Datafiles in backup set 24430

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2014089    20/04/2020 23:55:24              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




RMAN>

 

Look above that the last incremental backup not generated a new level 0, it is the backupset 24430 (and backuppiece 24431).

 

Task inside ZDLRA

If we enter inside of ZDLRA, we can see that the task responsible to index the backupiece 23286 is in state ORDEING_WAIT. You can use query over ra_task and check using the state column..

 

SQL> select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, ERROR_COUNT, INTERRUPT_COUNT, BP_KEY,BS_KEY,DF_KEY,VB_KEY from rasys.ra_task where db_unique_name = 'ORCL18C' and state = 'ORDERING_WAIT' order by 5,2,7,10,11,12,13;




   TASK_ID TASK_TYPE       STATE                     WAITING_ON     DB_KEY DB_UNIQUE_NAME                 CREATION_TIME                       ERROR_COUNT INTERRUPT_COUNT     BP_KEY     BS_KEY     DF_KEY     VB_KEY

---------- --------------- ------------------------- ---------- ---------- ------------------------------ ----------------------------------- ----------- --------------- ---------- ---------- ---------- ----------

     40203 INDEX_BACKUP    ORDERING_WAIT                             15993 ORCL18C                        20-APR-20 11.55.26.779191 PM +02:00           0               1      24431




SQL>

 

And there is no incident for this error or state:

 

SQL> select incident_id, error_text last_seen from ra_incident_log where db_unique_name = 'ORCL18C' and status not in ('FIXED', 'RESET') order by last_seen desc;




no rows selected




SQL>

 

Unfortunately, this occurs for ZDLRA, even if there is a task in the ORDEING_WAIT state, it is not reported as an error. And if you think about, the virtual full backup it is not generated and the feature that it is the virtual full backup is not in place for these datafiles.
But even with tasks in this state, you will not be unprotected, the backup is there and can be restored.

 

ZDLRA Internals

 

If we check inside of ZDLRA we can check more details (this is one example how to navigate through internal tables of ZDLRA rman catalog and find the information – same that you find with list backupset inside rman):

 

SQL> select bs_key, db_key, pdb_key from bp where bp_key = 24431;




    BS_KEY     DB_KEY    PDB_KEY

---------- ---------- ----------

     24430      15993      16000




SQL> select * from rc_database where db_key = 15993;




    DB_KEY  DBINC_KEY       DBID NAME     RESETLOGS_CHANGE# RESETLOGS FINAL_CHANGE#

---------- ---------- ---------- -------- ----------------- --------- -------------

     15993      15994  558466555 ORCL18C            1477662 11-AUG-19




SQL> select df_key from df where dbinc_key = 15994 and  file# = 12;




    DF_KEY

----------

     16026




SQL> select bdf_key, ckp_scn from bdf where bs_key = 24430;




   BDF_KEY    CKP_SCN

---------- ----------

     24432    2014089




SQL>

###################################################################

RMAN> list backupset 24430;







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24430   Incr 1  256.00K    SBT_TAPE    00:00:00     20/04/2020 23:55:24

        BP Key: 24431   Status: AVAILABLE  Compressed: NO  Tag: TAG20200420T235524

        Handle: ORCL18C_a5uu5e3c_1_1   Media: Recovery Appliance (ZDLRAS1)

  List of Datafiles in backup set 24430

  Container ID: 3, PDB Name: ORCL18P

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2014089    20/04/2020 23:55:24              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




RMAN>

 

Using the BP table we can find the DB_KEY, and with that, we can go to RC_DATABASE to get the incarnation key (DBINC_KEY). With that, at DF table we can discover the DF_KEY for the datafile 12 and for this database incarnation. And with the DBF table, we can find the CKP_SCN for this datafile.
But if we go to the BLOCKS tables (ZDLRA table that stores the index – and “more or less” the virtual full), there are no database blocks with the SCN 2014089 for the backupset 24430.  If you want to understand who it works, you can read my post about this.

 

SQL> select * from blocks where df_key = 16026 and scn >= 2014089;




no rows selected




SQL>

 

And if we check for SCN 2013972 (there came from backup was put in disk), nothing too (as expected):

 

SQL> select * from blocks where df_key = 16026 and scn >= 2013972;




no rows selected




SQL>

 

As if we check with the last SCN 2013810 that are know by ZDLRA (last virtual full backup, backupset 24359), we can see which blocks are there:

 

SQL> select * from blocks where df_key = 16026 and scn >= 2013810 order by scn, chunkno ;




    DF_KEY    BLOCKNO        SCN     CKP_ID    CHUNKNO    COFFSET       USED  DBINC_KEY     ENDBLK

---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

     16026          1    2013810    2013810      25601      33121        338      15994

     16026          0    2013810    2013810      25601       8192      24576      15994




SQL>

 

And there is just PLANS for this virtual backup (nothing generated for the next existing backups):

 

SQL> select VB_KEY, DF_KEY, CKP_SCN, SRCBP_KEY, VCBP_KEY from vbdf where CKP_SCN >= 2013810 and df_key = 16026;




    VB_KEY     DF_KEY    CKP_SCN  SRCBP_KEY   VCBP_KEY

---------- ---------- ---------- ---------- ----------

     24354      16026    2013810      24293      24356




SQL> select * from plans_details where VB_KEY = 24354;




    DF_KEY       TYPE     VB_KEY    BLKRANK    BLOCKNO    CHUNKNO    NUMBLKS    COFFSET   NUMBYTES

---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

     16026          1      24354          1          0      25601          1       8192      24576

     16026          1      24354          1          2      16385          2      32788        256

     16026          1      24354          1          4      14337        132      33045      14000

     16026          1      24354          1        136      16385          3      33044        553

     16026          1      24354          1        139      23553          1      32788        327

     16026          1      24354          1        140      16385          2      33786        264

     16026          1      24354          1        142      25601          1      32788        333

     16026          1      24354          1        143      16385          1      34182        132

     16026          1      24354          1        191      14337          3      47045        252

     16026          1      24354          1 4294967295      25601          1      33121        338




10 rows selected.




SQL>

 

As you can figure out, ZDLRA can’t fill the gap to create the virtual full backup.

 

Recurring error

If you not solve the problem, and continue to ingest backups, the task will remain in ORDERING_WAIT:

 

RMAN> BACKUP INCREMENTAL LEVEL 1 DEVICE TYPE SBT FILESPERSET 1 DATAFILE 12;




Starting backup at 21/04/2020 00:04:00

using channel ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 21/04/2020 00:04:01

channel ORA_SBT_TAPE_1: finished piece 1 at 21/04/2020 00:04:04

piece handle=ORCL18C_a7uu5ejh_1_1 tag=TAG20200421T000400 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03

Finished backup at 21/04/2020 00:04:04




Starting Control File and SPFILE Autobackup at 21/04/2020 00:04:04

piece handle=c-558466555-20200421-00 comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 21/04/2020 00:04:13




RMAN> list backup of datafile 12 completed after "sysdate - 5/1440";







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24511   Incr 1  256.00K    SBT_TAPE    00:00:01     21/04/2020 00:04:02

        BP Key: 24512   Status: AVAILABLE  Compressed: NO  Tag: TAG20200421T000400

        Handle: ORCL18C_a7uu5ejh_1_1   Media: Recovery Appliance (ZDLRAS1)

  List of Datafiles in backup set 24511

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2021757    21/04/2020 00:04:01              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




RMAN>

####################################

SQL> select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, ERROR_COUNT, INTERRUPT_COUNT, BP_KEY,BS_KEY,DF_KEY,VB_KEY from rasys.ra_task where db_unique_name = 'ORCL18C' and state = 'ORDERING_WAIT' order by 5,2,7,10,11,12,13;




   TASK_ID TASK_TYPE       STATE                     WAITING_ON     DB_KEY DB_UNIQUE_NAME                 CREATION_TIME                       ERROR_COUNT INTERRUPT_COUNT     BP_KEY     BS_KEY     DF_KEY     VB_KEY

---------- --------------- ------------------------- ---------- ---------- ------------------------------ ----------------------------------- ----------- --------------- ---------- ---------- ---------- ----------

     40203 INDEX_BACKUP    ORDERING_WAIT                             15993 ORCL18C                        20-APR-20 11.55.26.779191 PM +02:00           0               1      24431

     40210 INDEX_BACKUP    ORDERING_WAIT                             15993 ORCL18C                        21-APR-20 12.04.05.207342 AM +02:00           0               1      24512




SQL>

 

Even if you try to do a cumulative incremental backup, the problem will be the same:

 

RMAN> BACKUP CUMULATIVE INCREMENTAL LEVEL 1 DEVICE TYPE SBT FILESPERSET 1 DATAFILE 12;




Starting backup at 21/04/2020 00:07:13

using channel ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 21/04/2020 00:07:13

channel ORA_SBT_TAPE_1: finished piece 1 at 21/04/2020 00:07:20

piece handle=ORCL18C_a9uu5eph_1_1 tag=TAG20200421T000713 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:07

Finished backup at 21/04/2020 00:07:20




Starting Control File and SPFILE Autobackup at 21/04/2020 00:07:20

piece handle=c-558466555-20200421-01 comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 21/04/2020 00:07:28




RMAN> list backup of datafile 12 completed after "sysdate - 2/1440";







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24604   Incr 1  256.00K    SBT_TAPE    00:00:02     21/04/2020 00:07:15

        BP Key: 24605   Status: AVAILABLE  Compressed: NO  Tag: TAG20200421T000713

        Handle: ORCL18C_a9uu5eph_1_1   Media: Recovery Appliance (ZDLRAS1)

  List of Datafiles in backup set 24604

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2021959    21/04/2020 00:07:13              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




RMAN>

#############################################

SQL> select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, ERROR_COUNT, INTERRUPT_COUNT, BP_KEY,BS_KEY,DF_KEY,VB_KEY from rasys.ra_task where db_unique_name = 'ORCL18C' and state = 'ORDERING_WAIT' order by 5,2,7,10,11,12,13;




   TASK_ID TASK_TYPE       STATE                     WAITING_ON     DB_KEY DB_UNIQUE_NAME                 CREATION_TIME                       ERROR_COUNT INTERRUPT_COUNT     BP_KEY     BS_KEY     DF_KEY     VB_KEY

---------- --------------- ------------------------- ---------- ---------- ------------------------------ ----------------------------------- ----------- --------------- ---------- ---------- ---------- ----------

     40203 INDEX_BACKUP    ORDERING_WAIT                             15993 ORCL18C                        20-APR-20 11.55.26.779191 PM +02:00           0               1      24431

     40210 INDEX_BACKUP    ORDERING_WAIT                             15993 ORCL18C                        21-APR-20 12.04.05.207342 AM +02:00           0               1      24512

     40215 INDEX_BACKUP    ORDERING_WAIT                             15993 ORCL18C                        21-APR-20 12.07.18.140618 AM +02:00           0               1      24605




SQL>

 

Solving ORDERING_WAIT

There are two ways to solve the issue, for both the idea is the same: ingest the database blocks inside of ZDLRA. And we do this performing backup.

 

FOR RECOVER OF TAG

The first is to use the command “BACKUP [CUMULATIVE] INCREMENTAL LEVEL 1 … FOR RECOVER OF TAG ‘<TAG>’”. The idea here is to create one incremental backup that recovers since one specific tag. You can check the documentation for more details if you want, it exists since Oracle 10g.
As you can imagine, the critical point here is to define the correct TAG do be used as a reference. And in this case, the tag is the last full backup (virtual or no) that it is inside of ZDLRA. Doing this, we ingest all changed blocks and fill the gap that is holding the task.
In this case, I used normal incremental. Look that the tag is from the last virtual full backup that is inside of ZDLRA for this datafile:

 

RMAN> BACKUP INCREMENTAL LEVEL 1 DEVICE TYPE SBT FOR RECOVER OF TAG 'TAG20200420T235158' DATAFILE 12;




Starting backup at 21/04/2020 00:12:17

using channel ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 21/04/2020 00:12:17

channel ORA_SBT_TAPE_1: finished piece 1 at 21/04/2020 00:12:20

piece handle=ORCL18C_abuu5f31_1_1 tag=TAG20200420T235158 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03

Finished backup at 21/04/2020 00:12:20




Starting Control File and SPFILE Autobackup at 21/04/2020 00:12:20

piece handle=c-558466555-20200421-02 comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 21/04/2020 00:12:28




RMAN>

 

And we can see that a new virtual full backup was created with this incremental backup (look the last 5 backupsets):

 

RMAN> list backup of datafile 12 completed after "sysdate - 40/1440";







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24283   Incr 1  40.00K     SBT_TAPE    00:00:04     20/04/2020 23:49:26

        BP Key: 24284   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T234921

        Handle: VB$_1891149551_24282I   Media:

  List of Datafiles in backup set 24283

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2013573    20/04/2020 23:49:22              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24287   Incr 0  40.00K     SBT_TAPE    00:00:04     20/04/2020 23:49:26

        BP Key: 24288   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T234921

        Handle: VB$_1891149551_24282_12   Media:

  List of Datafiles in backup set 24287

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   0  Incr 2013573    20/04/2020 23:49:22              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24355   Incr 1  40.00K     SBT_TAPE    00:00:03     20/04/2020 23:52:02

        BP Key: 24356   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T235158

        Handle: VB$_1891149551_24354I   Media:

  List of Datafiles in backup set 24355

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2013810    20/04/2020 23:51:59              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24359   Incr 0  40.00K     SBT_TAPE    00:00:03     20/04/2020 23:52:02

        BP Key: 24360   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T235158

        Handle: VB$_1891149551_24354_12   Media:

  List of Datafiles in backup set 24359

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   0  Incr 2013810    20/04/2020 23:51:59              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24391   Incr 0  1.18M      DISK        00:00:00     20/04/2020 23:54:01

        BP Key: 24394   Status: AVAILABLE  Compressed: NO  Tag: BKP-DBF-TO-DISK

        Piece Name: /tmp/a3uu5e0p_1_1

  List of Datafiles in backup set 24391

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   0  Incr 2013972    20/04/2020 23:54:01              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24430   Incr 1  256.00K    SBT_TAPE    00:00:00     20/04/2020 23:55:24

        BP Key: 24431   Status: AVAILABLE  Compressed: NO  Tag: TAG20200420T235524

        Handle: ORCL18C_a5uu5e3c_1_1   Media: Recovery Appliance (ZDLRAS1)

  List of Datafiles in backup set 24430

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2014089    20/04/2020 23:55:24              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24511   Incr 1  256.00K    SBT_TAPE    00:00:01     21/04/2020 00:04:02

        BP Key: 24512   Status: AVAILABLE  Compressed: NO  Tag: TAG20200421T000400

        Handle: ORCL18C_a7uu5ejh_1_1   Media: Recovery Appliance (ZDLRAS1)

  List of Datafiles in backup set 24511

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2021757    21/04/2020 00:04:01              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24604   Incr 1  256.00K    SBT_TAPE    00:00:02     21/04/2020 00:07:15

        BP Key: 24605   Status: AVAILABLE  Compressed: NO  Tag: TAG20200421T000713

        Handle: ORCL18C_a9uu5eph_1_1   Media: Recovery Appliance (ZDLRAS1)

  List of Datafiles in backup set 24604

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2021959    21/04/2020 00:07:13              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24814   Incr 1  40.00K     SBT_TAPE    00:00:03     21/04/2020 00:12:20

        BP Key: 24815   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T235158

        Handle: VB$_1891149551_24813I   Media:

  List of Datafiles in backup set 24814

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   1  Incr 2025613    21/04/2020 00:12:17              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

24818   Incr 0  40.00K     SBT_TAPE    00:00:03     21/04/2020 00:12:20

        BP Key: 24819   Status: AVAILABLE  Compressed: YES  Tag: TAG20200420T235158

        Handle: VB$_1891149551_24813_12   Media:

  List of Datafiles in backup set 24818

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  12   0  Incr 2025613    21/04/2020 00:12:17              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf




RMAN>

 

And inside of ZDLRA, we can see that task previously in ORDERING_WAIT finished (check the BP_KEY column):

 

SQL> select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, COMPLETION_TIME, ERROR_COUNT, INTERRUPT_COUNT, BP_KEY,BS_KEY,DF_KEY,VB_KEY from rasys.ra_task where task_id IN (40203,40210,40215);




   TASK_ID TASK_TYPE       STATE                     WAITING_ON     DB_KEY DB_UNIQUE_NAME                 CREATION_TIME                       COMPLETION_TIME                     ERROR_COUNT INTERRUPT_COUNT     BP_KEY     BS_KEY     DF_KEY     VB_KEY

---------- --------------- ------------------------- ---------- ---------- ------------------------------ ----------------------------------- ----------------------------------- ----------- --------------- ---------- ---------- ---------- ----------

     40203 INDEX_BACKUP    COMPLETED                                 15993 ORCL18C                        20-APR-20 11.55.26.779191 PM +02:00 21-APR-20 12.13.09.253916 AM +02:00           0               1      24431

     40210 INDEX_BACKUP    COMPLETED                                 15993 ORCL18C                        21-APR-20 12.04.05.207342 AM +02:00 21-APR-20 12.13.21.663869 AM +02:00           0               1      24512

     40215 INDEX_BACKUP    COMPLETED                                 15993 ORCL18C                        21-APR-20 12.07.18.140618 AM +02:00 21-APR-20 12.13.42.179087 AM +02:00           0               1      24605




SQL>

 

And if we check for the BLOCKS for this datafile, we can see that was registered new that are higher with the last full backup before the error, and they go until the last backup made

 

SQL> select * from blocks where df_key = 16026 and scn >= 2013810 order by scn, chunkno ;




    DF_KEY    BLOCKNO        SCN     CKP_ID    CHUNKNO    COFFSET       USED  DBINC_KEY     ENDBLK

---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

     16026          0    2013810    2013810      25601       8192      24576      15994

     16026          1    2013810    2013810      25601      33121        338      15994

     16026        142    2021942    2025613      26625      32788        414      15994

     16026          1    2025613    2025613      26625      33202        336      15994

     16026          0    2025613    2025613      26625       8192      24576      15994




SQL>

 

And we can see that now exists PLANS for the backupset of virtual full backup that exists before the error (SCN 2013810) and after we fix (SCN 2025613):

 

SQL> select VB_KEY, DF_KEY, CKP_SCN, SRCBP_KEY, VCBP_KEY from vbdf where CKP_SCN >= 2013810 and DF_KEY = 16026;




    VB_KEY     DF_KEY    CKP_SCN  SRCBP_KEY   VCBP_KEY

---------- ---------- ---------- ---------- ----------

     24354      16026    2013810      24293      24356

     24813      16026    2025613      24706      24815




SQL>

SQL> select * from plans_details where VB_KEY IN (24354,24813) order by VB_KEY,BLOCKNO;




    DF_KEY       TYPE     VB_KEY    BLKRANK    BLOCKNO    CHUNKNO    NUMBLKS    COFFSET   NUMBYTES

---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

     16026          1      24354          1          0      25601          1       8192      24576

     16026          1      24354          1          2      16385          2      32788        256

     16026          1      24354          1          4      14337        132      33045      14000

     16026          1      24354          1        136      16385          3      33044        553

     16026          1      24354          1        139      23553          1      32788        327

     16026          1      24354          1        140      16385          2      33786        264

     16026          1      24354          1        142      25601          1      32788        333

     16026          1      24354          1        143      16385          1      34182        132

     16026          1      24354          1        191      14337          3      47045        252

     16026          1      24354          1 4294967295      25601          1      33121        338

     16026          1      24813          1          0      26625          1       8192      24576

     16026          1      24813          1          2      16385          2      32788        256

     16026          1      24813          1          4      14337        132      33045      14000

     16026          1      24813          1        136      16385          3      33044        553

     16026          1      24813          1        139      23553          1      32788        327

     16026          1      24813          1        140      16385          2      33786        264

     16026          1      24813          1        142      26625          1      32788        414

     16026          1      24813          1        143      16385          1      34182        132

     16026          1      24813          1        191      14337          3      47045        252

     16026          1      24813          1 4294967295      26625          1      33202        336




20 rows selected.




SQL>

 

So, this means that the incremental backup that we made with FOR RECOVERY OF TAG was ingested and used to fix the needed gap.
And if we try to recover the datafile 12, we can do without a problem. Check that the used backup to recover was the last virtual full backup generated from the “RECOVERY OF TAG” command:

 

RMAN> run{

2> ALTER PLUGGABLE DATABASE ORCL18P CLOSE IMMEDIATE INSTANCES=ALL;

3> RESTORE DATAFILE 12;

4> RECOVER DATAFILE 12;

5> ALTER PLUGGABLE DATABASE ORCL18P OPEN INSTANCES=ALL;

6> }




Statement processed

starting full resync of recovery catalog

full resync complete




Starting restore at 21/04/2020 00:57:53

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=88 device type=DISK

allocated channel: ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: SID=70 device type=SBT_TAPE

channel ORA_SBT_TAPE_1: RA Library (ZDLRAS1) SID=A3C1D44670B428B0E053010310AC5DA9




channel ORA_SBT_TAPE_1: starting datafile backup set restore

channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set

channel ORA_SBT_TAPE_1: restoring datafile 00012 to /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf

channel ORA_SBT_TAPE_1: reading from backup piece VB$_1891149551_24813_12

channel ORA_SBT_TAPE_1: piece handle=VB$_1891149551_24813_12 tag=TAG20200420T235158

channel ORA_SBT_TAPE_1: restored backup piece 1

channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:00:25

Finished restore at 21/04/2020 00:58:21




Starting recover at 21/04/2020 00:58:21

using channel ORA_DISK_1

using channel ORA_SBT_TAPE_1




starting media recovery

media recovery complete, elapsed time: 00:00:00




Finished recover at 21/04/2020 00:58:22




Statement processed

starting full resync of recovery catalog

full resync complete




RMAN>

 

Unfortunately, backup tags for ZDLRA can be tricky when you directly specify it during the backup phase. They can be the same and the usage “FOR TAG” can be more difficult to define. One option is to merge and execute the command BACKUP CUMULATIVE INCREMENTAL LEVEL 1 DEVICE TYPE SBT FOR RECOVER OF TAG ‘<TAG>’ DATAFILE XX.
Doing this, the command will pick up all the blocks from the last full backup that have the tag that you defined. The result is the same:

 

RMAN> BACKUP CUMULATIVE INCREMENTAL LEVEL 1 DEVICE TYPE SBT FOR RECOVER OF TAG 'TAG20200419T232006' DATAFILE 12;




Starting backup at 19/04/2020 23:38:44

using channel ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 19/04/2020 23:38:44

channel ORA_SBT_TAPE_1: finished piece 1 at 19/04/2020 23:38:59

piece handle=ORCL18C_97uu2oo4_1_1 tag=TAG20200419T232006 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:15

Finished backup at 19/04/2020 23:38:59




Starting Control File and SPFILE Autobackup at 19/04/2020 23:39:00

piece handle=c-558466555-20200419-04 comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 19/04/2020 23:39:16




RMAN>

 

BACKUP FULL

The other option to solve the ORDERING_WAIT is to do a full backup of the datafile. With this, all the blocks are read and ingested at ZDLRA.
The procedure is the same as above and the result the same. The only point is that for huge files this can take a long time (since it is full) and the above incremental approach can be more suitable.

 

Monitoring the Tasks

 

So, as you can see above, the ORDERING_WAIT state can have a lot of collateral effects for you ZDLRA. Unfortunately, this not generates incidents that are reported, you need to write a query to check this directly at ra_task table.
Whatever the method that you choose to solve the problem (RECOVERY OF TAG, or FULL BACKUP) always verify if the new virtual full backup is generated. It is a good practice to do this to avoid errors and do a double cross-check over the tasks
It is a simple query, and a simple monitoring thing to do. But this will avoid a huge problem.
 
Reference:
 

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”


LuxOUG – Virtual Tech Days
Category: Database Author: Andre Luiz Dutra Ontalba (Board Member) Date: 5 years ago Comments: 0
First LuxOUG Virtual Tech Days event for the Oracle community.
 
We will hold our first online event, covering various technologies such as DevOps, Engineered System, Middleware, Cloud and others.
 
 Event Schedule:

Day 1 – 22/06

Speaker: Toon Koppelaars
Session:  Database Core Performance Principles.
Speaker: Piet Visser
Session: Partitioning – Positives and Pitfalls. (database, development) – this one will include an SQL demo.
Speaker: Bruno Reis
Session: Beginner-friendly Python for Oracle DBAs.
Speaker: Heli Helskyaho
Session: Introduction to Machine Learning.

Day 2 – 23/06

Speaker: Kamran Agayev
Session: From DBA to Data Engineer – How to survive a career transition.
Speaker: Nikitas Xenakis
Session: Building a Highly Available & Scalable Logistics Platform with Oracle 19c & Goldengate 19c.
Speaker: Franky Weber
Session: Cheating your application code with Oracle Database.
Speaker: Rodrigo Jorge
Session: Scanning Oracle Database for Malicious Changes.

Day 3 – 24/06

Speaker: Sandesh Rao 
Session: Introduction to AutoML and Data Science using the Oracle Autonomous Database.
Speaker: Robert Marz
Session: Oracle Cloud Infrastructure – Network Setup for DBAs.
Speaker: Alex Zaballa
Session: Exploring All Options to Move your Oracle Databases to the Oracle Cloud.
Speaker: Mariami Kupatadze
Session: Main components, memory structures, physical and logical structures and more.

Day 4 – 25/06

 
Speaker: Erik Van Roon 
Session:  Handling errors during bulk DML operations.
Speaker: Mohamed Houri 
Session:  Cursor Optimization under Adaptive and Extended Cursor Sharing.
Speaker: Y V Ravi Kumar
Session:  Oracle Sharding Technical Deep Dive.
Speaker: Lonneke Dikmans
Session: Oracle Blockchain Platform – a case study.
===================================================================================
For registration and participation in the virtual event room, please CLICK HERE
 
Enrollment open until 20/06. 
 
After this period the event will be broadcast on our Youtube channel – CLICK HERE
 
We have split the videos by speaker this is the playlist for LuxOUG Virtual Tech Days: CLICK HERE
====================================================================================
 
Presentations for download :
 
Speaker: Heli Helskyaho – PDF
Speaker: Mariami Kupatadze – PDF
Speaker: Toon Koppelaars – PDF
Speaker: Y V Ravi Kumar – PDF
Speaker: Rodrigo Jorge – PDF
Speaker: Erik Van Roon – PDF and Scripts 
Speaker: Alex Zaballa – PDF
Speaker: Bruno Reis – PDF
Speaker: Robert Marz – PDF
Speaker: Mohamed Houri  – PDF
Speaker: Sandesh Rao – PDF
Speaker: Lonneke Dikmans – PDF
Speaker: Piet Visser – PDF
Speaker: Franky Weber – PDF
 
 
 
 
See you at the event
 
 
LuxOUG Board

How to upgrade ODA Patch: 18.8.0.0 for Virtualized System
Category: Engineer System Author: Andre Luiz Dutra Ontalba (Board Member) Date: 5 years ago Comments: 0

How to upgrade ODA Patch: 18.8.0.0 for Virtualized System

Introduction

The goal of this document is to describe step by step how to upgrade ODA Virtualized System.

Prerequisites

For this Upgrade the ODA must be at version 18.3

Oracle Database Appliance Documentation (Check the last version of the patch)

Start Upgrade

1 – Backup of ODA_BASE (Both Nodes): It can take up to 2h00 hours.

From DOM0:
As root user:
Node 1:

oakcli stop oda_base


mkdir -p /backup/odax58duts1/odax58duts1_date +"%Y%m%d"


nohup tar -cvzf oakDom1.odax58duts1_dom0.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1 &

After complete the backup:

oakcli start oda_base

Node 2:

oakcli stop oda_base


mkdir -p /backup/odax58duts2/odax58duts2_date +"%Y%m%d"


nohup tar -cvzf oakDom1.odax58duts2_dom0.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1 &

After complete the backup:

oakcli start oda_base

2 – Download patch in a shared directory or separated directory in both ODA servers:

Os User: root

mkdir -p /backup/patchODA2020

Download all required files to this directory:

/backup/patchODA2020

There is 2 .zip files to be download patch (30518438):

p30518438_188000_Linux-x86-64_1of2.zip


p30518438_188000_Linux-x86-64_2of2.zip

Note: We must guarantee some minimum amount of free space (20GB) in the ODA BASE, file systems: “/” and “/u01“
– Purge old logs from ODA, with root user:

oakcli manage cleanrepo --ver 18.3.0.0.0


oakcli manage cleanrepo --ver 18.6.0.0.0


/usr/local/bin/purgeODALog -orcl 20 -tfa 10 -osw 10 -oak 10

– Clean up old patches from GRID_HOME, affects “/u01“:

su - grid


. oraenv


+ASM1 or +ASM2


cd $ORACLE_HOME/OPatch


./opatch util cleanup
– Clean up old patches from ORACLE_HOME, affects “/u01“:

su - oracle


. oraenv <SID>


cd $ORACLE_HOME/OPatch


./opatch util cleanup

Note:
– It must be performed this clean in every ORACLE_HOME versions that exists in the server.
– Look at “/home/oracle“, “/home/grid” and “/tmp” to perform some clean up and release some space in the file system “/“.

3 – Unpack downloaded patch in both ODA nodes:

cd /backup/patchODA2020


oakcli unpack -package p30518438_188000_Linux-x86-64_1of2.zip


oakcli unpack -package p30518438_188000_Linux-x86-64_2of2.zip

4 – Verify and Validate Upgrade ODA components (S.O):

export EXTRA_OS_RPMS_LOC=/backup/patchODA2020


oakcli validate -c ospatch -ver 18.8.0.0.0

Note:
Solving rpm’s conflict for version 18.8.0.0.0.

e.g: With errors and conflicts

BEGIN OUTPUT:
NODE 1:

[root@odax58duts1 patchODA2020]# oakcli validate -c ospatch -ver 18.8.0.0.0


INFO: Validating the OS patch for the version 18.8.0.0.0


INFO: 2020-06-06 14:06:16: Performing a dry run for OS patching


ERROR: 2020-06-06 14:06:31: Unable to run the command : /usr/bin/yum --exclude=kmod-mpt2sas,ibutils-libs,dapl,libcxgb3,libipathverbs,libmthca,libnes,ofed-docs update --disablerepo=* --enablerepo=ODA_REPOS_LOC -y


ERROR: 2020-06-06 14:06:31: Loaded plugins: rhnplugin, ulninfo, versionlock


This system is not registered with ULN.


You can use uln_register to register.


ULN support will be disabled.


Repository ol6_latest is listed more than once in the configuration


Repository ol6_addons is listed more than once in the configuration


Repository ol6_UEK_latest is listed more than once in the configuration


Setting up Update Process


Resolving Dependencies


--> Running transaction check


---> Package cpupowerutils.x86_64 0:1.3-2.el6 will be updated


---> Package cpupowerutils.x86_64 0:1.3-2.0.1.el6 will be an update


---> Package cups-libs.x86_64 1:1.4.2-79.el6 will be updated


---> Package cups-libs.x86_64 1:1.4.2-81.el6_10 will be an update


---> Package dbus.x86_64 1:1.2.24-9.0.1.el6 will be updated


---> Package dbus.x86_64 1:1.2.24-11.0.1.el6_10 will be an update


---> Package dbus-libs.x86_64 1:1.2.24-9.0.1.el6 will be updated


---> Package dbus-libs.x86_64 1:1.2.24-11.0.1.el6_10 will be an update


---> Package dracut.noarch 0:004-411.0.3.el6 will be updated


---> Package dracut.noarch 0:004-411.0.4.el6 will be an update


---> Package dracut-kernel.noarch 0:004-411.0.3.el6 will be updated


---> Package dracut-kernel.noarch 0:004-411.0.4.el6 will be an update


---> Package glibc.x86_64 0:2.12-1.212.0.2.el6 will be updated


---> Package glibc.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update


---> Package glibc-common.x86_64 0:2.12-1.212.0.2.el6 will be updated


---> Package glibc-common.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update


---> Package glibc-devel.i686 0:2.12-1.212.0.2.el6 will be updated


---> Package glibc-devel.x86_64 0:2.12-1.212.0.2.el6 will be updated


---> Package glibc-devel.i686 0:2.12-1.212.0.3.el6_10.3 will be an update


---> Package glibc-devel.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update


---> Package glibc-headers.x86_64 0:2.12-1.212.0.2.el6 will be updated


---> Package glibc-headers.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update


---> Package initscripts.x86_64 0:9.03.61-1.0.3.el6 will be updated


---> Package initscripts.x86_64 0:9.03.61-1.0.6.el6 will be an update


---> Package kernel-headers.x86_64 0:2.6.32-754.11.1.el6 will be updated


---> Package kernel-headers.x86_64 0:2.6.32-754.18.2.el6 will be an update


---> Package kernel-uek.x86_64 0:4.1.12-124.33.4.el6uek will be installed


---> Package kernel-uek-firmware.noarch 0:4.1.12-124.33.4.el6uek will be installed


---> Package kexec-tools.x86_64 0:2.0.7-1.0.27.el6 will be updated


---> Package kexec-tools.x86_64 0:2.0.7-1.0.28.el6 will be an update


---> Package ksplice.x86_64 0:1.0.38-1.el6 will be updated


---> Package ksplice.x86_64 0:1.0.43-1.el6 will be an update


---> Package ksplice-core0.x86_64 0:1.0.38-1.el6 will be updated


---> Package ksplice-core0.x86_64 0:1.0.43-1.el6 will be an update


---> Package ksplice-offline.x86_64 0:1.0.38-1.el6 will be updated


---> Package ksplice-offline.x86_64 0:1.0.43-1.el6 will be an update


---> Package ksplice-tools.x86_64 0:1.0.38-1.el6 will be updated


---> Package ksplice-tools.x86_64 0:1.0.43-1.el6 will be an update


---> Package libgudev1.x86_64 0:147-2.73.0.1.el6_8.2 will be updated


---> Package libgudev1.x86_64 0:147-2.73.0.2.el6_8.2 will be an update


---> Package libudev.x86_64 0:147-2.73.0.1.el6_8.2 will be updated


---> Package libudev.x86_64 0:147-2.73.0.2.el6_8.2 will be an update


---> Package mailx.x86_64 0:12.4-8.el6_6 will be updated


---> Package mailx.x86_64 0:12.4-10.el6_10 will be an update


---> Package openssl.x86_64 0:1.0.1e-57.0.6.el6 will be updated


---> Package openssl.x86_64 0:1.0.1e-58.0.1.el6_10 will be an update


---> Package oracle-ofed-release.x86_64 0:1.0.0-50.el6 will be updated


---> Package oracle-ofed-release.x86_64 0:1.0.0-51.el6 will be an update


---> Package perf.x86_64 0:2.6.32-754.11.1.el6 will be updated


---> Package perf.x86_64 0:2.6.32-754.18.2.el6 will be an update


---> Package python.x86_64 0:2.6.6-66.0.1.el6_8 will be updated


---> Package python.x86_64 0:2.6.6-68.0.1.el6_10 will be an update


---> Package python-libs.x86_64 0:2.6.6-66.0.1.el6_8 will be updated


---> Package python-libs.x86_64 0:2.6.6-68.0.1.el6_10 will be an update


---> Package rdma.noarch 2:3.10-3.0.40.el6 will be updated


---> Package rdma.noarch 2:3.10-3.0.41.el6 will be an update


---> Package samba.x86_64 0:3.6.23-51.0.1.el6 will be updated


---> Package samba.x86_64 0:3.6.23-52.0.1.el6_10 will be an update


---> Package samba-client.x86_64 0:3.6.23-51.0.1.el6 will be updated


---> Package samba-client.x86_64 0:3.6.23-52.0.1.el6_10 will be an update


---> Package samba-common.x86_64 0:3.6.23-51.0.1.el6 will be updated


---> Package samba-common.x86_64 0:3.6.23-52.0.1.el6_10 will be an update


---> Package samba-winbind.x86_64 0:3.6.23-51.0.1.el6 will be updated


---> Package samba-winbind.x86_64 0:3.6.23-52.0.1.el6_10 will be an update


---> Package samba-winbind-clients.x86_64 0:3.6.23-51.0.1.el6 will be updated


---> Package samba-winbind-clients.x86_64 0:3.6.23-52.0.1.el6_10 will be an update


---> Package sudo.x86_64 0:1.8.6p3-29.el6_9 will be updated


---> Package sudo.x86_64 0:1.8.6p3-29.0.1.el6_10.2 will be an update


---> Package tzdata.noarch 0:2019a-1.el6 will be updated


---> Package tzdata.noarch 0:2019c-1.el6 will be an update


---> Package tzdata-java.noarch 0:2018e-3.el6 will be updated


---> Package tzdata-java.noarch 0:2019c-1.el6 will be an update


---> Package udev.x86_64 0:147-2.73.0.1.el6_8.2 will be updated


---> Package udev.x86_64 0:147-2.73.0.2.el6_8.2 will be an update


---> Package vim-common.x86_64 2:7.4.629-5.el6_8.1 will be updated


---> Package vim-common.x86_64 2:7.4.629-5.el6_10.2 will be an update


---> Package vim-enhanced.x86_64 2:7.4.629-5.el6_8.1 will be updated


---> Package vim-enhanced.x86_64 2:7.4.629-5.el6_10.2 will be an update


---> Package vim-minimal.x86_64 2:7.4.629-5.el6_8.1 will be updated


---> Package vim-minimal.x86_64 2:7.4.629-5.el6_10.2 will be an update


---> Package xorg-x11-server-Xorg.x86_64 0:1.17.4-17.0.1.el6 will be updated


---> Package xorg-x11-server-Xorg.x86_64 0:1.17.4-17.0.2.el6 will be an update


---> Package xorg-x11-server-common.x86_64 0:1.17.4-17.0.1.el6 will be updated


---> Package xorg-x11-server-common.x86_64 0:1.17.4-17.0.2.el6 will be an update


--> Finished Dependency Resolution


Dependencies Resolved


================================================================================


 Package                 Arch    Version                   Repository      Size


================================================================================


Installing:


 kernel-uek              x86_64  4.1.12-124.33.4.el6uek    ODA_REPOS_LOC   42 M


 kernel-uek-firmware     noarch  4.1.12-124.33.4.el6uek    ODA_REPOS_LOC  2.6 M


Updating:


 cpupowerutils           x86_64  1.3-2.0.1.el6             ODA_REPOS_LOC   77 k


 cups-libs               x86_64  1:1.4.2-81.el6_10         ODA_REPOS_LOC  322 k


 dbus                    x86_64  1:1.2.24-11.0.1.el6_10    ODA_REPOS_LOC  211 k


 dbus-libs               x86_64  1:1.2.24-11.0.1.el6_10    ODA_REPOS_LOC  127 k


 dracut                  noarch  004-411.0.4.el6           ODA_REPOS_LOC  129 k


 dracut-kernel           noarch  004-411.0.4.el6           ODA_REPOS_LOC   29 k


 glibc                   x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  3.8 M


 glibc-common            x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC   14 M


 glibc-devel             i686    2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  992 k


 glibc-devel             x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  991 k


 glibc-headers           x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  620 k


 initscripts             x86_64  9.03.61-1.0.6.el6         ODA_REPOS_LOC  952 k


 kernel-headers          x86_64  2.6.32-754.18.2.el6       ODA_REPOS_LOC  4.6 M


 kexec-tools             x86_64  2.0.7-1.0.28.el6          ODA_REPOS_LOC  339 k


 ksplice                 x86_64  1.0.43-1.el6              ODA_REPOS_LOC  9.1 k


 ksplice-core0           x86_64  1.0.43-1.el6              ODA_REPOS_LOC  271 k


 ksplice-offline         x86_64  1.0.43-1.el6              ODA_REPOS_LOC  7.9 k


 ksplice-tools           x86_64  1.0.43-1.el6              ODA_REPOS_LOC   92 k


 libgudev1               x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC   65 k


 libudev                 x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC   78 k


 mailx                   x86_64  12.4-10.el6_10            ODA_REPOS_LOC  235 k


 openssl                 x86_64  1.0.1e-58.0.1.el6_10      ODA_REPOS_LOC  1.5 M


 oracle-ofed-release     x86_64  1.0.0-51.el6              ODA_REPOS_LOC   16 k


 perf                    x86_64  2.6.32-754.18.2.el6       ODA_REPOS_LOC  4.8 M


 python                  x86_64  2.6.6-68.0.1.el6_10       ODA_REPOS_LOC   76 k


 python-libs             x86_64  2.6.6-68.0.1.el6_10       ODA_REPOS_LOC  5.3 M


 rdma                    noarch  2:3.10-3.0.41.el6         ODA_REPOS_LOC   76 k


 samba                   x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  5.1 M


 samba-client            x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC   11 M


 samba-common            x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC   10 M


 samba-winbind           x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  2.2 M


 samba-winbind-clients   x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  2.0 M


 sudo                    x86_64  1.8.6p3-29.0.1.el6_10.2   ODA_REPOS_LOC  712 k


 tzdata                  noarch  2019c-1.el6               ODA_REPOS_LOC  507 k


 tzdata-java             noarch  2019c-1.el6               ODA_REPOS_LOC  188 k


 udev                    x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC  360 k


 vim-common              x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  6.7 M


 vim-enhanced            x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  1.0 M


 vim-minimal             x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  421 k


 xorg-x11-server-Xorg    x86_64  1.17.4-17.0.2.el6         ODA_REPOS_LOC  1.4 M


 xorg-x11-server-common  x86_64  1.17.4-17.0.2.el6         ODA_REPOS_LOC   51 k


Transaction Summary


================================================================================


Install       2 Package(s)


Upgrade      41 Package(s)


Total download size: 126 M


Downloading Packages:


--------------------------------------------------------------------------------


Total                                           190 MB/s | 126 MB     00:00


Running rpm_check_debug


Running Transaction Test


Transaction Check Error:


  file /usr/bin/ldd from install of glibc-common-2.12-1.212.0.3.el6_10.3.x86_64 conflicts with file from package glibc-common-2.12-1.212.0.3.el6_10.3.i686


  file /usr/lib/locale/locale-archive.tmpl from install of glibc-common-2.12-1.212.0.3.el6_10.3.x86_64 conflicts with file from package glibc-common-2.12-1.212.0.3.el6_10.3.i686


Error Summary


-------------


WARNING: 2020-06-06 14:06:31: OS Upgrade is not successful. Need to resolve conflicts


INFO: 2020-06-06 14:06:31: Copy the required RPMs to a location and set EXTRA_OS_RPMS_LOC to that location

Here we need to solve the dependency problem, in this case we will remove the package

[root@odax58duts1 patchODA2020]# rpm -e glibc-common-2.12-1.212.0.3.el6_10.3.i686 --nodeps


You have new mail in /var/spool/mail/root


[root@odax58duts1 patchODA2020]# oakcli validate -c ospatch -ver 18.8.0.0.0


INFO: Validating the OS patch for the version 18.8.0.0.0


INFO: 2020-06-06 14:09:51: Performing a dry run for OS patching


INFO: 2020-06-06 14:10:09: No conflict detected during the OS update, dry run check.

NODE 2:

[root@odax58duts2 patchODA2020]# oakcli validate -c ospatch -ver 18.8.0.0.0


INFO: Validating the OS patch for the version 18.8.0.0.0


INFO: 2020-06-06 14:15:59: Performing a dry run for OS patching


ERROR: 2020-06-06 14:16:18: Unable to run the command : /usr/bin/yum --exclude=kmod-mpt2sas,ibutils-libs,dapl,libcxgb3,libipathverbs,libmthca,libnes,ofed-docs update --disablerepo=* --enablerepo=ODA_REPOS_LOC -y


ERROR: 2020-06-06 14:16:18: Loaded plugins: rhnplugin, ulninfo, versionlock


This system is not registered with ULN.


You can use uln_register to register.


ULN support will be disabled.


Repository ol6_latest is listed more than once in the configuration


Repository ol6_addons is listed more than once in the configuration


Repository ol6_UEK_latest is listed more than once in the configuration


Setting up Update Process


Resolving Dependencies


--> Running transaction check


---> Package cpupowerutils.x86_64 0:1.3-2.el6 will be updated


---> Package cpupowerutils.x86_64 0:1.3-2.0.1.el6 will be an update


---> Package cups-libs.x86_64 1:1.4.2-79.el6 will be updated


---> Package cups-libs.x86_64 1:1.4.2-81.el6_10 will be an update


---> Package dbus.x86_64 1:1.2.24-9.0.1.el6 will be updated


---> Package dbus.x86_64 1:1.2.24-11.0.1.el6_10 will be an update


---> Package dbus-libs.x86_64 1:1.2.24-9.0.1.el6 will be updated


---> Package dbus-libs.x86_64 1:1.2.24-11.0.1.el6_10 will be an update


---> Package dracut.noarch 0:004-411.0.3.el6 will be updated


---> Package dracut.noarch 0:004-411.0.4.el6 will be an update


---> Package dracut-kernel.noarch 0:004-411.0.3.el6 will be updated


---> Package dracut-kernel.noarch 0:004-411.0.4.el6 will be an update


---> Package glibc.x86_64 0:2.12-1.212.0.2.el6 will be updated


---> Package glibc.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update


---> Package glibc-devel.i686 0:2.12-1.212.0.2.el6 will be updated


---> Package glibc-devel.x86_64 0:2.12-1.212.0.2.el6 will be updated


---> Package glibc-devel.i686 0:2.12-1.212.0.3.el6_10.3 will be an update


---> Package glibc-devel.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update


---> Package glibc-headers.x86_64 0:2.12-1.212.0.2.el6 will be updated


---> Package glibc-headers.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update


---> Package initscripts.x86_64 0:9.03.61-1.0.3.el6 will be updated


---> Package initscripts.x86_64 0:9.03.61-1.0.6.el6 will be an update


---> Package kernel-headers.x86_64 0:2.6.32-754.11.1.el6 will be updated


---> Package kernel-headers.x86_64 0:2.6.32-754.18.2.el6 will be an update


---> Package kernel-uek.x86_64 0:4.1.12-124.33.4.el6uek will be installed


---> Package kernel-uek-firmware.noarch 0:4.1.12-124.33.4.el6uek will be installed


---> Package kexec-tools.x86_64 0:2.0.7-1.0.27.el6 will be updated


---> Package kexec-tools.x86_64 0:2.0.7-1.0.28.el6 will be an update


---> Package ksplice.x86_64 0:1.0.38-1.el6 will be updated


---> Package ksplice.x86_64 0:1.0.43-1.el6 will be an update


---> Package ksplice-core0.x86_64 0:1.0.38-1.el6 will be updated


---> Package ksplice-core0.x86_64 0:1.0.43-1.el6 will be an update


---> Package ksplice-offline.x86_64 0:1.0.38-1.el6 will be updated


---> Package ksplice-offline.x86_64 0:1.0.43-1.el6 will be an update


---> Package ksplice-tools.x86_64 0:1.0.38-1.el6 will be updated


---> Package ksplice-tools.x86_64 0:1.0.43-1.el6 will be an update


---> Package libgudev1.x86_64 0:147-2.73.0.1.el6_8.2 will be updated


---> Package libgudev1.x86_64 0:147-2.73.0.2.el6_8.2 will be an update


---> Package libudev.x86_64 0:147-2.73.0.1.el6_8.2 will be updated


---> Package libudev.x86_64 0:147-2.73.0.2.el6_8.2 will be an update


---> Package mailx.x86_64 0:12.4-8.el6_6 will be updated


---> Package mailx.x86_64 0:12.4-10.el6_10 will be an update


---> Package openssl.x86_64 0:1.0.1e-57.0.6.el6 will be updated


---> Package openssl.x86_64 0:1.0.1e-58.0.1.el6_10 will be an update


---> Package oracle-ofed-release.x86_64 0:1.0.0-50.el6 will be updated


---> Package oracle-ofed-release.x86_64 0:1.0.0-51.el6 will be an update


---> Package perf.x86_64 0:2.6.32-754.11.1.el6 will be updated


---> Package perf.x86_64 0:2.6.32-754.18.2.el6 will be an update


---> Package python.x86_64 0:2.6.6-66.0.1.el6_8 will be updated


---> Package python.x86_64 0:2.6.6-68.0.1.el6_10 will be an update


---> Package python-libs.x86_64 0:2.6.6-66.0.1.el6_8 will be updated


---> Package python-libs.x86_64 0:2.6.6-68.0.1.el6_10 will be an update


---> Package rdma.noarch 2:3.10-3.0.40.el6 will be updated


---> Package rdma.noarch 2:3.10-3.0.41.el6 will be an update


---> Package samba.x86_64 0:3.6.23-51.0.1.el6 will be updated


---> Package samba.x86_64 0:3.6.23-52.0.1.el6_10 will be an update


---> Package samba-client.x86_64 0:3.6.23-51.0.1.el6 will be updated


---> Package samba-client.x86_64 0:3.6.23-52.0.1.el6_10 will be an update


---> Package samba-common.x86_64 0:3.6.23-51.0.1.el6 will be updated


---> Package samba-common.x86_64 0:3.6.23-52.0.1.el6_10 will be an update


---> Package samba-winbind.x86_64 0:3.6.23-51.0.1.el6 will be updated


---> Package samba-winbind.x86_64 0:3.6.23-52.0.1.el6_10 will be an update


---> Package samba-winbind-clients.x86_64 0:3.6.23-51.0.1.el6 will be updated


---> Package samba-winbind-clients.x86_64 0:3.6.23-52.0.1.el6_10 will be an update


---> Package sudo.x86_64 0:1.8.6p3-29.el6_9 will be updated


---> Package sudo.x86_64 0:1.8.6p3-29.0.1.el6_10.2 will be an update


---> Package tzdata.noarch 0:2019a-1.el6 will be updated


---> Package tzdata.noarch 0:2019c-1.el6 will be an update


---> Package tzdata-java.noarch 0:2018e-3.el6 will be updated


---> Package tzdata-java.noarch 0:2019c-1.el6 will be an update


---> Package udev.x86_64 0:147-2.73.0.1.el6_8.2 will be updated


---> Package udev.x86_64 0:147-2.73.0.2.el6_8.2 will be an update


---> Package vim-common.x86_64 2:7.4.629-5.el6_8.1 will be updated


---> Package vim-common.x86_64 2:7.4.629-5.el6_10.2 will be an update


---> Package vim-enhanced.x86_64 2:7.4.629-5.el6_8.1 will be updated


---> Package vim-enhanced.x86_64 2:7.4.629-5.el6_10.2 will be an update


---> Package vim-minimal.x86_64 2:7.4.629-5.el6_8.1 will be updated


---> Package vim-minimal.x86_64 2:7.4.629-5.el6_10.2 will be an update


---> Package xorg-x11-server-Xorg.x86_64 0:1.17.4-17.0.1.el6 will be updated


---> Package xorg-x11-server-Xorg.x86_64 0:1.17.4-17.0.2.el6 will be an update


---> Package xorg-x11-server-common.x86_64 0:1.17.4-17.0.1.el6 will be updated


---> Package xorg-x11-server-common.x86_64 0:1.17.4-17.0.2.el6 will be an update


--> Finished Dependency Resolution


Dependencies Resolved


================================================================================


 Package                 Arch    Version                   Repository      Size


================================================================================


Installing:


 kernel-uek              x86_64  4.1.12-124.33.4.el6uek    ODA_REPOS_LOC   42 M


 kernel-uek-firmware     noarch  4.1.12-124.33.4.el6uek    ODA_REPOS_LOC  2.6 M


Updating:


 cpupowerutils           x86_64  1.3-2.0.1.el6             ODA_REPOS_LOC   77 k


 cups-libs               x86_64  1:1.4.2-81.el6_10         ODA_REPOS_LOC  322 k


 dbus                    x86_64  1:1.2.24-11.0.1.el6_10    ODA_REPOS_LOC  211 k


 dbus-libs               x86_64  1:1.2.24-11.0.1.el6_10    ODA_REPOS_LOC  127 k


 dracut                  noarch  004-411.0.4.el6           ODA_REPOS_LOC  129 k


 dracut-kernel           noarch  004-411.0.4.el6           ODA_REPOS_LOC   29 k


 glibc                   x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  3.8 M


 glibc-devel             i686    2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  992 k


 glibc-devel             x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  991 k


 glibc-headers           x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  620 k


 initscripts             x86_64  9.03.61-1.0.6.el6         ODA_REPOS_LOC  952 k


 kernel-headers          x86_64  2.6.32-754.18.2.el6       ODA_REPOS_LOC  4.6 M


 kexec-tools             x86_64  2.0.7-1.0.28.el6          ODA_REPOS_LOC  339 k


 ksplice                 x86_64  1.0.43-1.el6              ODA_REPOS_LOC  9.1 k


 ksplice-core0           x86_64  1.0.43-1.el6              ODA_REPOS_LOC  271 k


 ksplice-offline         x86_64  1.0.43-1.el6              ODA_REPOS_LOC  7.9 k


 ksplice-tools           x86_64  1.0.43-1.el6              ODA_REPOS_LOC   92 k


 libgudev1               x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC   65 k


 libudev                 x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC   78 k


 mailx                   x86_64  12.4-10.el6_10            ODA_REPOS_LOC  235 k


 openssl                 x86_64  1.0.1e-58.0.1.el6_10      ODA_REPOS_LOC  1.5 M


 oracle-ofed-release     x86_64  1.0.0-51.el6              ODA_REPOS_LOC   16 k


 perf                    x86_64  2.6.32-754.18.2.el6       ODA_REPOS_LOC  4.8 M


 python                  x86_64  2.6.6-68.0.1.el6_10       ODA_REPOS_LOC   76 k


 python-libs             x86_64  2.6.6-68.0.1.el6_10       ODA_REPOS_LOC  5.3 M


 rdma                    noarch  2:3.10-3.0.41.el6         ODA_REPOS_LOC   76 k


 samba                   x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  5.1 M


 samba-client            x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC   11 M


 samba-common            x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC   10 M


 samba-winbind           x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  2.2 M


 samba-winbind-clients   x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  2.0 M


 sudo                    x86_64  1.8.6p3-29.0.1.el6_10.2   ODA_REPOS_LOC  712 k


 tzdata                  noarch  2019c-1.el6               ODA_REPOS_LOC  507 k


 tzdata-java             noarch  2019c-1.el6               ODA_REPOS_LOC  188 k


 udev                    x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC  360 k


 vim-common              x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  6.7 M


 vim-enhanced            x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  1.0 M


 vim-minimal             x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  421 k


 xorg-x11-server-Xorg    x86_64  1.17.4-17.0.2.el6         ODA_REPOS_LOC  1.4 M


 xorg-x11-server-common  x86_64  1.17.4-17.0.2.el6         ODA_REPOS_LOC   51 k


Transaction Summary


================================================================================


Install       2 Package(s)


Upgrade      40 Package(s)


Total download size: 112 M


Downloading Packages:


--------------------------------------------------------------------------------


Total                                           187 MB/s | 112 MB     00:00


Running rpm_check_debug


ERROR with rpm_check_debug vs depsolve:


glibc = 2.12-1.212.0.1.el6 is needed by (installed) nscd-2.12-1.212.0.1.el6.x86_64


** Found 11 pre-existing rpmdb problem(s), 'yum check' output follows:


glibc-2.12-1.212.0.2.el6.x86_64 has missing requires of glibc-common = ('0', '2.12', '1.212.0.2.el6')


glibc-2.12-1.212.0.3.el6_10.3.i686 is a duplicate with glibc-2.12-1.212.0.2.el6.x86_64


libgcc-4.4.7-23.0.1.el6.x86_64 is a duplicate with libgcc-4.4.7-18.el6.i686


nscd-2.12-1.212.0.1.el6.x86_64 has missing requires of glibc = ('0', '2.12', '1.212.0.1.el6')


oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of libnfsodm18.so()(64bit)


oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(GridDefParams)


oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(oakosdiskinfo)


oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(oaksharedstorageinfo)


oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(oakstoragetopology)


oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(ol5_to_ol6_upgrade)


oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(s_GridSteps)


Your transaction was saved, rerun it with:


 yum load-transaction /tmp/yum_save_tx-2020-06-06-14-16PkHNuJ.yumtx


WARNING: 2020-06-06 14:16:18: OS Upgrade is not successful. Need to resolve conflicts


INFO: 2020-06-06 14:16:18: Copy the required RPMs to a location and set EXTRA_OS_RPMS_LOC to that location

Here we need to solve the dependency problem, in this case we will remove the package

[root@odax58duts2 patchODA2020]# rpm -e nscd-2.12-1.212.0.1.el6 --nodeps


[root@odax58duts2 patchODA2020]# oakcli validate -c ospatch -ver 18.8.0.0.0


INFO: Validating the OS patch for the version 18.8.0.0.0


INFO: 2020-06-06 14:16:38: Performing a dry run for OS patching


INFO: 2020-06-06 14:16:55: No conflict detected during the OS update, dry run check.

Now we have the environment validated for updating the operating system

5 – Verify which components it will be required to update:

oakcli update -patch 18.8.0.0.0 --verify

e.g:

[root@odax58duts1 patchODA2020]#   oakcli update -patch 18.8.0.0.0 --verify


INFO: 2020-06-06 14:27:34: Reading the metadata file now...


                Component Name            Installed Version         Proposed Patch Version


                ---------------           ------------------        -----------------


                Controller_INT            11.05.03.00               Up-to-date


                Controller_EXT            11.05.03.00               Up-to-date


                Expander                  001E                      Up-to-date


                SSD_SHARED                944A                      Up-to-date


                HDD_LOCAL                 A7E0                      Up-to-date


                HDD_SHARED                A7E0                      Up-to-date


                ILOM                      4.0.4.40 r130079          5.0.0.20 r133445


                BIOS                      25080100                  Up-to-date


                IPMI                      1.8.15.0                  Up-to-date


                HMP                       2.4.5.0.1                 Up-to-date


                OAK                       18.7.0.0.0                18.8.0.0.0


                OL                        6.10                      Up-to-date


                OVM                       3.4.4                     Up-to-date


                GI_HOME                   18.7.0.0.190716           18.8.0.0.191015


                DB_HOME {


                [ OraDb12102_home1 ]      12.1.0.2.190716           12.1.0.2.191015


                [ OraDb11204_home1 ]      11.2.0.4.190716           11.2.0.4.191015


                [ OraDB18Home1 ]          18.7.1.0.191015           18.8.0.0.191015


                [ OraDb11203_home2 ]      11.2.0.3.15               No-update


                             }


[root@odax58duts2 patchODA2020]#   oakcli update -patch 18.8.0.0.0 --verify


INFO: 2020-06-06 14:27:39: Reading the metadata file now...


                Component Name            Installed Version         Proposed Patch Version


                ---------------           ------------------        -----------------


                Controller_INT            11.05.03.00               Up-to-date


                Controller_EXT            11.05.03.00               Up-to-date


                Expander                  001E                      Up-to-date


                SSD_SHARED                944A                      Up-to-date


                HDD_LOCAL                 A7E0                      Up-to-date


                HDD_SHARED                A7E0                      Up-to-date


                ILOM                      4.0.4.40 r130079          5.0.0.20 r133445


                BIOS                      25080100                  Up-to-date


                IPMI                      1.8.15.0                  Up-to-date


                HMP                       2.4.5.0.1                 Up-to-date


                OAK                       18.7.0.0.0                18.8.0.0.0


                OL                        6.10                      Up-to-date


                OVM                       3.4.4                     Up-to-date


                GI_HOME                   18.7.0.0.190716           18.8.0.0.191015


                DB_HOME {


                [ OraDb12102_home1 ]      12.1.0.2.190716           12.1.0.2.191015


                [ OraDb11204_home1 ]      11.2.0.4.190716           11.2.0.4.191015


                [ OraDB18Home1 ]          18.7.1.0.191015           18.8.0.0.191015


                [ OraDb11203_home2 ]      11.2.0.3.15               No-update

Note:

– Stop all databases and clusterware resource before patch ILOM in both servers

6 – Update ILOM

I prefer to first update ILOM in case there is a problem during the patch I can access the environment via ILOM.
 
If you want to update ILOM with the patch, it may be your choice as well.
 
 
Note:  Upgrade ILOM: Both ILOM nodes, but not in the same time.
 
Download ILOM Patch separately: Sun Server X5-8 (ILOM 5.0.0.20 133445)
Custom File Name: p30802633_300_Generic.zip
The firmware are inside of this .zip file in the following path:
C:\Users\andre.ontalba\Downloads\p30802633_300_Generic\Oracle_Server_X5-8-3.0.0.91223-FIRMWARE_PACK\Firmware\service-processor\ILOM-5_0_0_20_r133445-Oracle_Server_X5-4_X5-8.pkg
You must copy this file to the following directory in the server to use for transfer this package. In my case dutsLinux1.

/patch/ILOM

This file must be owned by “oracle” user and group “oinstall“
Connect to the gateway machine: dutsLinux1

ssh xxxx@dutsLinux1

Connect to the first ILOM from the gateway machine:

ssh root@odax58duts1-ilom


Enter remote user password: **********

Check current version:
e.g:
Type:

  -> version


SP firmware 4.0.4.40


SP firmware build number: 130079


SP firmware date: Thu May 07 09:54:31 CST 2019


SP filesystem version: 0.2.10

Stopping DOM0 and ODA_BASE: (Connect to the ILOM):

stop /SYS

Check status:

show /SYS

e.g: Section “Properties –> power_state = Off”

  Properties:


      type = Host System


      ipmi_name = /SYS


      product_name = SUN SERVER X5-8


      product_part_number = XXXXXXXXXXXXXXX


      product_serial_number = XXXXXXXXXXXXXXX


      product_manufacturer = Oracle Corporation


      fault_state = OK


      clear_fault_action = (none)


      power_state = Off

Load new ILOM image:

load -source scp://xxxx@dutsLinux1/ILOM/ ILOM-5_0_0_20_r133445-Oracle_Server_X5-4_X5-8.pkg

e.g:

load -source scp://root@dutsLinux1/ILOM/ ILOM-5_0_0_20_r133445-Oracle_Server_X5-4_X5-8.pkg


Enter remote user password: **********


NOTE: An upgrade takes several minutes to complete. ILOM


      will enter a special mode to load new firmware. No


      other tasks can be performed in ILOM until the


      firmware upgrade is complete and ILOM is reset.


    You can choose to postpone the server BIOS upgrade until the


    next server poweroff. If you do not do that, you should


    perform a clean shutdown of the server before continuing.


Are you sure you want to load the specified file (y/n)? y


Preserve existing SP configuration (y/n)? y


Preserve existing BIOS configuration (y/n)? y


Delay BIOS upgrade until next server poweroff or reset (y/n)? n


...

After the automatic reboot performed in the ILOM, you can validate the new firmware and BIOS version:

version


Hostname: odax58duts1-ilom


-> version


SP firmware 5.0.0.20


SP firmware build number: 133445


SP firmware date: Thr Feb 06 09:54:31 CST 2020


SP filesystem version: 0.2.10


show /SYS/MB/BIOS

/SYS/MB/BIOS

  Targets:


  Properties:


      type = BIOS


      ipmi_name = MB/BIOS


      fru_name = SYSTEM BIOS


      fru_manufacturer = AMERICAN MEGATRENDS


      fru_version = 25080100


      fru_part_number = APTIO

Start the DOM0 and ODA_BASE after upgrade ILOM:

start /SYS


exit

Note:  Repeat this procedure on the second node

7 – Upgrade ODA Servers

Note: Run only from first server and make sure that is the master node: oakcli show ismaster ⇒ “OAKD is in Master Mode”

oakcli update -patch 18.8.0.0.0 –server

e.g:

[root@odax58duts2 patchODA2020]# oakcli update -patch 18.8.0.0.0 –server

This procedure can take between 2 to 3 hours to execute on both nodes.

INFO: DB, ASM, Clusterware may be stopped during the patch if required


INFO: Both Nodes may get rebooted automatically during the patch if required


Do you want to continue: [Y/N]?: y


INFO: User has confirmed for the reboot


INFO: Patch bundle must be unpacked on the second Node also before applying the patch


Did you unpack the patch bundle on the second Node? : [Y/N]? : y


INFO: All the VMs except the ODABASE will be shutdown forcefully if needed


Do you want to continue : [Y/N]? : y

To be able to monitor the application of the patch we can see the logs in the directories below
Log file directory node 1:   /opt/oracle/oak/log/odax58duts1/patch/18.8.0.0.0/
Log file directory node 2:   /opt/oracle/oak/log/odax58duts2/patch/18.8.0.0.0/

Let’s check how is the server update
Node 1

[root@odax58duts1 patchODA2020]# oakcli show version -detail


INFO: 2020-06-06 17:17:34: Reading the metadata file now...


                Component Name            Installed Version         Proposed Patch Version


                ---------------           ------------------        -----------------


                Controller_INT            11.05.03.00               Up-to-date


                Controller_EXT            11.05.03.00               Up-to-date


                Expander                  001E                      Up-to-date


                SSD_SHARED                944A                      Up-to-date


                HDD_LOCAL                 A7E0                      Up-to-date


                HDD_SHARED                A7E0                      Up-to-date


                ILOM                      5.0.0.20 r133445          Up-to-date


                BIOS                      25080100                  Up-to-date


                IPMI                      1.8.15.0                  Up-to-date


                HMP                       2.4.5.0.1                 Up-to-date


                OAK                       18.8.0.0.0                Up-to-date


                OL                        6.10                      Up-to-date


                OVM                       3.4.4                     Up-to-date


                GI_HOME                   18.8.0.0.191015           Up-to-date


                DB_HOME {


                [ OraDb12102_home1 ]      12.1.0.2.190716           12.1.0.2.191015


                [ OraDb11204_home1 ]      11.2.0.4.190716           11.2.0.4.191015


                [ OraDB18Home1 ]          18.7.1.0.191015           18.8.0.0.191015


                [ OraDb11203_home2 ]      11.2.0.3.15               No-update

Node 2

[root@odax58duts2 patchODA2020]#   oakcli show version -detail


INFO: 2020-06-06 17:18:39: Reading the metadata file now...


                Component Name            Installed Version         Proposed Patch Version


                ---------------           ------------------        -----------------


                Controller_INT            11.05.03.00               Up-to-date


                Controller_EXT            11.05.03.00               Up-to-date


                Expander                  001E                      Up-to-date


                SSD_SHARED                944A                      Up-to-date


                HDD_LOCAL                 A7E0                      Up-to-date


                HDD_SHARED                A7E0                      Up-to-date


                ILOM                      5.0.0.20 r133445          Up-to-date


                BIOS                      25080100                  Up-to-date


                IPMI                      1.8.15.0                  Up-to-date


                HMP                       2.4.5.0.1                 Up-to-date


                OAK                       18.8.0.0.0                Up-to-date


                OL                        6.10                      Up-to-date


                OVM                       3.4.4                     Up-to-date


                GI_HOME                   18.8.0.0.191015           Up-to-date


                DB_HOME {


                [ OraDb12102_home1 ]      12.1.0.2.190716           12.1.0.2.191015


                [ OraDb11204_home1 ]      11.2.0.4.190716           11.2.0.4.191015


                [ OraDB18Home1 ]          18.7.1.0.191015           18.8.0.0.191015


                [ OraDb11203_home2 ]      11.2.0.3.15               No-update

8 – ODA Patch: Database Binaries

Now is time to apply patch on the Oracle Database Binaries (11.2.0.4,12.1,12.2,18.).
Get the database list and Oracle Home before patch:

oakcli show databases

First step is stop TFA (Both Nodes with root)

tfactl stop

To apply the patch in the Oracle Binaries: (Execute Only From First Node)

oakcli update -patch 18.8.0.0.0 --database


[root@odax58duts1 18.8.0.0.0]# oakcli update -patch 18.8.0.0.0 --database


INFO: Running pre-install scripts


INFO: Running  prepatching on node 0


INFO: Running  prepatching on node 1


INFO: Completed pre-install scripts


...


...


INFO: 2020-06-06 18:51:24: ------------------Patching DB-------------------------


INFO: 2020-06-06 18:51:24: Getting all the possible Database Homes for patching


...


INFO: 2020-06-06 18:52:03: Patching 11.2.0.4 Database Homes on the Node odax58duts1


Found the following 11.2.0.4 homes possible for patching:


HOME_NAME                      HOME_LOCATION


---------                      -------------


OraDb11204_home1               /u01/app/oracle/product/11.2.0.4/dbhome_1


[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]


Would you like to patch all the above homes: Y | N ? : Y


INFO: 2020-06-06 18:52:15: Updating OPATCH


Verifying Opatch version for home:</u01/app/oracle/product/11.2.0.4/dbhome_1>.


Expecting version:<11.2.0.3.22>


Opatch version on node <odax58duts1> is <11.2.0.3.22>


Opatch version on node <odax58duts2> is <11.2.0.3.22>


INFO: 2020-06-06 18:53:41: Performing the conflict checks...


SUCCESS: 2020-06-06 18:53:53: Conflict checks passed for all the Homes


INFO: 2020-06-06 18:53:53: Checking if the patch is already applied on any of the Homes


INFO: 2020-06-06 18:53:58: Home is not Up-to-date


SUCCESS: 2020-06-06 18:53:59: Successfully stopped the Database consoles


SUCCESS: 2020-06-06 18:54:06: Successfully stopped the EM agents


INFO: 2020-06-06 18:54:11: Applying the patch on oracle home : /u01/app/oracle/product/11.2.0.4/dbhome_1 ...


SUCCESS: 2020-06-06 18:56:20: Successfully applied the patch on the Home : /u01/app/oracle/product/11.2.0.4/dbhome_1


SUCCESS: 2020-06-06 18:56:20: Successfully started the Database consoles


SUCCESS: 2020-06-06 18:56:20: Successfully started the EM Agents


INFO: 2020-06-06 18:56:23: Patching 11.2.0.4 Database Homes on the Node odax58duts2


...


INFO: 2020-06-06 19:00:02: Patching 12.1.0.2 Database Homes on the Node odax58duts1


Found the following 12.1.0.2 homes possible for patching:


HOME_NAME                      HOME_LOCATION


---------                      -------------


OraDb12102_home1               /u01/app/oracle/product/12.1.0.2/dbhome_1


[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]


Would you like to patch all the above homes: Y | N ? : Y


INFO: 2020-06-06 19:06:38: Updating OPATCH


Verifying Opatch version for home:</u01/app/oracle/product/12.1.0.2/dbhome_1>.


Expecting version:<12.2.0.1.18>


Opatch version on node <odax58duts1> is <12.2.0.1.18>


Opatch version on node <odax58duts2> is <12.2.0.1.18>


INFO: 2020-06-06 19:07:37: Rolling back patches on 12.1.0.2.x home if required...


INFO: 2020-06-06 19:07:43: Checking if any patches need to be rolled back on </u01/app/oracle/product/12.1.0.2/dbhome_1>


INFO: 2020-06-06 19:11:35: Performing the conflict checks...


SUCCESS: 2020-06-06 19:11:59: Conflict checks passed for all the Homes


INFO: 2020-06-06 19:11:59: Checking if the patch is already applied on any of the Homes


INFO: 2020-06-06 19:12:11: Home is not Up-to-date


SUCCESS: 2020-06-06 19:12:13: Successfully stopped the Database consoles


SUCCESS: 2020-06-06 19:12:19: Successfully stopped the EM agents


INFO: 2020-06-06 19:12:25: Applying patch on /u01/app/oracle/product/12.1.0.2/dbhome_1 Homes


INFO: 2020-06-06 19:12:25: It may take upto 15 mins. Please wait...


SUCCESS: 2020-06-06 19:17:19: Successfully applied the patch on the Home : /u01/app/oracle/product/12.1.0.2/dbhome_1


SUCCESS: 2020-06-06 19:17:19: Successfully started the Database consoles


SUCCESS: 2020-06-06 19:17:19: Successfully started the EM Agents


INFO: 2020-06-06 19:17:23: Patching 12.1.0.2 Database Homes on the Node odax58duts2


...


INFO: 2020-06-06 19:27:08: Patching 18.0.0.0 Database Homes on the Node odax58duts1


Found the following 18.0.0.0 homes possible for patching:


HOME_NAME                      HOME_LOCATION


---------                      -------------


OraDB18Home1                   /u01/app/oracle/product/18.0.0.0


[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]


Would you like to patch all the above homes: Y | N ? : Y


INFO: 2020-06-06 19:27:15: Updating OPATCH


Verifying Opatch version for home:</u01/app/oracle/product/18.0.0.0>.


Expecting version:<12.2.0.1.18>


Opatch version on node <odax58duts1> is <12.2.0.1.18>


Opatch version on node <odax58duts2> is <12.2.0.1.18>


INFO: 2020-06-06 19:27:26: Rolling back patches on 18.x home if required...


INFO: 2020-06-06 19:27:33: Checking if any patches need to be rolled back on </u01/app/oracle/product/18.0.0.0>


INFO: 2020-06-06 19:28:57: Performing the conflict checks...


SUCCESS: 2020-06-06 19:29:57: Conflict checks passed for all the Homes


INFO: 2020-06-06 19:29:57: Checking if the patch is already applied on any of the Homes


INFO: 2020-06-06 19:30:36: Home is not Up-to-date


SUCCESS: 2020-06-06 19:30:38: Successfully stopped the Database consoles


SUCCESS: 2020-06-06 19:30:44: Successfully stopped the EM agents


INFO: 2020-06-06 19:30:49: Applying patch on /u01/app/oracle/product/18.0.0.0 Homes


INFO: 2020-06-06 19:30:49: It may take up to 15 mins. Please wait...


SUCCESS: 2020-06-06 19:40:34: Successfully applied the patch on the Home : /u01/app/oracle/product/18.0.0.0


SUCCESS: 2020-06-06 19:40:34: Successfully started the Database consoles


SUCCESS: 2020-06-06 19:40:34: Successfully started the EM Agents


INFO: 2020-06-06 19:40:37: Patching 18.0.0.0 Database Homes on the Node odax58duts2


INFO: DB patching summary on node: odax58duts1


SUCCESS: 2020-06-06 19:52:28:  Successfully applied the patch on the Home /u01/app/oracle/product/11.2.0.4/dbhome_1


SUCCESS: 2020-06-06 19:52:28:  Successfully applied the patch on the Home /u01/app/oracle/product/12.1.0.2/dbhome_1


SUCCESS: 2020-06-06 19:52:28:  Successfully applied the patch on the Home /u01/app/oracle/product/18.0.0.0


INFO: DB patching summary on node: odax58duts2


SUCCESS: 2020-06-06 19:52:28:  Successfully applied the patch on the Home /u01/app/oracle/product/11.2.0.4/dbhome_1


SUCCESS: 2020-06-06 19:52:28:  Successfully applied the patch on the Home /u01/app/oracle/product/12.1.0.2/dbhome_1


SUCCESS: 2020-06-06 19:52:28:  Successfully applied the patch on the Home /u01/app/oracle/product/18.0.0.0


INFO: Executing /tmp/pending_actions on both nodes


You have new mail in /var/spool/mail/root


[root@odax58duts1 18.8.0.0.0]#

Let’s check how is the database update
Node 1

[root@odax58duts1 patchODA2020]# oakcli show version -detail


INFO: 2020-06-06 19:55:34: Reading the metadata file now...


                Component Name            Installed Version         Proposed Patch Version


                ---------------           ------------------        -----------------


                Controller_INT            11.05.03.00               Up-to-date


                Controller_EXT            11.05.03.00               Up-to-date


                Expander                  001E                      Up-to-date


                SSD_SHARED                944A                      Up-to-date


                HDD_LOCAL                 A7E0                      Up-to-date


                HDD_SHARED                A7E0                      Up-to-date


                ILOM                      5.0.0.20 r133445          Up-to-date


                BIOS                      25080100                  Up-to-date


                IPMI                      1.8.15.0                  Up-to-date


                HMP                       2.4.5.0.1                 Up-to-date


                OAK                       18.8.0.0.0                Up-to-date


                OL                        6.10                      Up-to-date


                OVM                       3.4.4                     Up-to-date


                GI_HOME                   18.8.0.0.191015           Up-to-date


                DB_HOME {


                [ OraDb12102_home1 ]      12.1.0.2.191015           Up-to-date


                [ OraDb11204_home1 ]      11.2.0.4.191015           Up-to-date


                [ OraDB18Home1 ]          18.8.0.0.191015           Up-to-date


                [ OraDb11203_home2 ]      11.2.0.3.15               No-update


                             }

Node 2

[root@odax58duts2 patchODA2020]#   oakcli show version -detail


INFO: 2020-06-06 19:56:39: Reading the metadata file now...


                Component Name            Installed Version         Proposed Patch Version


                ---------------           ------------------        -----------------


                Controller_INT            11.05.03.00               Up-to-date


                Controller_EXT            11.05.03.00               Up-to-date


                Expander                  001E                      Up-to-date


                SSD_SHARED                944A                      Up-to-date


                HDD_LOCAL                 A7E0                      Up-to-date


                HDD_SHARED                A7E0                      Up-to-date


                ILOM                      5.0.0.20 r133445          Up-to-date


                BIOS                      25080100                  Up-to-date


                IPMI                      1.8.15.0                  Up-to-date


                HMP                       2.4.5.0.1                 Up-to-date


                OAK                       18.8.0.0.0                Up-to-date


                OL                        6.10                      Up-to-date


                OVM                       3.4.4                     Up-to-date


                GI_HOME                   18.8.0.0.191015           Up-to-date


                DB_HOME {


                [ OraDb12102_home1 ]      12.1.0.2.191015           Up-to-date


                [ OraDb11204_home1 ]      11.2.0.4.191015           Up-to-date


                [ OraDB18Home1 ]          18.8.0.0.191015           Up-to-date


                [ OraDb11203_home2 ]      11.2.0.3.15               No-update

9- Apply DATAPATCH/CATBUNDLE in databases 11.2 / 12.1 / 12.2 and 18.8

export NLS_LANG=AMERICAN_AMERICA.AL32UTF8


$ORACLE_HOME/OPatch/datapatch -verbose

Note: It is required set up the NLS_LANG to “AMERICAN_AMERICA.AL32UTF8” variable in order to avoid a BUG during the DATAPATCH in the Oracle Database 12.1

Reference Documents:

Oracle Database Appliance – 18.2, 12.X, and 2.X Supported ODA Versions & Known Issues (Doc ID 888888.1)
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.8/cmtxn/patching-oda.html#GUID-49F5F510-3A38-4E6A-B915-FCBCD36CDDDB

I hope I helped with this ODA upgrade procedure
Andre Ontalba

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited  to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”


1 … 6 7 8 9 10 … 32
LUXOUG – LUXEMBOURG ORACLE USERS GROUP
© 2025 LUXOUG – LUXEMBOURG ORACLE USERS GROUP