Blog
opatch fails with Error: ‘Archive Action: Source file “$ORACLE_HOME/.patch_storage/…” does not exist.’
Category: Database,Engineer System Author: Andre Luiz Dutra Ontalba (Board Member) Date: 5 years ago Comments: 0

opatch fails with Error: 'Archive Action: Source file "$ORACLE_HOME/.patch_storage/..." does not exist.'

 
Another quick article about a problem I had yesterday during an update patch in ODA. I found an error during patch in the Oracle Binary.

 

[Sep 14, 2020 2:40:58 PM] [INFO] add CopyAction for olsrelod.sql
[Sep 14, 2020 2:40:58 PM] [INFO] OPatchSessionHelper::sortOnOverlay() Given list - 25897615 25034396 26477255 20370037 21688501 18430870 27435440 24425998
[Sep 14, 2020 2:40:58 PM] [INFO] size of PatchObject list: 8
[Sep 14, 2020 2:40:59 PM] [INFO] Patch 24425998:
Achive Action: Directory "/u01/app/oracle/product/12.1.0.2/dbhome_3/.patch_storage/24425998_Sep_28_2016_12_31_24" does not exists or is not readable.
'oracle.rdbms, 12.1.0.2.0': Cannot update file '/u01/app/oracle/product/12.1.0.2/dbhome_3/lib/libserver12.a' with '/ksfd.o'
[Sep 14, 2020 2:40:59 PM] [INFO] Prerequisite check "CheckRollbackable" on auto-rollback patches failed.
The details are:

Patch 24425998:
Achive Action: Directory "/u01/app/oracle/product/12.1.0.2/dbhome_3/.patch_storage/24425998_Sep_28_2016_12_31_24" does not exists or is not readable.
'oracle.rdbms, 12.1.0.2.0': Cannot update file '/u01/app/oracle/product/12.1.0.2/dbhome_3/lib/libserver12.a' with '/ksfd.o'
[Sep 14, 2020 2:40:59 PM] [SEVERE] OUI-67073:UtilSession failed: Prerequisite check "CheckRollbackable" on auto-rollback patches failed.
[Sep 14, 2020 2:40:59 PM] [INFO] --------------------------------------------------------------------------------
[Sep 14, 2020 2:40:59 PM] [INFO] The following warnings have occurred during OPatch execution:
[Sep 14, 2020 2:40:59 PM] [INFO] 1) OUI-67303:
Patches [ 25897615 25034396 26477255 20370037 21688501 18430870 27435440 24425998 ] will be rolled back.
[Sep 14, 2020 2:40:59 PM] [INFO] 2) OUI-67303:
Patches [ 25897615 25034396 26477255 20370037 21688501 18430870 27435440 24425998 ] will be rolled back.
[Sep 14, 2020 2:40:59 PM] [INFO] --------------------------------------------------------------------------------
[Sep 14, 2020 2:40:59 PM] [INFO] Finishing UtilSession at Mon Sep 14 14:40:59 CEST 2020
[Sep 14, 2020 2:40:59 PM] [INFO] Log file location: /u01/app/oracle/product/12.1.0.2/dbhome_3/cfgtoollogs/opatchauto/core/opatch/opatch2020-09-14_14-39-20PM_1.log

 

According to the document: opatch fails with Error: ‘Archive Action: Source file “$ ORACLE_HOME / .patch_storage / …” does not exist.’ or ‘Achive Action: Directory “$ ORACLE_HOME / .patch_storage / …” does not exist or is not readable’. (Doc ID 1244414.1)
The reason for this is:
Files needed to rollback existing subset patch (es) are missing from $ORACLE_HOME/.patch_storage.

BACKGROUND
==============
When an Oracle software patch is installed, the first step is to place an unmodified copy of each affected $ORACLE_HOME file into a directory under the “$ORACLE_HOME/.patch_storage”.  These file copies will be used if the software patch is ever manually, or automatically rolled back.

 

Well after I saw this error, I was sure that nothing had been removed before the patch.
 
The only supported solutions:

 

  1. The missing directories and files can be restored from a backup of the ORACLE_HOME.
  2.  
  3. If no backups exist then re-install the $ORACLE_HOME.
  4.  
  5. Clone from another ORACLE_HOME of a like installation.

 

 

In my case I always have the backup of the following directories $ORACLE_HOME/inventory/oneoffs and    $ORACLE_HOME/.patch_storage.
 
I found the patch folder /.patch_storage/24425998_Sep_28_2016_12_31_24 that was needed to perform the rollback and after that the patch was applied successfully.
 
2020-09-15 07:47:04 Patch 29972716 is successfully already applied on the Home: /u01/app/oracle/product/12.1.0.2/db_home3
2020-09-15 07:47:04 SUCCESS: Successfully applied the patch on the Home : /u01/app/oracle/product/12.1.0.2/db_home4, /u01/app/oracle/product/12.1.0.2/db_home1, /u01/app/oracle/product/12.1.0.2/db_home3.

 

I hope this helps you!!!
 
Stay tuned, following on twitter @aontalba and on Linkedin

 

Andre Luiz Dutra Ontalba

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited  to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”


rootupgrade.sh Fails with CRS-1136: Rejecting the rolling upgrade mode change because the cluster is being patched
Category: Database,Engineer System Author: Andre Luiz Dutra Ontalba (Board Member) Date: 5 years ago Comments: 0

rootupgrade.sh Fails with CRS-1136: Rejecting the rolling upgrade mode change because the cluster is being patched

Yesterday during an ODA upgrade, I came across an error during the cluster upgrade process, where this error was presented.

 

.
.
2020/03/03 14:34:00 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
2020/03/03 14:34:04 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2020/03/03 14:34:32 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/crsctl start rollingupgrade 18.0.0.0.0'
CRS-1136: Rejecting the rolling upgrade mode change because the cluster is being patched.
CRS-4000: Command Start failed, or completed with errors.
2020/03/03 14:34:32 CLSRSC-511: failed to set Oracle Clusterware and ASM to rolling migration mode
Died at /u01/app/18.0.0.0/grid/crs/install/oraasm.pm line 1455.

 

Well following Oracle’s note rootupgrade.sh Fails with CRS-1136: Rejecting the rolling upgrade mode change because the cluster is being patched (Doc ID 2494827.1)    .
 
I found the solution to the problem by following the steps below.
 
Run the commands below to identify the versions of crs, releasepatch and softwarepatch to see if there are any differences.

 

bash-4.3# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [2660242823].
bash-4.3#

bash-4.3# crsctl query crs releasepatch
Oracle Clusterware release patch level is [1953265745] and the complete list of patches [23600818 26839277 27001739 27105253 27128906 27144050 27335416 ] have been applied on the local node.
bash-4.3#

bash-4.3# crsctl query crs softwarepatch
Oracle Clusterware patch level on node odatest1 is [1953265745]

 

We can see that the crs has a different version than the releasepatch and softwarepatch.
 
Well done that we will fix the problem.
 
1 – Run stop rollingpatch as root user, which will update OCR with correct values
<GRID_HOME>/bin/crsctl stop rollingpatch  

 

root@odatest1:~# /u01/app/12.1.0.2/grid/bin/crsctl stop rollingpatch
CRS-1161: The cluster was successfully patched to patch level [1953265745].
root@odatest1:~# 

 

2 – Verify software/release patch levels and retry rootupgrade.sh.

 

bash-4.3# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1953265745].
bash-4.3#

bash-4.3# crsctl query crs releasepatch
Oracle Clusterware release patch level is [1953265745] and the complete list of patches [23600818 26839277 27001739 27105253 27128906 27144050 27335416 ] have been applied on the local node.
bash-4.3#

bash-4.3# crsctl query crs softwarepatch
Oracle Clusterware patch level on node odatest1 is [1953265745]

 

root@odatest1:~# /u01/app/18.0.0.0/grid/rootupgrade.sh







.
.
2020/03/03 15:34:00 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
2020/03/03 15:34:04 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2020/03/03 15:34:32 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/crsctl start rollingupgrade 18.0.0.0.0'
CRS-1131: The cluster was successfully set to rolling upgrade mode.
2020/03/03 15:35:10 CLSRSC-482: Running command: '/u01/app/18.0.0.0/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/12.1.0.2/grid -oldCRSVersion 12.1.0.2.0 -firstNode true -startRolling false '

ASM configuration upgraded in local node successfully.

2020/03/03 15:34:20 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode

.
.

2020/03/03 15:54:00 CLSRSC-595: Executing upgrade step 8 of 19: 'UpgradeNode'.

2020/03/03 15:54:04 CLSRSC-474: Initiating upgrade of resource types

2020/03/03 15:56:20 CLSRSC-475: Upgrade of resource types successfully initiated.

2020/03/03 15:56:44 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.

2020/03/03 15:57:05 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

 

 

I hope this helps you!!!

 

Stay tuned, following on twitter @aontalba and on Linkedin
 
Andre Luiz Dutra Ontalba

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited  to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”


Feature – Auto-tune Detached Block Volumes
Category: Cloud Author: Andre Luiz Dutra Ontalba (Board Member) Date: 5 years ago Comments: 0

Feature: Auto-tune Detached Block Volumes

Last month Oracle launched a great feature that helps with block volume costs.Let’s see a little bit about this feature.
 
You can now tune the performance of your detached volumes to the Lower Cost setting automatically. With this new capability, while your volumes stay in a detached state you can achieve significant cost savings.
 
When you’re ready to use them for your workloads, simply attach them, and their performance and cost are automatically and instantaneously adjusted to the performance setting you originally configured. 
 
When you enable this feature for your volumes, the volume is monitored and changed to the Lower Cost performance option automatically when it is disconnected and this setting is maintained until you reconnect it. This feature now comes integrated with the storage platform. You can take advantage of this with a click on the console or using a command line option in the CLI for each of your volumes.
 
Auto-tune for detached volumes capability is generally available for all existing and new boot and block volumes in all regions globally, on the Console, and through CLI, SDK, API, and Terraform.
 

 

Enabling and Managing Auto-tune for Detached Volumes

Enabling the auto-tune feature for detached volumes is straightforward with a click in the Oracle Cloud Infrastructure Console.
To enable auto-tune for a volume, on the Block Volume Details screen of the Console, click Edit and slide the Auto-Tune Performance setting to On for the volume. The Edit dialog window is also revised as part of this feature update.When you enable this feature for your volumes, the volume is monitored and changed to the Lower Cost performance option automatically when it is disconnected, and maintained in this setting until you reconnect it. This feature now comes integrated with the storage platform. You can take advantage of this with a click on the console or using a command line option in the CLI for each of your volumes.

 

When the Auto-Tune Performance is set to On for a detached volume, the auto-tuning takes effect after 24 hours. After that, if the volume is still detached, its performance and cost is lowered to the Lower Cost setting automatically.
After Auto-Tune is applied
After 24 hours
When the volume is attached again, its performance is set to the Default Performance setting immediately and automatically.

 

𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬

 

  • Build-in feature – New volumes or Existing ones
  • Applicable to Block & Boot Volumes
  • Available in all Regions.
  • Reduce Operational Expenses
  • Once enabled Auto-tune it takes 24 hours to be effective.
  • Done via Console, CLI, SDK, API, Terraform
  • Switched Automatically Lower cost Performance Option when detached from Instance.
  • When attached again it gets to previous defined settings.
  • Optional Feature – You have to enabled it while creating Volumes
 

 

I hope this helps you!!!
Andre Luiz Dutra Ontalba

 

 

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited  to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”


HOW TO REMOVE HAIP ON ODA 18.8.0.0.0
Category: Engineer System Author: Andre Luiz Dutra Ontalba (Board Member) Date: 5 years ago Comments: 0

HOW TO REMOVE HAIP ON ODA 18.8.0.0.0

I needed to remove HAIP from ODA after migrating to version 18.8.0.0.0 and decided to prepare this procedure.
 
This action plan should only require one clusterware restart vs. patching which can result in two or three clusterware restarts.
 
Let’s go to the procedure.
1. Backup gpnp profile

 

[grid@testoda1 peer]$ cd /u01/app/18.0.0.0/grid/gpnp/'hostname'/profiles/peer
[grid@testoda1 peer]$ cp -p profile.xml profile.xml.bkp
[grid@testoda2 peer]$ /u01/app/18.0.0.0/grid/gpnp/'hostname'/profiles/peer
[grid@testoda2 peer]$ cp -p profile.xml profile.xml.bkp
2. Get the cluster_interconnect interfaces (only on node0)
[grid@testoda1 ~]$ /u01/app/18.0.0.0/grid/bin/oifcfg getif

btbond1 10.32.16.0 global public
p1p1 192.168.16.0 global cluster_interconnect,asm
p1p2 192.168.17.0 global cluster_interconnect,asm

Please note: That private interface names might be different depending on the model, and/or ODA version which was used to deploy the machine, etc.

For the rest of this note, we are using p1p1/p1p2 as an example in the steps below.

 
3. Backup existing ifcfg- files
[root@testoda1 ~]# cd /etc/sysconfig/network-scripts
[root@testoda1 network-scripts]# cp ifcfg-p1p1 backupifcfgFiles/ifcfg-p1p1.bak
[root@testoda1 network-scripts]# cp ifcfg-p1p2 backupifcfgFiles/ifcfg-p1p2.bak
[root@testoda2 ~]# cd /etc/sysconfig/network-scripts
[root@testoda2 network-scripts]# cp ifcfg-p1p1 backupifcfgFiles/ifcfg-p1p1.bak
[root@testoda2 network-scripts]# cp ifcfg-p1p2 backupifcfgFiles/ifcfg-p1p2.bak
4. Create ifcfg-icbond0 and modify ifcfg- files
[root@testoda1 network-scripts]# vi /etc/sysconfig/network-scripts/ifcfg-icbond0

# This file is automatically created by the ODA software.

DEVICE=icbond0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
TYPE=BOND
IPV6INIT=no
NM_CONTROLLED=no
PEERDNS=no
MTU=9000
BONDING_OPTS=”mode=active-backup miimon=100 primary=p1p1″
IPADDR=192.168.16.24
NETMASK=255.255.255.0

[root@testoda1 network-scripts]# vi /etc/sysconfig/network-scripts/ifcfg-p1p1

# This file is automatically created by the ODA software.

DEVICE=p1p1
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
USERCTL=no

# disable generic and large receive offloads on all interfaces,
# to prevent known problems, specifically in bridge configurations.

ETHTOOL_OFFLOAD_OPTS=”lro off gro off”
IPV6INIT=no
PEERDNS=no
BOOTPROTO=none
MASTER=icbond0
SLAVE=yes
MTU=9000

[root@testoda1 network-scripts]# vi /etc/sysconfig/network-scripts/ifcfg-p1p2

/etc/sysconfig/network-scripts/ifcfg-p1p2

# This file is automatically created by the ODA software.

DEVICE=p1p2
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
USERCTL=no

# disable generic and large receive offloads on all interfaces,
# to prevent known problems, specifically in bridge configurations.

ETHTOOL_OFFLOAD_OPTS=”lro off gro off”
IPV6INIT=no
PEERDNS=no
BOOTPROTO=none
MASTER=icbond0
SLAVE=yes
MTU=9000

[root@testoda2 network-scripts]# vi /etc/sysconfig/network-scripts/ifcfg-icbond0

# This file is automatically created by the ODA software.

DEVICE=icbond0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
TYPE=BOND
IPV6INIT=no
NM_CONTROLLED=no
PEERDNS=no
MTU=9000
BONDING_OPTS=”mode=active-backup miimon=100 primary=p1p1″
IPADDR=192.168.16.25
NETMASK=255.255.255.0

[root@testoda2 network-scripts]# vi /etc/sysconfig/network-scripts/ifcfg-p1p1

# This file is automatically created by the ODA software.

DEVICE=p1p1
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
USERCTL=no

# disable generic and large receive offloads on all interfaces,
# to prevent known problems, specifically in bridge configurations.

ETHTOOL_OFFLOAD_OPTS=”lro off gro off”
IPV6INIT=no
PEERDNS=no
BOOTPROTO=none
MASTER=icbond0
SLAVE=yes
MTU=9000

[root@testoda2 network-scripts]# vi /etc/sysconfig/network-scripts/ifcfg-p1p2

# This file is automatically created by the ODA software.

DEVICE=p1p2
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
USERCTL=no

# disable generic and large receive offloads on all interfaces,
# to prevent known problems, specifically in bridge configurations.

ETHTOOL_OFFLOAD_OPTS=”lro off gro off”
IPV6INIT=no
PEERDNS=no
BOOTPROTO=none
MASTER=icbond0
SLAVE=yes
MTU=9000

 
5. Creating/replacing init.ora-s for APX instances
[grid@testoda1]$ echo "+APX1.cluster_interconnects='192.168.16.24'" > $ORACLE_HOME/dbs/init+APX1.ora

[grid@testoda2]$ echo “+APX2.cluster_interconnects=’192.168.16.25′” > $ORACLE_HOME/dbs/init+APX2.ora

6. Stop the Clusterware on node2
[root@testoda2 ~]# /u01/app/18.0.0.0/grid/bin/crsctl stop crs -f
7. Set the new, bonded cluster_interconnect interface and remove p1p1/p1p2 interfaces from the configuration (only on node0)
[grid@testoda1 ~]$ oifcfg setif -global icbond0/192.168.16.0:cluster_interconnect,asm

[grid@testoda1 ~]$ oifcfg getif

btbond1 10.209.244.0 global public
p1p1 192.168.16.0 global cluster_interconnect,asm
p1p2 192.168.17.0 global cluster_interconnect,asm
icbond0 192.168.16.0 global cluster_interconnect,asm

[grid@testoda1 ~]$ oifcfg delif -global p1p1/192.168.16.0

[grid@testoda1 ~]$ oifcfg delif -global p1p2/192.168.17.0

[grid@testoda1 ~]$ oifcfg getif

btbond1 10.209.244.0 global public
icbond0 192.168.16.0 global cluster_interconnect,asm

8. Remove HAIP dependency in ora.asm
[root@testoda1 ~]# /u01/app/18.0.0.0/grid/bin/crsctl modify res ora.cluster_interconnect.haip -attr ENABLED=0 -init

[root@testoda2 ~]# /u01/app/18.0.0.0/grid/bin/crsctl modify res ora.cluster_interconnect.haip -attr ENABLED=0 -init

[root@testoda1 ~]# /u01/app/18.0.0.0/grid/bin/crsctl modify res ora.asm -attr “START_DEPENDENCIES=’hard(ora.cssd,ora.ctssd) pullup(ora.cssd,ora.ctssd) weak(ora.drivers.acfs)’, STOP_DEPENDENCIES=’hard(intermediate:ora.cssd)'” -init

[root@testoda2 ~]# /u01/app/18.0.0.0/grid/bin/crsctl modify res ora.asm -attr “START_DEPENDENCIES=’hard(ora.cssd,ora.ctssd) pullup(ora.cssd,ora.ctssd) weak(ora.drivers.acfs)’, STOP_DEPENDENCIES=’hard(intermediate:ora.cssd)'” -init

9. Removing ora.cluster_interconnect.haip resource
[root@testoda1 ~]# /u01/app/18.0.0.0/grid/bin/crsctl delete resource ora.cluster_interconnect.haip -init –f

[root@testoda2 ~]# /u01/app/18.0.0.0/grid/bin/crsctl delete resource ora.cluster_interconnect.haip -init –f

10. Stop the Clusterware on node1
[root@testoda1 ~]# /u01/app/18.0.0.0/grid/bin/crsctl stop crs -f
11. Restart the network
[root@testoda1 network-scripts]# service network restart

[root@testoda1 network-scripts]# ifconfig -a

[root@testoda1 network-scripts]# cat /proc/net/bondinf/icbond0

[root@testoda2 network-scripts]# service network restart

[root@testoda2 network-scripts]# ifconfig -a

[root@testoda2 network-scripts]# cat /proc/net/bondinf/icbond0

 
12. Restart the Clusterware
[root@testoda1 ~]# /u01/app/18.0.0.0/grid/bin/crsctl start crs

[root@testoda2 ~]# /u01/app/18.0.0.0/grid/bin/crsctl start crs

13. Restart dcs-agent to rediscover the interfaces automatically
[grid@testoda1 ~]# /opt/oracle/dcs/bin/restartagent.sh

[grid@testoda2 ~]# /opt/oracle/dcs/bin/restartagent.sh

14. Checking the cluster service after removing the service
[root@testoda1 ~]#

[root@testoda1 ~]# /u01/app/18.0.0.0/grid/bin/crsctl check cluster -all
**************************************************************
testoda1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
testoda2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
I hope I helped with this procedure
Andre Luiz Dutra Ontalba
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited  to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”


1 … 4 5 6 7 8 … 32
LUXOUG – LUXEMBOURG ORACLE USERS GROUP
© 2025 LUXOUG – LUXEMBOURG ORACLE USERS GROUP