Blog – Page 9 – LUXOUG – LUXEMBOURG ORACLE USERS GROUP

How to upgrade ODA Patch: 18.8.0.0 for Virtualized System

Category: Engineer System Author: Andre Luiz Dutra Ontalba (Board Member) Date: 6 years ago Comments: 0

How to upgrade ODA Patch: 18.8.0.0 for Virtualized System

Introduction

The goal of this document is to describe step by step how to upgrade ODA Virtualized System.

Prerequisites

For this Upgrade the ODA must be at version 18.3

Oracle Database Appliance Documentation (Check the last version of the patch)

Start Upgrade

1 – Backup of ODA_BASE (Both Nodes): It can take up to 2h00 hours.

From DOM0:

As root user:

Node 1:

oakcli stop oda_base

mkdir -p /backup/odax58duts1/odax58duts1_date +"%Y%m%d"

nohup tar -cvzf oakDom1.odax58duts1_dom0.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1 &

After complete the backup:

oakcli start oda_base

Node 2:

oakcli stop oda_base

mkdir -p /backup/odax58duts2/odax58duts2_date +"%Y%m%d"

nohup tar -cvzf oakDom1.odax58duts2_dom0.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1 &

After complete the backup:

oakcli start oda_base

2 – Download patch in a shared directory or separated directory in both ODA servers:

Os User: root

mkdir -p /backup/patchODA2020

Download all required files to this directory:

/backup/patchODA2020

There is 2 .zip files to be download patch (30518438):

p30518438_188000_Linux-x86-64_1of2.zip

p30518438_188000_Linux-x86-64_2of2.zip

Note: We must guarantee some minimum amount of free space (20GB) in the ODA BASE, file systems: “/” and “/u01“

– Purge old logs from ODA, with root user:

oakcli manage cleanrepo --ver 18.3.0.0.0

oakcli manage cleanrepo --ver 18.6.0.0.0

/usr/local/bin/purgeODALog -orcl 20 -tfa 10 -osw 10 -oak 10

– Clean up old patches from GRID_HOME, affects “/u01“:

su - grid

. oraenv

+ASM1 or +ASM2

cd $ORACLE_HOME/OPatch

./opatch util cleanup

– Clean up old patches from ORACLE_HOME, affects “/u01“:

su - oracle
. oraenv <SID>
cd $ORACLE_HOME/OPatch
./opatch util cleanup

Note:
– It must be performed this clean in every ORACLE_HOME versions that exists in the server.
– Look at “/home/oracle“, “/home/grid” and “/tmp” to perform some clean up and release some space in the file system “/“.

3 – Unpack downloaded patch in both ODA nodes:

cd /backup/patchODA2020
oakcli unpack -package p30518438_188000_Linux-x86-64_1of2.zip
oakcli unpack -package p30518438_188000_Linux-x86-64_2of2.zip

4 – Verify and Validate Upgrade ODA components (S.O):

export EXTRA_OS_RPMS_LOC=/backup/patchODA2020

oakcli validate -c ospatch -ver 18.8.0.0.0

Note:

Solving rpm’s conflict for version 18.8.0.0.0.

e.g: With errors and conflicts

BEGIN OUTPUT:

NODE 1:

[root@odax58duts1 patchODA2020]# oakcli validate -c ospatch -ver 18.8.0.0.0
INFO: Validating the OS patch for the version 18.8.0.0.0
INFO: 2020-06-06 14:06:16: Performing a dry run for OS patching
ERROR: 2020-06-06 14:06:31: Unable to run the command : /usr/bin/yum --exclude=kmod-mpt2sas,ibutils-libs,dapl,libcxgb3,libipathverbs,libmthca,libnes,ofed-docs update --disablerepo=* --enablerepo=ODA_REPOS_LOC -y
ERROR: 2020-06-06 14:06:31: Loaded plugins: rhnplugin, ulninfo, versionlock
This system is not registered with ULN.
You can use uln_register to register.
ULN support will be disabled.
Repository ol6_latest is listed more than once in the configuration
Repository ol6_addons is listed more than once in the configuration
Repository ol6_UEK_latest is listed more than once in the configuration
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package cpupowerutils.x86_64 0:1.3-2.el6 will be updated
---> Package cpupowerutils.x86_64 0:1.3-2.0.1.el6 will be an update
---> Package cups-libs.x86_64 1:1.4.2-79.el6 will be updated
---> Package cups-libs.x86_64 1:1.4.2-81.el6_10 will be an update
---> Package dbus.x86_64 1:1.2.24-9.0.1.el6 will be updated
---> Package dbus.x86_64 1:1.2.24-11.0.1.el6_10 will be an update
---> Package dbus-libs.x86_64 1:1.2.24-9.0.1.el6 will be updated
---> Package dbus-libs.x86_64 1:1.2.24-11.0.1.el6_10 will be an update
---> Package dracut.noarch 0:004-411.0.3.el6 will be updated
---> Package dracut.noarch 0:004-411.0.4.el6 will be an update
---> Package dracut-kernel.noarch 0:004-411.0.3.el6 will be updated
---> Package dracut-kernel.noarch 0:004-411.0.4.el6 will be an update
---> Package glibc.x86_64 0:2.12-1.212.0.2.el6 will be updated
---> Package glibc.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update
---> Package glibc-common.x86_64 0:2.12-1.212.0.2.el6 will be updated
---> Package glibc-common.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update
---> Package glibc-devel.i686 0:2.12-1.212.0.2.el6 will be updated
---> Package glibc-devel.x86_64 0:2.12-1.212.0.2.el6 will be updated
---> Package glibc-devel.i686 0:2.12-1.212.0.3.el6_10.3 will be an update
---> Package glibc-devel.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update
---> Package glibc-headers.x86_64 0:2.12-1.212.0.2.el6 will be updated
---> Package glibc-headers.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update
---> Package initscripts.x86_64 0:9.03.61-1.0.3.el6 will be updated
---> Package initscripts.x86_64 0:9.03.61-1.0.6.el6 will be an update
---> Package kernel-headers.x86_64 0:2.6.32-754.11.1.el6 will be updated
---> Package kernel-headers.x86_64 0:2.6.32-754.18.2.el6 will be an update
---> Package kernel-uek.x86_64 0:4.1.12-124.33.4.el6uek will be installed
---> Package kernel-uek-firmware.noarch 0:4.1.12-124.33.4.el6uek will be installed
---> Package kexec-tools.x86_64 0:2.0.7-1.0.27.el6 will be updated
---> Package kexec-tools.x86_64 0:2.0.7-1.0.28.el6 will be an update
---> Package ksplice.x86_64 0:1.0.38-1.el6 will be updated
---> Package ksplice.x86_64 0:1.0.43-1.el6 will be an update
---> Package ksplice-core0.x86_64 0:1.0.38-1.el6 will be updated
---> Package ksplice-core0.x86_64 0:1.0.43-1.el6 will be an update
---> Package ksplice-offline.x86_64 0:1.0.38-1.el6 will be updated
---> Package ksplice-offline.x86_64 0:1.0.43-1.el6 will be an update
---> Package ksplice-tools.x86_64 0:1.0.38-1.el6 will be updated
---> Package ksplice-tools.x86_64 0:1.0.43-1.el6 will be an update
---> Package libgudev1.x86_64 0:147-2.73.0.1.el6_8.2 will be updated
---> Package libgudev1.x86_64 0:147-2.73.0.2.el6_8.2 will be an update
---> Package libudev.x86_64 0:147-2.73.0.1.el6_8.2 will be updated
---> Package libudev.x86_64 0:147-2.73.0.2.el6_8.2 will be an update
---> Package mailx.x86_64 0:12.4-8.el6_6 will be updated
---> Package mailx.x86_64 0:12.4-10.el6_10 will be an update
---> Package openssl.x86_64 0:1.0.1e-57.0.6.el6 will be updated
---> Package openssl.x86_64 0:1.0.1e-58.0.1.el6_10 will be an update
---> Package oracle-ofed-release.x86_64 0:1.0.0-50.el6 will be updated
---> Package oracle-ofed-release.x86_64 0:1.0.0-51.el6 will be an update
---> Package perf.x86_64 0:2.6.32-754.11.1.el6 will be updated
---> Package perf.x86_64 0:2.6.32-754.18.2.el6 will be an update
---> Package python.x86_64 0:2.6.6-66.0.1.el6_8 will be updated
---> Package python.x86_64 0:2.6.6-68.0.1.el6_10 will be an update
---> Package python-libs.x86_64 0:2.6.6-66.0.1.el6_8 will be updated
---> Package python-libs.x86_64 0:2.6.6-68.0.1.el6_10 will be an update
---> Package rdma.noarch 2:3.10-3.0.40.el6 will be updated
---> Package rdma.noarch 2:3.10-3.0.41.el6 will be an update
---> Package samba.x86_64 0:3.6.23-51.0.1.el6 will be updated
---> Package samba.x86_64 0:3.6.23-52.0.1.el6_10 will be an update
---> Package samba-client.x86_64 0:3.6.23-51.0.1.el6 will be updated
---> Package samba-client.x86_64 0:3.6.23-52.0.1.el6_10 will be an update
---> Package samba-common.x86_64 0:3.6.23-51.0.1.el6 will be updated
---> Package samba-common.x86_64 0:3.6.23-52.0.1.el6_10 will be an update
---> Package samba-winbind.x86_64 0:3.6.23-51.0.1.el6 will be updated
---> Package samba-winbind.x86_64 0:3.6.23-52.0.1.el6_10 will be an update
---> Package samba-winbind-clients.x86_64 0:3.6.23-51.0.1.el6 will be updated
---> Package samba-winbind-clients.x86_64 0:3.6.23-52.0.1.el6_10 will be an update
---> Package sudo.x86_64 0:1.8.6p3-29.el6_9 will be updated
---> Package sudo.x86_64 0:1.8.6p3-29.0.1.el6_10.2 will be an update
---> Package tzdata.noarch 0:2019a-1.el6 will be updated
---> Package tzdata.noarch 0:2019c-1.el6 will be an update
---> Package tzdata-java.noarch 0:2018e-3.el6 will be updated
---> Package tzdata-java.noarch 0:2019c-1.el6 will be an update
---> Package udev.x86_64 0:147-2.73.0.1.el6_8.2 will be updated
---> Package udev.x86_64 0:147-2.73.0.2.el6_8.2 will be an update
---> Package vim-common.x86_64 2:7.4.629-5.el6_8.1 will be updated
---> Package vim-common.x86_64 2:7.4.629-5.el6_10.2 will be an update
---> Package vim-enhanced.x86_64 2:7.4.629-5.el6_8.1 will be updated
---> Package vim-enhanced.x86_64 2:7.4.629-5.el6_10.2 will be an update
---> Package vim-minimal.x86_64 2:7.4.629-5.el6_8.1 will be updated
---> Package vim-minimal.x86_64 2:7.4.629-5.el6_10.2 will be an update
---> Package xorg-x11-server-Xorg.x86_64 0:1.17.4-17.0.1.el6 will be updated
---> Package xorg-x11-server-Xorg.x86_64 0:1.17.4-17.0.2.el6 will be an update
---> Package xorg-x11-server-common.x86_64 0:1.17.4-17.0.1.el6 will be updated
---> Package xorg-x11-server-common.x86_64 0:1.17.4-17.0.2.el6 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
 Package                 Arch    Version                   Repository      Size
================================================================================
Installing:
 kernel-uek              x86_64  4.1.12-124.33.4.el6uek    ODA_REPOS_LOC   42 M
 kernel-uek-firmware     noarch  4.1.12-124.33.4.el6uek    ODA_REPOS_LOC  2.6 M
Updating:
 cpupowerutils           x86_64  1.3-2.0.1.el6             ODA_REPOS_LOC   77 k
 cups-libs               x86_64  1:1.4.2-81.el6_10         ODA_REPOS_LOC  322 k
 dbus                    x86_64  1:1.2.24-11.0.1.el6_10    ODA_REPOS_LOC  211 k
 dbus-libs               x86_64  1:1.2.24-11.0.1.el6_10    ODA_REPOS_LOC  127 k
 dracut                  noarch  004-411.0.4.el6           ODA_REPOS_LOC  129 k
 dracut-kernel           noarch  004-411.0.4.el6           ODA_REPOS_LOC   29 k
 glibc                   x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  3.8 M
 glibc-common            x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC   14 M
 glibc-devel             i686    2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  992 k
 glibc-devel             x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  991 k
 glibc-headers           x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  620 k
 initscripts             x86_64  9.03.61-1.0.6.el6         ODA_REPOS_LOC  952 k
 kernel-headers          x86_64  2.6.32-754.18.2.el6       ODA_REPOS_LOC  4.6 M
 kexec-tools             x86_64  2.0.7-1.0.28.el6          ODA_REPOS_LOC  339 k
 ksplice                 x86_64  1.0.43-1.el6              ODA_REPOS_LOC  9.1 k
 ksplice-core0           x86_64  1.0.43-1.el6              ODA_REPOS_LOC  271 k
 ksplice-offline         x86_64  1.0.43-1.el6              ODA_REPOS_LOC  7.9 k
 ksplice-tools           x86_64  1.0.43-1.el6              ODA_REPOS_LOC   92 k
 libgudev1               x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC   65 k
 libudev                 x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC   78 k
 mailx                   x86_64  12.4-10.el6_10            ODA_REPOS_LOC  235 k
 openssl                 x86_64  1.0.1e-58.0.1.el6_10      ODA_REPOS_LOC  1.5 M
 oracle-ofed-release     x86_64  1.0.0-51.el6              ODA_REPOS_LOC   16 k
 perf                    x86_64  2.6.32-754.18.2.el6       ODA_REPOS_LOC  4.8 M
 python                  x86_64  2.6.6-68.0.1.el6_10       ODA_REPOS_LOC   76 k
 python-libs             x86_64  2.6.6-68.0.1.el6_10       ODA_REPOS_LOC  5.3 M
 rdma                    noarch  2:3.10-3.0.41.el6         ODA_REPOS_LOC   76 k
 samba                   x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  5.1 M
 samba-client            x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC   11 M
 samba-common            x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC   10 M
 samba-winbind           x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  2.2 M
 samba-winbind-clients   x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  2.0 M
 sudo                    x86_64  1.8.6p3-29.0.1.el6_10.2   ODA_REPOS_LOC  712 k
 tzdata                  noarch  2019c-1.el6               ODA_REPOS_LOC  507 k
 tzdata-java             noarch  2019c-1.el6               ODA_REPOS_LOC  188 k
 udev                    x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC  360 k
 vim-common              x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  6.7 M
 vim-enhanced            x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  1.0 M
 vim-minimal             x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  421 k
 xorg-x11-server-Xorg    x86_64  1.17.4-17.0.2.el6         ODA_REPOS_LOC  1.4 M
 xorg-x11-server-common  x86_64  1.17.4-17.0.2.el6         ODA_REPOS_LOC   51 k
Transaction Summary
================================================================================
Install       2 Package(s)
Upgrade      41 Package(s)
Total download size: 126 M
Downloading Packages:
--------------------------------------------------------------------------------
Total                                           190 MB/s | 126 MB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Check Error:
  file /usr/bin/ldd from install of glibc-common-2.12-1.212.0.3.el6_10.3.x86_64 conflicts with file from package glibc-common-2.12-1.212.0.3.el6_10.3.i686
  file /usr/lib/locale/locale-archive.tmpl from install of glibc-common-2.12-1.212.0.3.el6_10.3.x86_64 conflicts with file from package glibc-common-2.12-1.212.0.3.el6_10.3.i686
Error Summary
-------------
WARNING: 2020-06-06 14:06:31: OS Upgrade is not successful. Need to resolve conflicts
INFO: 2020-06-06 14:06:31: Copy the required RPMs to a location and set EXTRA_OS_RPMS_LOC to that location

Here we need to solve the dependency problem, in this case we will remove the package

[root@odax58duts1 patchODA2020]# rpm -e glibc-common-2.12-1.212.0.3.el6_10.3.i686 --nodeps

You have new mail in /var/spool/mail/root

[root@odax58duts1 patchODA2020]# oakcli validate -c ospatch -ver 18.8.0.0.0

INFO: Validating the OS patch for the version 18.8.0.0.0

INFO: 2020-06-06 14:09:51: Performing a dry run for OS patching

INFO: 2020-06-06 14:10:09: No conflict detected during the OS update, dry run check.

NODE 2:

[root@odax58duts2 patchODA2020]# oakcli validate -c ospatch -ver 18.8.0.0.0
INFO: Validating the OS patch for the version 18.8.0.0.0
INFO: 2020-06-06 14:15:59: Performing a dry run for OS patching
ERROR: 2020-06-06 14:16:18: Unable to run the command : /usr/bin/yum --exclude=kmod-mpt2sas,ibutils-libs,dapl,libcxgb3,libipathverbs,libmthca,libnes,ofed-docs update --disablerepo=* --enablerepo=ODA_REPOS_LOC -y
ERROR: 2020-06-06 14:16:18: Loaded plugins: rhnplugin, ulninfo, versionlock
This system is not registered with ULN.
You can use uln_register to register.
ULN support will be disabled.
Repository ol6_latest is listed more than once in the configuration
Repository ol6_addons is listed more than once in the configuration
Repository ol6_UEK_latest is listed more than once in the configuration
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package cpupowerutils.x86_64 0:1.3-2.el6 will be updated
---> Package cpupowerutils.x86_64 0:1.3-2.0.1.el6 will be an update
---> Package cups-libs.x86_64 1:1.4.2-79.el6 will be updated
---> Package cups-libs.x86_64 1:1.4.2-81.el6_10 will be an update
---> Package dbus.x86_64 1:1.2.24-9.0.1.el6 will be updated
---> Package dbus.x86_64 1:1.2.24-11.0.1.el6_10 will be an update
---> Package dbus-libs.x86_64 1:1.2.24-9.0.1.el6 will be updated
---> Package dbus-libs.x86_64 1:1.2.24-11.0.1.el6_10 will be an update
---> Package dracut.noarch 0:004-411.0.3.el6 will be updated
---> Package dracut.noarch 0:004-411.0.4.el6 will be an update
---> Package dracut-kernel.noarch 0:004-411.0.3.el6 will be updated
---> Package dracut-kernel.noarch 0:004-411.0.4.el6 will be an update
---> Package glibc.x86_64 0:2.12-1.212.0.2.el6 will be updated
---> Package glibc.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update
---> Package glibc-devel.i686 0:2.12-1.212.0.2.el6 will be updated
---> Package glibc-devel.x86_64 0:2.12-1.212.0.2.el6 will be updated
---> Package glibc-devel.i686 0:2.12-1.212.0.3.el6_10.3 will be an update
---> Package glibc-devel.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update
---> Package glibc-headers.x86_64 0:2.12-1.212.0.2.el6 will be updated
---> Package glibc-headers.x86_64 0:2.12-1.212.0.3.el6_10.3 will be an update
---> Package initscripts.x86_64 0:9.03.61-1.0.3.el6 will be updated
---> Package initscripts.x86_64 0:9.03.61-1.0.6.el6 will be an update
---> Package kernel-headers.x86_64 0:2.6.32-754.11.1.el6 will be updated
---> Package kernel-headers.x86_64 0:2.6.32-754.18.2.el6 will be an update
---> Package kernel-uek.x86_64 0:4.1.12-124.33.4.el6uek will be installed
---> Package kernel-uek-firmware.noarch 0:4.1.12-124.33.4.el6uek will be installed
---> Package kexec-tools.x86_64 0:2.0.7-1.0.27.el6 will be updated
---> Package kexec-tools.x86_64 0:2.0.7-1.0.28.el6 will be an update
---> Package ksplice.x86_64 0:1.0.38-1.el6 will be updated
---> Package ksplice.x86_64 0:1.0.43-1.el6 will be an update
---> Package ksplice-core0.x86_64 0:1.0.38-1.el6 will be updated
---> Package ksplice-core0.x86_64 0:1.0.43-1.el6 will be an update
---> Package ksplice-offline.x86_64 0:1.0.38-1.el6 will be updated
---> Package ksplice-offline.x86_64 0:1.0.43-1.el6 will be an update
---> Package ksplice-tools.x86_64 0:1.0.38-1.el6 will be updated
---> Package ksplice-tools.x86_64 0:1.0.43-1.el6 will be an update
---> Package libgudev1.x86_64 0:147-2.73.0.1.el6_8.2 will be updated
---> Package libgudev1.x86_64 0:147-2.73.0.2.el6_8.2 will be an update
---> Package libudev.x86_64 0:147-2.73.0.1.el6_8.2 will be updated
---> Package libudev.x86_64 0:147-2.73.0.2.el6_8.2 will be an update
---> Package mailx.x86_64 0:12.4-8.el6_6 will be updated
---> Package mailx.x86_64 0:12.4-10.el6_10 will be an update
---> Package openssl.x86_64 0:1.0.1e-57.0.6.el6 will be updated
---> Package openssl.x86_64 0:1.0.1e-58.0.1.el6_10 will be an update
---> Package oracle-ofed-release.x86_64 0:1.0.0-50.el6 will be updated
---> Package oracle-ofed-release.x86_64 0:1.0.0-51.el6 will be an update
---> Package perf.x86_64 0:2.6.32-754.11.1.el6 will be updated
---> Package perf.x86_64 0:2.6.32-754.18.2.el6 will be an update
---> Package python.x86_64 0:2.6.6-66.0.1.el6_8 will be updated
---> Package python.x86_64 0:2.6.6-68.0.1.el6_10 will be an update
---> Package python-libs.x86_64 0:2.6.6-66.0.1.el6_8 will be updated
---> Package python-libs.x86_64 0:2.6.6-68.0.1.el6_10 will be an update
---> Package rdma.noarch 2:3.10-3.0.40.el6 will be updated
---> Package rdma.noarch 2:3.10-3.0.41.el6 will be an update
---> Package samba.x86_64 0:3.6.23-51.0.1.el6 will be updated
---> Package samba.x86_64 0:3.6.23-52.0.1.el6_10 will be an update
---> Package samba-client.x86_64 0:3.6.23-51.0.1.el6 will be updated
---> Package samba-client.x86_64 0:3.6.23-52.0.1.el6_10 will be an update
---> Package samba-common.x86_64 0:3.6.23-51.0.1.el6 will be updated
---> Package samba-common.x86_64 0:3.6.23-52.0.1.el6_10 will be an update
---> Package samba-winbind.x86_64 0:3.6.23-51.0.1.el6 will be updated
---> Package samba-winbind.x86_64 0:3.6.23-52.0.1.el6_10 will be an update
---> Package samba-winbind-clients.x86_64 0:3.6.23-51.0.1.el6 will be updated
---> Package samba-winbind-clients.x86_64 0:3.6.23-52.0.1.el6_10 will be an update
---> Package sudo.x86_64 0:1.8.6p3-29.el6_9 will be updated
---> Package sudo.x86_64 0:1.8.6p3-29.0.1.el6_10.2 will be an update
---> Package tzdata.noarch 0:2019a-1.el6 will be updated
---> Package tzdata.noarch 0:2019c-1.el6 will be an update
---> Package tzdata-java.noarch 0:2018e-3.el6 will be updated
---> Package tzdata-java.noarch 0:2019c-1.el6 will be an update
---> Package udev.x86_64 0:147-2.73.0.1.el6_8.2 will be updated
---> Package udev.x86_64 0:147-2.73.0.2.el6_8.2 will be an update
---> Package vim-common.x86_64 2:7.4.629-5.el6_8.1 will be updated
---> Package vim-common.x86_64 2:7.4.629-5.el6_10.2 will be an update
---> Package vim-enhanced.x86_64 2:7.4.629-5.el6_8.1 will be updated
---> Package vim-enhanced.x86_64 2:7.4.629-5.el6_10.2 will be an update
---> Package vim-minimal.x86_64 2:7.4.629-5.el6_8.1 will be updated
---> Package vim-minimal.x86_64 2:7.4.629-5.el6_10.2 will be an update
---> Package xorg-x11-server-Xorg.x86_64 0:1.17.4-17.0.1.el6 will be updated
---> Package xorg-x11-server-Xorg.x86_64 0:1.17.4-17.0.2.el6 will be an update
---> Package xorg-x11-server-common.x86_64 0:1.17.4-17.0.1.el6 will be updated
---> Package xorg-x11-server-common.x86_64 0:1.17.4-17.0.2.el6 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
 Package                 Arch    Version                   Repository      Size
================================================================================
Installing:
 kernel-uek              x86_64  4.1.12-124.33.4.el6uek    ODA_REPOS_LOC   42 M
 kernel-uek-firmware     noarch  4.1.12-124.33.4.el6uek    ODA_REPOS_LOC  2.6 M
Updating:
 cpupowerutils           x86_64  1.3-2.0.1.el6             ODA_REPOS_LOC   77 k
 cups-libs               x86_64  1:1.4.2-81.el6_10         ODA_REPOS_LOC  322 k
 dbus                    x86_64  1:1.2.24-11.0.1.el6_10    ODA_REPOS_LOC  211 k
 dbus-libs               x86_64  1:1.2.24-11.0.1.el6_10    ODA_REPOS_LOC  127 k
 dracut                  noarch  004-411.0.4.el6           ODA_REPOS_LOC  129 k
 dracut-kernel           noarch  004-411.0.4.el6           ODA_REPOS_LOC   29 k
 glibc                   x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  3.8 M
 glibc-devel             i686    2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  992 k
 glibc-devel             x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  991 k
 glibc-headers           x86_64  2.12-1.212.0.3.el6_10.3   ODA_REPOS_LOC  620 k
 initscripts             x86_64  9.03.61-1.0.6.el6         ODA_REPOS_LOC  952 k
 kernel-headers          x86_64  2.6.32-754.18.2.el6       ODA_REPOS_LOC  4.6 M
 kexec-tools             x86_64  2.0.7-1.0.28.el6          ODA_REPOS_LOC  339 k
 ksplice                 x86_64  1.0.43-1.el6              ODA_REPOS_LOC  9.1 k
 ksplice-core0           x86_64  1.0.43-1.el6              ODA_REPOS_LOC  271 k
 ksplice-offline         x86_64  1.0.43-1.el6              ODA_REPOS_LOC  7.9 k
 ksplice-tools           x86_64  1.0.43-1.el6              ODA_REPOS_LOC   92 k
 libgudev1               x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC   65 k
 libudev                 x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC   78 k
 mailx                   x86_64  12.4-10.el6_10            ODA_REPOS_LOC  235 k
 openssl                 x86_64  1.0.1e-58.0.1.el6_10      ODA_REPOS_LOC  1.5 M
 oracle-ofed-release     x86_64  1.0.0-51.el6              ODA_REPOS_LOC   16 k
 perf                    x86_64  2.6.32-754.18.2.el6       ODA_REPOS_LOC  4.8 M
 python                  x86_64  2.6.6-68.0.1.el6_10       ODA_REPOS_LOC   76 k
 python-libs             x86_64  2.6.6-68.0.1.el6_10       ODA_REPOS_LOC  5.3 M
 rdma                    noarch  2:3.10-3.0.41.el6         ODA_REPOS_LOC   76 k
 samba                   x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  5.1 M
 samba-client            x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC   11 M
 samba-common            x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC   10 M
 samba-winbind           x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  2.2 M
 samba-winbind-clients   x86_64  3.6.23-52.0.1.el6_10      ODA_REPOS_LOC  2.0 M
 sudo                    x86_64  1.8.6p3-29.0.1.el6_10.2   ODA_REPOS_LOC  712 k
 tzdata                  noarch  2019c-1.el6               ODA_REPOS_LOC  507 k
 tzdata-java             noarch  2019c-1.el6               ODA_REPOS_LOC  188 k
 udev                    x86_64  147-2.73.0.2.el6_8.2      ODA_REPOS_LOC  360 k
 vim-common              x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  6.7 M
 vim-enhanced            x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  1.0 M
 vim-minimal             x86_64  2:7.4.629-5.el6_10.2      ODA_REPOS_LOC  421 k
 xorg-x11-server-Xorg    x86_64  1.17.4-17.0.2.el6         ODA_REPOS_LOC  1.4 M
 xorg-x11-server-common  x86_64  1.17.4-17.0.2.el6         ODA_REPOS_LOC   51 k
Transaction Summary
================================================================================
Install       2 Package(s)
Upgrade      40 Package(s)
Total download size: 112 M
Downloading Packages:
--------------------------------------------------------------------------------
Total                                           187 MB/s | 112 MB     00:00
Running rpm_check_debug
ERROR with rpm_check_debug vs depsolve:
glibc = 2.12-1.212.0.1.el6 is needed by (installed) nscd-2.12-1.212.0.1.el6.x86_64
** Found 11 pre-existing rpmdb problem(s), 'yum check' output follows:
glibc-2.12-1.212.0.2.el6.x86_64 has missing requires of glibc-common = ('0', '2.12', '1.212.0.2.el6')
glibc-2.12-1.212.0.3.el6_10.3.i686 is a duplicate with glibc-2.12-1.212.0.2.el6.x86_64
libgcc-4.4.7-23.0.1.el6.x86_64 is a duplicate with libgcc-4.4.7-18.el6.i686
nscd-2.12-1.212.0.1.el6.x86_64 has missing requires of glibc = ('0', '2.12', '1.212.0.1.el6')
oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of libnfsodm18.so()(64bit)
oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(GridDefParams)
oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(oakosdiskinfo)
oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(oaksharedstorageinfo)
oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(oakstoragetopology)
oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(ol5_to_ol6_upgrade)
oak-18.7.0.0.0_LINUX.X64_190915-1.x86_64 has missing requires of perl(s_GridSteps)
Your transaction was saved, rerun it with:
 yum load-transaction /tmp/yum_save_tx-2020-06-06-14-16PkHNuJ.yumtx
WARNING: 2020-06-06 14:16:18: OS Upgrade is not successful. Need to resolve conflicts
INFO: 2020-06-06 14:16:18: Copy the required RPMs to a location and set EXTRA_OS_RPMS_LOC to that location

Here we need to solve the dependency problem, in this case we will remove the package

[root@odax58duts2 patchODA2020]# rpm -e nscd-2.12-1.212.0.1.el6 --nodeps

[root@odax58duts2 patchODA2020]# oakcli validate -c ospatch -ver 18.8.0.0.0

INFO: Validating the OS patch for the version 18.8.0.0.0

INFO: 2020-06-06 14:16:38: Performing a dry run for OS patching

INFO: 2020-06-06 14:16:55: No conflict detected during the OS update, dry run check.

Now we have the environment validated for updating the operating system

5 – Verify which components it will be required to update:

oakcli update -patch 18.8.0.0.0 --verify

e.g:

[root@odax58duts1 patchODA2020]# oakcli update -patch 18.8.0.0.0 --verify

INFO: 2020-06-06 14:27:34: Reading the metadata file now...

Component Name Installed Version Proposed Patch Version

--------------- ------------------ -----------------

Controller_INT 11.05.03.00 Up-to-date

Controller_EXT 11.05.03.00 Up-to-date

Expander 001E Up-to-date

SSD_SHARED 944A Up-to-date

HDD_LOCAL A7E0 Up-to-date

HDD_SHARED A7E0 Up-to-date

ILOM 4.0.4.40 r130079 5.0.0.20 r133445

BIOS 25080100 Up-to-date

IPMI 1.8.15.0 Up-to-date

HMP 2.4.5.0.1 Up-to-date

OAK 18.7.0.0.0 18.8.0.0.0

OL 6.10 Up-to-date

OVM 3.4.4 Up-to-date

GI_HOME 18.7.0.0.190716 18.8.0.0.191015

DB_HOME {

[ OraDb12102_home1 ] 12.1.0.2.190716 12.1.0.2.191015

[ OraDb11204_home1 ] 11.2.0.4.190716 11.2.0.4.191015

[ OraDB18Home1 ] 18.7.1.0.191015 18.8.0.0.191015

[ OraDb11203_home2 ] 11.2.0.3.15 No-update

}

[root@odax58duts2 patchODA2020]# oakcli update -patch 18.8.0.0.0 --verify

INFO: 2020-06-06 14:27:39: Reading the metadata file now...

Component Name Installed Version Proposed Patch Version

--------------- ------------------ -----------------

Controller_INT 11.05.03.00 Up-to-date

Controller_EXT 11.05.03.00 Up-to-date

Expander 001E Up-to-date

SSD_SHARED 944A Up-to-date

HDD_LOCAL A7E0 Up-to-date

HDD_SHARED A7E0 Up-to-date

ILOM 4.0.4.40 r130079 5.0.0.20 r133445

BIOS 25080100 Up-to-date

IPMI 1.8.15.0 Up-to-date

HMP 2.4.5.0.1 Up-to-date

OAK 18.7.0.0.0 18.8.0.0.0

OL 6.10 Up-to-date

OVM 3.4.4 Up-to-date

GI_HOME 18.7.0.0.190716 18.8.0.0.191015

DB_HOME {

[ OraDb12102_home1 ] 12.1.0.2.190716 12.1.0.2.191015

[ OraDb11204_home1 ] 11.2.0.4.190716 11.2.0.4.191015

[ OraDB18Home1 ] 18.7.1.0.191015 18.8.0.0.191015

[ OraDb11203_home2 ] 11.2.0.3.15 No-update

Note:

– Stop all databases and clusterware resource before patch ILOM in both servers

6 – Update ILOM

I prefer to first update ILOM in case there is a problem during the patch I can access the environment via ILOM.

If you want to update ILOM with the patch, it may be your choice as well.

Note: Upgrade ILOM: Both ILOM nodes, but not in the same time.

Download ILOM Patch separately: Sun Server X5-8 (ILOM 5.0.0.20 133445)

Custom File Name: p30802633_300_Generic.zip

The firmware are inside of this .zip file in the following path:

C:\Users\andre.ontalba\Downloads\p30802633_300_Generic\Oracle_Server_X5-8-3.0.0.91223-FIRMWARE_PACK\Firmware\service-processor\ILOM-5_0_0_20_r133445-Oracle_Server_X5-4_X5-8.pkg

You must copy this file to the following directory in the server to use for transfer this package. In my case dutsLinux1.

/patch/ILOM

This file must be owned by “oracle” user and group “oinstall“

Connect to the gateway machine: dutsLinux1

ssh xxxx@dutsLinux1

Connect to the first ILOM from the gateway machine:

ssh root@odax58duts1-ilom

Enter remote user password: **********

Check current version:

e.g:

Type:

-> version

SP firmware 4.0.4.40

SP firmware build number: 130079

SP firmware date: Thu May 07 09:54:31 CST 2019

SP filesystem version: 0.2.10

Stopping DOM0 and ODA_BASE: (Connect to the ILOM):

stop /SYS

Check status:

show /SYS

e.g: Section “Properties –> power_state = Off”

  Properties:
      type = Host System
      ipmi_name = /SYS
      product_name = SUN SERVER X5-8
      product_part_number = XXXXXXXXXXXXXXX
      product_serial_number = XXXXXXXXXXXXXXX
      product_manufacturer = Oracle Corporation
      fault_state = OK
      clear_fault_action = (none)
      power_state = Off

Load new ILOM image:

load -source scp://xxxx@dutsLinux1/ILOM/ ILOM-5_0_0_20_r133445-Oracle_Server_X5-4_X5-8.pkg

e.g:

load -source scp://root@dutsLinux1/ILOM/ ILOM-5_0_0_20_r133445-Oracle_Server_X5-4_X5-8.pkg

Enter remote user password: **********

NOTE: An upgrade takes several minutes to complete. ILOM

will enter a special mode to load new firmware. No

other tasks can be performed in ILOM until the

firmware upgrade is complete and ILOM is reset.

You can choose to postpone the server BIOS upgrade until the

next server poweroff. If you do not do that, you should

perform a clean shutdown of the server before continuing.

Are you sure you want to load the specified file (y/n)? y

Preserve existing SP configuration (y/n)? y

Preserve existing BIOS configuration (y/n)? y

Delay BIOS upgrade until next server poweroff or reset (y/n)? n

...

After the automatic reboot performed in the ILOM, you can validate the new firmware and BIOS version:

version

Hostname: odax58duts1-ilom

-> version

SP firmware 5.0.0.20

SP firmware build number: 133445

SP firmware date: Thr Feb 06 09:54:31 CST 2020

SP filesystem version: 0.2.10

show /SYS/MB/BIOS

/SYS/MB/BIOS

  Targets:
  Properties:
      type = BIOS
      ipmi_name = MB/BIOS
      fru_name = SYSTEM BIOS
      fru_manufacturer = AMERICAN MEGATRENDS
      fru_version = 25080100
      fru_part_number = APTIO

Start the DOM0 and ODA_BASE after upgrade ILOM:

start /SYS

exit

Note: Repeat this procedure on the second node

7 – Upgrade ODA Servers

Note: Run only from first server and make sure that is the master node: oakcli show ismaster ⇒ “OAKD is in Master Mode”

oakcli update -patch 18.8.0.0.0 –server

e.g:

[root@odax58duts2 patchODA2020]# oakcli update -patch 18.8.0.0.0 –server

This procedure can take between 2 to 3 hours to execute on both nodes.

INFO: DB, ASM, Clusterware may be stopped during the patch if required

INFO: Both Nodes may get rebooted automatically during the patch if required

Do you want to continue: [Y/N]?: y

INFO: User has confirmed for the reboot

INFO: Patch bundle must be unpacked on the second Node also before applying the patch

Did you unpack the patch bundle on the second Node? : [Y/N]? : y

INFO: All the VMs except the ODABASE will be shutdown forcefully if needed

Do you want to continue : [Y/N]? : y

To be able to monitor the application of the patch we can see the logs in the directories below

Log file directory node 1: /opt/oracle/oak/log/odax58duts1/patch/18.8.0.0.0/

Log file directory node 2: /opt/oracle/oak/log/odax58duts2/patch/18.8.0.0.0/

Let’s check how is the server update

Node 1

[root@odax58duts1 patchODA2020]# oakcli show version -detail

INFO: 2020-06-06 17:17:34: Reading the metadata file now...

Component Name Installed Version Proposed Patch Version

--------------- ------------------ -----------------

Controller_INT 11.05.03.00 Up-to-date

Controller_EXT 11.05.03.00 Up-to-date

Expander 001E Up-to-date

SSD_SHARED 944A Up-to-date

HDD_LOCAL A7E0 Up-to-date

HDD_SHARED A7E0 Up-to-date

ILOM 5.0.0.20 r133445 Up-to-date

BIOS 25080100 Up-to-date

IPMI 1.8.15.0 Up-to-date

HMP 2.4.5.0.1 Up-to-date

OAK 18.8.0.0.0 Up-to-date

OL 6.10 Up-to-date

OVM 3.4.4 Up-to-date

GI_HOME 18.8.0.0.191015 Up-to-date

DB_HOME {

[ OraDb12102_home1 ] 12.1.0.2.190716 12.1.0.2.191015

[ OraDb11204_home1 ] 11.2.0.4.190716 11.2.0.4.191015

[ OraDB18Home1 ] 18.7.1.0.191015 18.8.0.0.191015

[ OraDb11203_home2 ] 11.2.0.3.15 No-update

Node 2

[root@odax58duts2 patchODA2020]# oakcli show version -detail

INFO: 2020-06-06 17:18:39: Reading the metadata file now...

Component Name Installed Version Proposed Patch Version

--------------- ------------------ -----------------

Controller_INT 11.05.03.00 Up-to-date

Controller_EXT 11.05.03.00 Up-to-date

Expander 001E Up-to-date

SSD_SHARED 944A Up-to-date

HDD_LOCAL A7E0 Up-to-date

HDD_SHARED A7E0 Up-to-date

ILOM 5.0.0.20 r133445 Up-to-date

BIOS 25080100 Up-to-date

IPMI 1.8.15.0 Up-to-date

HMP 2.4.5.0.1 Up-to-date

OAK 18.8.0.0.0 Up-to-date

OL 6.10 Up-to-date

OVM 3.4.4 Up-to-date

GI_HOME 18.8.0.0.191015 Up-to-date

DB_HOME {

[ OraDb12102_home1 ] 12.1.0.2.190716 12.1.0.2.191015

[ OraDb11204_home1 ] 11.2.0.4.190716 11.2.0.4.191015

[ OraDB18Home1 ] 18.7.1.0.191015 18.8.0.0.191015

[ OraDb11203_home2 ] 11.2.0.3.15 No-update

8 – ODA Patch: Database Binaries

Now is time to apply patch on the Oracle Database Binaries (11.2.0.4,12.1,12.2,18.).

Get the database list and Oracle Home before patch:

oakcli show databases

First step is stop TFA (Both Nodes with root)

tfactl stop

To apply the patch in the Oracle Binaries: (Execute Only From First Node)

oakcli update -patch 18.8.0.0.0 --database

[root@odax58duts1 18.8.0.0.0]# oakcli update -patch 18.8.0.0.0 --database

INFO: Running pre-install scripts

INFO: Running prepatching on node 0

INFO: Running prepatching on node 1

INFO: Completed pre-install scripts

...

INFO: 2020-06-06 18:51:24: ------------------Patching DB-------------------------

INFO: 2020-06-06 18:51:24: Getting all the possible Database Homes for patching

...

INFO: 2020-06-06 18:52:03: Patching 11.2.0.4 Database Homes on the Node odax58duts1

Found the following 11.2.0.4 homes possible for patching:

HOME_NAME HOME_LOCATION

--------- -------------

OraDb11204_home1 /u01/app/oracle/product/11.2.0.4/dbhome_1

[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]

Would you like to patch all the above homes: Y | N ? : Y

INFO: 2020-06-06 18:52:15: Updating OPATCH

Verifying Opatch version for home:</u01/app/oracle/product/11.2.0.4/dbhome_1>.

Expecting version:<11.2.0.3.22>

Opatch version on node <odax58duts1> is <11.2.0.3.22>

Opatch version on node <odax58duts2> is <11.2.0.3.22>

INFO: 2020-06-06 18:53:41: Performing the conflict checks...

SUCCESS: 2020-06-06 18:53:53: Conflict checks passed for all the Homes

INFO: 2020-06-06 18:53:53: Checking if the patch is already applied on any of the Homes

INFO: 2020-06-06 18:53:58: Home is not Up-to-date

SUCCESS: 2020-06-06 18:53:59: Successfully stopped the Database consoles

SUCCESS: 2020-06-06 18:54:06: Successfully stopped the EM agents

INFO: 2020-06-06 18:54:11: Applying the patch on oracle home : /u01/app/oracle/product/11.2.0.4/dbhome_1 ...

SUCCESS: 2020-06-06 18:56:20: Successfully applied the patch on the Home : /u01/app/oracle/product/11.2.0.4/dbhome_1

SUCCESS: 2020-06-06 18:56:20: Successfully started the Database consoles

SUCCESS: 2020-06-06 18:56:20: Successfully started the EM Agents

INFO: 2020-06-06 18:56:23: Patching 11.2.0.4 Database Homes on the Node odax58duts2

...

INFO: 2020-06-06 19:00:02: Patching 12.1.0.2 Database Homes on the Node odax58duts1

Found the following 12.1.0.2 homes possible for patching:

HOME_NAME HOME_LOCATION

--------- -------------

OraDb12102_home1 /u01/app/oracle/product/12.1.0.2/dbhome_1

[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]

Would you like to patch all the above homes: Y | N ? : Y

INFO: 2020-06-06 19:06:38: Updating OPATCH

Verifying Opatch version for home:</u01/app/oracle/product/12.1.0.2/dbhome_1>.

Expecting version:<12.2.0.1.18>

Opatch version on node <odax58duts1> is <12.2.0.1.18>

Opatch version on node <odax58duts2> is <12.2.0.1.18>

INFO: 2020-06-06 19:07:37: Rolling back patches on 12.1.0.2.x home if required...

INFO: 2020-06-06 19:07:43: Checking if any patches need to be rolled back on </u01/app/oracle/product/12.1.0.2/dbhome_1>

INFO: 2020-06-06 19:11:35: Performing the conflict checks...

SUCCESS: 2020-06-06 19:11:59: Conflict checks passed for all the Homes

INFO: 2020-06-06 19:11:59: Checking if the patch is already applied on any of the Homes

INFO: 2020-06-06 19:12:11: Home is not Up-to-date

SUCCESS: 2020-06-06 19:12:13: Successfully stopped the Database consoles

SUCCESS: 2020-06-06 19:12:19: Successfully stopped the EM agents

INFO: 2020-06-06 19:12:25: Applying patch on /u01/app/oracle/product/12.1.0.2/dbhome_1 Homes

INFO: 2020-06-06 19:12:25: It may take upto 15 mins. Please wait...

SUCCESS: 2020-06-06 19:17:19: Successfully applied the patch on the Home : /u01/app/oracle/product/12.1.0.2/dbhome_1

SUCCESS: 2020-06-06 19:17:19: Successfully started the Database consoles

SUCCESS: 2020-06-06 19:17:19: Successfully started the EM Agents

INFO: 2020-06-06 19:17:23: Patching 12.1.0.2 Database Homes on the Node odax58duts2

...

INFO: 2020-06-06 19:27:08: Patching 18.0.0.0 Database Homes on the Node odax58duts1

Found the following 18.0.0.0 homes possible for patching:

HOME_NAME HOME_LOCATION

--------- -------------

OraDB18Home1 /u01/app/oracle/product/18.0.0.0

[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]

Would you like to patch all the above homes: Y | N ? : Y

INFO: 2020-06-06 19:27:15: Updating OPATCH

Verifying Opatch version for home:</u01/app/oracle/product/18.0.0.0>.

Expecting version:<12.2.0.1.18>

Opatch version on node <odax58duts1> is <12.2.0.1.18>

Opatch version on node <odax58duts2> is <12.2.0.1.18>

INFO: 2020-06-06 19:27:26: Rolling back patches on 18.x home if required...

INFO: 2020-06-06 19:27:33: Checking if any patches need to be rolled back on </u01/app/oracle/product/18.0.0.0>

INFO: 2020-06-06 19:28:57: Performing the conflict checks...

SUCCESS: 2020-06-06 19:29:57: Conflict checks passed for all the Homes

INFO: 2020-06-06 19:29:57: Checking if the patch is already applied on any of the Homes

INFO: 2020-06-06 19:30:36: Home is not Up-to-date

SUCCESS: 2020-06-06 19:30:38: Successfully stopped the Database consoles

SUCCESS: 2020-06-06 19:30:44: Successfully stopped the EM agents

INFO: 2020-06-06 19:30:49: Applying patch on /u01/app/oracle/product/18.0.0.0 Homes

INFO: 2020-06-06 19:30:49: It may take up to 15 mins. Please wait...

SUCCESS: 2020-06-06 19:40:34: Successfully applied the patch on the Home : /u01/app/oracle/product/18.0.0.0

SUCCESS: 2020-06-06 19:40:34: Successfully started the Database consoles

SUCCESS: 2020-06-06 19:40:34: Successfully started the EM Agents

INFO: 2020-06-06 19:40:37: Patching 18.0.0.0 Database Homes on the Node odax58duts2

INFO: DB patching summary on node: odax58duts1

SUCCESS: 2020-06-06 19:52:28: Successfully applied the patch on the Home /u01/app/oracle/product/11.2.0.4/dbhome_1

SUCCESS: 2020-06-06 19:52:28: Successfully applied the patch on the Home /u01/app/oracle/product/12.1.0.2/dbhome_1

SUCCESS: 2020-06-06 19:52:28: Successfully applied the patch on the Home /u01/app/oracle/product/18.0.0.0

INFO: DB patching summary on node: odax58duts2

SUCCESS: 2020-06-06 19:52:28: Successfully applied the patch on the Home /u01/app/oracle/product/11.2.0.4/dbhome_1

SUCCESS: 2020-06-06 19:52:28: Successfully applied the patch on the Home /u01/app/oracle/product/12.1.0.2/dbhome_1

SUCCESS: 2020-06-06 19:52:28: Successfully applied the patch on the Home /u01/app/oracle/product/18.0.0.0

INFO: Executing /tmp/pending_actions on both nodes

You have new mail in /var/spool/mail/root

[root@odax58duts1 18.8.0.0.0]#

Let’s check how is the database update

Node 1

[root@odax58duts1 patchODA2020]# oakcli show version -detail

INFO: 2020-06-06 19:55:34: Reading the metadata file now...

Component Name Installed Version Proposed Patch Version

--------------- ------------------ -----------------

Controller_INT 11.05.03.00 Up-to-date

Controller_EXT 11.05.03.00 Up-to-date

Expander 001E Up-to-date

SSD_SHARED 944A Up-to-date

HDD_LOCAL A7E0 Up-to-date

HDD_SHARED A7E0 Up-to-date

ILOM 5.0.0.20 r133445 Up-to-date

BIOS 25080100 Up-to-date

IPMI 1.8.15.0 Up-to-date

HMP 2.4.5.0.1 Up-to-date

OAK 18.8.0.0.0 Up-to-date

OL 6.10 Up-to-date

OVM 3.4.4 Up-to-date

GI_HOME 18.8.0.0.191015 Up-to-date

DB_HOME {

[ OraDb12102_home1 ] 12.1.0.2.191015 Up-to-date

[ OraDb11204_home1 ] 11.2.0.4.191015 Up-to-date

[ OraDB18Home1 ] 18.8.0.0.191015 Up-to-date

[ OraDb11203_home2 ] 11.2.0.3.15 No-update

}

Node 2

[root@odax58duts2 patchODA2020]# oakcli show version -detail

INFO: 2020-06-06 19:56:39: Reading the metadata file now...

Component Name Installed Version Proposed Patch Version

--------------- ------------------ -----------------

Controller_INT 11.05.03.00 Up-to-date

Controller_EXT 11.05.03.00 Up-to-date

Expander 001E Up-to-date

SSD_SHARED 944A Up-to-date

HDD_LOCAL A7E0 Up-to-date

HDD_SHARED A7E0 Up-to-date

ILOM 5.0.0.20 r133445 Up-to-date

BIOS 25080100 Up-to-date

IPMI 1.8.15.0 Up-to-date

HMP 2.4.5.0.1 Up-to-date

OAK 18.8.0.0.0 Up-to-date

OL 6.10 Up-to-date

OVM 3.4.4 Up-to-date

GI_HOME 18.8.0.0.191015 Up-to-date

DB_HOME {

[ OraDb12102_home1 ] 12.1.0.2.191015 Up-to-date

[ OraDb11204_home1 ] 11.2.0.4.191015 Up-to-date

[ OraDB18Home1 ] 18.8.0.0.191015 Up-to-date

[ OraDb11203_home2 ] 11.2.0.3.15 No-update

9- Apply DATAPATCH/CATBUNDLE in databases 11.2 / 12.1 / 12.2 and 18.8

export NLS_LANG=AMERICAN_AMERICA.AL32UTF8

$ORACLE_HOME/OPatch/datapatch -verbose

Note: It is required set up the NLS_LANG to “AMERICAN_AMERICA.AL32UTF8” variable in order to avoid a BUG during the DATAPATCH in the Oracle Database 12.1

Reference Documents:

Oracle Database Appliance – 18.2, 12.X, and 2.X Supported ODA Versions & Known Issues (Doc ID 888888.1)

https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.8/cmtxn/patching-oda.html#GUID-49F5F510-3A38-4E6A-B915-FCBCD36CDDDB

I hope I helped with this ODA upgrade procedure

Andre Ontalba

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”

DB_UNIQUE_NAME, PDB, and Data Guard

Category: Database Author: Fernando Simon (Board Member) Date: 6 years ago Comments: 0

DB_UNIQUE_NAME, PDB, and Data Guard

When you change the parameters for the database is possible to specify the db_unique_name and allow more control where you want to apply/use it. This is very useful to limit the scope, but you need to be aware of some collateral effects. Even not present at the official doc, you can use it. But check here some details that you need to take care of.

Unplug and Plug

The environment below is a simple 19c single instance database. There, exists 3 PDB’s and you can see the db_unique_name for this CDB:

SQL> show pdbs




    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

         2 PDB$SEED                       READ ONLY  NO

         3 PDBDG                          READ WRITE NO

         5 PDBMOVE                        READ WRITE NO

SQL> show parameter db_unique_name;




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_unique_name                       string      or19dg

SQL>

And we can connect in the PDBMOVE and change the parameters. First, as usual, normal set without any special parameter:

SQL> alter system set open_cursors = 300 scope = spfile;




System altered.




SQL>

But, we can specify the db_unique_name for alter system. ALTER SYSTEM… DB_UNIQUE_NAME = ‘VALUE’. Here you can see that I used the same that as defined for root level.

SQL> alter system set sessions = 100  scope = spfile db_unique_name = 'or19dg';




System altered.




SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'or19dg';




System altered.




SQL>

Detail #1 here, if you specify the DB_UNIQUE_NAME the scope can be JUST spfile. Otherwise you will receive error.

SQL> alter system set cursor_sharing = FORCE scope = both db_unique_name = 'or19dg';

alter system set cursor_sharing = FORCE scope = both db_unique_name = 'or19dg'

*

ERROR at line 1:

ORA-65147: DB_UNIQUE_NAME specified without SPFILE scope

SQL>

In this example I set two parameters for PDB level and you can see at PDB_SPFILE$ inside of root cdb that values was set:

SQL> alter session set container = cdb$root;




Session altered.




SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2549618825;




DB_UNIQ_NAME                   NAME                           VALUE$                                      PDB_UID

------------------------------ ------------------------------ ---------------------------------------- ----------

*                              open_cursors                   300                                      2549618825

or19dg                         cursor_sharing                 'FORCE'                                  2549618825

or19dg                         sessions                       100                                      2549618825




SQL>

And after the reload of the database the values are set:

[oracle@orcl19p ~]$ srvctl stop database -d or19dg

[oracle@orcl19p ~]$ srvctl start database -d or19dg

[oracle@orcl19p ~]$

[oracle@orcl19p ~]$ sqlplus / as sysdba




SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 1 17:33:29 2020

Version 19.5.0.0.0




Copyright (c) 1982, 2019, Oracle.  All rights reserved.







Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.5.0.0.0




SQL> show pdbs




    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

         2 PDB$SEED                       READ ONLY  NO

         3 PDBDG                          READ WRITE NO

         5 PDBMOVE                        READ WRITE NO

SQL> alter session set container = PDBMOVE;




Session altered.




SQL> show parameter cursor_sharing;




NAME                                 TYPE                             VALUE

------------------------------------ -------------------------------- ------------------------------

cursor_sharing                       string                           FORCE

SQL>

UNPLUG

But if I need to unplug the PDB, the values will be exported too, right? Yes, but no as you imagine.

So, doing a simple unplug to XML:

SQL> alter pluggable database PDBMOVE close immediate;




Pluggable database altered.




SQL> alter pluggable database PDBMOVE unplug into '/tmp/pdbmove_par.xml';




Pluggable database altered.




SQL> drop pluggable database PDBMOVE keep datafiles;




Pluggable database dropped.




SQL>

And checking the file for these parameters at generated XML:

[oracle@orcl19p ~]$ cat /tmp/pdbmove_par.xml |grep open_cursors

      <spfile>*.open_cursors=300#HWM:300,</spfile>

[oracle@orcl19p ~]$ cat /tmp/pdbmove_par.xml |grep cursor_sharing

      <spfile>*.cursor_sharing='FORCE'</spfile>

[oracle@orcl19p ~]$ cat /tmp/pdbmove_par.xml |grep sessions

      <spfile>*.sessions=100#HWM:100,flag:1,</spfile>

[oracle@orcl19p ~]$

As you can see here, the parameters was exported, but the parameter changed to “*”. The definition to specific db_unique_name was cleared.

And if we try to plug again the same PDB, we can see that parameter was loaded as “*”:

SQL> create  pluggable database PDBMOVE USING '/tmp/pdbmove_par.xml' NOCOPY TEMPFILE REUSE;




Pluggable database created.




SQL> col name format a30

SQL> col VALUE$ format a40

SQL> set linesize 250

SQL> select CON_ID,DBID,CON_UID,GUID,NAME from v$pdbs;




    CON_ID       DBID    CON_UID GUID                             NAME

---------- ---------- ---------- -------------------------------- ------------------------------

         2  826464235  826464235 9B151A78B6DB533AE0533205A00AFC30 PDB$SEED

         3 1340021208 1340021208 9B155010BEAC661BE0533205A00AF21B PDBDG

         4 2549618825 3364812106 A7082B6C610C5E1DE0533205A00AF7FE PDBMOVE




SQL> show pdbs




    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

         2 PDB$SEED                       READ ONLY  NO

         3 PDBDG                          READ WRITE NO

         4 PDBMOVE                        MOUNTED

SQL> alter pluggable database PDBMOVE open;




Pluggable database altered.




SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 3364812106;




DB_UNIQ_NAME                   NAME                           VALUE$                                      PDB_UID

------------------------------ ------------------------------ ---------------------------------------- ----------

*                              cursor_sharing                 'FORCE'                                  3364812106

*                              open_cursors                   300                                      3364812106

*                              sessions                       100                                      3364812106




SQL>

DATAGUARD

As you can imagine, using the db_unique_name for an alter system can affect the dataguard environments. If you know, the db_unique_name is different between primary and standby. So, if you define some parameter, depending on the way that you made, after you, switchover/failover can occur that parameter returns to the default value.

It is not the case that this is wrong, maybe you need to set some parameters for one server/side in specific. Maybe because of less memory, less CPU power, or whatever the reason.

In this example, I have gold19c as primary, and golds19c as standby. And I have the GOLD19P as used PDB here:

[oracle@goldpn1 ~]$ dgmgrl sys/oracle@gold19c

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Mon Jun 1 19:13:39 2020

Version 19.7.0.0.0




Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.




Welcome to DGMGRL, type "help" for information.

Connected to "GOLD19C"

Connected as SYSDBA.

DGMGRL> show configuration;




Configuration - gold19c




  Protection Mode: MaxAvailability

  Members:

  gold19c  - Primary database

    golds19c - Physical standby database




Fast-Start Failover:  Disabled




Configuration Status:

SUCCESS   (status updated 72 seconds ago)




DGMGRL> exit

[oracle@goldpn1 ~]$

[oracle@goldpn1 ~]$ sqlplus / as sysdba




SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 1 19:16:50 2020

Version 19.7.0.0.0




Copyright (c) 1982, 2020, Oracle.  All rights reserved.







Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.7.0.0.0




SQL> show parameter db_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_name                              string      gold19c

SQL> show parameter db_unique_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_unique_name                       string      GOLD19C

SQL>

SQL> show pdbs




    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

         2 PDB$SEED                       READ ONLY  NO

         3 GOLD19P                        READ WRITE NO

SQL> col name format a30

SQL> col VALUE$ format a40

SQL> set linesize 250

SQL> select CON_ID,DBID,CON_UID,GUID,NAME from v$pdbs;




    CON_ID       DBID    CON_UID GUID                             NAME

---------- ---------- ---------- -------------------------------- ------------------------------

         2 2661745537 2661745537 A6B13C68753F63D3E0530A07A00A6303 PDB$SEED

         3 2141275600 2141275600 A6B19FBF38306073E0530C07A00ACE20 GOLD19P




SQL>

And to set the parameters I used the same than before, with and without db_unique_name:

SQL> alter session set container = GOLD19P;




Session altered.




SQL> alter system set open_cursors = 300 scope = spfile;




System altered.




SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'GOLD19C';




System altered.




SQL>

And as you can see, the parameter was stored.

SQL> alter session set container = cdb$root;




Session altered.




SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2141275600;




DB_UNIQ_NAME                   NAME                           VALUE$                                      PDB_UID

------------------------------ ------------------------------ ---------------------------------------- ----------

*                              open_cursors                   300                                      2141275600

GOLD19C                        cursor_sharing                 'FORCE'                                  2141275600




SQL>

But if I made the switchover:

[oracle@goldpn1 ~]$ dgmgrl sys/oracle@gold19c

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Mon Jun 1 19:24:15 2020

Version 19.7.0.0.0




Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.




Welcome to DGMGRL, type "help" for information.

Connected to "GOLD19C"

Connected as SYSDBA.

DGMGRL> switchover to golds19c;

Performing switchover NOW, please wait...

Operation requires a connection to database "golds19c"

Connecting ...

Connected to "GOLDS19C"

Connected as SYSDBA.

New primary database "golds19c" is opening...

Oracle Clusterware is restarting database "gold19c" ...

Connected to "GOLD19C"

Connected to "GOLD19C"

Switchover succeeded, new primary is "golds19c"

DGMGRL>

You can see that now, the value for the parameter cursor_sharing now have the default value because the db_unique_name does not hit what was defined:

[oracle@goldsn1 ~]$ sqlplus / as sysdba




SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 1 19:35:01 2020

Version 19.7.0.0.0




Copyright (c) 1982, 2020, Oracle.  All rights reserved.







Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.7.0.0.0




SQL> show parameter db_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_name                              string      gold19c

SQL> show parameter db_unique_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_unique_name                       string      GOLDS19C

SQL>

SQL> alter session set container = GOLD19P;




Session altered.




SQL> show parameter cursor_sharing;




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

cursor_sharing                       string      EXACT

SQL> show parameter open_cursors;




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

open_cursors                         integer     300

SQL>

But you can see that the parameter is defined at PDB_SPFILE$:

SQL> alter session set container = cdb$root;




Session altered.




SQL>

SQL> col name format a30

SQL> col VALUE$ format a40

SQL> set linesize 250

SQL> select CON_ID,DBID,CON_UID,GUID,NAME from v$pdbs;




    CON_ID       DBID    CON_UID GUID                             NAME

---------- ---------- ---------- -------------------------------- ------------------------------

         2 2661745537 2661745537 A6B13C68753F63D3E0530A07A00A6303 PDB$SEED

         3 2141275600 2141275600 A6B19FBF38306073E0530C07A00ACE20 GOLD19P




SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2141275600;




DB_UNIQ_NAME                   NAME                           VALUE$                                      PDB_UID

------------------------------ ------------------------------ ---------------------------------------- ----------

*                              open_cursors                   300                                      2141275600

GOLD19C                        cursor_sharing                 'FORCE'                                  2141275600




SQL>

As showed, the value at standby is not defied, but as told before this can be expected behaviour. Or no. If you want to set the parameter and it became value in both, primary and standby, you don’t need to specify the db_unique_name, otherwise when the switchover/failover occurs the value will not be there. And, this can lead to some unexpected behaviours (at application side as and example) and need to troubleshoot (and until find this about set parameter, can be a long path).

Others points

The db_unique_name option does not check what you specify, so, take care to set the correct value. As you can see below I defined it as SIMON, and was accepted and saved:

SQL> alter session set container = GOLD19P;

Session altered.

SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'SIMON';

System altered.

SQL> alter session set container = cdb$root;

Session altered.

SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2141275600;

DB_UNIQ_NAME                   NAME                           VALUE$                                      PDB_UID

------------------------------ ------------------------------ ---------------------------------------- ----------

*                              open_cursors                   300                                      2141275600

GOLD19C                        cursor_sharing                 'FORCE'                                  2141275600

SIMON                          cursor_sharing                 'FORCE'                                  2141275600

SQL>

And to reset, the process is similar. As you see below, you can specify the db_unique_name to delete specific. But if you specify nothing, the reset will be done for all.

SQL> alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C';

System altered.

SQL> alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C';

alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C'

*

ERROR at line 1:

ORA-32010: cannot find entry to delete in SPFILE

SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'GOLD19C';

System altered.

SQL> alter system reset cursor_sharing scope = spfile;

System altered.

SQL> alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C';

alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C'

*

ERROR at line 1:

ORA-32010: cannot find entry to delete in SPFILE

SQL>

Conclusion

Sometimes a simple definition can lead to some strange behaviors. Think that your application is running ok and after a switchover change completely the way how it works (cursor_sharing as above). And until you dig the solution can take time.

Db_unique_name for set parameter needs to be used carefully to avoid these cases. It is not the case that you never need to use it, sometimes this definition needs to be used. You can use it to prepare everything BEFORE the switchover to avoid some error or to tune the database since the beginning.

References

Initialization parameters in a Multitenant database – FAQ and Examples (Doc ID 2101638.1)

Initialization parameters in a Multitenant database – Facts and additional information (Doc ID 2101596.1)

2.7 Managing Initialization Parameters Using a Server Parameter File

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”

New Feature – Per-Second Billing for Compute and Autonomous Database.

Category: Cloud Author: Andre Luiz Dutra Ontalba (Board Member) Date: 6 years ago Comments: 0

New Feature - Per-Second Billing for Compute and Autonomous Database.

Oracle this month announced a new form of billing for Compute Instance and Autonomous Database with that the Oracle Cloud has a stronger platform for legacy applications moving to cloud and a new family of cloud native applications that rely on microservices and dynamic scaling.

Compute instances are now billed per second of usage, rather than per hour. This helps you reduce costs when using instances for short amounts of time. Virtual machine (VM) instances have a minimum charge of one minute. Bare metal instances have a minimum charge of one hour. After the first minute (for VMs) or the first hour (for bare metal instances), usage is billed in one-second increments.

With this billing model, usage of Compute and Autonomous Database is billed per-second. All prices continue to be quoted on an hourly basis.

Here are some details about this billing model:

All virtual machine (VM) Compute instances, including those with graphical processing unit (GPU) chips, are now billed per-second with a one-minute minimum.

All bare metal Compute instances, including those with GPU chips, are now billed per-second with a one-hour minimum.

Autonomous Data Warehouse and Autonomous Transaction Processing usage is now billed per-second with a one-minute minimum.

Windows OS images are now billed per-second with a one-minute minimum.

Microsoft SQL Server images available in the Oracle Cloud Marketplace are now billed per-second with a 744-hour (one month) minimum.

The workloads that will see the biggest impact from this change are those that stop and start frequently, and those that run for short durations. Here are some examples of significant benefits:

Application and database development
High-performance computing
Continuous integration/continuous development(CI/CD)
Big boost to Cloud Native applications – Microservices & Dynamic Scaling

I believe that with this, Oracle increasingly invests and provides more resources and different ways for its customers to be able to plan and prepare their environments for the cloud.

Reference: https://docs.cloud.oracle.com/

I hope this short article has helped.

Andre Ontalba

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”

Exadata and ZDLRA, Disable HAIP

Category: Engineer System Author: Fernando Simon (Board Member) Date: 6 years ago Comments: 0

Exadata and ZDLRA, Disable HAIP

HAIP (High Availability IP) is not supported for the Exadata environment but can occur (if you did not create the cluster using OEDA) that HAIP became in use. And this particularity true for ZDLRA. So, during the upgrade from the previous version (12.2) to a higher version, it is needed to remove HAIP.

Usually, when we upgrading from 12.2 to 18c the HAIP is removed from Exadata. If the upgrade is from 12.1, and HAIP is there, it continues and is not removed by the upgrade process. If you are using HAIP and your GI is 12.1, this procedure as-is described here can’t be used (need some adaptation), because of some requirements from ASM+ACFS+DB. But since this is a preliminary step from a GI upgrade, the focus is to disable and remove it from GI.

The HAIP is not needed for Exadata because by architecture the InfiniBand network already defines (per server) two IP’s to avoid the single point of failure. So, it is not needed to create an additional layer (HAIP and virtual IP), that does the same that already exists by network design.

*Image removed from Oracle Presentation: Oracle Real Application Clusters 19c- Best Practices and Internals- EMEA Tour Oct 2019

This procedure can be used for a non-Exadata environment too. But before continue it is important to be aware of some details with ACFS. If you use ACFS, be careful since the HAIP IP is used in the ASM proxy and needs to be updated; and if ACFS is in place, this procedure is valid just for GI 12.2+.

Environment

In this scenario, I am removing HAIP from ZDLRA GI. The HAIP has configured automatically and we are using ACFS.

The focus is to disable HAIP and remove it from CRS. Another detail is that we need to remove the dependencies between the CRS resources (like ASM that need HAIP) too.

One important detail is that since we are removing the HAIP, we need to swap the HAIP IP that it is used by services (like ASM Proxy) for something that exists. So, you need to have high availability IP’s to use. For Exadata, we use the interconnect IP’s, but if you are using in another environment, be careful that your network has the needed requirements (throughput, failover, and others).

GI Upgrade

During the GI upgrade from 12.2 to 19c we need to run the runcluvfy and it will detect if HAIP is enabled:

....

....

Checks did not pass for the following nodes:

        zeroserv02,zeroserv01







Failures were encountered during execution of CVU verification request "stage -pre crsinst".




Verifying Node Connectivity ...FAILED

zeroserv02: PRVG-11068 : Highly Available IP (HAIP) is enabled on the nodes

            "zeroserv01,zeroserv02".




zeroserv01: PRVG-11068 : Highly Available IP (HAIP) is enabled on the nodes

            "zeroserv01,zeroserv02".




Verifying RPM Package Manager database ...INFORMATION

PRVG-11250 : The check "RPM Package Manager database" was not performed because

it needs 'root' user privileges.







CVU operation performed:      stage -pre crsinst

Date:                         Dec 9, 2019 1:38:46 PM

CVU home:                     /u01/app/19.0.0.0/grid/

User:                         oracle

[root@zeroserv01 ~]#

Remove HAIP

The steps need to be executed in order to avoid errors during the procedure. Unfortunately, it is needed to have one maintenance window due to the unavailability of the services. The running databases need to be shutdown due to the CRS restart by an example.

Bellow, the steps are summarized and explained to be followed. Please be aware of the IP’s involved (they will be different in your case).

1 – Shutdown services

So, this first point is to shutdown all databases running in this cluster:

[oracle@zeroserv01 ~]$ srvctl stop database -d zdlras -o immediate

[oracle@zeroserv01 ~]$

2- Check HAIP, ASM Proxy, and ACFS

It is needed to check if HAIP is enabled. Another detail is check for ASM proxy, and ACFS too:

[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/crsctl stat res ora.cluster_interconnect.haip -t -init

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.cluster_interconnect.haip

      1        ONLINE  ONLINE       zeroserv01               STABLE

--------------------------------------------------------------------------------

[oracle@zeroserv01 ~]$

[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/crsctl stat res ora.proxy_advm -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.proxy_advm

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

--------------------------------------------------------------------------------

[oracle@zeroserv01 ~]$

[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/crsctl stat res -w "TYPE = ora.acfs.type" -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.catalog.raadmin.acfs

               ONLINE  ONLINE       zeroserv01               mounted on /raacfs/r

                                                             aadmin,STABLE

               ONLINE  ONLINE       zeroserv02               mounted on /raacfs/r

                                                             aadmin,STABLE

ora.catalog.raosbadmin.acfs

               ONLINE  ONLINE       zeroserv01               mounted on /osbcat,S

                                                             TABLE

               ONLINE  ONLINE       zeroserv02               mounted on /osbcat,S

                                                             TABLE

--------------------------------------------------------------------------------

[oracle@zeroserv01 ~]$

3 – Check New IP

The next step is to check the new IP that will be used. Since ASM does not use HAIP, we can (at Exadata) pickup the same used IP:

[oracle@zeroserv01 ~]$ export ORACLE_SID=+ASM1

[oracle@zeroserv01 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid

[oracle@zeroserv01 ~]$ export PATH=$ORACLE_HOME/bin:$PATH

[oracle@zeroserv01 ~]$

[oracle@zeroserv01 ~]$ sqlplus / as sysasm




SQL*Plus: Release 12.2.0.1.0 Production on Mon Dec 9 15:00:55 2019




Copyright (c) 1982, 2016, Oracle.  All rights reserved.







Connected to:

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production




SQL> show parameter cluster_interconnects




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

cluster_interconnects                string      192.168.10.1:192.168.10.2

SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

[oracle@zeroserv01 ~]$

[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/oifcfg getif

bondeth0  10.208.68.0  global  public

ib0  192.168.8.0  global  cluster_interconnect,asm

ib1  192.168.8.0  global  cluster_interconnect,asm

[oracle@zeroserv01 ~]$

In this case, the IP that will substitute HAIP IP is 192.168.10.1/192.168.10.2. And, as you can see above, they are from the interconnect network.

4 – Fixing IP for ASM Proxy

The next step is to fix the IP that is used by the ASM Proxy instance. But default, it pickup the HAIP IP during the startup. The idea here is to force the same IP than the ASM instance for cluster_interconnect parameter.

Connecting at APX instance

Connect at APX instance, check the parameters (and see that are without values). After that we create one init file from memory to have the backup:

[oracle@zeroserv01 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid

[oracle@zeroserv01 ~]$ export PATH=$ORACLE_HOME/bin:$PATH

[oracle@zeroserv01 ~]$ export ORACLE_SID=+APX1

[oracle@zeroserv01 ~]$

[oracle@zeroserv01 ~]$ sqlplus / as sysasm




SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:20:59 2019




Copyright (c) 1982, 2016, Oracle.  All rights reserved.







Connected to:

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production




SQL>  show parameter instance_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

instance_name                        string      +APX1

SQL>  show parameter instance_type




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

instance_type                        string      ASMPROXY

SQL> show parameter cluster_interconnects




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

cluster_interconnects                string

SQL> create pfile = '/tmp/pfileapx1' from memory;




File created.




SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

[oracle@zeroserv01 ~]$

Changing the IP

For ASM Proxy instance we need to set the IP for cluster_interconnect parameter with the same IP used by ASM instance.

To do that, we need to set it in init for ASM Proxy. But since the ASM Proxy instance doesn’t have one, we create one at $GI_HOME/dbs with the connect name (look below the folder and instance name). Below I set just the cluster to interconnect parameter and dedicated to the instance running in this node.

[oracle@zeroserv01 ~]$ echo "+APX1.cluster_interconnects='192.168.10.1:192.168.10.2'" > $ORACLE_HOME/dbs/init+APX1.ora

[oracle@zeroserv01 ~]$

[oracle@zeroserv01 ~]$ cat $ORACLE_HOME/dbs/init+APX1.ora

+APX1.cluster_interconnects='192.168.10.1:192.168.10.2'

[oracle@zeroserv01 ~]$

Restart the ASM Proxy

After creating the init, we restart the ASM Proxy in this node.

[root@zeroserv01 ~]# export ORACLE_HOME=/u01/app/12.2.0.1/grid

[root@zeroserv01 ~]# export PATH=$ORACLE_HOME/bin:$PATH

[root@zeroserv01 ~]#

[root@zeroserv01 ~]# $ORACLE_HOME/bin/srvctl stop asm -proxy -node zeroserv01 -force

[root@zeroserv01 ~]#

[root@zeroserv01 ~]# $ORACLE_HOME/bin/srvctl start asm -proxy -node zeroserv01

[root@zeroserv01 ~]#

[root@zeroserv01 ~]# ps -ef |grep APX

oracle   267811      1  0 10:43 ?        00:00:00 apx_pmon_+APX1

oracle   267813      1  0 10:43 ?        00:00:00 apx_clmn_+APX1

oracle   267815      1  0 10:43 ?        00:00:00 apx_psp0_+APX1

oracle   267820      1  1 10:43 ?        00:00:00 apx_vktm_+APX1

oracle   267824      1  0 10:43 ?        00:00:00 apx_gen0_+APX1

oracle   267826      1  0 10:43 ?        00:00:00 apx_mman_+APX1

oracle   267830      1  0 10:43 ?        00:00:00 apx_gen1_+APX1

oracle   267834      1  0 10:43 ?        00:00:00 apx_diag_+APX1

oracle   267836      1  0 10:43 ?        00:00:00 apx_dskm_+APX1

oracle   267838      1  0 10:43 ?        00:00:00 apx_pman_+APX1

oracle   267840      1  0 10:43 ?        00:00:00 apx_dia0_+APX1

oracle   267842      1  0 10:43 ?        00:00:00 apx_lreg_+APX1

oracle   267845      1  0 10:43 ?        00:00:00 apx_pxmn_+APX1

oracle   267847      1  0 10:43 ?        00:00:00 apx_rbal_+APX1

oracle   267849      1  0 10:43 ?        00:00:00 apx_vbg0_+APX1

oracle   267851      1  0 10:43 ?        00:00:00 apx_vdbg_+APX1

oracle   267853      1  0 10:43 ?        00:00:00 apx_vubg_+APX1

root     267979  32720  0 10:43 pts/0    00:00:00 grep --color=auto APX

[root@zeroserv01 ~]#

Look that instance restarted correctly and it is up.

And we can check the parameter if it is OK with the IP:

[oracle@zeroserv01 ~]$ sqlplus / as sysasm




SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:51:59 2019




Copyright (c) 1982, 2016, Oracle.  All rights reserved.







Connected to:

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production




SQL> show parameter cluster_interconnects




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

cluster_interconnects                string      192.168.10.1:192.168.10.2

SQL> show parameter instance_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

instance_name                        string      +APX1

SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

[oracle@zeroserv01 ~]$

If the instance not startup (or the IP’s are wrong) you need to check the init file that you created in $GIHOME/dbs folder.

Restart ACFS

Since we restarted the ASM Proxy instance, the ACFS mountpoints went down and need to be up again:

[root@zeroserv01 ~]# $ORACLE_HOME/bin/crsctl start res -w "TYPE = ora.acfs.type" -n zeroserv01

CRS-2672: Attempting to start 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01'

CRS-2672: Attempting to start 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01'

CRS-2676: Start of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01' succeeded

CRS-2672: Attempting to start 'ora.catalog.raosbadmin.acfs' on 'zeroserv01'

CRS-2676: Start of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01' succeeded

CRS-2672: Attempting to start 'ora.catalog.raadmin.acfs' on 'zeroserv01'

CRS-2676: Start of 'ora.catalog.raosbadmin.acfs' on 'zeroserv01' succeeded

CRS-2676: Start of 'ora.catalog.raadmin.acfs' on 'zeroserv01' succeeded

[root@zeroserv01 ~]#

I started up using the CRS resource for that.

Other Nodes

After it is Ok for node 1, we can do it for other nodes. Here, since I made for ZDLRA, I have just the second node to fix ASM Proxy. Be careful with the IP’s used, they are specific for this node and remember to use the correct init file.

[root@zeroserv02 ~]# su - oracle

Last login: Tue Dec 10 10:53:02 CET 2019

[oracle@zeroserv02 ~]$ export ORACLE_SID=+ASM2

[oracle@zeroserv02 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid

[oracle@zeroserv02 ~]$ export PATH=$ORACLE_HOME/bin:$PATH

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$ sqlplus / as sysasm




SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:56:24 2019




Copyright (c) 1982, 2016, Oracle.  All rights reserved.







Connected to:

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production




SQL> show parameter cluster_interconnects




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

cluster_interconnects                string      192.168.10.3:192.168.10.4

SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid

[oracle@zeroserv02 ~]$ export PATH=$ORACLE_HOME/bin:$PATH

[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX1

[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX2

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$ sqlplus / as sysasm




SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:57:15 2019




Copyright (c) 1982, 2016, Oracle.  All rights reserved.







Connected to:

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production




SQL>

SQL> show parameter instance_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

instance_name                        string      +APX2

SQL> show parameter cluster_interconnects




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

cluster_interconnects                string

SQL> create pfile = '/tmp/pfileapx2' from memory;




File created.




SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$ ls -l $ORACLE_HOME/dbs/init?APX*

ls: cannot access /u01/app/12.2.0.1/grid/dbs/init?APX*: No such file or directory

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$ echo "+APX2.cluster_interconnects='192.168.10.3:192.168.10.4'" > $ORACLE_HOME/dbs/init+APX2.ora

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$ cat $ORACLE_HOME/dbs/init+APX2.ora

+APX2.cluster_interconnects='192.168.10.3:192.168.10.4'

[oracle@zeroserv02 ~]$

[oracle@zeroserv02 ~]$ exit   

[root@zeroserv02 ~]# export ORACLE_HOME=/u01/app/12.2.0.1/grid

[root@zeroserv02 ~]# export PATH=$ORACLE_HOME/bin:$PATH

[root@zeroserv02 ~]#

[root@zeroserv02 ~]#

[root@zeroserv02 ~]# $ORACLE_HOME/bin/srvctl stop asm -proxy -node zeroserv02 -force

[root@zeroserv02 ~]#

[root@zeroserv02 ~]#

[root@zeroserv02 ~]# $ORACLE_HOME/bin/srvctl start asm -proxy -node zeroserv02

[root@zeroserv02 ~]#

[root@zeroserv02 ~]#

[root@zeroserv02 ~]#

[root@zeroserv02 ~]# $ORACLE_HOME/bin/crsctl start res -w "TYPE = ora.acfs.type" -n zeroserv02

CRS-2672: Attempting to start 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02'

CRS-2672: Attempting to start 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02'

CRS-2676: Start of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02' succeeded

CRS-2672: Attempting to start 'ora.catalog.raosbadmin.acfs' on 'zeroserv02'

CRS-2676: Start of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02' succeeded

CRS-2672: Attempting to start 'ora.catalog.raadmin.acfs' on 'zeroserv02'

CRS-2676: Start of 'ora.catalog.raosbadmin.acfs' on 'zeroserv02' succeeded

CRS-2676: Start of 'ora.catalog.raadmin.acfs' on 'zeroserv02' succeeded

[root@zeroserv02 ~]#

[root@zeroserv02 ~]#

[root@zeroserv02 ~]# su - oracle

Last login: Tue Dec 10 10:54:10 CET 2019

[oracle@zeroserv02 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid

[oracle@zeroserv02 ~]$ export PATH=$ORACLE_HOME/bin:$PATH

[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX1

[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX2




[oracle@zeroserv02 ~]$ sqlplus / as sysasm




SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 11:00:46 2019




Copyright (c) 1982, 2016, Oracle.  All rights reserved.







Connected to:

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production




SQL> show parameter cluster_interconnects




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

cluster_interconnects                string      192.168.10.3:192.168.10.4

SQL> show parameter instance_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

instance_name                        string      +APX2

SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

You have new mail in /var/spool/mail/oracle

[oracle@zeroserv02 ~]$

5 – ASM Dependency

Since the ASM since the CRS depends on HAIP we need to remove this dependency. Here we have a tricky part. It is needed to completely change the dependencies for ASM, setting it to CRS/CSS directly.

First, at first node we check the current dependency for START and STOP:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res ora.asm -p |grep START_DEPENDENCIES=

START_DEPENDENCIES=hard(ora.ASMNET1LSNR_ASM.lsnr,ora.ASMNET2LSNR_ASM.lsnr) weak(ora.LISTENER.lsnr) pullup(ora.ASMNET1LSNR_ASM.lsnr,ora.ASMNET2LSNR_ASM.lsnr) dispersion:active(site:type:ora.asm.type)

[root@zeroserv01 ~]#

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res ora.asm -p |grep STOP_DEPENDENCIES=

STOP_DEPENDENCIES=hard(intermediate:ora.ASMNET1LSNR_ASM.lsnr,intermediate:ora.ASMNET2LSNR_ASM.lsnr)

[root@zeroserv01 ~]#

And now we change it. Look at the parameters values. They are completely different, and need to be with these specific values:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd) pullup(ora.cssd,ora.ctssd) weak(ora.drivers.acfs)',STOP_DEPENDENCIES='hard(intermediate:ora.cssd)'" -init

[root@zeroserv01 ~]#

If you see closely the HAIP it is not listed there as a dependency, setting with the values, we completely remove HAIP dependency and inheritance.

And we need to do the same for the other node:

[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd) pullup(ora.cssd,ora.ctssd) weak(ora.drivers.acfs)',STOP_DEPENDENCIES='hard(intermediate:ora.cssd)'" -init

[root@zeroserv02 ~]#

6 – Disable HAIP Resource

The next step is to disable the HAIP resource from startup at CRS. We do this in both nodes.

Node 1:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=0" -init

[root@zeroserv01 ~]#

Node 2:

[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=0" -init

[root@zeroserv02 ~]#

But check that we do not stop yet the HAIP resource. It is needed to remain up at this moment.

If we check with the init for CRS, it is still there in node 1:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.asm

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.cluster_interconnect.haip

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.crf

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.crsd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.cssd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.cssdmonitor

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.ctssd

      1        ONLINE  ONLINE       zeroserv01               OBSERVER,STABLE

ora.diskmon

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.drivers.acfs

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.drivers.oka

      1        OFFLINE OFFLINE                               STABLE

ora.evmd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.gipcd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.gpnpd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.mdnsd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.storage

      1        ONLINE  ONLINE       zeroserv01               STABLE

--------------------------------------------------------------------------------

[root@zeroserv01 ~]#

And for the other node too:

[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.asm

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.cluster_interconnect.haip

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.crf

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.crsd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.cssd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.cssdmonitor

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.ctssd

      1        ONLINE  ONLINE       zeroserv02               ACTIVE:0,STABLE

ora.diskmon

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.drivers.acfs

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.drivers.oka

      1        OFFLINE OFFLINE                               STABLE

ora.evmd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.gipcd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.gpnpd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.mdnsd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.storage

      1        ONLINE  ONLINE       zeroserv02               STABLE

--------------------------------------------------------------------------------

[root@zeroserv02 ~]#

7 – Restart CRS

After we configure the dependencies for ASM we can restart CRS to shutfown HAIP (and test the changes that we made). I prefer to execute, first, in just one node and if everything goes well, I do for the others.

Stop Cluster

So, first the STOP CLUSTER in the first node:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop cluster -f

CRS-2673: Attempting to stop 'ora.crsd' on 'zeroserv01'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'zeroserv01'

CRS-2673: Attempting to stop 'ora.catalog.raadmin.acfs' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.catalog.raosbadmin.acfs' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.DELTA.dg' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.CATALOG.dg' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.cvu' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.chad' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.chad' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.qosmserver' on 'zeroserv01'

CRS-2677: Stop of 'ora.DELTA.dg' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.CATALOG.dg' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv01'

CRS-2677: Stop of 'ora.cvu' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.asm' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv01'

CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.scan2.vip' on 'zeroserv01'

CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.scan3.vip' on 'zeroserv01'

CRS-2677: Stop of 'ora.catalog.raosbadmin.acfs' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01'

CRS-2677: Stop of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.catalog.raadmin.acfs' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01'

CRS-2677: Stop of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.proxy_advm' on 'zeroserv01'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.zeroserv01.vip' on 'zeroserv01'

CRS-2677: Stop of 'ora.scan3.vip' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.qosmserver' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.scan2.vip' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.zeroserv01.vip' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.proxy_advm' on 'zeroserv01' succeeded

CRS-2675: Stop of 'ora.chad' on 'zeroserv02' failed

CRS-2679: Attempting to clean 'ora.chad' on 'zeroserv02'

CRS-2675: Stop of 'ora.chad' on 'zeroserv01' failed

CRS-2679: Attempting to clean 'ora.chad' on 'zeroserv01'

CRS-2681: Clean of 'ora.chad' on 'zeroserv02' succeeded

CRS-2681: Clean of 'ora.chad' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.mgmtdb' on 'zeroserv01'

CRS-2677: Stop of 'ora.mgmtdb' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'zeroserv01'

CRS-2677: Stop of 'ora.MGMTLSNR' on 'zeroserv01' succeeded

CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'zeroserv02'

CRS-2672: Attempting to start 'ora.qosmserver' on 'zeroserv02'

CRS-2672: Attempting to start 'ora.scan2.vip' on 'zeroserv02'

CRS-2672: Attempting to start 'ora.scan3.vip' on 'zeroserv02'

CRS-2672: Attempting to start 'ora.cvu' on 'zeroserv02'

CRS-2672: Attempting to start 'ora.zeroserv01.vip' on 'zeroserv02'

CRS-2676: Start of 'ora.cvu' on 'zeroserv02' succeeded

CRS-2676: Start of 'ora.scan2.vip' on 'zeroserv02' succeeded

CRS-2676: Start of 'ora.zeroserv01.vip' on 'zeroserv02' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv02'

CRS-2676: Start of 'ora.scan3.vip' on 'zeroserv02' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv02'

CRS-2676: Start of 'ora.MGMTLSNR' on 'zeroserv02' succeeded

CRS-2672: Attempting to start 'ora.mgmtdb' on 'zeroserv02'

CRS-2676: Start of 'ora.qosmserver' on 'zeroserv02' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv02' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv02' succeeded

CRS-2676: Start of 'ora.mgmtdb' on 'zeroserv02' succeeded

CRS-2672: Attempting to start 'ora.chad' on 'zeroserv02'

CRS-2676: Start of 'ora.chad' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'zeroserv01'

CRS-2677: Stop of 'ora.ons' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'zeroserv01'

CRS-2677: Stop of 'ora.net1.network' on 'zeroserv01' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'zeroserv01' has completed

CRS-2677: Stop of 'ora.crsd' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.evmd' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.storage' on 'zeroserv01'

CRS-2677: Stop of 'ora.storage' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv01'

CRS-2677: Stop of 'ora.ctssd' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.evmd' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.asm' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'zeroserv01'

CRS-2677: Stop of 'ora.cssd' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.diskmon' on 'zeroserv01'

CRS-2677: Stop of 'ora.diskmon' on 'zeroserv01' succeeded

[root@zeroserv01 ~]#

Stop/Start CRS

And if everything was successful, we can stop CRS. This is needed because HAIP is from init from CRS:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop crs -f

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.crf' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'zeroserv01'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'zeroserv01'

CRS-2677: Stop of 'ora.drivers.acfs' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.crf' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'zeroserv01'

CRS-2677: Stop of 'ora.mdnsd' on 'zeroserv01' succeeded

CRS-2677: Stop of 'ora.gipcd' on 'zeroserv01' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'zeroserv01' has completed

CRS-4133: Oracle High Availability Services has been stopped.

[root@zeroserv01 ~]#

After that we can start again CRS in this node:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

[root@zeroserv01 ~]#

Check CRS Init

And after some time we can see if the HAIP not restarted (as expected):

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.asm

      1        ONLINE  ONLINE       zeroserv01               Started,STABLE

ora.cluster_interconnect.haip

      1        OFFLINE OFFLINE                               STABLE

ora.crf

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.crsd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.cssd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.cssdmonitor

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.ctssd

      1        ONLINE  ONLINE       zeroserv01               ACTIVE:0,STABLE

ora.diskmon

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.drivers.acfs

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.drivers.oka

      1        OFFLINE OFFLINE                               STABLE

ora.evmd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.gipcd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.gpnpd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.mdnsd

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.storage

      1        ONLINE  ONLINE       zeroserv01               STABLE

--------------------------------------------------------------------------------

[root@zeroserv01 ~]#

As you can see above, the HAIP not started during the init of CRS. If the HAIP still up, please check the topic 5.

Other nodes

After that, we can do the same in the other nodes.

[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop cluster -f

CRS-2673: Attempting to stop 'ora.crsd' on 'zeroserv02'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'zeroserv02'

CRS-2673: Attempting to stop 'ora.DELTA.dg' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.catalog.raadmin.acfs' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.chad' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.catalog.raosbadmin.acfs' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv02'

CRS-2677: Stop of 'ora.DELTA.dg' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.CATALOG.dg' on 'zeroserv02'

CRS-2677: Stop of 'ora.CATALOG.dg' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv02'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.zeroserv02.vip' on 'zeroserv02'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'zeroserv02'

CRS-2677: Stop of 'ora.catalog.raadmin.acfs' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02'

CRS-2677: Stop of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02' succeeded

CRS-2677: Stop of 'ora.catalog.raosbadmin.acfs' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02'

CRS-2677: Stop of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.proxy_advm' on 'zeroserv02'

CRS-2677: Stop of 'ora.asm' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv02'

CRS-2677: Stop of 'ora.zeroserv02.vip' on 'zeroserv02' succeeded

CRS-2677: Stop of 'ora.scan1.vip' on 'zeroserv02' succeeded

CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv02' succeeded

CRS-2677: Stop of 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv02' succeeded

CRS-2677: Stop of 'ora.proxy_advm' on 'zeroserv02' succeeded

CRS-2677: Stop of 'ora.chad' on 'zeroserv02' succeeded

CRS-2672: Attempting to start 'ora.scan1.vip' on 'zeroserv01'

CRS-2672: Attempting to start 'ora.zeroserv02.vip' on 'zeroserv01'

CRS-2676: Start of 'ora.zeroserv02.vip' on 'zeroserv01' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'zeroserv01' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv01'

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv01' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'zeroserv02'

CRS-2677: Stop of 'ora.ons' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'zeroserv02'

CRS-2677: Stop of 'ora.net1.network' on 'zeroserv02' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'zeroserv02' has completed

CRS-2677: Stop of 'ora.crsd' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.evmd' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.storage' on 'zeroserv02'

CRS-2677: Stop of 'ora.storage' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv02'

CRS-2677: Stop of 'ora.ctssd' on 'zeroserv02' succeeded

CRS-2677: Stop of 'ora.evmd' on 'zeroserv02' succeeded

CRS-2677: Stop of 'ora.asm' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'zeroserv02'

CRS-2677: Stop of 'ora.cssd' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.diskmon' on 'zeroserv02'

CRS-2677: Stop of 'ora.diskmon' on 'zeroserv02' succeeded

[root@zeroserv02 ~]#

[root@zeroserv02 ~]#

[root@zeroserv02 ~]#

[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop crs -f

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.crf' on 'zeroserv02'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'zeroserv02'

CRS-2677: Stop of 'ora.drivers.acfs' on 'zeroserv02' succeeded

CRS-2677: Stop of 'ora.crf' on 'zeroserv02' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'zeroserv02'

CRS-2677: Stop of 'ora.mdnsd' on 'zeroserv02' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'zeroserv02' succeeded

CRS-2677: Stop of 'ora.gipcd' on 'zeroserv02' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'zeroserv02' has completed

CRS-4133: Oracle High Availability Services has been stopped.

[root@zeroserv02 ~]#

[root@zeroserv02 ~]#

[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

[root@zeroserv02 ~]#

[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.asm

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.cluster_interconnect.haip

      1        OFFLINE OFFLINE                               STABLE

ora.crf

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.crsd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.cssd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.cssdmonitor

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.ctssd

      1        ONLINE  ONLINE       zeroserv02               ACTIVE:0,STABLE

ora.diskmon

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.drivers.acfs

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.drivers.oka

      1        OFFLINE OFFLINE                               STABLE

ora.evmd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.gipcd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.gpnpd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.mdnsd

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.storage

      1        ONLINE  ONLINE       zeroserv02               STABLE

--------------------------------------------------------------------------------

[root@zeroserv02 ~]#

8 – Check if everything is UP

After doing the changes in both nodes we can see if everything is up and running:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ob_dbfs

               OFFLINE OFFLINE      zeroserv01               STABLE

               OFFLINE OFFLINE      zeroserv02               STABLE

ora.ASMNET1LSNR_ASM.lsnr

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

ora.ASMNET2LSNR_ASM.lsnr

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

ora.CATALOG.RAADMIN.advm

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

ora.CATALOG.RAOSBADMIN.advm

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

ora.CATALOG.dg

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

ora.DELTA.dg

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

ora.catalog.raadmin.acfs

               ONLINE  ONLINE       zeroserv01               mounted on /raacfs/r

                                                             aadmin,STABLE

               ONLINE  ONLINE       zeroserv02               mounted on /raacfs/r

                                                             aadmin,STABLE

ora.catalog.raosbadmin.acfs

               ONLINE  ONLINE       zeroserv01               mounted on /osbcat,S

                                                             TABLE

               ONLINE  ONLINE       zeroserv02               mounted on /osbcat,S

                                                             TABLE

ora.chad

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

ora.net1.network

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

ora.ons

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

ora.proxy_advm

               ONLINE  ONLINE       zeroserv01               STABLE

               ONLINE  ONLINE       zeroserv02               STABLE

rep_dbfs

               OFFLINE OFFLINE      zeroserv01               STABLE

               OFFLINE OFFLINE      zeroserv02               STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.LISTENER_SCAN2.lsnr

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.LISTENER_SCAN3.lsnr

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.MGMTLSNR

      1        ONLINE  ONLINE       zeroserv01               192.168.10.1 192.168

                                                             .10.2,STABLE

ora.asm

      1        ONLINE  ONLINE       zeroserv01               Started,STABLE

      2        ONLINE  ONLINE       zeroserv02               Started,STABLE

ora.cvu

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.mgmtdb

      1        ONLINE  ONLINE       zeroserv01               Open,STABLE

ora.qosmserver

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       zeroserv02               STABLE

ora.scan2.vip

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.scan3.vip

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.zdlras.db

      1        OFFLINE OFFLINE                               STABLE

      2        OFFLINE OFFLINE                               Instance Shutdown,ST

                                                             ABLE

ora.zeroserv01.vip

      1        ONLINE  ONLINE       zeroserv01               STABLE

ora.zeroserv02.vip

      1        ONLINE  ONLINE       zeroserv02               STABLE

--------------------------------------------------------------------------------

[root@zeroserv01 ~]#

As you can see, everything is up. ASM is online in both nodes, as the listeners for ASM too.

9 – Start Databases

To finish, we need to restart all databases that were stopped at topic 1.

[oracle@zeroserv01 ~]$ srvctl start database -d zdlras

[oracle@zeroserv01 ~]$

Clean up

After you complete the GI upgrade for 19c, you can remove the ASM Proxy init file (from $GI_HOME/dbs) that was created before. It is not needed anymore because the upgrade removes completely the HAIP from CRS because it is not in use.

Conclusion

As told before, HAIP is not supported by the Exadata environment. If you check the definition for HAIP and the hardware design for the Exadata/Engineering System, it is completely redundant. The InfiniBand network and the dual-channel/ports have the same effect (even better)

But unfortunately, HAIP can be up for several reasons (like ZDLRA) and because of the way that ASM Proxy starts and pick up the interconnect IP, HAIP can be selected. Remove HAIP it is a little tricky because of the ASM dependency, we need to set some specific parameters.

As you saw before, there are several steps. But you can use and adapt them to your environment. If it is Exadata/Engineering System (like ZDLRA) they can be used almost as-is (just check the IPs). If it is not, read and adapt.

References:

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”

How to upgrade ODA Patch: 18.8.0.0 for Virtualized System

Introduction

The goal of this document is to describe step by step how to upgrade ODA Virtualized System.

Prerequisites

For this Upgrade the ODA must be at version 18.3

Start Upgrade

1 – Backup of ODA_BASE (Both Nodes): It can take up to 2h00 hours.

From DOM0:

As root user:

Node 1:

After complete the backup:

Node 2:

After complete the backup:

2 – Download patch in a shared directory or separated directory in both ODA servers:

Os User: root

There is 2 .zip files to be download patch (30518438):

Note: We must guarantee some minimum amount of free space (20GB) in the ODA BASE, file systems: “/” and “/u01“

– Purge old logs from ODA, with root user:

– Clean up old patches from GRID_HOME, affects “/u01“:

– Clean up old patches from ORACLE_HOME, affects “/u01“:

Note: – It must be performed this clean in every ORACLE_HOME versions that exists in the server. – Look at “/home/oracle“, “/home/grid” and “/tmp” to perform some clean up and release some space in the file system “/“.

3 – Unpack downloaded patch in both ODA nodes:

4 – Verify and Validate Upgrade ODA components (S.O):

Note:

Solving rpm’s conflict for version 18.8.0.0.0.

e.g: With errors and conflicts

BEGIN OUTPUT:

NODE 1:

Here we need to solve the dependency problem, in this case we will remove the package

NODE 2:

Here we need to solve the dependency problem, in this case we will remove the package

Now we have the environment validated for updating the operating system

5 – Verify which components it will be required to update:

e.g:

Note:

– Stop all databases and clusterware resource before patch ILOM in both servers

6 – Update ILOM

I prefer to first update ILOM in case there is a problem during the patch I can access the environment via ILOM.

If you want to update ILOM with the patch, it may be your choice as well.

Note: Upgrade ILOM: Both ILOM nodes, but not in the same time.

Download ILOM Patch separately: Sun Server X5-8 (ILOM 5.0.0.20 133445)

Custom File Name: p30802633_300_Generic.zip

The firmware are inside of this .zip file in the following path:

C:\Users\andre.ontalba\Downloads\p30802633_300_Generic\Oracle_Server_X5-8-3.0.0.91223-FIRMWARE_PACK\Firmware\service-processor\ILOM-5_0_0_20_r133445-Oracle_Server_X5-4_X5-8.pkg

You must copy this file to the following directory in the server to use for transfer this package. In my case dutsLinux1.

This file must be owned by “oracle” user and group “oinstall“

Connect to the gateway machine: dutsLinux1

Connect to the first ILOM from the gateway machine:

Check current version:

e.g:

Type:

Stopping DOM0 and ODA_BASE: (Connect to the ILOM):

Check status:

e.g: Section “Properties –> power_state = Off”

Load new ILOM image:

e.g:

After the automatic reboot performed in the ILOM, you can validate the new firmware and BIOS version:

/SYS/MB/BIOS

Start the DOM0 and ODA_BASE after upgrade ILOM:

Note: Repeat this procedure on the second node

7 – Upgrade ODA Servers

Note: Run only from first server and make sure that is the master node: oakcli show ismaster ⇒ “OAKD is in Master Mode”

e.g:

This procedure can take between 2 to 3 hours to execute on both nodes.

To be able to monitor the application of the patch we can see the logs in the directories below

Log file directory node 1: /opt/oracle/oak/log/odax58duts1/patch/18.8.0.0.0/

Log file directory node 2: /opt/oracle/oak/log/odax58duts2/patch/18.8.0.0.0/

Let’s check how is the server update

Node 1

Node 2

8 – ODA Patch: Database Binaries

Now is time to apply patch on the Oracle Database Binaries (11.2.0.4,12.1,12.2,18.).

Get the database list and Oracle Home before patch:

First step is stop TFA (Both Nodes with root)

To apply the patch in the Oracle Binaries: (Execute Only From First Node)

Let’s check how is the database update

Node 1

Node 2

9- Apply DATAPATCH/CATBUNDLE in databases 11.2 / 12.1 / 12.2 and 18.8

Note: It is required set up the NLS_LANG to “AMERICAN_AMERICA.AL32UTF8” variable in order to avoid a BUG during the DATAPATCH in the Oracle Database 12.1

Note:
– It must be performed this clean in every ORACLE_HOME versions that exists in the server.
– Look at “/home/oracle“, “/home/grid” and “/tmp” to perform some clean up and release some space in the file system “/“.