ODA REIMAGE

The idea to reimage ODA is to refresh the environment without the need to jump from one by one to reach the last available version, or even rescue the system from S.O. failure/crash. The process to do a reimage can be check in the official documentation but unfortunately can be very tricky because the information (the order and steps) are not 100% clear. The idea is to show you how to reimage using version 18 (18.3 in this example), that represents the last available.

 

In resume the process is executed in the order:
  • ILOM: Boot the ISO
  • Prepare to create the appliance
  • Upload GI and DB base version to the repository
  • Linux: Create the appliance
  • Firmware and patch
  • Create Oracle Homes and the databases
  • Finish and clean the install
The environment in this scenario is:
  • ODA HA: Oda HA version with two nodes and with SSD disk available.
  • Reimage with 18 version: 18.3 version at the moment that reimages was done, but can use any 18.x version
  • Databases: 11, 12, 18 for Oracle Home and available versions
Before even start the reimage you have two steps. The first is to verify the hardware against failures. Disks, memory, CPU, MB, whatever error that exists need to be clean before the start. If no, you can hit error during the process. To check about hw error, you can log in the ILOM in the browser and verify for some incidents/errors (this vary from version to version of ODA, but usually the errors are visible in the first page of ILOM). Or you can log in in the console for ILOM and execute the command “show faulty”. Whatever the mode, check in both nodes. If you have a connection with the system execute smartctl –a /dev/DISK and query about errors and look in the column “Total uncorrected errors”. Another option is check if the disk is in warning state using the command smartctl -q errorsonly -H -l xerror /dev/DISK; if you have the report will be similar to “SMART Health Status: WARNING: ascq=0x97 [asc=b, ascq=97]”. For the errors, open SR and wait for the maintenance from Oracle HW team.

 

The second is save/backup every config that you need. If you did not lose the system backup from /etc folder it is a good start. Another hint is execute “ifconfig” and “route” to have one output from IP’s that you have and the routing table (vlan list it is important too if used). For Oracle files, backup tnsnames.ora, listener.ora, and sqlnet.ora for every OH or GI that you have. These files are example, depend on every system and needs. Adjust if you need something additional.

 

ILOM

 

The process to reimage start putting the ISO in the ilom and booting both nodes with that. After the boot, the install runs automatically, and you have a Linux to create the appliance.
Before starting the reimage one explanation since the machine was already in use it needs to cleanup the disks. The first option is cleanup before the reimage, using the software from the currently running linux/oda/oak. Or running the cleanup after the reimage finish and before creating the appliance (the option that I will use here).

 

The steps to reimage from ILOM are done in this order:
  • Login ILOM both nodes using the browser
  • Navigate to “Host Management” menu and choose “Host Control” submenu.
    1. Select the option “CDROM” in the list and after click in SAVE.
  • Navigate to “Remote Control” menu and choose “Redirection” submenu.
    1. Click on the option “Launch Remote Control Console”
    2. You can receive some confirmation to execute some java (depend on how your machine has java configured) or ask to agree about the certificate. Always click yes and agree or continue.
  • In the newly opened window select the menu “KVMS” and after the submenu “Storage”.
  • In the “Storage Devices” do:
    1. Click on the “Add”
    2. Select the ISO file and after click in the option “Select”
    3. After, verify if the ISO appeared in the “Path” and the “Device Type” is “ISO Image” in the Storage Devices menu. Select the line with the ISO and after click in the option “Connect”.
    4. After you click in the “Connect” the information marked in red will appear in the screen and the menu now will appear as “Disconnect”. You just need to click in the option “OK”.
    5. DO NOT CLONE the Console.
  • Go back to the Browser and click in the menu “Host Management” and after in the submenu “Power Control”
    1. Select the option “Power Cycle” and after click in the “Save” option.
    2. Maybe you will receive on a question if you are sure about the reboot. Just click Ok.
  • You will see in the console:
    1. Reboot
    2. The process will boot the ISO
    3. Install the new ODA Image
In the end, you have the Linux console

 

All the steps above you can see in the Image gallery below. All the images are linked with the step mentioned.

 

 

 
Remember to do this in both nodes. You can do this both at the same time. The installation time depends more from the network because involve uploading the ISO from machine to ODA. The postscript part during the installation tends to take a lot of time.
Please does not close the Remote Console because this kills the ISO mounting device and the installation will crash. There is no problem if the browser login receives timeout, the installation will continue.
One important thing from this part is that does not update the ILOM SP or even other firmware for HW. This will be executed in the next steps.
Recently Oracle changed the ODA base page and removed the links to ISO files. But the patch 27604623 contains the Oracle Database Appliance Bare-metal ISO Image to download. Just to be clear, I am reimaging for baremetal (not OVM).

 

Prepare to create the appliance

After you “reimage” you need to configure and create the appliance. But before we need to execute some steps. Now, to connect in the Linux console we have two options: same java console opened to do the reimage, or, ssh to ilom ip and start the console (start /SP/console). I will use the last option because allow to work (copy and paste commands) more smoothly. Example (root/welcome1 are user/password that is the default for reimaged ODA):

 

Oracle(R) Integrated Lights Out Manager




Version 3.2.9.23 r116695




Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.




Warning: HTTPS certificate is set to factory default.




Hostname: odak1-ilom




-> start /SP/console

Are you sure you want to start /SP/console (y/n)? y




Serial console started.  To stop, type ESC (







Oracle Linux Server release 6.10

Kernel 4.1.12-124.18.6.el6uek.x86_64 on an x86_64




test0 login: root

[root@test0 ~]#


Cleanup

As told before, we need to clean the headers for ODA. I start checking the headers with command “cleanup.pl -checkHeader”:

 

[root@test0 ~]# /opt/oracle/oak/onecmd/cleanup.pl -checkHeader

Command may take few minutes...

OS Disk               Disk Type    OAK Header   ASM Header(p1)       ASM Header(p2)

 /dev/sda                HDD        Found           Found                Found

 /dev/sdb                HDD        Found           Found                Found

 /dev/sdaa               HDD        Found           Found                Found

 /dev/sdab               HDD        Found           Found                Found

 /dev/sdac               HDD        Found           Found                Found

 /dev/sdad               HDD        Found           Found                Found

 /dev/sdae               HDD        Found           Found                Found

 /dev/sdaf               HDD        Found           Found                Found

 /dev/sdag               HDD        Found           Found                Found

 /dev/sdah               HDD        Found          Erased               Erased

 /dev/sdai               HDD        Found           Found                Found

 /dev/sdaj               HDD        Found           Found                Found

 /dev/sdak               HDD        Found           Found                Found

 /dev/sdal               HDD        Found           Found                Found

 /dev/sdam               HDD        Found           Found                Found

 /dev/sdan               HDD        Found           Found                Found

 /dev/sdao               SSD        Found           Found       p2 not created

 /dev/sdap               SSD        Found           Found       p2 not created

 /dev/sdaq               SSD        Found           Found       p2 not created

 /dev/sdar               SSD        Found           Found       p2 not created

 /dev/sdas               SSD        Found           Found       p2 not created

 /dev/sdat               SSD        Found           Found       p2 not created

 /dev/sdau               SSD        Found           Found       p2 not created

 /dev/sdav               SSD        Found           Found       p2 not created

[root@test0 ~]#


Look above that some ASM Header was found. To clean we execute “/opt/oracle/oak/onecmd/cleanup.pl”  with option “-erasedata”. The cleanup script executes more than just erase disk headers, it cleans all the configurations (hostname, ips, users) and because of this needs to be executed in both nodes from HA env. This equalizes all nodes with the same config and avoid an error when you create the appliance. Example:

 

[root@test0 ~]# /opt/oracle/oak/onecmd/cleanup.pl -erasedata

INFO: *******************************************************************

INFO: ** Starting process to cleanup provisioned host oak1             **

INFO: *******************************************************************

WARNING: Secure Erase is an irrecoverable process. All data on the disk

WARNING: will be erased, and cannot be recovered by any means. On X3-2,

WARNING: X4-2, and X5-2 HA, the secure erase process can take more than

WARNING: 10 hours. If you need this data, then take a complete backup

WARNING: before proceeding.

Do you want to continue (yes/no) : yes

INFO:

....

....

INFO: Executing </opt/oracle/oak/bin/odaeraser.py -a -f -v >

Start erasing disks on the system

On some platforms, this will take several hours to finish, please wait

Do you want to continue (yes|no) [yes] ? yes

Number of disks are processing: 0




Disk     Vendor     Model                    Erase method         Status       Time(seconds)

e0_pd_00 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_01 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_02 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_03 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_04 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_05 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_06 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_07 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_08 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_09 HGST       H7280A520SUN8.0T         SCSI cryp[  777.976350] reboot: Restarting system

...

...

After node reboot you can check the disk readers (look that the cleanup will change the hostname to “oak”):

 

[root@oak2 ~]# /opt/oracle/oak/onecmd/cleanup.pl -checkHeader

Command may take few minutes...

OS Disk               Disk Type    OAK Header   ASM Header(p1)       ASM Header(p2)

 /dev/sda                HDD       Erased          Erased              UnKnown

 /dev/sdb                HDD       Erased          Erased              UnKnown

 /dev/sdaa               HDD       Erased          Erased              UnKnown

 /dev/sdab               HDD       Erased          Erased              UnKnown

 /dev/sdac               HDD       Erased          Erased              UnKnown

 /dev/sdad               HDD       Erased          Erased              UnKnown

 /dev/sdae               HDD       Erased          Erased              UnKnown

 /dev/sdaf               HDD       Erased          Erased              UnKnown

 /dev/sdag               HDD       Erased          Erased              UnKnown

 /dev/sdah               HDD       Erased          Erased              UnKnown

 /dev/sdai               HDD       Erased          Erased              UnKnown

 /dev/sdaj               HDD       Erased          Erased              UnKnown

 /dev/sdak               HDD       Erased          Erased              UnKnown

 /dev/sdal               HDD       Erased          Erased              UnKnown

 /dev/sdam               HDD       Erased          Erased              UnKnown

 /dev/sdan               HDD       Erased          Erased              UnKnown

 /dev/sdao               SSD       Erased          Erased              UnKnown

 /dev/sdap               SSD       Erased          Erased              UnKnown

 /dev/sdaq               SSD       Erased          Erased              UnKnown

 /dev/sdar               SSD       Erased          Erased              UnKnown

 /dev/sdas               SSD       Erased          Erased              UnKnown

 /dev/sdat               SSD       Erased          Erased              UnKnown

 /dev/sdau               SSD       Erased          Erased              UnKnown

 /dev/sdav               SSD       Erased          Erased              UnKnown



First network

After the cleanup we can create the basic network. To do that (need to do in both nodes), just execute the script “/opt/oracle/dcs/bin/odacli” with parameter “configure-firstnet”. Look:

 

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli configure-firstnet

Select the Interface to configure the network on (bond0 bond1) [bond0]:

Configure DHCP on bond0 (yes/no) [no]:

INFO: You have chosen Static configuration

Use VLAN on bond0 (yes/no) [no]:yes

Configure VLAN on bond0, input VLAN ID [2 - 4094] 2999

INFO: using network interface bond0.2999

Enter the IP address to configure : 200.200.67.100

Enter the Netmask address to configure : 255.255.255.0

Enter the Gateway address to configure[200.200.67.1] :

INFO: Restarting the network

Shutting down interface bond0:  [  OK  ]

Shutting down interface bond1:  [  OK  ]

Shutting down interface ib0:  [  OK  ]

Shutting down interface ib1:  [  OK  ]

Shutting down interface ibbond0:  [  OK  ]

Shutting down loopback interface:  [  OK  ]

Bringing up loopback interface:  [  OK  ]

Bringing up interface bond0:  [  OK  ]

Bringing up interface bond1:  [  OK  ]

Bringing up interface ibbond0:  Determining if ip address 192.168.16.24 is already in use for device ibbond0...

[  OK  ]

Bringing up interface bond0.2999:  Determining if ip address 200.200.67.100 is already in use for device bond0.2999...

[  OK  ]

INFO: Restarting the DCS agent

initdcsagent stop/waiting

initdcsagent start/running, process 23750

[root@oak1 ~]#

And in the second node:

 

[root@oak2 ~]# /opt/oracle/dcs/bin/odacli configure-firstnet

Select the Interface to configure the network on (bond0 bond1) [bond0]:

Configure DHCP on bond0 (yes/no) [no]:

INFO: You have chosen Static configuration

Use VLAN on bond0 (yes/no) [no]:yes

Configure VLAN on bond0, input VLAN ID [2 - 4094] 2999

INFO: using network interface bond0.2999

Enter the IP address to configure : 200.200.67.101

Enter the Netmask address to configure : 255.255.255.0

Enter the Gateway address to configure[200.200.67.1] :

INFO: Restarting the network

Shutting down interface bond0:  [  OK  ]

Shutting down interface bond1:  [  OK  ]

Shutting down interface ib0:  [  OK  ]

Shutting down interface ib1:  [  OK  ]

Shutting down interface ibbond0:  [  OK  ]

Shutting down loopback interface:  [  OK  ]

Bringing up loopback interface:  [  OK  ]

Bringing up interface bond0:  [  OK  ]

Bringing up interface bond1:  [  OK  ]

Bringing up interface ibbond0:  Determining if ip address 192.168.16.25 is already in use for device ibbond0...

[  OK  ]

Bringing up interface bond0.2999:  Determining if ip address 200.200.67.101 is already in use for device bond0.2999...

[  OK  ]

INFO: Restarting the DCS agent

initdcsagent stop/waiting

initdcsagent start/running, process 10775

[root@oak2 ~]#



The most important is to define correctly the vlan (if used, as in this case). You can rerun the command if needed. After that, you can exit the ILOM console and do ssh to the ODA node.

Upload GI and DB base version to the repository

After you configure the first network you need to upload the files that will fill the internal ODA repository and allow the “create appliance script” do initiate the ASM and base Oracle Home. There are some several steps, but you will execute in just one node.
Again, Oracle recently changed the ODA base page (https://support.oracle.com/epmos/faces/DocContentDisplay?id=888888.1) and removed the links to files with clone version for OH and GI. But the patch numbers are patch 27604593 for GI and patch 27604558 for RDBMS. Bot are for baremetal image, if you are using VM installation, they are others (but you can find then patch search inside Oracle MOS.
The first step is to upload the files “p27604558_183000_Linux-x86-64.zip”and “p27604593_183000_Linux-x86-64.zip” to the first node and unzip. Remember that the file name will change if you are reimage using another version. Here I used in “/tmp”

 

[root@oak1 ~]# mkdir /tmp/deploy

[root@oak1 ~]# cd /tmp/deploy

[root@oak1 deploy]# ###### EXECUTE THE SCP OR COPY FILES TO THE FOLDER########

[root@oak1 deploy]# unzip -qa p27604558_183000_Linux-x86-64.zip

[root@oak1 deploy]# unzip -qa p27604593_183000_Linux-x86-64.zip

replace README.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: A

[root@oak1 deploy]#

[root@oak1 deploy]#

[root@oak1 deploy]# ls -l

total 20559868

-rw-r--r-- 1 root root 4322872013 Dec 18 08:24 odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip

-rwxr-xr-x 1 root root 6193475789 Dec 18 08:03 odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip

-rw-r--r-- 1 root root 4322873299 May 22 08:37 p27604558_183000_Linux-x86-64.zip

-rw-r--r-- 1 root root 6193477129 May 22 08:35 p27604593_183000_Linux-x86-64.zip

-rw-r--r-- 1 root root       2044 Dec 18 08:05 README.txt

[root@oak1 deploy]#


After that we can upload these file to the internal ODA repository:

 

[root@oak1 deploy]# /opt/oracle/dcs/bin/odacli update-repository -f /tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip

{

  "jobId" : "b66269aa-e007-4304-8074-7ba7e9c8c5c2",

  "status" : "Created",

  "message" : "/tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 08:51:14 AM UTC",

  "resourceList" : [ ],

  "description" : "Repository Update",

  "updatedTime" : "May 22, 2019 08:51:14 AM UTC"

}

[root@oak1 deploy]#


Look that one job will be created, and you can check the progress for the job with the “/opt/oracle/dcs/bin/odacli describe-job -i” passing as parameter the jobId:

 

[root@oak1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i b66269aa-e007-4304-8074-7ba7e9c8c5c2

 /opt/oracle/dcs/bin/odacli describe-job -i b66269aa-e007-430




Job details

----------------------------------------------------------------

                     ID:  b66269aa-e007-4304-8074-7ba7e9c8c5c2

            Description:  Repository Update

                 Status:  Running

                Created:  May 22, 2019 8:51:14 AM UTC

                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Check AvailableSpace                     May 22, 2019 8:51:14 AM UTC         May 22, 2019 8:51:14 AM UTC         Success

Setting up ssh equivalance               May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:51:15 AM UTC         Success

Copy BundleFile                          May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:51:15 AM UTC         Running




[root@oak1 deploy]#

[root@oak1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i b66269aa-e007-4304-8074-7ba7e9c8c5c2




Job details

----------------------------------------------------------------

                     ID:  b66269aa-e007-4304-8074-7ba7e9c8c5c2

            Description:  Repository Update

                 Status:  Success

                Created:  May 22, 2019 8:51:14 AM UTC

                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Check AvailableSpace                     May 22, 2019 8:51:14 AM UTC         May 22, 2019 8:51:14 AM UTC         Success

Setting up ssh equivalance               May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:51:15 AM UTC         Success

Copy BundleFile                          May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:52:21 AM UTC         Success

Validating CopiedFile                    May 22, 2019 8:52:24 AM UTC         May 22, 2019 8:52:48 AM UTC         Success

Unzip bundle                             May 22, 2019 8:52:48 AM UTC         May 22, 2019 8:54:17 AM UTC         Success

Unzip bundle                             May 22, 2019 8:54:17 AM UTC         May 22, 2019 8:55:47 AM UTC         Success

Delete PatchBundles                      May 22, 2019 8:55:47 AM UTC         May 22, 2019 8:55:48 AM UTC         Success

Removing ssh keys                        May 22, 2019 8:55:48 AM UTC         May 22, 2019 8:55:49 AM UTC         Success




[root@oak1 deploy]#


Create appliance

After you clean the disk headers, prepare the first network and upload the files in the repository you can create the appliance. This means that you will configure the operational system, groups, users, folders, GI, ASM, and Oracle database. There are two ways to create, one is using the web interface (this replace the old java app that creates the XML file), and the other is from CLI. Unfortunately, until I wrote this, if you use VLAN you can’t create the appliance with the web interface, just with CLI.
The CLI uses the script “/opt/oracle/dcs/bin/odacli create-appliance” that receive a JSON file as a parameter. There is not some much information the parameter for this JSON file, just the readme with some examples and one web page with other examples. But in both, no real good explanation to each option, look:
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxd/create-appliance-using-json-file.html#GUID-42250FD2-EA91-4457-9ED7-CA3A2A863B40

 

Fortunately, most of the options are self-explained, other parts I will explain below. First, the full JSON file that I used:

 

[root@oak1 ~]# cat odas.json

{

  "instance" : {

    "name" : "odas",

    "instanceBaseName" : "odas",

    "systemPassword" : null,

    "dbEdition" : "EE",

    "timeZone" : "Europe/Luxembourg",

    "ntpServers" : [ "200.200.13.125" ],

    "dnsServers" : [ "200.200.1.32", "200.200.1.33" ],

    "domainName" : "xxxx.xxxx.xxx",

    "isRoleSeparated" : true,

    "osUserGroup" : {

      "groups" : [ {

        "groupId" : 1001,

        "groupName" : "oinstall",

        "groupRole" : "oinstall"

      }, {

        "groupId" : 1003,

        "groupName" : "dbaoper",

        "groupRole" : "dbaoper"

      }, {

        "groupId" : 1002,

        "groupName" : "dba",

        "groupRole" : "dba"

      }, {

        "groupId" : 1006,

        "groupName" : "asmadmin",

        "groupRole" : "asmadmin"

      }, {

        "groupId" : 1005,

        "groupName" : "asmoper",

        "groupRole" : "asmoper"

      }, {

        "groupId" : 1004,

        "groupName" : "asmdba",

        "groupRole" : "asmdba"

      } ],

      "users" : [ {

        "userId" : 1001,

        "userName" : "oracle",

        "userRole" : "oracleUser"

      }, {

        "userId" : 1000,

        "userName" : "grid",

        "userRole" : "gridUser"

      } ]

    },

    "objectStoreCredentials" : null

  },

  "nodes" : [ {

    "nodeNumber" : "0",

    "nodeName" : "odas1",

    "localDisk" : null,

    "network" : [ {

      "name" : null,

      "nicName" : "bond0.2999",

      "vlanId" : null,

      "ipAddress" : "200.200.67.100",

      "subNetMask" : "255.255.255.0",

      "gateway" : "200.200.67.1",

      "interfaceType" : null,

      "networkType" : [ "Public" ],

      "isDefaultNetwork" : true

    } ],

    "ilom" : null

  }, {

    "nodeNumber" : "1",

    "nodeName" : "odas2",

    "localDisk" : null,

    "network" : [ {

      "name" : null,

      "nicName" : "bond0.2999",

      "vlanId" : null,

      "ipAddress" : "200.200.67.101",

      "subNetMask" : "255.255.255.0",

      "gateway" : "200.200.67.1",

      "interfaceType" : null,

      "networkType" : [ "Public" ],

      "isDefaultNetwork" : true

    } ],

    "ilom" : null

  } ],

  "grid" : {

    "diskGroup" : [ {

      "diskGroupName" : "DATA",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 90

    }, {

      "diskGroupName" : "RECO",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 10

    }, {

      "diskGroupName" : "REDO",

      "redundancy" : "HIGH",

      "disks" : null,

      "diskPercentage" : null

    }, {

      "diskGroupName" : "FLASH",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : null

    } ],

    "scan" : {

      "scanName" : "odas-scan",

      "ipAddresses" : [ "200.200.67.104", "200.200.67.105" ]

    },

    "vip" : [ {

      "nodeNumber" : "0",

      "vipName" : "odas1-vip",

      "ipAddress" : "200.200.67.102"

    }, {

      "nodeNumber" : "1",

      "vipName" : "odas2-vip",

      "ipAddress" : "200.200.67.103"

    } ],

    "language" : "en",

    "enableAFD" : "TRUE",

    "gridVersion" : null

  },

  "database" : {

    "dbName" : "mfod",

    "databaseUniqueName" : "mfod_ora",

    "dbVersion" : "18.3.0.0.180717",

    "dbHomeId" : null,

    "instanceOnly" : false,

    "registerOnly" : null,

    "isCdb" : true,

    "pdBName" : "pdb1",

    "pdbAdminuserName" : "pdbuser",

    "enableTDE" : false,

    "adminPassword" : "WWeeeeee99##",

    "dbType" : "RAC",

    "dbTargetNodeNumber" : null,

    "dbClass" : "OLTP",

    "dbShape" : "odb1",

    "dbStorage" : "ASM",

    "dbRedundancy" : null,

    "dbCharacterSet" : {

      "characterSet" : "AL32UTF8",

      "nlsCharacterset" : "AL16UTF16",

      "dbTerritory" : "AMERICA",

      "dbLanguage" : "AMERICAN"

    },

    "dbConsoleEnable" : false,

    "backupConfigId" : null,

    "backupConfigName" : null,

    "dbOnFlashStorage" : true,

    "dbEdition" : "EE",

    "rmanBkupPassword" : null,

    "dbDomainName" : null,

    "enableFlashCache" : null,

    "level0BackupDay" : null

  },

  "asr" : null

}

[root@oak1 ~]#


The file starts with a basic config where you define the name parameter and the instanceBaseName that will define some basic internal identification parameters. One example is the time zone.

The first that needs attention is isRoleSeparated and osUserGroup. If you choose the first as false, you don’t need the specify grid users. Otherwise, is needed to specify it.

For nodes option, you specify the hostname, ips, and gateway. Be careful with the nickname that is the same name that you configured the first network. Some steps above. The ilom parameter, since I am not changing now (and I think that in reimage scenario you do not change at this moment), remains null.

The grid parameter defines the disk division. You can choose arbitrary values for diskPercentage and disks that you will select for each one if needed. For disk, left undefined that ODA will divide automatically and accordingly. The key point here is that even, by default, FLASH and REDO diskgroups are created with default values (check that I even defined disk percentage for each one), if you forgot to specify, they are not created and the disk will not be used. I already wrote about one problem in this particular part and you can read here: http://www.fernandosimon.com/blog/oda-json-and-flash/

The database parameter defines the information for database creation. You have the basic like database name, unique name, cdb (or no), pdb name. The most tricky here are dbVersion (that need to be the same from that you uploaded in the repository), adminPassword (that will be used for root password after create the appliance), dbShape (that is the same – you choose one – for ODA available shapes), and dbStorage (that you choose between ASM and ACFS, 11 just support ACFS). Again, for ASM and ACFS I already wrote about that here: http://www.fernandosimon.com/blog/oda-acfs-and-asm-dilemma/

Important hint, if you want to create a database in another version that 18c you need to upload the files/clone images in the repository before.

To create just execute:

 

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli create-appliance -r odas.json

{

  "instance" : {

    "name" : "odas",

    "instanceBaseName" : "odas",

    "systemPassword" : null,

    "dbEdition" : "EE",

    "timeZone" : "Europe/Luxembourg",

    "ntpServers" : [ "200.200.13.125" ],

    "dnsServers" : [ "200.200.1.32", "200.200.1.33" ],

    "domainName" : "xxxx.xxxx.xxx",

    "isRoleSeparated" : true,

    "osUserGroup" : {

      "groups" : [ {

        "groupId" : 1001,

        "groupName" : "oinstall",

        "groupRole" : "oinstall"

      }, {

        "groupId" : 1003,

        "groupName" : "dbaoper",

        "groupRole" : "dbaoper"

      }, {

        "groupId" : 1002,

        "groupName" : "dba",

        "groupRole" : "dba"

      }, {

        "groupId" : 1006,

        "groupName" : "asmadmin",

        "groupRole" : "asmadmin"

      }, {

        "groupId" : 1005,

        "groupName" : "asmoper",

        "groupRole" : "asmoper"

      }, {

        "groupId" : 1004,

        "groupName" : "asmdba",

        "groupRole" : "asmdba"

      } ],

      "users" : [ {

        "userId" : 1001,

        "userName" : "oracle",

        "userRole" : "oracleUser"

      }, {

        "userId" : 1000,

        "userName" : "grid",

        "userRole" : "gridUser"

      } ]

    },

    "objectStoreCredentials" : null

  },

  "nodes" : [ {

    "nodeNumber" : "0",

    "nodeName" : "odas1",

    "localDisk" : null,

    "network" : [ {

      "name" : null,

      "nicName" : "bond0.2999",

      "vlanId" : null,

      "ipAddress" : "200.200.67.100",

      "subNetMask" : "255.255.255.0",

      "gateway" : "200.200.67.1",

      "interfaceType" : null,

      "networkType" : [ "Public" ],

      "isDefaultNetwork" : true

    } ],

    "ilom" : null

  }, {

    "nodeNumber" : "1",

    "nodeName" : "odas2",

    "localDisk" : null,

    "network" : [ {

      "name" : null,

      "nicName" : "bond0.2999",

      "vlanId" : null,

      "ipAddress" : "200.200.67.101",

      "subNetMask" : "255.255.255.0",

      "gateway" : "200.200.67.1",

      "interfaceType" : null,

      "networkType" : [ "Public" ],

      "isDefaultNetwork" : true

    } ],

    "ilom" : null

  } ],

  "grid" : {

    "diskGroup" : [ {

      "diskGroupName" : "DATA",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 90

    }, {

      "diskGroupName" : "RECO",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 10

    }, {

      "diskGroupName" : "REDO",

      "redundancy" : "HIGH",

      "disks" : null,

      "diskPercentage" : null

    }, {

      "diskGroupName" : "FLASH",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : null

    } ],

    "scan" : {

      "scanName" : "odas-scan",

      "ipAddresses" : [ "200.200.67.104", "200.200.67.105" ]

    },

    "vip" : [ {

      "nodeNumber" : "0",

      "vipName" : "odas1-vip",

      "ipAddress" : "200.200.67.102"

    }, {

      "nodeNumber" : "1",

      "vipName" : "odas2-vip",

      "ipAddress" : "200.200.67.103"

    } ],

    "language" : "en",

    "enableAFD" : "TRUE",

    "gridVersion" : null

  },

  "database" : {

    "dbName" : "mfod",

    "databaseUniqueName" : "mfod_ora",

    "dbVersion" : "18.3.0.0.180717",

    "dbHomeId" : null,

    "instanceOnly" : false,

    "registerOnly" : null,

    "isCdb" : true,

    "pdBName" : "pdb1",

    "pdbAdminuserName" : "pdbuser",

    "enableTDE" : false,

    "adminPassword" : "WWeeeeee99##",

    "dbType" : "RAC",

    "dbTargetNodeNumber" : null,

    "dbClass" : "OLTP",

    "dbShape" : "odb1",

    "dbStorage" : "ASM",

    "dbRedundancy" : null,

    "dbCharacterSet" : {

      "characterSet" : "AL32UTF8",

      "nlsCharacterset" : "AL16UTF16",

      "dbTerritory" : "AMERICA",

      "dbLanguage" : "AMERICAN"

    },

    "dbConsoleEnable" : false,

    "backupConfigId" : null,

    "backupConfigName" : null,

    "dbOnFlashStorage" : true,

    "dbEdition" : "EE",

    "rmanBkupPassword" : null,

    "dbDomainName" : null,

    "enableFlashCache" : null,

    "level0BackupDay" : null

  },

  "asr" : null

}

{

  "jobId" : "f07961fa-4192-489b-9a55-53165a012376",

  "status" : "Created",

  "message" : null,

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 09:03:36 AM UTC",

  "resourceList" : [ ],

  "description" : "Provisioning service creation",

  "updatedTime" : "May 22, 2019 09:03:36 AM UTC"

}

[root@oak1 ~]#


Look that you will receive one jobId if everything is fine. To follow just describe the job:

 

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i f07961fa-4192-489b-9a55-53165a012376




Job details

----------------------------------------------------------------

                     ID:  f07961fa-4192-489b-9a55-53165a012376

            Description:  Provisioning service creation

                 Status:  Running

                Created:  May 22, 2019 11:03:36 AM CEST

                Message:




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

networks updation                        May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

updating network                         May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

Setting up Vlan                          May 22, 2019 11:03:38 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

networks updation                        May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

updating network                         May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

Setting up Vlan                          May 22, 2019 11:03:43 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmdba'creation            May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmadmin'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'dba'creation               May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'dbaoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'oinstall'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS user 'grid'creation                   May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

OS user 'oracle'creation                 May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

SSH equivalance setup                    May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

Grid home creation                       May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Running

Creating GI home directories             May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Success

Cloning Gi home                          May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Running




[root@oak1 ~]#



And after some time:
 
[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i f07961fa-4192-489b-9a55-53165a012376




Job details

----------------------------------------------------------------

                     ID:  f07961fa-4192-489b-9a55-53165a012376

            Description:  Provisioning service creation

                 Status:  Success

                Created:  May 22, 2019 11:03:36 AM CEST

                Message:




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

networks updation                        May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

updating network                         May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

Setting up Vlan                          May 22, 2019 11:03:38 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

networks updation                        May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

updating network                         May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

Setting up Vlan                          May 22, 2019 11:03:43 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmdba'creation            May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmadmin'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'dba'creation               May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'dbaoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'oinstall'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS user 'grid'creation                   May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

OS user 'oracle'creation                 May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

SSH equivalance setup                    May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

Grid home creation                       May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:08:00 AM CEST       Success

Creating GI home directories             May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Success

Cloning Gi home                          May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:06:36 AM CEST       Success

Cloning Gi home                          May 22, 2019 11:06:36 AM CEST       May 22, 2019 11:07:58 AM CEST       Success

Updating GiHome version                  May 22, 2019 11:07:58 AM CEST       May 22, 2019 11:08:00 AM CEST       Success

Updating GiHome version                  May 22, 2019 11:07:58 AM CEST       May 22, 2019 11:08:00 AM CEST       Success

Storage discovery                        May 22, 2019 11:08:00 AM CEST       May 22, 2019 11:19:03 AM CEST       Success

Grid stack creation                      May 22, 2019 11:19:03 AM CEST       May 22, 2019 11:53:45 AM CEST       Success

Configuring GI                           May 22, 2019 11:19:03 AM CEST       May 22, 2019 11:20:08 AM CEST       Success

Running GI root scripts                  May 22, 2019 11:20:08 AM CEST       May 22, 2019 11:32:18 AM CEST       Success

Running GI config assistants             May 22, 2019 11:37:22 AM CEST       May 22, 2019 11:40:04 AM CEST       Success

Setting AUDIT SYSLOG LEVEL               May 22, 2019 11:50:06 AM CEST       May 22, 2019 11:50:06 AM CEST       Success

Post cluster OAKD configuration          May 22, 2019 11:53:45 AM CEST       May 22, 2019 11:59:11 AM CEST       Success

Disk group 'RECO'creation                May 22, 2019 11:59:20 AM CEST       May 22, 2019 11:59:38 AM CEST       Success

Disk group 'REDO'creation                May 22, 2019 11:59:38 AM CEST       May 22, 2019 11:59:47 AM CEST       Success

Disk group 'FLASH'creation               May 22, 2019 11:59:47 AM CEST       May 22, 2019 11:59:58 AM CEST       Success

Volume 'commonstore'creation             May 22, 2019 11:59:58 AM CEST       May 22, 2019 12:00:48 PM CEST       Success

ACFS File system 'DATA'creation          May 22, 2019 12:00:48 PM CEST       May 22, 2019 12:01:13 PM CEST       Success

Database home creation                   May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:05:28 PM CEST       Success

Validating dbHome available space        May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:01:13 PM CEST       Success

Validating dbHome available space        May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:01:13 PM CEST       Success

Creating DbHome Directory                May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:01:13 PM CEST       Success

Extract DB clones                        May 22, 2019 12:01:14 PM CEST       May 22, 2019 12:04:12 PM CEST       Success

Clone Db home                            May 22, 2019 12:04:12 PM CEST       May 22, 2019 12:05:11 PM CEST       Success

Enable DB options                        May 22, 2019 12:05:11 PM CEST       May 22, 2019 12:05:27 PM CEST       Success

Run Root DB scripts                      May 22, 2019 12:05:27 PM CEST       May 22, 2019 12:05:27 PM CEST       Success

Provisioning service creation            May 22, 2019 12:05:28 PM CEST       May 22, 2019 12:23:12 PM CEST       Success

Database Creation                        May 22, 2019 12:05:28 PM CEST       May 22, 2019 12:20:27 PM CEST       Success

Change permission for xdb wallet files   May 22, 2019 12:20:27 PM CEST       May 22, 2019 12:20:27 PM CEST       Success

Add Startup Trigger to Open all PDBS     May 22, 2019 12:20:27 PM CEST       May 22, 2019 12:20:28 PM CEST       Success

SqlPatch upgrade                         May 22, 2019 12:21:53 PM CEST       May 22, 2019 12:23:08 PM CEST       Success

Running dbms_stats init_package          May 22, 2019 12:23:08 PM CEST       May 22, 2019 12:23:11 PM CEST       Success

updating the Database version            May 22, 2019 12:23:11 PM CEST       May 22, 2019 12:23:12 PM CEST       Success

users tablespace creation                May 22, 2019 12:23:12 PM CEST       May 22, 2019 12:23:15 PM CEST       Success

Install TFA                              May 22, 2019 12:23:16 PM CEST       May 22, 2019 12:28:08 PM CEST       Success




[root@oak1 ~]#


You can follow the log too if you want to see more detailed (a lot) info. To do just go to directory /opt/oracle/dcs/log/ and check the file dcs-agent.log. Everything will be there, and it is really importing if some step fails. Here you can see one example of the output.
After the end of the job you can check the details for the appliance:

 

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-system




Appliance Information

----------------------------------------------------------------

                     ID: 6ce235cf-effc-4748-9d0f-ac246e9ee819

               Platform: X5-2-HA

        Data Disk Count: 24

         CPU Core Count: 36

                Created: May 22, 2019 11:03:36 AM CEST




System Information

----------------------------------------------------------------

                   Name: odas

            Domain Name: xxxx.xxxx.xxx

              Time Zone: Europe/Luxembourg

             DB Edition: EE

            DNS Servers: 200.200.1.32 200.200.1.33

            NTP Servers: 200.200.13.125




Disk Group Information

----------------------------------------------------------------

DG Name                   Redundancy                Percentage

------------------------- ------------------------- ------------

Data                      Normal                    90

Reco                      Normal                    10

Redo                      High                      100

Flash                     Normal                    100




[root@oak1 ~]#

[root@oak1 ~]# odaadmcli show diskgroup

DiskGroups

----------

DATA

FLASH

RECO

REDO

[root@oak1 ~]#

[root@oak1 ~]# su - grid

[grid@odas1 ~]$ asmcmd

ASMCMD> lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU   Total_MB    Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  NORMAL  N         512             512   4096  4194304  108019712  108007188          6751232        50627978              0             Y  DATA/

MOUNTED  NORMAL  N         512             512   4096  4194304    1525760    1518672           381440          568616              0             N  FLASH/

MOUNTED  NORMAL  N         512             512   4096  4194304   11997184   11995736           749824         5622956              0             N  RECO/

MOUNTED  HIGH    N         512             512   4096  4194304     762880     749908           190720          186396              0             N  REDO/

ASMCMD> exit

[grid@odas1 ~]$

[grid@odas1 ~]$ logout

[root@oak1 ~]#

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli list-dbstorages




ID                                       Type   DBUnique Name        Status

---------------------------------------- ------ -------------------- ----------

d26bf34f-9379-4872-839b-ae0af6926aae     Asm    mfod_ora             Configured

[root@oak1 ~]#

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-dbstorage -i d26bf34f-9379-4872-839b-ae0af6926aae

Database Storage details

----------------------------------------------------------------

                     ID: d26bf34f-9379-4872-839b-ae0af6926aae

                DB Name: mfod

          DBUnique Name: mfod_ora

         DB Resource ID: 938b616a-99ed-4466-8d76-820d3489517d

           Storage Type: ASM

                   DATA:

                         Location: +FLASH/mfod_ora

                       Used Space: 3.21GB

                       Free Space: 741.53GB

                   REDO:

                         Location: +REDO/mfod_ora

                       Used Space: 4.06GB

                       Free Space: 244.11GB

                   RECO:

                         Location: +RECO/mfod_ora

                       Used Space: 334.0MB

                       Free Space: 5.72TB

                  State: ResourceState(status=Configured)

                Created: May 22, 2019 11:03:36 AM CEST

            UpdatedTime: May 22, 2019 12:01:13 PM CEST

[root@oak1 ~]#


After that you reboot both nodes to have a fresh start with all the configuration up and running:

 

[root@oak1 ~]# reboot




Broadcast message from root@odas1

        (/dev/ttyS0) at 12:37 ...




The system is going down for reboot NOW!

[root@oak1 ~]#

       

Oracle Linux Server release 6.10

Kernel 4.1.12-124.18.6.el6uek.x86_64 on an x86_64




odas1 login: root

Password:

Last login: Wed May 22 11:10:13 from odap1.xxxx.xxxx.xxx

[root@odas1 ~]#


As an example, you can add vlan to ODA if needed (after the reboot):

 

[root@odas1 ~]# odaadmcli show vlan

        NAME                     ID    INTERFACE   CONFIG_TYPE IP_ADDRESS      NETMASK         GATEWAY         NODENUM

        Public-network           2999  bond0       Public      200.200.67.100   255.255.255.0   200.200.67.1     0

        Public-network           2999  bond0       Public      200.200.67.101   255.255.255.0   200.200.67.1     1

[root@odas1 ~]#

[root@odas1 ~]# cat /proc/net/vlan/config

VLAN Dev name    | VLAN ID

Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD

bond0.2999     | 2999  | bond0

[root@odas1 ~]#

[root@odas1 ~]# odaadmcli create vlan VLAN_3001 -vlanid 3001 -if bond0 -node 0 -setuptype database -ip 200.200.131.58 -netmask 255.255.255.0 -gateway 200.200.131.1




Created Vlan : VLAN_3001

[root@odas1 ~]# odaadmcli create vlan VLAN_3010 -vlanid 3010 -if bond0 -node 0 -setuptype database -ip 200.190.3.58 -netmask 255.255.255.0 -gateway 200.190.3.1




Created Vlan : VLAN_3010

[root@odas1 ~]# odaadmcli create vlan VLAN_3030 -vlanid 3030 -if bond0 -node 0 -setuptype management -ip 200.200.192.156 -netmask 255.255.255.0 -gateway 200.200.192.1




Created Vlan : VLAN_3030

[root@odas1 ~]# odaadmcli create vlan VLAN_3002 -vlanid 3002 -if bond1 -node 0 -setuptype backup -ip 200.200.68.128 -netmask 255.255.255.0 -gateway 200.200.68.1




Created Vlan : VLAN_3002

[root@odas1 ~]# odaadmcli create vlan VLAN_3001 -vlanid 3001 -if bond0 -node 1 -setuptype database -ip 200.200.131.59 -netmask 255.255.255.0 -gateway 200.200.131.1




Created Vlan : VLAN_3001

[root@odas1 ~]# odaadmcli create vlan VLAN_3010 -vlanid 3010 -if bond0 -node 1 -setuptype database -ip 200.190.3.59 -netmask 255.255.255.0 -gateway 200.190.3.1




Created Vlan : VLAN_3010

[root@odas1 ~]# odaadmcli create vlan VLAN_3030 -vlanid 3030 -if bond0 -node 1 -setuptype management -ip 200.200.192.157 -netmask 255.255.255.0 -gateway 200.200.192.1




Created Vlan : VLAN_3030

[root@odas1 ~]# odaadmcli create vlan VLAN_3002 -vlanid 3002 -if bond1 -node 1 -setuptype backup -ip 200.200.68.129 -netmask 255.255.255.0 -gateway 200.200.68.1




Created Vlan : VLAN_3002

[root@odas1 ~]#

[root@odas1 ~]#

[root@odas1 ~]# odaadmcli show vlan

        NAME                     ID    INTERFACE   CONFIG_TYPE IP_ADDRESS      NETMASK         GATEWAY         NODENUM

        Public-network           2999  bond0       Public      200.200.67.100   255.255.255.0   200.200.67.1     0

        Public-network           2999  bond0       Public      200.200.67.101   255.255.255.0   200.200.67.1     1

        VLAN_3002                3002  bond1       backup      200.200.68.128   255.255.255.0   200.200.68.1     0

        VLAN_3002                3002  bond1       backup      200.200.68.129   255.255.255.0   200.200.68.1     1

        VLAN_3001                3001  bond0       database    200.200.131.58   255.255.255.0   200.200.131.1    0

        VLAN_3001                3001  bond0       database    200.200.131.59   255.255.255.0   200.200.131.1    1

        VLAN_3010                3010  bond0       database    200.190.3.58     255.255.255.0   200.190.3.1      0

        VLAN_3010                3010  bond0       database    200.190.3.59     255.255.255.0   200.190.3.1      1

        VLAN_3030                3030  bond0       management  200.200.192.156  255.255.255.0   200.200.192.1    0

        VLAN_3030                3030  bond0       management  200.200.192.157  255.255.255.0   200.200.192.1    1

[root@odas1 ~]#


Firmware and patch

As described in the documentation, the reimage process does not update firmware and we need to this after creating the appliance. This point ins controversy, because there is no information when need to do this, you can run before or after creating the appliance. Here I made after the create, but if you are in really old version and have a lot of old firmware running maybe you can run before the create to avoid issues.
The funny part is that the firmware’s are inside of one the default patch file. Again, you need to search in patch search page because Oracle removed from 88888.1 page these links (look the path number in the example below). So, the procedure to update firmware is the same to apply a patch in ODA and need to download the same patch version. As usual, the first step is to upload the files:

 

[root@odas1 ~]# cd /tmp/deploy/

[root@odas1 deploy]#

[root@odas1 deploy]# unzip -qa p28864490_183000_Linux-x86-64_1of3.zip

[root@odas1 deploy]# unzip -qa p28864490_183000_Linux-x86-64_2of3.zip

[root@odas1 deploy]# unzip -qa p28864490_183000_Linux-x86-64_3of3.zip

[root@odas1 deploy]#

[root@odas1 deploy]# rm -rf p*zip

[root@odas1 deploy]#

[root@odas1 deploy]# ls -l

total 15441928

-rw-r--r-- 1 root root 2767978208 Dec 13 14:50 oda-sm-18.3.0.0.0-181205-server1of3.zip

-rwxr-xr-x 1 root root 6835610213 Dec 13 15:08 oda-sm-18.3.0.0.0-181205-server2of3.zip

-rwxr-xr-x 1 root root 6193475789 Dec 13 15:22 oda-sm-18.3.0.0.0-181205-server3of3.zip

-rw-r--r-- 1 root root       1155 Dec 13 15:22 README.txt

[root@odas1 deploy]#

[root@odas1 deploy]#


If you want to check what are you running, just execute odacli describe-component (look ilom version as an example):

 

[root@odas1 ~]# odacli describe-component

System Version

---------------

18.3.0.0.0




System node Name

---------------

odas1




Local System Version

---------------

18.3.0.0.0




Component                                Installed Version    Available Version

---------------------------------------- -------------------- --------------------

OAK                                       18.3.0.0.0            up-to-date




GI                                        18.3.0.0.180717       up-to-date




DB                                        18.3.0.0.180717       up-to-date




DCSAGENT                                  18.3.0.0.0            up-to-date




ILOM                                      3.2.9.23.r116695      4.0.2.26.b.r125868




BIOS                                      30110000              30130500




OS                                        6.10                  up-to-date




FIRMWARECONTROLLER {

[ c0 ]                                    4.650.00-7176         up-to-date

[ c1,c2 ]                                 13.00.00.00           up-to-date

}




FIRMWAREEXPANDER                          0018                  up-to-date




FIRMWAREDISK {

[ c0d0,c0d1 ]                             A7E0                  up-to-date

[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date

c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,

c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,

c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,

c2d12,c2d13,c2d14,c2d15 ]

[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date

c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,

c2d20,c2d21,c2d22,c2d23 ]

}




System node Name

---------------

odas2




Local System Version

---------------

18.3.0.0.0




Component                                Installed Version    Available Version

---------------------------------------- -------------------- --------------------

OAK                                       18.3.0.0.0            up-to-date




GI                                        18.3.0.0.180717       up-to-date




DB                                        18.3.0.0.180717       up-to-date




DCSAGENT                                  18.3.0.0.0            up-to-date




ILOM                                      3.2.9.23.r116695      4.0.2.26.b.r125868




BIOS                                      30110000              30130500




OS                                        6.10                  up-to-date




FIRMWARECONTROLLER {

[ c0 ]                                    4.650.00-7176         up-to-date

[ c1,c2 ]                                 13.00.00.00           up-to-date

}




FIRMWAREEXPANDER                          0018                  up-to-date




FIRMWAREDISK {

[ c0d0,c0d1 ]                             A7E0                  up-to-date

[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date

c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,

c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,

c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,

c2d12,c2d13,c2d14,c2d15 ]

[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date

c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,

c2d20,c2d21,c2d22,c2d23 ]

}




root@odas1 ~]#

 

 

After that you can upload the files in the repository (look the path to the files):

 

[root@odas1 deploy]# odacli update-repository -f /tmp/deploy/oda-sm-18.3.0.0.0-181205-server1of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server2of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server3of3.zip

{

  "jobId" : "cdfd2e72-777b-4ccb-a670-17b88c7cc102",

  "status" : "Created",

  "message" : "/tmp/deploy/oda-sm-18.3.0.0.0-181205-server1of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server2of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server3of3.zip",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 15:01:03 PM CEST",

  "resourceList" : [ ],

  "description" : "Repository Update",

  "updatedTime" : "May 22, 2019 15:01:03 PM CEST"

}

[root@odas1 deploy]#

[root@odas1 deploy]#

[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i cdfd2e72-777b-4ccb-a670-17b88c7cc102




Job details

----------------------------------------------------------------

                     ID:  cdfd2e72-777b-4ccb-a670-17b88c7cc102

            Description:  Repository Update

                 Status:  Success

                Created:  May 22, 2019 3:01:03 PM CEST

                Message:  /tmp/deploy/oda-sm-18.3.0.0.0-181205-server1of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server2of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server3of3.zip




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Check AvailableSpace                     May 22, 2019 3:01:03 PM CEST        May 22, 2019 3:01:03 PM CEST        Success

Setting up ssh equivalance               May 22, 2019 3:01:03 PM CEST        May 22, 2019 3:01:03 PM CEST        Success

Copy BundleFile                          May 22, 2019 3:01:03 PM CEST        May 22, 2019 3:02:47 PM CEST        Success

Validating CopiedFile                    May 22, 2019 3:02:49 PM CEST        May 22, 2019 3:03:23 PM CEST        Success

Unzip bundle                             May 22, 2019 3:03:23 PM CEST        May 22, 2019 3:05:43 PM CEST        Success

Unzip bundle                             May 22, 2019 3:05:43 PM CEST        May 22, 2019 3:08:08 PM CEST        Success

Delete PatchBundles                      May 22, 2019 3:08:08 PM CEST        May 22, 2019 3:08:11 PM CEST        Success

Removing ssh keys                        May 22, 2019 3:08:11 PM CEST        May 22, 2019 3:08:26 PM CEST        Success




[root@odas1 deploy]#

[root@odas1 deploy]#

 

After the success to upload you first update the storage using odacli update-storage command to upload to desired version (the version are in the filename or check what you downloaded from MOS):

 

[root@odas1 ~]# odacli update-storage -v 18.3.0.0.0 --rolling

{

  "jobId" : "18d1e5f8-7c31-4ce7-8241-f7b2ed5673cd",

  "status" : "Created",

  "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 15:25:24 PM CEST",

  "resourceList" : [ ],

  "description" : "Storage Firmware Patching",

  "updatedTime" : "May 22, 2019 15:25:24 PM CEST"

}

[root@odas1 ~]#

[root@odas1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i 18d1e5f8-7c31-4ce7-8241-f7b2ed5673cd




Job details

----------------------------------------------------------------

                     ID:  18d1e5f8-7c31-4ce7-8241-f7b2ed5673cd

            Description:  Storage Firmware Patching

                 Status:  Success

                Created:  May 22, 2019 3:25:24 PM CEST

                Message:




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Applying Firmware Disk Patches           May 22, 2019 3:25:24 PM CEST        May 22, 2019 3:25:45 PM CEST        Success

Applying Firmware Controller Patches     May 22, 2019 3:25:45 PM CEST        May 22, 2019 3:25:58 PM CEST        Success




[root@odas1 ~]#
After you can update the server (running odacli update-server). Here, since you are “updating” to the same version that you are running, the steps related to yum does not execute and just firmware part runs:

 

[root@odas1 ~]# odacli update-server -v 18.3.0.0.0

{

  "jobId" : "8bc7cc15-a543-4e26-ade4-8dd578ea67ba",

  "status" : "Created",

  "message" : "Success of Server Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 15:29:15 PM CEST",

  "resourceList" : [ ],

  "description" : "Server Patching",

  "updatedTime" : "May 22, 2019 15:29:15 PM CEST"

}

[root@odas1 ~]#

[root@odas1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i 8bc7cc15-a543-4e26-ade4-8dd578ea67ba




Job details

----------------------------------------------------------------

                     ID:  8bc7cc15-a543-4e26-ade4-8dd578ea67ba

            Description:  Server Patching

                 Status:  Running

                Created:  May 22, 2019 3:29:15 PM CEST

                Message:




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Patch location validation                May 22, 2019 3:29:16 PM CEST        May 22, 2019 3:29:16 PM CEST        Success

Patch location validation                May 22, 2019 3:29:16 PM CEST        May 22, 2019 3:29:16 PM CEST        Success

dcs-controller upgrade                   May 22, 2019 3:29:19 PM CEST        May 22, 2019 3:29:19 PM CEST        Success

dcs-controller upgrade                   May 22, 2019 3:29:19 PM CEST        May 22, 2019 3:29:20 PM CEST        Success

Patch location validation                May 22, 2019 3:29:20 PM CEST        May 22, 2019 3:29:21 PM CEST        Success

Patch location validation                May 22, 2019 3:29:20 PM CEST        May 22, 2019 3:29:21 PM CEST        Success

dcs-cli upgrade                          May 22, 2019 3:29:21 PM CEST        May 22, 2019 3:29:21 PM CEST        Success

dcs-cli upgrade                          May 22, 2019 3:29:21 PM CEST        May 22, 2019 3:29:21 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:29 PM CEST        May 22, 2019 3:29:36 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:29:37 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:29:37 PM CEST        Success

Updating YumPluginVersionLock rpm        May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:29:37 PM CEST        Success

Applying OS Patches                      May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:31:38 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:31:38 PM CEST        May 22, 2019 3:31:39 PM CEST        Success

Applying HMP Patches                     May 22, 2019 3:31:39 PM CEST        May 22, 2019 3:31:39 PM CEST        Success

Patch location validation                May 22, 2019 3:31:40 PM CEST        May 22, 2019 3:31:40 PM CEST        Success

Patch location validation                May 22, 2019 3:31:40 PM CEST        May 22, 2019 3:31:40 PM CEST        Success

oda-hw-mgmt upgrade                      May 22, 2019 3:31:40 PM CEST        May 22, 2019 3:31:41 PM CEST        Success

oda-hw-mgmt upgrade                      May 22, 2019 3:31:41 PM CEST        May 22, 2019 3:31:41 PM CEST        Success

OSS Patching                             May 22, 2019 3:31:41 PM CEST        May 22, 2019 3:31:43 PM CEST        Success

Applying Firmware Disk Patches           May 22, 2019 3:32:02 PM CEST        May 22, 2019 3:32:15 PM CEST        Success

Applying Firmware Expander Patches       May 22, 2019 3:32:27 PM CEST        May 22, 2019 3:32:34 PM CEST        Success

Applying Firmware Controller Patches     May 22, 2019 3:32:47 PM CEST        May 22, 2019 3:32:55 PM CEST        Success

Checking Ilom patch Version              May 22, 2019 3:32:56 PM CEST        May 22, 2019 3:32:59 PM CEST        Success

Checking Ilom patch Version              May 22, 2019 3:32:59 PM CEST        May 22, 2019 3:33:01 PM CEST        Success

Patch location validation                May 22, 2019 3:33:01 PM CEST        May 22, 2019 3:33:02 PM CEST        Success

Patch location validation                May 22, 2019 3:33:01 PM CEST        May 22, 2019 3:33:02 PM CEST        Success

Save password in Wallet                  May 22, 2019 3:33:03 PM CEST        May 22, 2019 3:33:04 PM CEST        Success

Apply Ilom patch                         May 22, 2019 3:33:04 PM CEST        May 22, 2019 3:45:07 PM CEST        Success

Apply Ilom patch                         May 22, 2019 3:45:07 PM CEST        May 22, 2019 3:56:45 PM CEST        Success

Copying Flash Bios to Temp location      May 22, 2019 3:56:45 PM CEST        May 22, 2019 3:56:45 PM CEST        Success

Copying Flash Bios to Temp location      May 22, 2019 3:56:45 PM CEST        May 22, 2019 3:56:45 PM CEST        Success

Starting the clusterware                 May 22, 2019 3:58:41 PM CEST        May 22, 2019 4:00:09 PM CEST        Success

clusterware patch verification           May 22, 2019 4:00:09 PM CEST        May 22, 2019 4:00:10 PM CEST        Success

clusterware patch verification           May 22, 2019 4:00:09 PM CEST        May 22, 2019 4:00:10 PM CEST        Success

Patch location validation                May 22, 2019 4:00:10 PM CEST        May 22, 2019 4:00:10 PM CEST        Success

Patch location validation                May 22, 2019 4:00:10 PM CEST        May 22, 2019 4:00:10 PM CEST        Success

Opatch updation                          May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

Opatch updation                          May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

Patch conflict check                     May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

Patch conflict check                     May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

clusterware upgrade                      May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

clusterware upgrade                      May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

Updating GiHome version                  May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:12 PM CEST        Success

Updating GiHome version                  May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:12 PM CEST        Success

Update System version                    May 22, 2019 4:00:33 PM CEST        May 22, 2019 4:00:33 PM CEST        Success

Update System version                    May 22, 2019 4:00:33 PM CEST        May 22, 2019 4:00:33 PM CEST        Success

preRebootNode Actions                    May 22, 2019 4:00:33 PM CEST        May 22, 2019 4:00:33 PM CEST        Running




[root@odas1 ~]#

 

 

After, reboot both nodes!
As example check that ilom was updated:

 

[root@odas1 ~]# odacli describe-component

System Version

---------------

18.3.0.0.0




System node Name

---------------

odas1




Local System Version

---------------

18.3.0.0.0




Component                                Installed Version    Available Version

---------------------------------------- -------------------- --------------------

OAK                                       18.3.0.0.0            up-to-date




GI                                        18.3.0.0.180717       up-to-date




DB                                        18.3.0.0.180717       up-to-date




DCSAGENT                                  18.3.0.0.0            up-to-date




ILOM                                      4.0.2.26.b.r125868    up-to-date




BIOS                                      30130500              up-to-date




OS                                        6.10                  up-to-date




FIRMWARECONTROLLER {

[ c0 ]                                    4.650.00-7176         up-to-date

[ c1,c2 ]                                 13.00.00.00           up-to-date

}




FIRMWAREEXPANDER                          0018                  up-to-date




FIRMWAREDISK {

[ c0d0,c0d1 ]                             A7E0                  up-to-date

[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date

c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,

c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,

c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,

c2d12,c2d13,c2d14,c2d15 ]

[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date

c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,

c2d20,c2d21,c2d22,c2d23 ]

}




System node Name

---------------

odas2




Local System Version

---------------

18.3.0.0.0




Component                                Installed Version    Available Version

---------------------------------------- -------------------- --------------------

OAK                                       18.3.0.0.0            up-to-date




GI                                        18.3.0.0.180717       up-to-date




DB                                        18.3.0.0.180717       up-to-date




DCSAGENT                                  18.3.0.0.0            up-to-date




ILOM                                      4.0.2.26.b.r125868    up-to-date




BIOS                                      30130500              up-to-date




OS                                        6.10                  up-to-date




FIRMWARECONTROLLER {

[ c0 ]                                    4.650.00-7176         up-to-date

[ c1,c2 ]                                 13.00.00.00           up-to-date

}




FIRMWAREEXPANDER                          0018                  up-to-date




FIRMWAREDISK {

[ c0d0,c0d1 ]                             A7E0                  up-to-date

[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date

c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,

c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,

c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,

c2d12,c2d13,c2d14,c2d15 ]

[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date

c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,

c2d20,c2d21,c2d22,c2d23 ]

}







[root@odas1 ~]#

[root@odas1 ~]#
 

Create Oracle Homes and the databases

After update all the firmware’s you can upload the clone files for Oracle Homes versions that you want and create it. The procedure is the same than before, upload the files to (and just) ODA node 1 in somewhere and after that upload the repository and create the homes.
Upload the files and update the repository. Look below that I made one for each version:

 

[root@odas1 ~]# cd /tmp/deploy/

[root@odas1 deploy]#

[root@odas1 deploy]# unzip -qa p23494992_183000_Linux-x86-64.zip

[root@odas1 deploy]# rm p23494992_183000_Linux-x86-64.zip

rm: remove regular file `p23494992_183000_Linux-x86-64.zip'? y

[root@odas1 deploy]#

[root@odas1 deploy]# unzip -qa p23494997_183000_Linux-x86-64.zip

replace README.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: A

[root@odas1 deploy]# rm p23494997_183000_Linux-x86-64.zip

rm: remove regular file `p23494997_183000_Linux-x86-64.zip'? y

[root@odas1 deploy]#

[root@odas1 deploy]# odacli update-repository -f /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-12.1.0.2.zip

{

  "jobId" : "b88c4922-1107-41f5-b7cd-e1648093051b",

  "status" : "Created",

  "message" : "/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-12.1.0.2.zip",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 16:34:28 PM CEST",

  "resourceList" : [ ],

  "description" : "Repository Update",

  "updatedTime" : "May 22, 2019 16:34:28 PM CEST"

}

[root@odas1 deploy]#

[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i b88c4922-1107-41f5-b7cd-e1648093051b




Job details

----------------------------------------------------------------

                     ID:  b88c4922-1107-41f5-b7cd-e1648093051b

            Description:  Repository Update

                 Status:  Success

                Created:  May 22, 2019 4:34:28 PM CEST

                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-12.1.0.2.zip




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Check AvailableSpace                     May 22, 2019 4:34:28 PM CEST        May 22, 2019 4:34:28 PM CEST        Success

Setting up ssh equivalance               May 22, 2019 4:34:28 PM CEST        May 22, 2019 4:34:29 PM CEST        Success

Copy BundleFile                          May 22, 2019 4:34:29 PM CEST        May 22, 2019 4:34:54 PM CEST        Success

Validating CopiedFile                    May 22, 2019 4:34:54 PM CEST        May 22, 2019 4:35:07 PM CEST        Success

Unzip bundle                             May 22, 2019 4:35:07 PM CEST        May 22, 2019 4:35:42 PM CEST        Success

Unzip bundle                             May 22, 2019 4:35:46 PM CEST        May 22, 2019 4:36:23 PM CEST        Success

Delete PatchBundles                      May 22, 2019 4:36:24 PM CEST        May 22, 2019 4:36:25 PM CEST        Success

Removing ssh keys                        May 22, 2019 4:36:25 PM CEST        May 22, 2019 4:36:47 PM CEST        Success




[root@odas1 deploy]#

[root@odas1 deploy]# odacli update-repository -f /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-11.2.0.4.zip

{

  "jobId" : "6698aa6d-1a44-4b1c-bda1-1e691a41a133",

  "status" : "Created",

  "message" : "/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-11.2.0.4.zip",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 16:38:53 PM CEST",

  "resourceList" : [ ],

  "description" : "Repository Update",

  "updatedTime" : "May 22, 2019 16:38:53 PM CEST"

}

[root@odas1 deploy]#

[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i 6698aa6d-1a44-4b1c-bda1-1e691a41a133




Job details

----------------------------------------------------------------

                     ID:  6698aa6d-1a44-4b1c-bda1-1e691a41a133

            Description:  Repository Update

                 Status:  Success

                Created:  May 22, 2019 4:38:53 PM CEST

                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-11.2.0.4.zip




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Check AvailableSpace                     May 22, 2019 4:38:53 PM CEST        May 22, 2019 4:38:53 PM CEST        Success

Setting up ssh equivalance               May 22, 2019 4:38:53 PM CEST        May 22, 2019 4:38:53 PM CEST        Success

Copy BundleFile                          May 22, 2019 4:38:54 PM CEST        May 22, 2019 4:39:04 PM CEST        Success

Validating CopiedFile                    May 22, 2019 4:39:06 PM CEST        May 22, 2019 4:39:14 PM CEST        Success

Unzip bundle                             May 22, 2019 4:39:14 PM CEST        May 22, 2019 4:39:32 PM CEST        Success

Unzip bundle                             May 22, 2019 4:39:32 PM CEST        May 22, 2019 4:39:54 PM CEST        Success

Delete PatchBundles                      May 22, 2019 4:39:54 PM CEST        May 22, 2019 4:39:55 PM CEST        Success

Removing ssh keys                        May 22, 2019 4:39:55 PM CEST        May 22, 2019 4:39:55 PM CEST        Success




[root@odas1 deploy]#

[root@odas1 deploy]#

After that, you can create the Oracle home for each version using the command odacli create-dbhome. 12c:

 

[root@odas1 deploy]#

[root@odas1 deploy]# odacli create-dbhome -v 12.1.0.2.180717




Job details

----------------------------------------------------------------

                     ID:  5ddcdd49-2d1a-404d-b4b8-7ab79c5f9707

            Description:  Database Home OraDB12102_home1 creation with version :12.1.0.2.180717

                 Status:  Created

                Created:  May 22, 2019 4:42:58 PM CEST

                Message:  Create Database Home




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------




[root@odas1 deploy]#

[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i 5ddcdd49-2d1a-404d-b4b8-7ab79c5f9707




Job details

----------------------------------------------------------------

                     ID:  5ddcdd49-2d1a-404d-b4b8-7ab79c5f9707

            Description:  Database Home OraDB12102_home1 creation with version :12.1.0.2.180717

                 Status:  Success

                Created:  May 22, 2019 4:42:58 PM CEST

                Message:  Create Database Home




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Setting up ssh equivalance               May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success

Validating dbHome available space        May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success

Validating dbHome available space        May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success

Creating DbHome Directory                May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success

Extract DB clones                        May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:46:22 PM CEST        Success

Clone Db home                            May 22, 2019 4:46:22 PM CEST        May 22, 2019 4:47:32 PM CEST        Success

Enable DB options                        May 22, 2019 4:47:32 PM CEST        May 22, 2019 4:47:41 PM CEST        Success

Run Root DB scripts                      May 22, 2019 4:47:41 PM CEST        May 22, 2019 4:47:42 PM CEST        Success

Removing ssh keys                        May 22, 2019 4:47:48 PM CEST        May 22, 2019 4:47:48 PM CEST        Success




[root@odas1 deploy]#

[root@odas1 deploy]#
And 11g:

 

[root@odas1 deploy]# odacli create-dbhome -v 11.2.0.4.180717




Job details

----------------------------------------------------------------

                     ID:  d4512833-3198-4384-b0d5-3588e3ec8cdb

            Description:  Database Home OraDB11204_home1 creation with version :11.2.0.4.180717

                 Status:  Created

                Created:  May 22, 2019 4:50:06 PM CEST

                Message:  Create Database Home




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------




[root@odas1 deploy]#

[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i d4512833-3198-4384-b0d5-3588e3ec8cdb




Job details

----------------------------------------------------------------

                     ID:  d4512833-3198-4384-b0d5-3588e3ec8cdb

            Description:  Database Home OraDB11204_home1 creation with version :11.2.0.4.180717

                 Status:  Success

                Created:  May 22, 2019 4:50:06 PM CEST

                Message:  Create Database Home




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Setting up ssh equivalance               May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success

Validating dbHome available space        May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success

Validating dbHome available space        May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success

Creating DbHome Directory                May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success

Extract DB clones                        May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:51:59 PM CEST        Success

Clone Db home                            May 22, 2019 4:52:00 PM CEST        May 22, 2019 4:53:06 PM CEST        Success

Enable DB options                        May 22, 2019 4:53:06 PM CEST        May 22, 2019 4:53:11 PM CEST        Success

Run Root DB scripts                      May 22, 2019 4:53:11 PM CEST        May 22, 2019 4:53:11 PM CEST        Success

Removing ssh keys                        May 22, 2019 4:53:16 PM CEST        May 22, 2019 4:53:17 PM CEST        Success




[root@odas1 deploy]#

[root@odas1 deploy]#


And after that you can use the odacli list-dbhomes to see the options for database creation:

 

ID                                       Name                 DB Version                               Home Location                                 Status

---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------

886021e5-9bbf-4a14-9b50-398ddd00bfd0     OraDB18000_home1     18.3.0.0.180717                          /u01/app/oracle/product/18.0.0.0/dbhome_1     Configured

d791001f-b812-49fd-94f5-b11c5f580a30     OraDB12102_home1     12.1.0.2.180717                          /u01/app/oracle/product/12.1.0.2/dbhome_1     Configured

d162ff30-dc0d-4c4b-810e-c8f513dc9f62     OraDB11204_home1     11.2.0.4.180717                          /u01/app/oracle/product/11.2.0.4/dbhome_1     Configured




[root@odas1 deploy]#


As an example, if you want to delete the database that was created during the creation of appliance just execute odacli delete-database:

 

[root@odas1 ~]# odacli delete-database --dbName mfod

{

  "jobId" : "69adb079-e5e4-4d64-8b20-4eee73c65491",

  "status" : "Running",

  "message" : null,

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 16:17:27 PM CEST",

  "resourceList" : [ ],

  "description" : "Database service deletion with db name: mfod with id : 938b616a-99ed-4466-8d76-820d3489517d",

  "updatedTime" : "May 22, 2019 16:17:27 PM CEST"

}

[root@odas1 ~]#



Finish and clean the install

After all the steps listed until now, you have your ODA reimaged, updated, and with 18c, 12c and 11g available version for database creation. To finish you can update some files as needed. Examples of important files are /etc/fstab, /etc/sysctl.conf, /etc/sysconfig/network-scripts/route* and /etc/sysconfig/network-scripts/rules* (both related with routing tables for vlans – if needed), and add the listener for all the vlans that you created (if needed of course).
To finish, reboot both nodes and change the root password.
 
Reference that can be consulted to help you to understand the reimage and deployment procedure:
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/oracle-database-appliance-x7-2-deployment-and-users-guide.pdf         
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxd/oracle-database-appliance-x5-2-x4-2-x3-2-deployment-and-users-guide.pdf
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/odacli-comparison-chart.html#GUID-567E862F-F4A4-4E1E-A485-BD1945BEC673
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/postinstallation-tasks-oda.html#GUID-A256DFBD-9335-41D2-9BD5-A1BA6AAC4C1E
https://docs.oracle.com/cd/E80799_01/doc.121/e80521/GUID-4EAD207D-41FF-4FD7-8804-DE1B2EB7BE90.htm#CMTXG-GUID-E0CC0E23-A339-45D8-AFA2-09732DC6BC18
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/reimaging-oda.html#GUID-6DEDCA82-94B7-4155-9FBD-3B1FDC232FCC
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.5/cmtxl/create-appliance-using-json-file.html#GUID-42250FD2-EA91-4457-9ED7-CA3A2A863B40
https://docs.oracle.com/cd/E75550_01/doc.121/e77147/GUID-F99D3337-E913-405D-BE3F-85C31103A26A.htm#CMTAR-GUID-CF1FECF0-722F-4B43-82AA-525EA169B3E8