Recently I made an Exadata stack upgrade/update to the last 19.2 version (19.2.7.0.0.191012) released in October of 2019, and update the GI to the last 19c version (19.5.0.0.191015) and after that, I hade some issues to create 11G databases.

So, when I try to create an 11G RAC database, the error “File -oracka.ko- was not found” appears and creation fails. Here I want to share with you the workaround (since there is no solution yet) that I discovered and used to bypass the error.

The environment

The actual environment is:

• Grid Infrastructure: Version 19.5.0.0.191015

• Exadata domU: Version 19.2.7.0.0.191012 running kernel 4.1.12-124.30.1.el7uek.x86_64

• 11G Database: Version 11.2.0.4.180717

• ACFS: Used to store some files

oracka.ko

So, calling dbca:

[DEV-oracle@exsite1c1-]$ /u01/app/oracle/product/11.2.0.4/dbhome_1/bin/dbca -silent -createDatabase -templateName General_Purpose.dbc -gdbName D11TST19 -adminManaged -sid D11TST19 -sysPassword oracle11 -systemPassword oracle11 -characterSet WE8ISO8859P15 -emConfiguration NONE -storageType ASM -diskGroupName DATAC8 -recoveryGroupName RECOC8 -nodelist exsite1c1,exsite1c2 -sampleSchema false

Copying database files

100% complete

Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/D11TST19/D11TST19.log" for further details.

[DEV-oracle@exsite1c1-]$

And at the log:

[DEV-oracle@exsite1c1-]$ cat /u01/app/oracle/cfgtoollogs/dbca/D11TST19/D11TST19.log

Creating and starting Oracle instance

PRCR-1006 : Failed to add resource ora.d11tst19.db for d11tst19

PRCD-1094 : Failed to create start dependency for database d11tst19 on the ASM cluster filesystem where ACFS path /u01/app/oracle/product/11.2.0.4/dbhome_1 resides

PRCT-1129 : Execution of ACFS driver state check failed on local node with result Errors uncompressing file: oracka.zip.

File -oracka.ko- was not found.

Error # 50 - . Output: .

Errors uncompressing file: oracka.zip.

File -oracka.ko- was not found.

Error # 50 - . Output: .

ACFS-9200: Supported

DBCA_PROGRESS : 100%

[DEV-oracle@exsite1c1-]$

If you check for the trace of database creation:

[DEV-oracle@exsite1c1-]$ vi /u01/app/oracle/cfgtoollogs/dbca/D11TST19/trace.log

[Thread-166] [ 2019-11-07 08:30:37.860 CET ] [ASMFactoryImpl.isACFSSupported:954]  Entry: oraHome=/u01/app/oracle/product/11.2.0.4/dbhome_1, m_crsHome=/u01/app/19.0.0.0/grid

[Thread-166] [ 2019-11-07 08:30:37.860 CET ] [ASMFactoryImpl.isACFSSupported:958]  Checking if ACFS now...

[Thread-166] [ 2019-11-07 08:30:37.861 CET ] [USMDriverUtil.<init>:117]  Checking file exists for: /u01/app/19.0.0.0/grid/bin/acfsdriverstate

[Thread-166] [ 2019-11-07 08:30:37.862 CET ] [NativeSystem.isCmdScv:502]  isCmdScv: cmd=[]

[Thread-182] [ 2019-11-07 08:30:37.866 CET ] [StreamReader.run:61]  In StreamReader.run

[Thread-181] [ 2019-11-07 08:30:37.868 CET ] [StreamReader.run:61]  In StreamReader.run

[Thread-182] [ 2019-11-07 08:30:38.230 CET ] [StreamReader.run:65]  ERROR>Argument "2/8" isn't numeric in numeric ne (!=) at /u01/app/19.0.0.0/grid/lib/osds_acfsroot.pm line 2093.

[Thread-182] [ 2019-11-07 08:30:38.230 CET ] [StreamReader.run:65]  ERROR>Argument "2/8" isn't numeric in numeric ne (!=) at /u01/app/19.0.0.0/grid/lib/osds_acfsroot.pm line 2093.

[Thread-182] [ 2019-11-07 08:30:38.235 CET ] [StreamReader.run:65]  ERROR>error:  cannot delete old /u01/app/19.0.0.0/grid/usm/install/Oracle/EL7UEK/x86_64/4.1.12-112.16.4/4.1.12-112.16.4-x86_64/bin/oracka.ko

[Thread-182] [ 2019-11-07 08:30:38.235 CET ] [StreamReader.run:65]  ERROR>        Permission denied

[Thread-181] [ 2019-11-07 08:30:38.310 CET ] [StreamReader.run:65]  OUTPUT>Errors uncompressing file: oracka.zip.

[Thread-181] [ 2019-11-07 08:30:38.405 CET ] [StreamReader.run:65]  OUTPUT>File -oracka.ko- was not found.

[Thread-181] [ 2019-11-07 08:30:38.485 CET ] [StreamReader.run:65]  OUTPUT>Error # 50 - . Output: .

[Thread-182] [ 2019-11-07 08:30:38.504 CET ] [StreamReader.run:65]  ERROR>error:  cannot delete old /u01/app/19.0.0.0/grid/usm/install/Oracle/EL7UEK/x86_64/4.1.12-112.16.4/4.1.12-112.16.4-x86_64/bin/oracka.ko

[Thread-182] [ 2019-11-07 08:30:38.504 CET ] [StreamReader.run:65]  ERROR>        Permission denied

[Thread-181] [ 2019-11-07 08:30:38.587 CET ] [StreamReader.run:65]  OUTPUT>Errors uncompressing file: oracka.zip.

[Thread-181] [ 2019-11-07 08:30:38.667 CET ] [StreamReader.run:65]  OUTPUT>File -oracka.ko- was not found.

[Thread-181] [ 2019-11-07 08:30:38.753 CET ] [StreamReader.run:65]  OUTPUT>Error # 50 - . Output: .

[Thread-181] [ 2019-11-07 08:30:40.877 CET ] [StreamReader.run:65]  OUTPUT>ACFS-9200: Supported

[Thread-166] [ 2019-11-07 08:30:40.881 CET ] [UnixSystem.dorunRemoteExecCmd:3232]  retval = 0

[Thread-166] [ 2019-11-07 08:30:40.881 CET ] [UnixSystem.dorunRemoteExecCmd:3256]  exitValue = 0

[Thread-166] [ 2019-11-07 08:30:40.882 CET ] [USMDriverUtil.checkACFSState:267]  Printing ACFS output

[Thread-166] [ 2019-11-07 08:30:40.882 CET ] [USMDriverUtil.checkACFSState:268]  Errors uncompressing file: oracka.zip.

File -oracka.ko- was not found.

Error # 50 - . Output: .

Errors uncompressing file: oracka.zip.

File -oracka.ko- was not found.

Error # 50 - . Output: .

ACFS-9200: Supported

[Thread-166] [ 2019-11-07 08:30:40.882 CET ] [InstanceStepOPS.executeImpl:1014]  PRCR-1006 : Failed to add resource ora.d11tst19.db for d11tst19

PRCD-1094 : Failed to create start dependency for database d11tst19 on the ASM cluster filesystem where ACFS path /u01/app/oracle/product/11.2.0.4/dbhome_1 resides

PRCT-1129 : Execution of ACFS driver state check failed on local node with result Errors uncompressing file: oracka.zip.

File -oracka.ko- was not found.

Error # 50 - . Output: .

Errors uncompressing file: oracka.zip.

File -oracka.ko- was not found.

Error # 50 - . Output: .

ACFS-9200: Supported

[Thread-166] [ 2019-11-07 08:30:40.883 CET ] [BasicStep.configureSettings:304]  messageHandler being set=oracle.sysman.assistants.util.SilentMessageHandler@5a943dc4

[Thread-166] [ 2019-11-07 08:30:40.883 CET ] [BasicStep.configureSettings:304]  messageHandler being set=oracle.sysman.assistants.util.SilentMessageHandler@5a943dc4

oracle.sysman.assistants.util.step.StepExecutionException: PRCR-1006 : Failed to add resource ora.d11tst19.db for d11tst19

PRCD-1094 : Failed to create start dependency for database d11tst19 on the ASM cluster filesystem where ACFS path /u01/app/oracle/product/11.2.0.4/dbhome_1 resides

PRCT-1129 : Execution of ACFS driver state check failed on local node with result Errors uncompressing file: oracka.zip.

File -oracka.ko- was not found.

Error # 50 - . Output: .

Errors uncompressing file: oracka.zip.

File -oracka.ko- was not found.

Error # 50 - . Output: .

ACFS-9200: Supported

        at oracle.sysman.assistants.dbca.backend.InstanceStepOPS.executeImpl(InstanceStepOPS.java:1015)

        at oracle.sysman.assistants.util.step.BasicStep.execute(BasicStep.java:210)

        at oracle.sysman.assistants.util.step.BasicStep.callStep(BasicStep.java:251)

        at oracle.sysman.assistants.dbca.backend.DBEntryStep.executeImpl(DBEntryStep.java:229)

        at oracle.sysman.assistants.util.step.BasicStep.execute(BasicStep.java:210)

        at oracle.sysman.assistants.util.step.Step.execute(Step.java:140)

        at oracle.sysman.assistants.util.step.StepContext$ModeRunner.run(StepContext.java:2711)

        at java.lang.Thread.run(Thread.java:637)

[Thread-166] [ 2019-11-07 08:30:40.883 CET ] [SQLEngine.done:2189]  Done called

So, as you can see in the trace the process for database creation when detecting that ACFS is in place, tries to unzip some files (.ko, kernel objects) related to ACFS access but receive the error or that cannot delete old oracka.ko and for uncompressing file oracka.zip.

Source of the problem

Checking for the problem, we can see that access denied occurs for folder $GI_HOME/usm/install/Oracle/EL7UEK/x86_64/4.1.12-112.16.4/4.1.12-112.16.4-x86_64/bin/ inside of GI home. And as you can see is the same family of our domU kernel version. Inside os MOS there is no reference for this error, and after open the SR for Oracle was sent do dev team without ETA (as usual).

Because of that I continue to check and checked the folder:

[root@exsite1c1 ~]# cd /u01/app/19.0.0.0/grid/usm/install/Oracle/EL7UEK/x86_64/4.1.12-112.16.4/4.1.12-112.16.4-x86_64/bin/

[root@exsite1c1 bin]#

[root@exsite1c1 bin]# ls -l

total 151244

-rw-r--r-- 1 root oinstall  3085340 Sep  1 12:40 oracka.ko

-rw-r--r-- 1 root oinstall  2304100 Sep  1 12:44 oracka_mod_kga.ko

-rw-r--r-- 1 grid oinstall   623207 Oct  9 17:13 oracka_mod_kga.zip

-rw-r--r-- 1 grid oinstall   889060 Oct  9 17:14 oracka.zip

-rw-r--r-- 1 root oinstall 64000932 Sep  1 12:04 oracleacfs.ko

-rw-r--r-- 1 grid oinstall 19246647 Oct  9 17:13 oracleacfs.zip

-rw-r--r-- 1 root oinstall 27882452 Sep  1 11:25 oracleadvm.ko

-rw-r--r-- 1 grid oinstall  8777000 Oct  9 17:13 oracleadvm.zip

-rw-r--r-- 1 root oinstall  8776676 Sep  1 12:55 oracleafd.ko

-rw-r--r-- 1 grid oinstall  2879744 Oct  9 17:14 oracleafd.zip

-rw-r--r-- 1 root oinstall 10696156 Sep  1 11:04 oracleoks.ko

-rw-r--r-- 1 grid oinstall  3346594 Oct  9 17:14 oracleoks.zip

-rw-r--r-- 1 root oinstall  1757116 Sep  1 13:02 oracleolfs.ko

-rw-r--r-- 1 grid oinstall   567865 Oct  9 17:14 oracleolfs.zip

[root@exsite1c1 bin]#

As you can see, the GI Home came with some ko files already uncompressed and the permission for a group just as read. So, the delete and uncompressing by dbca will not work properly, and as a workaround to solve the issue, I changed the permission for folder bin and ko files to allow change for oinstall – made in all nodes of the cluster:

[root@exsite1c1 4.1.12-112.16.4-x86_64]# pwd

/u01/app/19.0.0.0/grid/usm/install/Oracle/EL7UEK/x86_64/4.1.12-112.16.4/4.1.12-112.16.4-x86_64

[root@exsite1c1 4.1.12-112.16.4-x86_64]#

[root@exsite1c1 4.1.12-112.16.4-x86_64]#

[root@exsite1c1 4.1.12-112.16.4-x86_64]# chmod g+w bin

[root@exsite1c1 4.1.12-112.16.4-x86_64]#

[root@exsite1c1 4.1.12-112.16.4-x86_64]#

[root@exsite1c1 4.1.12-112.16.4-x86_64]# ls -l

total 4

drwxrwxr-x 2 grid oinstall 4096 Nov  8 15:08 bin

[root@exsite1c1 4.1.12-112.16.4-x86_64]# cd bin

[root@exsite1c1 bin]# chown grid:oinstall oracka.ko oracka_mod_kga.ko oracleacfs.ko oracleadvm.ko oracleafd.ko oracleoks.ko oracleolfs.ko

[root@exsite1c1 bin]#

After executing this in both nodes, the dbca finish:

[DEV-oracle@exsite1c1-]$ /u01/app/oracle/product/11.2.0.4/dbhome_1/bin/dbca -silent -createDatabase -templateName General_Purpose.dbc -gdbName D11TST19 -adminManaged -sid D11TST19 -sysPassword oracle11 -systemPassword orcle11 -characterSet WE8ISO8859P15 -emConfiguration NONE -storageType ASM -diskGroupName DATAC8 -recoveryGroupName RECOC8 -nodelist exsite1c1,exsite1c2 -sampleSchema false

Creating and starting Oracle instance

1% complete

…

96% complete

100% complete

Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/D11TST19/D11TST19.log" for further details.

[DEV-oracle@exsite1c1-]$

[DEV-oracle@exsite1c1-]$ $ORACLE_HOME/bin/srvctl status database -d D11TST19

Instance D11TST191 is running on node exsite1c1

Instance D11TST192 is running on node exsite1c2

[DEV-oracle@exsite1c1-]$

Just to check the ko files and bin folder already came with wrong permissions in the original 19c packages:

[grid@exsite1c1 +ASM1]$ cd /u01/patches/grid

[grid@exsite1c1 +ASM1]$ unzip -q V982068-01.zip

[grid@exsite1c1 +ASM1]$ cd usm/install/Oracle/EL7UEK/x86_64/4.1.12-112.16.4/4.1.12-112.16.4-x86_64/bin

[grid@exsite1c1 +ASM1]$ ls -l

total 149116

-rw-r--r-- 1 grid oinstall  3085340 Feb 23  2019 oracka.ko

-rw-r--r-- 1 grid oinstall  2304100 Feb 23  2019 oracka_mod_kga.ko

-rw-r--r-- 1 grid oinstall   623214 Feb 23  2019 oracka_mod_kga.zip

-rw-r--r-- 1 grid oinstall   889106 Feb 23  2019 oracka.zip

-rw-r--r-- 1 grid oinstall 62740980 Feb 23  2019 oracleacfs.ko

-rw-r--r-- 1 grid oinstall 18881690 Feb 23  2019 oracleacfs.zip

-rw-r--r-- 1 grid oinstall 27485580 Feb 23  2019 oracleadvm.ko

-rw-r--r-- 1 grid oinstall  8643808 Feb 23  2019 oracleadvm.zip

-rw-r--r-- 1 grid oinstall  8773444 Feb 23  2019 oracleafd.ko

-rw-r--r-- 1 grid oinstall  2878816 Feb 23  2019 oracleafd.zip

-rw-r--r-- 1 grid oinstall 10655524 Feb 23  2019 oracleoks.ko

-rw-r--r-- 1 grid oinstall  3335912 Feb 23  2019 oracleoks.zip

-rw-r--r-- 1 grid oinstall  1757348 Feb 23  2019 oracleolfs.ko

-rw-r--r-- 1 grid oinstall   567876 Feb 23  2019 oracleolfs.zip

[grid@exsite1c1 +ASM1]$ cd ..

[grid@exsite1c1 +ASM1]$ ls -l

total 4

drwxr-xr-x 2 grid oinstall 4096 Apr 18  2019 bin

[grid@exsite1c1 +ASM1]$

The sum of all problems

This error for oracka.zip and oracka.ko occurs because there is a sum of little details. The release of GI 19c version already came with bad folder permission. The release update 19.5.0.0.191015 does not solve the issue too (and even the installation or rootupgrade script). This bin folder was checked by dbca because the kernel image for Exadata 19.2.7.0.0 (19.2.7.0.0.191012) comes with kernel 4.1.12-124.30.1.el7uek.x86_64 and the ACFS modules for this family are there. And to finish the sum of all problems, the ACFS was used and for that reason, it tries to use that kernel modules to check access.

So, since there is no solution at MOS (today while I am publishing this post), and there is no official workaround, was needed to find own solution. So, the workaround described here it is just temporary to allow 11g RAC database creation for this specific sun of Exadata Version + GI Home 19c + ACFS in use.

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”