RMAN, Allocate channel, CDB, and CLOSE, bug
Allocate channel for RMAN is used in various scenarios, most of the time is useful when you use tape as device type or you need to use some kind of format. The way to do the allocation not changed since a long time ago, but when you run the database in container mode you can hit a bug that turns your channel unusable. I will show you the bug and how to avoid it with a simple trick.
This bug hit every version since 12 and I discovered it last year when testing some scenarios, but I was able to test and post just recently. It just occurs for CDB databases and exists just one one-off solution published for 12.2. But there is one workaround more useful and works for every version.
The most interesting part is that everything that we made until now when allocate channel will not work. You can search in all doc available for allocate channel since 9i until 19c the first thing that you made after open the run{} is allocate channel. This is the default and recommended in the docs:
https://docs.oracle.com/en/database/oracle/oracle-database/19/rcmrf/ALLOCATE-CHANNEL.html#GUID-9320BFF7-0728-4B3D-85B9-2184557ECDCE,
https://docs.oracle.com/en/database/oracle/oracle-database/18/rcmrf/ALLOCATE-CHANNEL.html#GUID-9320BFF7-0728-4B3D-85B9-2184557ECDCE
https://docs.oracle.com/cd/A97630_01/server.920/a96566/rcmconc1.htm
My environment for this post is 18c database (last update from April 2019) with just one PDB:
[oracle@orcloel7 ~]$ sqlplus / as sysdba
SQL*Plus: Release 18.0.0.0.0 - Production on Sun Jun 30 23:24:23 2019
Version 18.6.0.0.0
Copyright (c) 1982, 2018, Oracle. All rights reserved.
Connected to:
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.6.0.0.0
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 ORCL18P READ WRITE NO
SQL> exit
Disconnected from Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.6.0.0.0
[oracle@orcloel7 ~]$
And for rman I show the schema and the backup for one datafile (#12) just to have smaller output.
RMAN> report schema;
Report of database schema for database with db_unique_name ORCL18C
List of Permanent Datafiles
===========================
File Size(MB) Tablespace RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1 870 SYSTEM YES /u01/app/oracle/oradata/ORCL18C/system01.dbf
3 490 SYSAUX NO /u01/app/oracle/oradata/ORCL18C/sysaux01.dbf
4 330 UNDOTBS1 YES /u01/app/oracle/oradata/ORCL18C/undotbs01.dbf
5 290 PDB$SEED:SYSTEM NO /u01/app/oracle/oradata/ORCL18C/pdbseed/system01.dbf
6 340 PDB$SEED:SYSAUX NO /u01/app/oracle/oradata/ORCL18C/pdbseed/sysaux01.dbf
7 5 USERS NO /u01/app/oracle/oradata/ORCL18C/users01.dbf
8 100 PDB$SEED:UNDOTBS1 NO /u01/app/oracle/oradata/ORCL18C/pdbseed/undotbs01.dbf
9 300 ORCL18P:SYSTEM YES /u01/app/oracle/oradata/ORCL18C/ORCL18P/system01.dbf
10 360 ORCL18P:SYSAUX NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/sysaux01.dbf
11 100 ORCL18P:UNDOTBS1 YES /u01/app/oracle/oradata/ORCL18C/ORCL18P/undotbs01.dbf
12 5 ORCL18P:USERS NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
List of Temporary Files
=======================
File Size(MB) Tablespace Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1 37 TEMP 32767 /u01/app/oracle/oradata/ORCL18C/temp01.dbf
2 62 PDB$SEED:TEMP 32767 /u01/app/oracle/oradata/ORCL18C/pdbseed/temp012019-05-25_23-51-33-086-PM.dbf
3 62 ORCL18P:TEMP 32767 /u01/app/oracle/oradata/ORCL18C/ORCL18P/temp01.dbf
RMAN> list backup of datafile 12 completed after "sysdate - 60/1440";
starting full resync of recovery catalog
full resync complete
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ----------------
4610 Incr 1 40.00K SBT_TAPE 00:00:02 02-07-2019_00-03
BP Key: 4611 Status: AVAILABLE Compressed: YES Tag: TAG20190701T235918
Handle: VB$_4142545763_4609I Media:
List of Datafiles in backup set 4610
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ---------------- ----------- ------ ----
12 1 Incr 1714757 02-07-2019_00-03 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ----------------
4614 Incr 0 36.50K SBT_TAPE 00:00:02 02-07-2019_00-03
BP Key: 4615 Status: AVAILABLE Compressed: YES Tag: TAG20190701T235918
Handle: VB$_4142545763_4609_12 Media:
List of Datafiles in backup set 4614
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ---------------- ----------- ------ ----
12 0 Incr 1714757 02-07-2019_00-03 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
RMAN>
But look the example below where I try to restore the datafile (maybe because was corrupted – whatever the reason):
RMAN> run{
2> ALLOCATE CHANNEL CH1 DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS "SBT_LIBRARY=/u01/app/oracle/product/18.6.0.0/dbhome_1/lib/libra.so, ENV=(RA_WALLET='location=file:/u01/app/oracle/product/18.6.0.0/dbhome_1/dbs/ra_zdlra credential_alias=zdlra-scan:1521/zdlra:orcl')";
ALTER PLUGGABLE DATABASE ORCL18P CLOSE IMMEDIATE INSTANCES=ALL;
RESTORE DATAFILE 12;
RECOVER DATAFILE 12;
}3> 4> 5> 6>
allocated channel: CH1
channel CH1: SID=72 device type=SBT_TAPE
channel CH1: RA Library (ZDLRA) SID=8CA6E4CA5A751068E053010310AC0A07
Statement processed
starting full resync of recovery catalog
full resync complete
Starting restore at 02-07-2019_00-18
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=72 device type=DISK
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 07/02/2019 00:18:08
RMAN-06026: some targets not found - aborting restore
RMAN-06100: no channel to restore a backup or copy of datafile 12
RMAN>
And now this:
RMAN> run{
ALTER PLUGGABLE DATABASE ORCL18P CLOSE IMMEDIATE INSTANCES=ALL;
ALLOCATE CHANNEL CH1 DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS "SBT_LIBRARY=/u01/app/oracle/product/18.6.0.0/dbhome_1/lib/libra.so, ENV=(RA_WALLET='location=file:/u01/app/oracle/product/18.6.0.0/dbhome_1/dbs/ra_zdlra credential_alias=zdlra-scan:1521/zdlra:orcl')";
RESTORE DATAFILE 12;
RECOVER DATAFILE 12;
} 2> 3> 4> 5> 6>
Statement processed
starting full resync of recovery catalog
full resync complete
allocated channel: CH1
channel CH1: SID=63 device type=SBT_TAPE
channel CH1: RA Library (ZDLRA) SID=8CA703A063E7127DE053010310ACC85B
Starting restore at 02-07-2019_00-18
channel CH1: starting datafile backup set restore
channel CH1: specifying datafile(s) to restore from backup set
channel CH1: restoring datafile 00012 to /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel CH1: reading from backup piece VB$_4142545763_4609_12
channel CH1: piece handle=VB$_4142545763_4609_12 tag=TAG20190701T235918
channel CH1: restored backup piece 1
channel CH1: restore complete, elapsed time: 00:00:25
Finished restore at 02-07-2019_00-19
Starting recover at 02-07-2019_00-19
starting media recovery
media recovery complete, elapsed time: 00:00:01
Finished recover at 02-07-2019_00-19
starting full resync of recovery catalog
full resync complete
released channel: CH1
RMAN> ALTER PLUGGABLE DATABASE ORCL18P OPEN INSTANCES=ALL;
Statement processed
starting full resync of recovery catalog
full resync complete
RMAN>
Saw the bug? YES, the bug occurs when you allocate the channel and after execute a close command. When you do the “close”, rman internally cleans everything related for the session (including allocated channels) and you can hit the “RMAN-06100: no channel to restore a backup” or others similar errors like RMAN-06012: channel: CH1 not allocated.
The bug is Bug 28076886 – RMAN: RMAN-06012: “channel: %s not allocated” after ALTER PLUGGABLE DATABASE CLOSE command (Doc ID 28076886.8)
https://support.oracle.com/epmos/faces/DocContentDisplay?id=28076886.8.
The error occurs even for disk channels too (look full example):
RMAN> RUN {
ALLOCATE CHANNEL CH1 DEVICE TYPE DISK;
ALTER PLUGGABLE DATABASE ORCL18P CLOSE;
RECOVER DATABASE PREVIEW;
RELEASE CHANNEL CH1;
}2> 3> 4> 5> 6>
allocated channel: CH1
channel CH1: SID=110 device type=DISK
Statement processed
starting full resync of recovery catalog
full resync complete
Starting recover at 01-07-2019_00-34
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=110 device type=DISK
using channel ORA_DISK_1
no channel to restore a backup or copy of archived log for thread 1 with sequence 5 and starting SCN of 1552035
no channel to restore a backup or copy of archived log for thread 1 with sequence 6 and starting SCN of 1608615
no channel to restore a backup or copy of archived log for thread 1 with sequence 7 and starting SCN of 1669357
no channel to restore a backup or copy of archived log for thread 1 with sequence 8 and starting SCN of 1681533
no channel to restore a backup or copy of archived log for thread 1 with sequence 9 and starting SCN of 1681555
no channel to restore a backup or copy of archived log for thread 1 with sequence 10 and starting SCN of 1682610
List of Archived Log Copies for database with db_unique_name ORCL18C
=====================================================================
Key Thrd Seq S Low Time
------- ---- ------- - ----------------
3516 1 11 A 26-05-2019_19-13
Name: /u01/app/oracle/oradata/ORCL18C/archivelog/2019_06_30/o1_mf_1_11_gklbz4oh_.arc
recovery will be done up to SCN 1603932
Media recovery start SCN is 1603932
Recovery must be done beyond SCN 1692620 to clear datafile fuzziness
Finished recover at 01-07-2019_00-35
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of release command at 07/01/2019 00:35:01
RMAN-06012: channel: CH1 not allocated
RMAN> RUN {
ALTER PLUGGABLE DATABASE ORCL18P CLOSE;
ALLOCATE CHANNEL CH1 DEVICE TYPE DISK;
RECOVER DATABASE PREVIEW;
RELEASE CHANNEL CH1;
}2> 3> 4> 5> 6>
Statement processed
starting full resync of recovery catalog
full resync complete
allocated channel: CH1
channel CH1: SID=110 device type=DISK
Starting recover at 01-07-2019_00-35
no channel to restore a backup or copy of archived log for thread 1 with sequence 5 and starting SCN of 1552035
no channel to restore a backup or copy of archived log for thread 1 with sequence 6 and starting SCN of 1608615
no channel to restore a backup or copy of archived log for thread 1 with sequence 7 and starting SCN of 1669357
no channel to restore a backup or copy of archived log for thread 1 with sequence 8 and starting SCN of 1681533
no channel to restore a backup or copy of archived log for thread 1 with sequence 9 and starting SCN of 1681555
no channel to restore a backup or copy of archived log for thread 1 with sequence 10 and starting SCN of 1682610
List of Archived Log Copies for database with db_unique_name ORCL18C
=====================================================================
Key Thrd Seq S Low Time
------- ---- ------- - ----------------
3516 1 11 A 26-05-2019_19-13
Name: /u01/app/oracle/oradata/ORCL18C/archivelog/2019_06_30/o1_mf_1_11_gklbz4oh_.arc
recovery will be done up to SCN 1603932
Media recovery start SCN is 1603932
Recovery must be done beyond SCN 1693086 to clear datafile fuzziness
Finished recover at 01-07-2019_00-36
released channel: CH1
RMAN>
But can be trickier to discover the failure. Look the example below that I try to backup datafile #12:
RMAN> run{
ALLOCATE CHANNEL CH1 DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS "SBT_LIBRARY=/u01/app/oracle/product/18.6.0.0/dbhome_1/lib/libra.so, ENV=(RA_WALLET='location=file:/u01/app/oracle/product/18.6.0.0/dbhome_1/dbs/ra_zdlra credential_alias=zdlra-scan:1521/zdlra:orcl')";
ALTER PLUGGABLE DATABASE ORCL18P CLOSE IMMEDIATE INSTANCES=ALL;
BACKUP INCREMENTAL LEVEL 1 DATAFILE 12;
}2> 3> 4> 5>
allocated channel: CH1
channel CH1: SID=156 device type=SBT_TAPE
channel CH1: RA Library (ZDLRA) SID=8CBA2C4600D309E8E053010310AC3E17
Statement processed
starting full resync of recovery catalog
full resync complete
Starting backup at 02-07-2019_23-25
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=106 device type=DISK
channel ORA_DISK_1: starting incremental level 1 datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel ORA_DISK_1: starting piece 1 at 02-07-2019_23-25
channel ORA_DISK_1: finished piece 1 at 02-07-2019_23-25
piece handle=/u01/app/oracle/oradata/ORCL18C/89BE7E168E3D28BDE053010310AC2497/backupset/2019_07_02/o1_mf_nnnd1_TAG20190702T232547_gkqlyx8j_.bkp tag=TAG20190702T232547 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 02-07-2019_23-25
Starting Control File and SPFILE Autobackup at 02-07-2019_23-25
piece handle=/u01/app/oracle/oradata/ORCL18C/autobackup/2019_07_02/o1_mf_s_1012605951_gkqlz1vm_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at 02-07-2019_23-25
RMAN> list backup of datafile 12 summary;
List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- ---------------- ------- ------- ---------- ---
4270 B 0 A SBT_TAPE 01-07-2019_00-42 1 1 YES TAG20190701T003820
4610 B 1 A SBT_TAPE 02-07-2019_00-03 1 1 YES TAG20190701T235918
4614 B 0 A SBT_TAPE 02-07-2019_00-03 1 1 YES TAG20190701T235918
5197 B 1 A DISK 02-07-2019_23-25 1 1 NO TAG20190702T232547
RMAN>
And now:
RMAN> run{
ALTER PLUGGABLE DATABASE ORCL18P CLOSE IMMEDIATE INSTANCES=ALL;
ALLOCATE CHANNEL CH1 DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS "SBT_LIBRARY=/u01/app/oracle/product/18.6.0.0/dbhome_1/lib/libra.so, ENV=(RA_WALLET='location=file:/u01/app/oracle/product/18.6.0.0/dbhome_1/dbs/ra_zdlra credential_alias=zdlra-scan:1521/zdlra:orcl')";
BACKUP INCREMENTAL LEVEL 1 DATAFILE 12;
}2> 3> 4> 5>
Statement processed
starting full resync of recovery catalog
full resync complete
allocated channel: CH1
channel CH1: SID=105 device type=SBT_TAPE
channel CH1: RA Library (ZDLRA) SID=8CBAA68CCD751232E053010310AC374B
Starting backup at 02-07-2019_23-44
channel CH1: starting incremental level 1 datafile backup set
channel CH1: specifying datafile(s) in backup set
input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel CH1: starting piece 1 at 02-07-2019_23-44
channel CH1: finished piece 1 at 02-07-2019_23-44
piece handle=ORCL18C_3pu5ma33_1_1 tag=TAG20190702T234435 comment=API Version 2.0,MMS Version 12.2.0.2
channel CH1: backup set complete, elapsed time: 00:00:03
Finished backup at 02-07-2019_23-44
Starting Control File and SPFILE Autobackup at 02-07-2019_23-44
piece handle=c-551670416-20190702-02 comment=API Version 2.0,MMS Version 12.2.0.2
Finished Control File and SPFILE Autobackup at 02-07-2019_23-44
released channel: CH1
RMAN>
Look that because of the bug, in the first example, the backup was made to disk and not in the tape as requested. In the last, you can see that backup was made correctly in tape. But as you saw, no error was reported.
Conclusion
As you saw a tricky bug can change everything that we already know (and need to do) for a daily basis activity. One simple “close” for a PDB can destroy your backup/restore policy that you use from decades. Besides that even without “RMAN-06100: no channel to restore a backup” or other similar errors like RMAN-06012: channel: CH1 not allocated you can have problems.
Luckily if you are running affected version (everything below 19.2) the workaround is very simple to deploy.
This post was published in my personal blog too: http://www.fernandosimon.com/blog/rman-allocate-channel-cdb-and-close-bug/
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”
