standby

I am bloging about experience we had using rman incremental backup for syncing standby database.

We had a standby database which lagged by considerable time and we decided to use rman incremental database to make it current. We have done this in past and used normal procedure of taking current_scn from V$database on standby and took rman incremental backup from this scn on primary database.

When we did the recovery it finished successfully but on starting MRP  it still required old archive log.
We tried manual recovery and still it was failing and asking for same old archive. This was strange and I ran following queries to find out the SCN

SELECT CURRENT_SCN FROM V$DATABASE;
CURRENT_SCN
 -----------
 10641805448174

Checking the minimum checkpoint_change# for V$datafile reported following SCN which was less then current_SCN.

SQL> select min(checkpoint_change#) from V$datafile;
 MIN(CHECKPOINT_CHANGE#)
 ------------------------------
 10641804610367

At this time I decided to check first_change# for the archive required by standby

select first_change# from GV$ARCHIVED_LOG where THREAD#=2 and SEQUENCE#=699;
 FIRST_CHANGE#
 ------------------------------
 10618920961649

This value was way less and was different then the initial current_scn which we used to take rman incremental backup. Checking on My Oracle Support , found note 836986.1 which recommended running following query

SQL> select min(fhscn) from x$kcvfh;
MIN(FHSCN)
 ----------------
 10618920961649

This matches with the scn from V$archived_log. Note recommends to take lower value of x$kcvfh and V$database.
So we took backup again with this SCN from primary. Since we have RAC database, we used channels on both instance

run
 {
 allocate channel c1 device type disk format '/VOL301/rmanbackup/stby_PROD_%U' connect 'sys/password@PROD1';
 allocate channel c2 device type disk format '/VOL301/rmanbackup/stby_PROD_%U' connect 'sys/password@PROD1';
 allocate channel c3 device type disk format '/VOL301/rmanbackup/stby_PROD_%U' connect 'sys/password@PROD2';
 allocate channel c4 device type disk format '/VOL301/rmanbackup/stby_PROD_%U' connect 'sys/password@PROD2';
 backup incremental from scn 10618920961649 database format '/VOL301/rmanbackup/stby_PROD_%U' tag 'Tag_INCR_PROD';
 }

We copied the backups and also restored the controlfile and cataloged the backups. Restoring now did the trick

run
 {
 allocate channel c1 device type disk ;
 allocate channel c2 device type disk ;
 allocate channel c3 device type disk ;
 allocate channel c4 device type disk ;
 recover database noredo;
 }

I think x$kcvfh should also be part of documentation as it wil help people check both scn before proceeding. Anyways this seems to be some special case as we have used this rman incremental approach lot of times and faced this first time.
If anyone has clue on what caused this, then I would be really interested to know it.