| Steve Rea's Oracle and SunGardHE Banner Tips, Tricks, and Scripts Back to Home |
Recovering from a database crash or hardware failure can take hours or days. But, with a Data Guard standby database, your users can be back up and running within minutes. This presentation shows you step-by-step how to implement and configure a standby database locally or on a remote site so that it's ready to take over your processing at a moment's notice with no data loss, just using sqlplus and the Oracle 9i or later that you probably have now.
The presentation given at SETA Central 2005 and the Arkansas Banner Users Group 2006 entitled "Crashes Happen - Downtime Won't - With Data Guard" is in DataGuard.ppt (about 170 K) If you don't have Microsoft PowerPoint on your PC, you can download the presentation along with the PowerPoint viewer (for Windows XP) at DataGuard_with_Viewer.zip (about 3 Meg).
Note: This is a work in
progress, which I will be adding to as we test our implementation (we went live
with Data Guard on our production database on 6/15/07, with our standby on a
remote site). However, there is enough information here for you to create
and begin testing your own Data Guard implementation.
Note 2: I am only covering implementing physical
standby databases here. Logical standby databases can only be implemented
in Banner for reporting purposes, not for failover or switchover purposes, since
Banner uses datatypes that aren't supported by
logical standby databases (such as LONG's). To see those tables with unsupported datatypes,
run the following (thanks to Lee Johnston at the University of West Florida for
this heads-up):
select distinct owner,table_name from
dba_logstdby_unsupported order by owner,table_name;
If you do implement logical standby's for Banner, you should exclude those
unsupported tables from the standby processing, such as using the following for
each of those tables listed above (read the chapters on logical standby's for
further information; thanks to April Sims at Southern Utah University for this
information):
exec dbms_logstdby.skip('DML','<owner>','<tablename>');
exec dbms_logstdby.skip('SCHEMA_DDL','<owner>','<tablename>');
For this particular Oracle 9iR2 Data Guard implementation, we will be creating
and testing a local physical standby database running in Maximum Availability mode [1.4]
for a primary database, then, recreating the standby database on
the remote site and testing it again there (to be added
later). For the local standby database (which, in this case, is not
on a separate local server), the same directory structure as the primary
database will be used here, except for the instance-level name (for example,
/data/oradata/PPRD2
contains the standby's datafiles paralleling the primary's /data/oradata/PPRD
directory).
If you want to skip the local testing, you could modify these steps to create
the standby directly on the remote site or on a separate local server (which, in
addition to FTP'ing instead of copying the files, would require that some of the
commands and "file name convert" parameters be changed to accomodate any
differences in directory structures on the remote site, the primary site's
tnsnames.ora file would have to point to the remote site for the standby, and
the remote site's listener.ora, tnsnames.ora, and sqlnet.ora files would have to
be set up).
For your own Data Guard implementation, you
might want to copy and paste this page into Notepad (with Word Wrap turned on),
then:
. oraenv
PPRD
|
. oraenv
PPRD2
|
select name,instance_name,database_role,status,open_mode from v$instance,v$database; |
. oraenv
PPRD
sqlplus "/ as sysdba"
create spfile from pfile;
shutdown immediate
startup
exit
|
. oraenv
PPRD
sqlplus "/ as sysdba"
select force_logging from v$database;
alter database force logging;
|
archive log list
should show:
Database log mode Archive Mode
Automatic archival Enabled
If the database is not in ARCHIVELOG mode:
If you are using an spfile:
alter system set log_archive_dest = '/orcl/oradata/PPRD/archivelogs' scope = spfile;
alter system set log_archive_format = 'arch_PPRD_%S.arc' scope = spfile;
alter system set log_archive_start = true scope = spfile;
Else, if you are using a pfile:
edit the $ORACLE_HOME/dbs/initPPRD.ora file to contain:
log_archive_dest = /orcl/oradata/PPRD/archivelogs
log_archive_format = arch_PPRD_%S.arc
log_archive_start = true
shutdown immediate
startup mount
alter database archivelog;
alter database open;
archive log list |
select records_used "Current Groups",records_total "Max Groups",
decode(sign(records_total-(records_used*2+1)),-1,LPAD('YES',21),LPAD('NO',21))
"Recreate MAXLOGFILES?"
from v$controlfile_record_section where type = 'REDO LOG';
|
alter database backup controlfile to trace;
select value from v$parameter where name = 'user_dump_dest';
!ls -ltr /pgms/oradata/PPRD/udump | tail
!vi /pgms/oradata/PPRD/udump/pprd_ora_475358.trc
Edit the latest .trc (textual control) file and remove all lines before the
STARTUP NOMOUNT line, change the maxlogfiles value from, say, 6 to 10, comment
out (put # in front of) the RECOVER command, and, for Oracle 9i and above,
remove the lines after ALTER DATABASE OPEN, or, if present, ALTER TABLESPACE
TEMP ADD TEMPFILE, and change all comment lines to start with dashes. The vi
commands to do these (for 9i+) are usually:
:1,/STARTUP NOMOUNT/-1d
:/MAXLOGFILES/s/6/10/
:/RECOVER DATABASE/s/^/# /
:/ALTER TABLESPACE TEMP/+2,$d
:1,$s/^#/--/
:wq
shutdown immediate
@/pgms/oradata/PPRD/udump/pprd_ora_475358.trc
|
(SID_DESC= (SID_NAME=PPRD2) (ORACLE_HOME=/pgms/oracle/product/v9203) ) |
myserver_pprd2 =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS =
(PROTOCOL = TCP)
(Host = 234.56.78.90) -- whatever host IP has PPRD2
(Port = 1521)
)
)
(CONNECT_DATA = (SID = PPRD2)
)
)
|
sqlnet.expire_time=2 |
PPRD2:/pgms/oracle/product/v9203:N |
On the primary database:
select distinct 'mkdir -p ' ||
replace(substr(file_name,1,instr(file_name,'/',-1)-1),'/PPRD','/PPRD2')
from dba_data_files
union
select distinct 'mkdir -p ' || replace(value,'/PPRD/','/PPRD2/') from v$parameter
where (name in ('background_dump_dest','user_dump_dest','core_dump_dest',
'audit_file_dest') or name like 'log_archive_dest%')
and name not like 'log_archive_dest_state%' and value is not null;
Run the resulting unix commands on the standby's site, for example for this
local standby:
host
mkdir -p /data/oradata/PPRD2
mkdir -p /ndxs/oradata/PPRD2
mkdir -p /orcl/oradata/PPRD2/archivelogs
mkdir -p /pgms/oradata/PPRD2/audit
mkdir -p /pgms/oradata/PPRD2/bdump
mkdir -p /pgms/oradata/PPRD2/cdump
mkdir -p /pgms/oradata/PPRD2/udump
exit
|
set pagesize 0 recsep off linesize 160 trimspool on feedback off
spool cp_standby.shl
select 'cp -p ' || name || ' ' || replace(name,'/PPRD/','/PPRD2/') from v$datafile
order by substr(name,instr(name,'/',-1));
spool off
!cat cp_standby.shl | sed '/^[^c][^p]/d' >cp_standbyx.shl; mv cp_standbyx.shl cp_standby.shl
|
If you are using an spfile:
create pfile='$ORACLE_HOME/dbs/initPPRD2.ora' from spfile;
Else, if you are using a pfile:
!cp -p $ORACLE_HOME/dbs/initPPRD.ora $ORACLE_HOME/dbs/initPPRD2.ora
|
control_files = (/orcl/oradata/PPRD2/ctrl_PPRD_01.ctl,
/pgms/oradata/PPRD2/ctrl_PPRD_02.ctl) # [3.2.6]
background_dump_dest = /pgms/oradata/PPRD2/bdump # [3.2.6]
core_dump_dest = /pgms/oradata/PPRD2/cdump # [3.2.6]
user_dump_dest = /pgms/oradata/PPRD2/udump # [3.2.6]
audit_file_dest = /pgms/oradata/PPRD2/audit # [3.2.6]
#log_archive_dest = /orcl/oradata/PPRD2/archivelogs
log_archive_dest_1 = 'LOCATION=/orcl/oradata/PPRD2/archivelogs MANDATORY' # [3.1.2,3.2.6,5.4.2,5.8.2.2,12]; for switchover
log_archive_dest_state_1 = ENABLE # [5.8.2.2]; for switchover
log_archive_dest_2 = 'SERVICE=myserver_pprd LGWR SYNC' # [5.7,5.8.2.2,12]; for switchover
log_archive_dest_state_2 = ENABLE # [5.8.2.2]; for switchover
standby_archive_dest = /orcl/oradata/PPRD2/archivelogs # [3.2.6,5.4.2,5.8.2.2]
standby_file_management = AUTO # [3.2.6,5.8.2.2]; or MANUAL for raw devices [8.4.1.2]
remote_archive_enable = TRUE # [3.2.6,5.3.2.1,5.8.2.2]; TRUE or RECEIVE, but must change RECEIVE to SEND on switchover
instance_name = PPRD2 # [3.2.6]
lock_name_space = PPRD2 # [3.2.6]; use when primary and standby on same system; same as instance_name
fal_server = myserver_pprd # [5.8.2.2,6.4.4]
fal_client = myserver_pprd2 # [5.8.2.2,6.4.4]
db_file_name_convert = ('/PPRD/','/PPRD2/') # [3.2.6,5.8.2.2]
log_file_name_convert = ('/PPRD/','/PPRD2/') # [3.2.6,5.8.2.2]
# log_archive_trace = 15 # [5.8.2.4.5,6.7.3]; to see progression of archiving of redo logs to the standby site
audit_trail = false # do not have auditing turned on in a standby database - can't audit to read-only database!
|
shutdown normal |
!sh cp_standby.shl |
startup select replace(value,'/PPRD/','/PPRD2/') from v$parameter where name = 'control_files'; alter database create standby controlfile as '/orcl/oradata/PPRD2/ctrl_PPRD_01.ctl'; exit |
cp -p /orcl/oradata/PPRD2/ctrl_PPRD_01.ctl /pgms/oradata/PPRD2/ctrl_PPRD_02.ctl |
lsnrctl reload |
lsnrctl reload |
rm $ORACLE_HOME/dbs/orapwPPRD2 orapwd file=$ORACLE_HOME/dbs/orapwPPRD2 password=<sys password> entries=5 |
. oraenv
PPRD2
sqlplus "/ as sysdba"
create spfile from pfile;
startup nomount
alter database mount standby database;
alter database recover managed standby database disconnect from session;
exit
|
#log_archive_dest = /orcl/oradata/PPRD/archivelogs
log_archive_dest_1 = 'LOCATION=/orcl/oradata/PPRD/archivelogs MANDATORY' # [3.1.2,5.8.2.1,12]
log_archive_dest_state_1 = ENABLE # [5.8.2.1]
log_archive_dest_2 = 'SERVICE=myserver_pprd2 LGWR SYNC' # [3.2.14,5.4.1,5.7,5.8.2.1,12]
log_archive_dest_state_2 = ENABLE # [3.2.14,5.4.1,5.8.2.1]
standby_archive_dest = /orcl/oradata/PPRD/archivelogs # [5.8.2.1]; for switchover
standby_file_management = AUTO # [5.8.2.1]; for switchover; or MANUAL for raw devices [8.4.1.2]
remote_archive_enable = TRUE # [5.3.2.1,5.8.2.1]; TRUE or SEND, but must change SEND to RECEIVE on switchover
instance_name = PPRD # [3.2.6]
lock_name_space = PPRD # [3.2.6]; use when primary and standby on same system; same as instance_name
fal_server = myserver_pprd2 # [5.8.2.1,6.4.4]; for switchover
fal_client = myserver_pprd # [5.8.2.1,6.4.4]; for switchover
db_file_name_convert = ('/PPRD2/','/PPRD/') # [5.8.2.1]; for switchover
log_file_name_convert = ('/PPRD2/','/PPRD/') # [5.8.2.1]; for switchover
# log_archive_trace = 15 # [5.8.2.4.5,6.7.3]; to see progression of archiving of redo logs to the standby site
|
. oraenv
PPRD
If you are using an spfile (the first alter system command removes log_archive_dest
from the spfile; for a description of the others, see the pfile initPPRD.ora above):
sqlplus "/ as sysdba"
shutdown normal
startup nomount
alter system reset log_archive_dest scope=spfile sid='*';
alter system set log_archive_dest_1 = 'LOCATION=/orcl/oradata/PPRD/archivelogs MANDATORY' scope=spfile;
alter system set log_archive_dest_state_1 = ENABLE scope=spfile;
alter system set log_archive_dest_2 = 'SERVICE=myserver_pprd2 LGWR SYNC' scope=spfile;
alter system set log_archive_dest_state_2 = ENABLE scope=spfile;
alter system set standby_archive_dest = '/orcl/oradata/PPRD/archivelogs' scope=spfile;
alter system set standby_file_management = AUTO scope=spfile;
alter system set remote_archive_enable = TRUE scope=spfile;
alter system set instance_name = PPRD scope=spfile;
alter system set lock_name_space = PPRD scope=spfile;
alter system set fal_server = myserver_pprd2 scope=spfile;
alter system set fal_client = myserver_pprd scope=spfile;
alter system set db_file_name_convert = '/PPRD2/','/PPRD/' scope=spfile;
alter system set log_file_name_convert = '/PPRD2/','/PPRD/' scope=spfile;
# alter system set log_archive_trace = 15 scope=spfile;
shutdown
startup
Else, if you are using a pfile:
sqlplus "/ as sysdba"
shutdown normal
startup
|
alter system switch logfile; |
On the primary database:
column o1 noprint
column o2 noprint
column maxgroup new_value maxgroup
select max(group#) maxgroup from v$logfile;
select group# o1,1 o2,'alter database add standby logfile group ' ||
to_char(group#+&maxgroup) || ' ('
from v$log
union all
select group#,2,' ''' || replace(replace(member,group#,group#+&maxgroup),
'PPRD/','PPRD2/stby_') || ''','
from v$logfile lf1 where member not in (select max(member) from v$logfile lf2
where lf1.group# = lf2.group#)
union all
select l1.group#,3,' ''' || replace(replace(member,l1.group#,l1.group#+&maxgroup),
'PPRD/','PPRD2/stby_') || ''') size ' || bytes / 1024 || 'K;'
from v$logfile lf1,v$log l1 where lf1.group# = l1.group#
and member = (select max(member) from v$logfile lf2 where lf1.group# = lf2.group#)
union all
select group#+1,1,'alter database add standby logfile group ' ||
to_char(group#+1+&maxgroup) || ' (' from v$log
where group# = &maxgroup
union all
select group#+1,2,' ''' || replace(replace(member,group#,group#+1+&maxgroup),
'PPRD/','PPRD2/stby_') || ''',' from v$logfile lf1 where group# = &maxgroup
and member not in (select max(member) from v$logfile lf2 where lf1.group# = lf2.group#)
union all
select l1.group#+1,3,' ''' || replace(replace(member,l1.group#,l1.group#+1+&maxgroup),
'PPRD/','PPRD2/stby_') || ''') size ' || bytes / 1024 || 'K;'
from v$logfile lf1,v$log l1 where lf1.group# = l1.group#
and l1.group# = &maxgroup
and member = (select max(member) from v$logfile lf2 where lf1.group# = lf2.group#)
order by 1,2,3;
exit
On the standby database, run the resulting sql from the above. (Note: This can't be done
until after archiving has been started by issuing "alter system switch logfile;" on the
primary.):
. oraenv
PPRD2
sqlplus "/ as sysdba"
alter database recover managed standby database cancel;
alter database open read only;
alter database add standby logfile group 4 (
'/orcl/oradata/PPRD2/stby_log_PPRD_4A.rdo',
'/orcl/oradata/PPRD2/stby_log_PPRD_4B.rdo') size 4096K;
alter database add standby logfile group 5 (
'/orcl/oradata/PPRD2/stby_log_PPRD_5A.rdo',
'/orcl/oradata/PPRD2/stby_log_PPRD_5B.rdo') size 4096K;
alter database add standby logfile group 6 (
'/orcl/oradata/PPRD2/stby_log_PPRD_6A.rdo',
'/orcl/oradata/PPRD2/stby_log_PPRD_6B.rdo') size 4096K;
alter database add standby logfile group 7 (
'/orcl/oradata/PPRD2/stby_log_PPRD_7A.rdo',
'/orcl/oradata/PPRD2/stby_log_PPRD_7B.rdo') size 4096K;
column member format a55
select vs.group#,vs.bytes,vl.member from v$standby_log vs,v$logfile vl
where vs.group# = vl.group# order by vs.group#,vl.member;
|
alter tablespace temp add tempfile '/data/oradata/PPRD2/temp_PPRD_01.dbf'
size 400064K reuse;
alter database recover managed standby database disconnect from session;
select * from v$tempfile;
exit
|
. oraenv
PPRD
sqlplus "/ as sysdba"
column o1 noprint
column o2 noprint
column maxgroup new_value maxgroup
select max(group#) maxgroup from v$logfile;
select group# o1,1 o2,'alter database add standby logfile group ' ||
to_char(group#+&maxgroup) || ' ('
from v$log
union all
select group#,2,' ''' || replace(replace(member,group#,group#+&maxgroup),
'PPRD/','PPRD/stby_') || ''','
from v$logfile lf1 where member not in (select max(member) from v$logfile lf2
where lf1.group# = lf2.group#)
union all
select l1.group#,3,' ''' || replace(replace(member,l1.group#,l1.group#+&maxgroup),
'PPRD/','PPRD/stby_') || ''') size ' || bytes / 1024 || 'K;'
from v$logfile lf1,v$log l1 where lf1.group# = l1.group#
and member = (select max(member) from v$logfile lf2 where lf1.group# = lf2.group#)
union all
select group#+1,1,'alter database add standby logfile group ' ||
to_char(group#+1+&maxgroup) || ' (' from v$log
where group# = &maxgroup
union all
select group#+1,2,' ''' || replace(replace(member,group#,group#+1+&maxgroup),
'PPRD/','PPRD/stby_') || ''',' from v$logfile lf1 where group# = &maxgroup
and member not in (select max(member) from v$logfile lf2 where lf1.group# = lf2.group#)
union all
select l1.group#+1,3,' ''' || replace(replace(member,l1.group#,l1.group#+1+&maxgroup),
'PPRD/','PPRD/stby_') || ''') size ' || bytes / 1024 || 'K;'
from v$logfile lf1,v$log l1 where lf1.group# = l1.group#
and l1.group# = &maxgroup
and member = (select max(member) from v$logfile lf2 where lf1.group# = lf2.group#)
order by 1,2,3;
Run the resulting sql from the above, for example:
alter database add standby logfile group 4 (
'/orcl/oradata/PPRD/stby_log_PPRD_4A.rdo',
'/orcl/oradata/PPRD/stby_log_PPRD_4B.rdo') size 4096K;
alter database add standby logfile group 5 (
'/orcl/oradata/PPRD/stby_log_PPRD_5A.rdo',
'/orcl/oradata/PPRD/stby_log_PPRD_5B.rdo') size 4096K;
alter database add standby logfile group 6 (
'/orcl/oradata/PPRD/stby_log_PPRD_6A.rdo',
'/orcl/oradata/PPRD/stby_log_PPRD_6B.rdo') size 4096K;
alter database add standby logfile group 7 (
'/orcl/oradata/PPRD/stby_log_PPRD_7A.rdo',
'/orcl/oradata/PPRD/stby_log_PPRD_7B.rdo') size 4096K;
column member format a55
select vs.group#,vs.bytes,vl.member from v$standby_log vs,v$logfile vl
where vs.group# = vl.group# order by vs.group#,vl.member;
|
alter session set nls_date_format = 'DD-MON-YYYY HH24:MI:SS';
select sysdate from dual;
alter system switch logfile;
!ls -ltr /orcl/oradata/PPRD2/archivelogs
You should see the archive logs that were received by the standby.
!tail /pgms/oradata/PPRD2/bdump/alert_PPRD2.log
You should see a message in the standby alert log such as:
Media Recovery Log /orcl/oradata/PPRD2/archivelogs/arch_PPRD_0000002866.arc
On the standby (8.5.3.1):
select process,status,thread#,sequence#,block#,blocks from v$managed_standby;
|
On the standby, see what logs are already there:
alter session set nls_date_format = 'DD-MON-YYYY HH24:MI:SS';
select sequence#, first_time, next_time, archived, applied
from v$archived_log order by sequence#;
On the primary, force a log switch:
alter system archive log current;
On the standby, see the new logs and if they have been applied (may need to wait
a few seconds before they are applied):
select sequence#, first_time, next_time, archived, applied
from v$archived_log order by sequence#;
On the primary, you can also see the new logs and if the standby has acknowledged
that they've been applied (again, may need to wait a few seconds before they
are acknowledged):
column name format a15
select name,sequence#, first_time, next_time, archived, applied
from v$archived_log where name not like '%/%' order by sequence#;
|
. oraenv
PPRD
sqlplus "/ as sysdba"
select value from v$parameter where name = 'log_archive_dest_2';
shutdown normal
(If that hangs, then do: shutdown immediate, startup, shutdown normal)
startup mount
alter database set standby database to maximize availability;
alter database open;
select protection_mode from v$database;
|
ORA-03113: end-of-file on communication channel |
exit sqlplus "/ as sysdba" startup mount alter database set standby database to maximize performance; alter database open; |
On the primary:
update spriden set spriden_first_name = 'James'
where spriden_pidm = 1234 and spriden_change_ind is null;
commit;
alter system switch logfile;
On the standby (may take a few seconds for the change to be applied):
select name,instance_name,database_role,status,open_mode from v$instance,v$database;
alter database recover managed standby database cancel;
alter database open read only;
select name,instance_name,database_role,status,open_mode from v$instance,v$database;
set pagesize 60
select * from spriden where spriden_pidm = 1234 and spriden_change_ind is null;
alter database recover managed standby database disconnect from session;
select name,instance_name,database_role,status,open_mode from v$instance,v$database;
On the primary:
update spriden set spriden_first_name = 'Stephen' where spriden_pidm = 1234
and spriden_change_ind is null;
commit;
alter system switch logfile;
On the standby:
alter database recover managed standby database cancel;
alter database open read only;
set pagesize 60
select * from spriden where spriden_pidm = 1234 and spriden_change_ind is null;
alter database recover managed standby database disconnect from session;
|
sqlplus "/ as sysdba" alter database recover managed standby database cancel; alter database open read only; connect myuserid/mypassword @myreport.sql connect / as sysdba alter database recover managed standby database disconnect from session; |
alter session set nls_date_format = 'DD-MON-YYYY HH24:MI:SS';
select sequence#, first_change#, next_change#, first_time, next_time
from v$archived_log where applied = 'NO' order by sequence#;
|
ORA-01552: cannot use system rollback segment for non-system tablespace |
alter system switch logfile; |
Start the standby databases and listeners before starting the primary database. Shut down the primary database (or defer sending transactions to the standby) before shutting down the standby database. |
If standby is currently in read-only access, terminate any active user sessions (run
the resulting SQL and commands generated below) and switch back to managed recovery:
select open_mode from v$database;
select 'alter system kill session ''' || sid || ',' || serial# || ''';'
from v$session where username is not null
and sid not in (select distinct sid from v$mystat);
select '!kill -9 ' || vp.spid
from v$session vs,v$process vp where vs.paddr = vp.addr
and vs.username is not null
and vs.sid not in (select distinct sid from v$mystat);
alter database recover managed standby database disconnect from session;
Then, cancel managed recovery (but see Note below):
select * from v$standby_log;
Should show all standby logs with a status of UNASSIGNED if archiving is
deferred (otherwise, the primary database is still sending transactions
to the standby - but this isn't always the case).
alter database recover managed standby database cancel;
shutdown immediate
Note: Before canceling managed recovery on the standby, it is suggested that you defer
archiving to the standby by doing the following on the primary. However, trying to
start up a primary database while archiving to the standby is set to "defer" causes an
"ORA-03113: end-of-file on communication channel" error (see further down below for
how to start up the primary if you encounter this problem). So, until this startup
problem is addressed and resolved by Oracle, don't defer archiving.
alter system set log_archive_dest_state_2 = DEFER;
alter system switch logfile;
|
On the standby (note: on startup, you will get the message: "ORA-01666: controlfile
is for a standby database"):
startup nomount
alter database mount standby database;
alter database recover managed standby database disconnect from session;
On the primary, re-enable archiving to the standby database, if needed:
alter system set log_archive_dest_state_2 = ENABLE;
alter system switch logfile;
|
startup nomount alter database mount standby database; alter database open read only; |
alter database recover managed standby database cancel; alter database open read only; |
Terminate any active user sessions on the standby database (except your own):
select 'alter system kill session ''' || sid || ',' || serial# || ''';'
from v$session where username is not null
and sid not in (select distinct sid from v$mystat);
select '!kill -9 ' || vp.spid
from v$session vs,v$process vp where vs.paddr = vp.addr
and vs.username is not null
and vs.sid not in (select distinct sid from v$mystat);
alter database recover managed standby database disconnect from session;
|
select decode(database_role,'PRIMARY','PRIMARY',decode(open_mode,
'MOUNTED',decode((select count(*) from v$managed_standby
where process like 'MRP%'),0,'CANCELLED','MANAGED RECOVERY'),
'READ ONLY','READ ONLY','UNKNOWN')) from v$database;
|
End all activities on the primary and standby database [7.1.2.1], probably just
doing on the primary database (PPRD):
select 'alter system kill session ''' || sid || ',' || serial# || ''';'
from v$session where username is not null
and sid not in (select distinct sid from v$mystat);
select '!kill -9 ' || vp.spid
from v$session vs,v$process vp where vs.paddr = vp.addr
and vs.username is not null
and vs.sid not in (select distinct sid from v$mystat);
Check primary database (PPRD) switchover status (on primary; looking for
"TO STANDBY"; but mine said "SESSIONS ACTIVE" (my sysdba session)) [7.2.1.1]:
select database_role,switchover_status from v$database;
Initiate switchover operation on the primary database (PPRD) [7.2.1.2]:
alter database commit to switchover to physical standby;
Shut down and restart the former primary instance (PPRD) as the new standby [7.2.1.3]:
shutdown immediate
if remote_archive_enable is set to SEND in the primary's init.ora file
(initPPRD.ora), change it to RECEIVE:
If you are using an spfile (do this after the startup below):
alter system set remote_archive_enable = RECEIVE scope = both;
Else, if you are using a pfile:
remote_archive_enable = RECEIVE
if audit_trail is set to anything but FALSE in the primary's init.ora file
(initPPRD.ora), change it to FALSE:
If you are using an spfile (do this after the startup below):
alter system set audit_trail = FALSE scope = both;
Else, if you are using a pfile:
audit_trail = FALSE
if log_archive_dest_state_2 is set to ENABLE in the primary's init.ora file
(initPPRD.ora), change it to DEFER (on pre-10g databases; added 8/23/07):
If you are using an spfile (do this after the startup below):
alter system set log_archive_dest_state_2 = DEFER scope = both;
Else, if you are using a pfile:
log_archive_dest_state_2 = DEFER
startup nomount
alter database mount standby database;
Check standby database (PPRD2) switchover status (on standby; looking for
"SWITCHOVER PENDING"; but mine said "TO PRIMARY") [7.2.1.4]:
select database_role,switchover_status from v$database;
Change the former standby instance (PPRD2) to the primary role, shut down, and
restart [7.2.1.5-6]:
alter database commit to switchover to primary;
shutdown normal
if remote_archive_enable is set to RECEIVE in the standby's init.ora file
(initPPRD2.ora), change it to SEND:
If you are using an spfile (do this after the startup below):
alter system set remote_archive_enable = SEND scope = both;
Else, if you are using a pfile:
remote_archive_enable = SEND
if audit_trail needs to be turned on for this "new" primary, change it
to TRUE (or whatever setting is needed) in the standby's init.ora file
(initPPRD2.ora):
If you are using an spfile (do this after the startup below):
alter system set audit_trail = TRUE scope = both;
Else, if you are using a pfile:
audit_trail = TRUE
if log_archive_dest_state_2 is set to DEFER in the standby's init.ora file
(initPPRD2.ora), change it to ENABLE (on pre-10g databases; added 8/23/07):
If you are using an spfile (do this after the startup below):
alter system set log_archive_dest_state_2 = ENABLE scope = both;
Else, if you are using a pfile:
log_archive_dest_state_2 = ENABLE
startup
select name,instance_name,database_role,status,open_mode from v$instance,v$database;
Start managed recovery on the new standby (on old primary) (PPRD) [7.2.1.7]:
alter database recover managed standby database disconnect from session;
select name,instance_name,database_role,status,open_mode from v$instance,v$database;
Write the current archive log on the new primary (on old standby) (PPRD2) [7.2.1.8]:
alter system archive log current;
Change tnsnames.ora entry on all application hosts (T:\APPS\ban6\orawin\NET80\ADMIN)
to point to the new primary (PPRD2) (or, just tell the users to use myserver_pprd2 as
the database to log into - make sure myserver_pprd2 is defined in the tnsnames.ora
file on the network; but, see "Multiple tnsnames addresses" for another option) [10.1.1]:
myserver_pprd =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS =
(PROTOCOL = TCP)
(Host = 234.56.78.90) -- whatever host IP has PPRD2
(Port = 1521)
)
)
(CONNECT_DATA = (SID = PPRD2) -- the new primary (old standby)
)
)
myserver_pprd2 =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS =
(PROTOCOL = TCP)
(Host = 123.45.67.89) -- whatever host IP has PPRD
(Port = 1521)
)
)
(CONNECT_DATA = (SID = PPRD) -- the new standby (old primary)
)
)
|
If running in maximum protection mode [7.1.3.1]:
select protection_mode from v$database;
alter database set standby database to maximize performance;
See if any archived redo log gaps exist [7.2.2.1.1]:
select thread#,low_sequence#,high_sequence# from v$archive_gap;
If any gaps exist:
Copy or transfer all missing archive logs from the primary system to the
standby system; then, for each (substituting filename):
alter database register physical logfile 'filename';
See if other missing archived redo logs exist (this actually refers to looking
at other standby sites, but, this can also be done on a single standby if you
could get the archive logs off the failed primary site) [7.2.2.1.2]:
select unique thread# as thread, max(sequence#) over (partition by thread#)
as last from v$archived_log;
!ls -ltr /orcl/oradata/PPRD/archivelogs | tail
If missing sequences are found:
Copy or transfer all missing archive logs from the other standby systems
to the first standby system; then, for each (substituting filename):
alter database register physical logfile 'filename';
Repeat steps 7.2.2.1.1 and 7.2.2.1.2 until no gaps remain [7.2.2.1.3].
Initiate failover operation on the standby database [7.2.2.1.4]. If the standby
database has standby redo logs and you have not manually registered any partial
archived redo logs, issue the following statement:
alter database recover managed standby database finish;
Otherwise, issue the following statement:
alter database recover managed standby database finish skip standby logfiles;
Then, convert the standby database to the primary role [7.2.2.1.5]:
alter database commit to switchover to primary;
shutdown immediate
(You might want to make a backup of this new primary database now, just in
case you have to recover it before the primary site is fixed and back up and
running.)
if audit_trail needs to be turned on for this "new" primary, change it
to TRUE (or whatever setting is needed) in the standby's init.ora file
(initPPRD2.ora):
If you are using an spfile (do this after the startup below):
alter system set audit_trail = TRUE scope = both;
Else, if you are using a pfile:
audit_trail = TRUE
startup
select name,instance_name,database_role,status,open_mode from v$instance,v$database;
Change tnsnames.ora entry on all application hosts (T:\APPS\ban6\orawin\NET80\ADMIN)
to point to the new primary (PPRD2) (or, just tell the users to use myserver_pprd2 as
the database to log into - make sure myserver_pprd2 is defined in the tnsnames.ora
file on the network; but, see "Multiple tnsnames addresses" for another option) [10.1.1]:
myserver_pprd =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS =
(PROTOCOL = TCP)
(Host = 234.56.78.90) -- whatever host IP has PPRD2
(Port = 1521)
)
)
(CONNECT_DATA = (SID = PPRD2) -- the new primary (old standby)
)
)
When the hardware or other problem is corrected, create a new "standby" database
on the primary site (as shown above), then do a switchover (also as shown above),
resulting in the primary database being back on the primary site, and the standby
database being on the standby site.
|
alter session set nls_date_format = 'DD-MON-YYYY HH24:MI:SS'; set recsep off column message format a59 select timestamp,message from v$dataguard_status; |
connect / as sysdba startup nomount column name format a25 column value format a52 select name,value from v$parameter where name like 'log_archive_dest_%'; alter system set log_archive_dest_state_2=ENABLE scope=both; alter database mount; alter database open; alter system set log_archive_dest_state_2=DEFER scope=both; |
connect / as sysdba startup mount alter database set standby database to maximize performance; alter database open; |
sqlplus "/ as sysdba" shutdown immediate startup mount alter database set standby database to maximize availability; alter database open; |
alter database recover managed standby database cancel; alter database create datafile '/data/oradata/PPRD2/devl_PPRD_02.dbf'; alter database recover managed standby database disconnect from session; |
grep -v "/[^/]*${BACKUP_SID}[^/].*/"
|
replace:
find / -name "*${BACKUP_SID}*" ! -type d 2>/dummy |
egrep '(\.ora$|\.dbf$|\.ctl$|\.rdo$)' | sort -t/ -k3 >backup_list.lst
with:
find / -name "*${BACKUP_SID}*" ! -type d 2>/dummy |
grep -v "/[^/]*${BACKUP_SID}[^/].*/" |
egrep '(\.ora$|\.dbf$|\.ctl$|\.rdo$)' | sort -t/ -k3 >backup_list.lst
|
myserver_pprd =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS =
(PROTOCOL = TCP)
(Host = 123.45.67.89) -- whatever host IP has PPRD
(Port = 1521)
)
(ADDRESS =
(PROTOCOL = TCP)
(Host = 234.56.78.90) -- whatever host IP has PPRD2
(Port = 1521)
)
)
(CONNECT_DATA = (SID = PPRD)
)
)
|
1) after:
PFILE=${ORACLE_HOME}/dbs/init${ORACLE_SID}.ora
add:
SPFILE=${ORACLE_HOME}/dbs/spfile${ORACLE_SID}.ora
2) replace:
if [ -f $PFILE ] ; then
with:
if [ -f $PFILE -o -f $SPFILE ] ; then
|
Change the primary database name the secondary instance name (PPRD2) before shutting it
down to do the copy (see 3.2.2.1 above):
shutdown normal
startup mount
!nid target=sys/<syspassword> dbname=PPRD2 setname=YES
answer Y
shutdown normal
Copy the primary database to the standby directory (see 3.2.2.2 above):
!sh cp_standby.shl
Create the standby control file (see 3.2.3 above):
startup
select replace(value,'/PPRD/','/PPRD2/') from v$parameter where name = 'control_files';
alter database create standby controlfile as '/orcl/oradata/PPRD2/ctrl_PPRD_01.ctl';
Change the primary database name back to the primary instance name (PPRD):
shutdown normal
!nid target=sys/<syspassword> dbname=PPRD setname=YES
answer Y
alter database open;
exit
|
Switching Primary and Standby for Disaster Planning Tests
Below is a step-by-step description of how we ran a Disaster Planning test to make sure that we could switch to the standby, run a real payroll (yikes!), and switch back to the primary. During this test, no one could access the original primary database (since it was in managed recovery mode), and the only ones that could get into Banner were the ones that knew the standby's connect string (StandbyServer_prod in this case). Since I was the only one with the password to the standby site, I ended up running some batch jobs for the test and FTP'ing the resulting files to our primary site (such as to print checks here and transfer the direct deposit file to the bank). We came across a few glitches, which are listed below, that were fairly easy to fix. Other than these, the payroll test went smoothly (although the Banner screens were slow). The standby payroll run was completed in a day, and we switched modes back to the original primary and original standby without having to recreate either of those databases. The glitches were:
Turn off the secondary standby database(s) on all other servers (except for
the main standby database):
OtherServer:
login oracle
PROD
sqlplus "/ as sysdba"
@dgshutdown.sql
exit
Shut down the Application Server after the next full backup until we get
ready to activate the standby for testing:
Access the Oracle Enterprise Manager:
http://<whatever Application Server
URL>:1811
User Name:
ias_admin
Password:
<our ias_admin password>
Click <whatever ias instance name>
Click Stop All
Click Yes button to confirm.
Turn off access to PROD:
PrimaryServer:
login oracle
PROD
sqlplus "/ as
sysdba"
alter system enable restricted session;
exit;
Turn off posting check and db status check in oracle's cron (these are our
procedures - you may have to turn off other cron jobs):
PrimaryServer:
login oracle
PROD
crontab -e
comment out
the following lines:
# 0,5,10,15,20,25,30,35,40,45,50,55 8-17 * * 1-5 /home/oracle/fgractg_check.shl
# 0,5,10,15,20,25,30,35,40,45,50,55 8-17 * * 1-5 /home/oracle/check_db_status.shl
:wq
crontab -l
Turn off posting check in jobsub's cron:
PrimaryServer:
login jobsub
crontab -e
comment out the following line:
# 0,10,20,30,40,50 8-16 * * 1-5 ./monitor.shl
:wq
crontab -l
Switch primary/standby modes on StandbyServer_prod and PrimaryServer_prod:
On the primary site, shut down daemon (one of our procedures), job
submission, posting and approvals (waiting about 5 minutes to complete) and any
other user processes:
PrimaryServer:
login oracle
PROD
rm /home/jobsub/PROD*
daemon_stop.shl
sqlplus "/ as sysdba"
@kill_user.sql
Kill the GURJOBS, GENLPRD, INTEGMGR, and COMMON processes (and any other user
processes).
Note: You may still see 'INACTIVE' sessions in v$session, but, the shutdown
below will clear them out.
Initiate switchover operation on the primary database, shut down, and restart
(also changing audit_trail to false, since we can't audit while in standby mode;
note: there is no "with session shutdown" prior to 10g):
alter system
set audit_trail=FALSE scope=spfile;
shutdown
immediate
startup
alter
database commit to switchover to physical standby with session shutdown;
shutdown
immediate
startup
nomount
alter
database mount standby database;
(leave
sqlplus running here while we work on the standby site)
Check standby database switchover status (on standby; looking for "SWITCHOVER
PENDING"):
StandbyServer:
login oracle
PROD
sqlplus "/ as sysdba"
select
database_role,switchover_status from v$database;
select
protection_mode from v$database;
Initiate switchover operation on the
standby database, shut down, and restart:
alter
database recover managed standby database disconnect from session;
alter
database commit to switchover to primary;
shutdown
immediate
startup
select
name,instance_name,database_role,status,open_mode from v$instance,v$database;
select * from
dba_temp_files;
If that shows no temp files allocated, then:
alter tablespace temp add tempfile '/data/oradata/PROD/temp_PROD_01.dbf'
size 400064K reuse;
(leave
sqlplus running here while we work on the primary site)
Start managed recovery on the new standby (on old primary):
PrimaryServer:
alter database recover managed
standby database disconnect from session;
select
name,instance_name,database_role,status,open_mode from v$instance,v$database;
exit
Write the current archive log on the new primary (on old standby):
StandbyServer:
alter system archive log current;
exit
See if it came through on the new standby (on old primary):
PrimaryServer:
ls -ltr /orcl/oradata/PROD/archivelogs
| tail
Start up jobsub and sleep/wake on StandbyServer (start_jobsub.shl and
start_sleep_wake.shl are our processes - you may have something similar):
start_jobsub.shl
cat /home/jobsub/gurjobsPROD.out
shows connected.
start_sleep_wake.shl
ps -ef | egrep '(fgractg|orappl)'
shows both fgractg and forappl are
running.
Turn on access to PROD:
StandbyServer:
login oracle
PROD
sqlplus "/ as sysdba"
select logins
from v$instance;
If it shows
RESTRICTED:
alter system disable restricted session;
exit
Start up the Application Server (leave them running for the payroll test):
Access the Oracle Enterprise Manager:
http://<whatever Application Server
URL>:1811
User Name:
ias_admin
Password:
<our ias_admin password>
Click <whatever ias instance name>
Click Start All
Can now connect to StandbyServer_prod in Banner Webforms INB. To connect to
Banner Self Service, use (note: we have a StandbyServer_prod DAD defined which
points to the standby):
https://<your URL>/pls/StandbyServer_prod/twbkwbis.P_WWWLogin
Run the disaster planning payroll test (done by payroll personnel).
To FTP a file from the StandbyServer server to PrimaryServer, such as
kgrigsby_phpmtim_148991.log:
StandbyServer:
cd /home/jobsub
ftp PrimaryServer.uaex.edu
jobsub
cd /home/jobsub
type ascii
put
kgrigsby_phpmtim_148991.log
bye
Had to recompile PHPCALC and its dependencies on StandbyServer before it
would work (just get PHPCALC working now, and compile the others on
StandbyServer later):
cd $BANNER_LINKS
export COBDIR=/usr/lib/cobol
export PATH=/usr/vac/bin:$PATH
gencmplc.shl
gencmpl.shl
paycmpl.shl
Then, run phpcalc.shl for payroll.
Run any other scripts through Secure FTP as needed, and FTP any files back to the primary site as needed.
Afterwards, compile all of the programs on StandbyServer for future runs (allcmpl.shl
is our script, which contains all of the Banner product compile scripts;
cescmp.shl and cescmpl2.shl are our scripts which compile our locally-developed
programs; runsqlplus and daemon are programs posted on my main web page):
cd $BANNER_LINKS
nohup allcmpl.shl >allcmpl.PROD.StandbyServer.log 2>&1 &
tail -f allcmpl.PROD.StandbyServer.log
make -f sctproc.mk runsqlplus
nohup cescmpl.shl >cescmpl.PROD.StandbyServer.log 2>&1 &
tail -f cescmpl.PROD.StandbyServer.log
grep 'Open fail' cescmpl.PROD.StandbyServer.log
nohup cescmpl2.shl >cescmpl2.PROD.StandbyServer.log 2>&1 &
tail -f cescmpl2.PROD.StandbyServer.log
grep 'Open fail' cescmpl2.PROD.StandbyServer.log
Compile E-Visions print software: eviadm, evippgrm, evilp:
cd $BANNER_HOME/evisions
make -f $BANNER_LINKS/sctproc.mk eviadm CHECKOPT=sqlcheck=limited
$EXE_HOME/eviadm
B - COMPILE FORMFUSION SERVER
SOFTWARE
Enter the password for EVISIONS
Version to compile and install:
1.6.5.326 (latest)
A. Compile/Execute all
5.) Standard ANSI C Compiler
X - EXIT INSTALL MENU
Compile daemon:
cd /home/common
rm -f daemon
make -f $BANNER_HOME/general/c/sctproc.mk CHECKOPT="sqlcheck=semantics
userid=<ID/password>" daemon
mv $BANNER_HOME/general/exe/daemon .
After the disaster planning payroll test, switch primary/standby modes again on StandbyServer_prod and PrimaryServer_prod to restore their original modes, and start up the secondary standby databases, if any:
On the standby site, shut down job submission, posting and approvals (waiting
about 5 minutes to complete) and any other user processes:
StandbyServer:
login oracle
PROD
rm /home/jobsub/PROD*
sqlplus "/ as sysdba"
@kill_user.sql
Kill the GENLPRD, GURJOBS, and any other user processes.
Initiate switchover operation on the primary database, shut down, and restart
(note: there is no "with session shutdown" prior to 10g):
alter
database commit to switchover to physical standby with session shutdown;
shutdown
immediate
startup
nomount
alter
database mount standby database;
(leave
sqlplus running here while we work on the primary site)
Check standby database switchover status (looking for "SWITCHOVER PENDING";
but mine said "TO PRIMARY"):
PrimaryServer:
login oracle
PROD
sqlplus "/ as sysdba"
select
database_role,switchover_status from v$database;
select
protection_mode from v$database;
Initiate switchover operation on the
standby database, shut down, and restart (also changing audit_trail back to
true):
alter
database commit to switchover to primary;
shutdown
immediate
startup
nomount
alter system
set audit_trail=TRUE scope=spfile;
shutdown
immediate
startup
select
name,instance_name,database_role,status,open_mode from v$instance,v$database;
(leave
sqlplus running here while we work on the primary site)
Start managed recovery on the original standby:
StandbyServer:
alter database recover managed
standby database disconnect from session;
select
name,instance_name,database_role,status,open_mode from v$instance,v$database;
exit
Write the current archive log on the original primary:
PrimaryServer:
alter system archive log current;
exit
Make a full backup of the primary database:
rmanbackup.shl PROD full
Start up jobsub and sleep/wake on PrimaryServer (these are our scripts - you
may have something similar):
login root
./jsbringup.shl
./swbringup.shl
Turn on posting check and db status check in oracle's cron:
PrimaryServer:
login oracle
PROD
crontab -e
uncomment the
following lines:
0,5,10,15,20,25,30,35,40,45,50,55 8-17 * * 1-5 /home/oracle/fgractg_check.shl
0,5,10,15,20,25,30,35,40,45,50,55 8-17 * * 1-5 /home/oracle/check_db_status.shl
:wq
crontab -l
Turn on posting check in jobsub's cron:
login jobsub
crontab -e
uncomment the following line:
0,10,20,30,40,50 8-16 * * 1-5 ./monitor.shl
:wq
crontab -l
Start up the secondary standby database(s), if any, and it will start
grabbing all of the archive logs from the original primary site since it was
shut down:
OtherServer:
login oracle
PROD
sqlplus "/ as sysdba"
@dgstartup.sql
exit
ls -ltr /ndxs/oradata/PROD/archivelogs
| tail -20
You Are Visitor Number |
This Page Was Last Updated on 09/23/09 |
Copyright © 2009 by Maristream.
All information, scripts, forms, and other material
on this web site are freely available to all Banner and Oracle Database Administrators,
Systems Administrators, Programmers, and others that may need it.
The webmaster who maintains this web site may be reached at srea@maristream.org. Visit our other web sites:
|
Maristream - New Product Research and Development |
Disclaimer: As with all software, especially where it affects your vital data, make sure that you examine theses scripts and that you understand what they do before you use them to see if they would have any adverse effect on your particular setup or database layout. Make a full backup of your database in case you have to revert to your original copy of the database before the scripts were run. Use these scripts at your own risk. As a condition of using these scripts, you agree to hold harmless both Maristream and Stephen Rea for any problems that they may cause or other situations that may arise from their use, and that neither Maristream nor I will be held liable for those consequences.