How to restore ASM based OCR after complete loss of the CRS diskgroup on Linux/Unix systems (11gR2)

How to restore ASM based OCR after complete loss of the CRS diskgroup on Linux/Unix systems 

This document assumes that the CRS diskgroup was completely lost, the name of the OCR diskgroup remains unchanged, however there may be a need to use a different diskgroup name, in which case the name of the OCR diskgroup would have to be modified in /etc/oracle/ocr.loc across all nodes prior to executing the following steps

Locate the latest automatic OCR backup

when using a non-shared CRS home, automatic OCR backups can be located on any node of the cluster, consequently all nodes need to be checked for the most recent backup:

$ ls -lrt $CRS_HOME/cdata/rac_cluster1/

Make sure the Grid Infrastructure is shutdown on all nodes

Given that the OCR diskgroup is missing, the GI stack will not be functional on any node, however there may still be various daemon processes running.  On each node shutdown the GI stack using the force (-f) option:

# $CRS_HOME/bin/crsctl stop crs -f

Start the CRS stack in exclusive mode

On the node that has the most recent OCR backup, log on as root and start CRS in exclusive mode, this mode will allow ASM to start & stay up without the presence of a Voting disk and without the CRS daemon process (crsd.bin) running.

11.2.0.1:

# $CRS_HOME/bin/crsctl start crs –excl

11.2.0.2 and above:

# $CRS_HOME/bin/crsctl start crs -excl -nocrs

Note

A new option '-nocrs' has been introduced with  11.2.0.2, which prevents the start of the ora.crsd resource. It is vital that this option is specified, otherwise the failure to start the ora.crsd resource will tear down ora.cluster_interconnect.haip, which in turn will cause ASM to crash.

Label the CRS disk for ASMLIB use

If using ASMLIB the disk to be used for the CRS disk group needs to stamped first, as user root do:

# /usr/sbin/oracleasm createdisk ASMD40 /dev/sdh1

Create the CRS diskgroup via sqlplus

The disk group can now be (re-)created via sqlplus from the grid user. The compatible.asm attribute must be set to 11.2 in order for the disk group to be used by CRS:

SQL> create diskgroup CRS external redundancy disk 'ORCL:ASMD40' attribute 'COMPATIBLE.ASM' = '11.2';

Restore the latest OCR backup

Now that the CRS disk group is created & mounted the OCR can be restored - must be done as the root user:

# cd $CRS_HOME/cdata/rac_cluster1/
# $CRS_HOME/bin/ocrconfig -restore backup00.ocr

Start the CRS daemon on the current node (11.2.0.1 only !)

Now that the OCR has been restored the CRS daemon can be started, this is needed to recreate the Voting file. Skip this step for 11.2.0.2.0.

# $CRS_HOME/bin/crsctl start res ora.crsd -init

Recreate the Voting file

The Voting file needs to be initialized in the CRS disk group:

# $CRS_HOME/bin/crsctl replace votedisk +CRS

Shutdown CRS

Since CRS is running in exclusive mode, it needs to be shutdown  to allow CRS to run on all nodes again. Use of the force (-f) option may be required:

# $CRS_HOME/bin/crsctl stop crs -f

Start CRS

As the root user submit the CRS startup on all cluster nodes:

# $CRS_HOME/bin/crsctl start crs

Verify CRS

To verify that CRS is fully functional again:

# $CRS_HOME/bin/crsctl check cluster -all

How to Recreate OCR/Voting Disk Accidentally Deleted

How to Recreate OCR/Voting Disk Accidentally Deleted (The note applies to 10gR2 and 11gR1)     

·         Shutdown the Oracle Clusterware stack on all the nodes using command crsctl stop crs as root user.

·         Backup the entire Oracle Clusterware home.

·         Execute <CRS_HOME>/install/rootdelete.sh on all nodes

·         Execute <CRS_HOME>/install/rootdeinstall.sh on the node which is supposed to be the first node

The following commands should return nothing

ps -e | grep -i 'ocs[s]d' 
ps -e | grep -i 'cr[s]d.bin' 
ps -e | grep -i 'ev[m]d.bin'     
                                  

·         Execute <CRS_HOME>/root.sh on first node

·         After successful root.sh execution on first node Execute root.sh on the rest of the nodes of the cluster

·         For 10gR2, use racgons; for 11g use onsconfig command. Using onsconfig stops and starts ONS so the changes take effect, while racgons doesn't do that so the changes won't take effect until ONS is restarted on all nodes. Examples for each are provided below.

For 10g : Execute as owner (generally oracle) of CRS_HOME command

<CRS_HOME>/bin/racgons add_config hostname1:port hostname2:port

For 11g : Execute as owner (generally oracle) of CRS_HOME command

<CRS_HOME>/install/onsconfig add_config hostname1:port hostname2:port

·         Execute as owner of CRS_HOME (generally oracle)  <CRS_HOME>/bin/oifcfg setif -global.

$ oifcfg setif -global  eth0/192.168.0.0:cluster_interconnect eth1/10.35.140.0:public

·         Add listener using netca. This may give errors if the listener.ora contains the entries already. If this is the case, move the listener.ora to /tmp from the $ORACLE_HOME/network/admin or from the $TNS_ADMIN directory if the TNS_ADMIN environmental is defined and then run netca. Add all the listeners that were added earlier.

·         Add ASM & database resource to the OCR using the appropriate srvctl add database command as the user who owns the ASM & database resource. Please ensure that this is not run as root user

·         Add  Instance, services using appropriate srvctl add commands. Please refer to the documentation for the exact commands.

Converting Single instance to RAC instance Database (rconfig)

Converting Single instance to RAC instance Database

There are different ways to convert a single instance database to RAC.

·         Manual Method

·         Using rconfig (from 10gR2)

·         Using DBCA

·         Using grid control

Notes

-          Install Oracle Clusterware on all nodes you intend to have as cluster members.

-          Eventually patch the clusterware software (the clusterware version need to be equal or higher than the rdbms version) as well as the rdbms software

-          During the "Install Oracle Database 10g or 11g Software with Real Application Clusters", make sure that you select a new Oracle home other than the one from which the single-instance database was running in case the single-instance database is running on one of the systems. Upgrade the RAC RDBMS software to the same version as the original single-instance database.

-          Configure the cluster listener, i.e. configure the listener through netca from the new cluster RDBMS Home. Eventually stop the single instance listeners when they are running on one of the clustered nodes in case they are using the same listener ports. Ideally, use the 'LISTENER' as name for that listener.

-          Prior 11g R2, it is recommended to run the listener from the RDBMS home and from11gR2 it is recommended to run the listener from the Grid Infrastructure home.

-          Move datafiles, redo logs, control file to a shared location on the cluster or ASM.

Convert Single-Instance database to RAC using rconfig

rconfig is a command line tool introduced in Oracle Database 10g R2 to convert Single-Instance 10g R2 Database to RAC(Real Application Cluster)

rconfig is located at $ORACLE_HOME/bin/. rconfig takes in a xml input file and convert the Single Instance database whose information is provided in the xml. 

a sample xml can be found $ORACLE_HOME/assistants/rconfig/sampleXMLs/ConvertToRAC.xml

rconfig performs following steps :

• Migrate the database to ASM storage (Only if ASM is specified as storage option in the configuration xml file)
• Create Database Instances on all nodes in the cluster 
• Configure Listener and NetService entries
• Configure and register CRS resources 
• Start the instances on all nodes in the cluster.

Prerequisites before we use rconfig

Before you convert a single-instance database to a RAC database using rconfig, ensure that the following conditions are met for each cluster node that you intend to make a RAC database node:

• Oracle Clusterware 10g Release 2 (10.2) is installed, configured, and running.
• Oracle Real Application Clusters 10g Release 2 (10.2) software is installed.
• The Oracle binary is enabled for RAC.
• Shared storage, either Oracle Cluster File System or ASM, is available and accessible from all nodes.
• User equivalence exists for the oracle user.

Test rconfig before converting to RAC

Oracle 10g R2 Install has provided a sample rconfig input XML file called ConvertToRAC.xml at directory $ORACLE_HOME/assistants/rconfig/sampleXMLs. Make a copy of the sample XML file & make your own customised copy of input giving your Instance specific details. The Sample XML file includes comments to make understand how to edit each variable.

For testing purpose you need to modify the "Convert verify = "ONLY" option in the xml file.Then run the tool :

$ ./rconfig convert.xml

The Convert verify option in the ConvertToRAC.xml file has three options:

• Convert verify="YES": rconfig performs checks to ensure that the prerequisites for single-instance to RAC conversion have been met before it starts conversion
• Convert verify="NO": rconfig does not perform prerequisite checks, and starts conversion
• Convert verify="ONLY" : rconfig only performs prerequisite checks; it does not start conversion after completing prerequisite checks

Convert the single instance to RAC single database

·         As the oracle user, go to the directory $ORACLE_HOME/assistants/rconfig/sampleXMLs, and open the file ConvertToRAC.xml using a text editor.

·         Review the ConvertToRAC.xml file, and modify the parameters as required for your system. The XML sample file contains comments that provide instructions for how to configure the file.

·         Go to the directory $ORACLE_HOME/bin, and use the following command to run rconfig, where input.xml is the name of the XML input file you configured

$  ./rconfig  ConvertToRAC.xml

Below are list of parameters you should updated in XML file :

Verify does a precheck to ensure all pre-requisites are met, before the conversion is attempted. 

     Convert verify="YES" 

Specify current OracleHome of non-rac database for SourceDBHome 

     SourceDBHome-- ORACLE_HOME path --/n:SourceDBHome

Specify OracleHome where the rac database should be configured. It can be same as SourceDBHome

     TargetDBHome--ORACLE_HOME path--/n:TargetDBHome

Specify SID of non-rac database and credential. User with sysdba role is required to perform conversion
      SourceDBInfo SID="DBname"

ASMInfo element is required only if the current non-rac database uses ASM Storage

      ASMInfo SID="+ASM1" 
      Password—ASM sys password--/n:Password

Specify the list of nodes that should have rac instances running. LocalNode should be the first node in this nodelist. 

     Node name="Node1name" 
     Node name="Node2name"

Specify prefix for rac instances. It can be same as the instance name for non-rac database or different. The instance number will be attached to this prefix. 

     InstancePrefix—Instance_without_number--/n:InstancePrefix

Specify port for the listener to be configured for rac database.If port="", alistener existing on localhost will be used for rac database.The listener will be extended to all nodes in the nodelist 

     Listener port="Port_Number"

Specify the type of storage to be used by rac database. Allowable values are CFS|ASM. The non-rac database should have same storage type. 

     SharedStorage type="ASM" 

Specify Database Area Location to be configured for rac database.If this field is left empty, current storage will be used for rac database. For CFS, this field will have directory path.

     TargetDatabaseArea-- --/n:TargetDatabaseArea

Specify Flash Recovery Area to be configured for rac database. If this field is left empty, current recovery area of non-rac database will be configured for rac database. If current database is not using recovery Area, the resulting rac database will not have a recovery area. 

     TargetFlashRecoveryArea-- --/n:TargetFlashRecoveryArea 

rconfig log files located

 

rconfig log files are located at  $ORACLE_HOME/cfgtoollogs/rconfig

Also one important thing to note is that rconfig re-writes the log file everytime you run the rconfig tool. So make sure have a copy in case you want to refer to the results of earlier run

Restart Rconfig

 

After running RCONFIG in Convert verify="YES" Mode, if you get any fatal error(e.g. Disk space not available, Some issue with 10g Parameters, Shared Storage Issues) which exits the rconfig tool & stops the conversion. You can restart the rconfig tool again by running the command " $ ./rconfig convert.xml ". This will perform the clean-up operation on converted Instance or will delete the files created by earlier run. It again starts the process of conversion from start i.e. taking rman backup of Single Instance. After the conversion you can check the filesystem to verify if there is anything left from the previous failed run.