Oracle 10G RAC release 2 implementations on Linux
Network configuration on AII RAC nodes
#vi/etc/hosts
127.0.0.1 localhost.localdomain localhost
#Public Network – (eth0)
192.168.1.100 linux1
192.168.1.101 linux2
#Private Interconnect – (eth1)
192.168.2.100 linux1-priv
192.168.2.101 linux2-priv
#Public virtual IP (VIP) addresses – (eth0:1)
192.168.1.200 linux1-vip
192.168.1.201 linux2-vip
#Private Storage Network for SAN – (eth1)
192.168.1.195 SAN
Setting Kernel Parameters on All RAC Nodes
#vi / etc/sysctl.conf
Net.core.rmem_default = 262144
Net.core.wmem_default = 262144
Net.core.rmem_max = 262144
Net.core.wmem_max = 262144
Kernel.shmmax = 2147483648
Kernel.shmall = 2097152
Kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
Setting shell limits for the oracle User all RAC Nodes
# vi / etc / security / limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
# vi / etc/pam.d/login
Session required /lib/security/pam_limits.so
Setting the correct date & time on all RAC nodes
Setting the date and time from linux1;
#date –s “9/2/2007 01:12:00”
Setting the date and time from linux2;
#date –s “9/2/200701:12:20”
Installing RPM Package on All RAC nodes
binutls-2.15.92.0.2-21
compat-db-4.1.25-9
compat-gcc-32-3.2.3-47.3
compat-gcc-32-c++-3.2.3-47.3
compat-libstdc++-33-3.2.3-47.3
compat-libgcc-296-2.96-132.7.2
control-center-2.8.0-12.rhel4.5
cpp-3.4.6-3
gcc-3.4.6-3
gcc-c++-3.4.6-3
glibc-2.3.4-2.25
glibc-common-2.3.4-2.25
glibc-devel-2.3.4-2.25
glibc-headers-2.3.4-2.25
glibc-kernheaders-2.4-9.1.98.EL
gnome-libs-1.4.1.2.90-44.1
libaio-0.3.105-2
libstdc++-devel-3.4.6-3
make-3.80-6.EL4
openmotif-2.2.3-10.RHEL4.5
openmotif21-2.1.30-11.RHEL4.6
pdksh-5.2.14-30.3
setarch-1.6-1
sysstat-5.0.5-11.rhel4
xscreensaver-4.18-5.rhel4.11
Configure SAN storage on All Nodes
# rpm – qa | grep iscsi
Iscsi-initiator-utils-4.0.3.0-5
# rpm –Uvh iscsi-initiator-utils-4.0.3.0-15.i386.rpm
# vi / etc/iscsi.conf
DiscoveryAddress = SAN
# service iscsi restart
# chkconfig --level 345 iscsi on
# fdisk / dev/sda (Create Partition only one node Linux1)
Command (m for help) : n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4) : 1
First cylinder (1-15134, default 1 ) : 1
Last cylinder or +size or +sizeM or +sizeK (1-15134, default 15134) : + 20000M
Command (m for help) : n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4) : 2
First cylinder (1-15134, default 1 ) :
Last cylinder or +size or +sizeM or +sizeK : + 50000M
Command (m for help) : n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4) : 3
First cylinder (1-15134, default 1 ) :
Last cylinder or +size or +sizeM or +sizeK : + 50000M
Command (m for help) : n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4) : 4
First cylinder (1-15134, default 1 ) :
Last cylinder or +size or +sizeM or +sizeK : + 100000M
Command (m for help) :W
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
#fdisk -1
It will display four partition on all nodes
/dev/sda1
/dev/sda2
/dev/sda3
/dev/sda4
Create Group and User for Oracle All RAC Nodes
# groupadd –g 501 oinstall
#useradd –m –u 501 –g oinstall –G oracle
# id oracle
Uid=501(oracle) gig=501(oinstall) groups=501(oinstall)
Set the password for the oracle account all RAC Nodes
# passwd oracle
Changing password for user oracle.
New UNIX password : XXXXXXXXXXX
Retype new UNIX password : XXXXXXXXXXX
Passwd : all authentication tokens updated sucessfully
Create the oracle base Directory All Nodes
# mkdir –p /u01/app/oracle/product/10.2.0/d_1
#chown –R oracle:oinstall /u01/app/oracle
#chmod –R 775 /u01/app/oracle
Create the oracle clustware Home Directory All Nodes
# mkdir –p /u01/app/crs
# chown –R oracle:oinstall /u01/app/crs
# chmod –R 775 /u01/app/crs
Create Mount Point for OCFS2 / clustware All Nodes
#mkdir –p /u02
# chown –R oracle:oinstall / u02
# chmod –R 775 /u02
Create Login Script for oracle User Account All nodes
# su – oracle
# vi.bash_profile
export ORACLE_SID=orcl1 /*second node change ORCL2*/
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=/Uo1/app/crs
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/network/jilib:$ORACLE_HOME/rdbms/jlib
export PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin
Configure the “hangcheck-timer” kernel module All RAC Nodes :
# vi / etc/modprobe.conf
Options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
Hangcheck-tick: This parameter defines the period of time between checks of system health. The default value is 60 seconds; oracle recommends setting it to 30 seconds. Hangcheck-margin: This parameter defines the maximum hang delay that should be tolerated before hangcheck-timer resets that RAC node. It defines the margin of error in seconds. The default value is 180 seconds; Oracle recommends setting it to 180 sconds.
Creating the RSA keys on All Oracle RAC Nodes
# rpm –qa |grep rsh*
# chkconfig rsh on
# chkconfig rlogin on
# service xinetd reload
# touch /etc/hosts.equiv
# chmod 600 /etc/hosts.equiv
# chown root:root /etc/hosts.equiv
#vi / etc/hosts.equiv
+linux1 oracle
+linux2 oracle
+linux1-priv oracle
+linux2-priv oracle
# mv / usr/Kerberos/bin/rsh/usr/Kerberos/bin/rsh.old
# mv / usr/Kerberos/bin/rlogin/usr/Kerberos/bin/rlogin.old
# mv / usr/Kerberos/bin/rcp/usr/Kerberos/bin/rcp.old
# which rsh
/usr/bin/rsh
Install and configure oracle cluster file system (OCFS2) All RAC Nodes
# rpm –Uvh ocfs2-2.6.9-55.EL-1.2.5-6.i686.rpm
# rpm –Uvh ocfs2console-1.2.4-1.i386.rpm
# rpm –Uvh ocfs2-tools-1.2.4-1.i386.rpm
Disable SELinux (RHEL4 U2 and higher ) All RAC Nodes:
# /usr/bin/system-config-securitylevel &
# init 6 (restart all servers)
Configure OCFS2 on All RAC Nodes
# ocfs2console &
1. Select [ Cluster] -> [ Configure Nodes…].
2. Click “Close” button on below window
3. Displaying “Node configuration” window
4. On the “node configuration” dialog , click the [ADD] button
5. This will bring up the “Add Node” dialog
6. In the “Add Node” dialog, enter the host name and IP address for the all the node one by one in the cluster. Leave the IP Port set to its default value of 7777.
Click [CLOSE] on the “node configuration” dialog.
After verifying all values are correct, exit the application using [ File] -> [Quit]. This needs to be performed on both Oracle RAC nodes in the cluster.
Check all RAC nodes OCFS configure
# more /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 192.168.2.100
number = 0
name = linux1
cluster = ocfs2
node:
ip_port = 7777
ip_address = 192.168.2.101
number = 1
name = linux2
cluster = ocfs2
cluster :
node_count = 2
name = ocfs2
# /etc/init.d/o2cb offline ocfs2
Stopping O2CB cluster ocfs2 : OK
The above command will offline the cluster we created, ocfs2.
# /etc/init.d/o2cb unload
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module “ocfs2_dlmfs” : OK
Unmounting configfs filesystem : OK
Unloading module “configfs” : OK
The above command will unload all OCFS2 modules.
# /etc/init.d/o2cb configure
Configuring the O2CB driver.
This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on boot. The current values will be shown in brackets (‘[]’). Hitting <enter> without typing an answer will keep that current value. Ctrl-C will abort.
Load O2CB driver on boot (y/n) [n] : y
Cluster to start on boot (Enter “none” to clear) [ocfs2] : ocfs2
Specify heartbeat dead threshold (>=7) [7] : 61
Specify network idle timeout in ms (>=5000) [10000] : 10000
Specify network keepalive delay in ms (>=1000) [5000] : 5000
Specify network reconnect delay in ms (>=2000) [2000] : 2000
Writing O2CS configuration : OK
Loading module “configfs” : OK
Mounting configfs filesystem at /config :OK
Loading module “ocfs2_nodemanager” :OK
Loading module “ocfs2_dlm” : OK
Loading module “ocfs_dlmfs” : OK
Mounting ocfs2_dlmfs filesystem at / dlm : ok
Starting O2CB cluster ocfs2 : OK
Format the OCFS2 file system only one node Linux1
# mkfs.ocfs2 –b 4k –C 32K –N 4 –L oracrsfiles /dev/sda1
Mount the OCFS2 File system All RAC nodes
# mount -t ocfs2 -0 datavolume,nointr –L “oracrsfiles” /u02
# vi/etc/fstab
LABEL = oracrsfiles /u02 ocfs _netdev,datavolume,nointr 0 0
# chkconfig –list o2cb
O2cb 0:off 1 : off 2: on 3:on 4:on 5:on 6:off
# chown oracle:oinstall / u02
# chmod 775 /u02
Create Directory for oracle clusterware Files any one node Linux1
# mkdir –p /u02/oradata/orcl
# chown –R oracle:oinstall /u02/oradata
# chmod –R 775 /u02/oradata
# Is –I /u02/oradata
total 4
drwxrwxr-x 2 oracle oinstall 4096 Sep 3 00:45 orcl
Reboot All RAC Nodes
# init 6
# mount
/dev/sde1 on / u02 type ocfs2 (rw, _netdev,datavolume,nointr,heartbeat=local)
# ca /proc/fs/ocfs2_nodemanager/hb_dead_threshold
61
Install and configure Automatic Storage Management (ASMLib 2.0) All RAC Nodes
# rpm -Uvh oracleasm-2.6.9-55.EL-2.0.3-1.i686.rpm
# rpm –Uvh oracleasmlib-2.0.2-1.i386.rpm
# rpm –Uvh oracleasm-support-2.0.3-1.i386.rpm
# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver
This will configure the on-boot properties of the oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets (‘[]’). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) [n] : y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration : [ OK ]
Creating /dev/oracleasm mount point : [ OK]
Loading module “oracleasm” : [ OK]
Mounting ASMlib driver filesystem : [ OK]
Scanning system for ASM disks: [ OK]
Create ASM Disks for oracle on any one node Linux1
# /etc/init.d/oracleasm createdisk VOL1/dev/sda2
Marking disk “/dev/sda2” as an ASM disk [ OK]
# /etc/init.d/oracleasm createdisk VOL2/dev/sda3
Marking disk “/dev/sda31” as an ASM disk [ OK]
# /etc/init.d/oracleasm createdisk VOL3/dev/sda4
Marking disk “/dev/sda4” as an ASM disk [ OK]
# /etc/init.d/oracleasm listdisks -All RAC nodes
VOL1
VOL2
VOL3
#init 6 on primary node Linux1
Login as oracle user on Linux1 and root user on Linux2
Install Oracle Clusterware 10g software from Linux1 as oracle user
$cd /software/clusterware
$ . / runInstallar
Click “Next” Button
Enter the appropriate name and path for the oracle Home and click the “Next” button.
Wait while the prerequisite checks are done. If you have any failures correct them and retry the tests before clicking the “Next” button
The “ Specify Cluster Configuration “ screen shows only the LINUX1 node in the cluster. Click the “ADD” button to continue.
Enter the details for the LINUX2 node and click the “OK” button.
Click the “NEXT” button to continue.
The “Specific Network Interface Usage” screen defines how each network interface will be used. Highlight the “eth0” interface and click the “Edit” button.
Set the “eth0” interface type to “Public” and click the “OK” button.
Leave the “eht1” interface as private and click the “Next” button.
Click the “External Redundancy” option, enter “/u02/oradata/orcl/ocrfile” as the OCR location and click the “Next” button. To have greater redundancy we would need to define another shared disk for an alternate location.
Click the “External Redundancy” option, enter “/u02/oradata/orcl/votingdisk” as the voting Disk location and click the “Next” button. To have greater redundancy we would need to define another shared disk for an alternate location.
On the “Summary” screen, click the “install” button to continue.
Wait while the installation takes place.
Once the install is complete , run the orainstRoot.sh and root.sh scripts on both nodes as directed on the following screen.
The output from the orainstRoot.sh file should look something like that listed below.
# cd /u01/app/oracle/oraInventory
# /orainstRoot.sh
Changing permissions of /u01/app/oracle/oraInventory to 770
Changing groupname of /u01/app/oracle/oraInventory to oinstall
The execution of the script is complete.
The output of the root.sh will vary a little depending on the node it is run on. The following text is the output from the Linux1 node.
# cd /u01/app/crs
# ./root.sh
WARNING: directory ‘/u01/app/crs/product/10.2.0’ is not owned by root
WARNING: directory ‘/u01/app/crs/product’ is not owned by root
WARNING: directory ‘/u01/app/crs’ is not owned by root
WARNING: directory ‘/u01/app’ is not owned by root
Checking to see if Oracle CRS stack is already configured.
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry Configuration upgraded successfully
WARNING: directory ‘/u01/app/crs’ is not owned by root
WARNING: directory ‘/u01/app’ is not owned by root
assigning default hostname Linux1 for node1
assigning default hostname Linux2 for node2
Successfully accumulated necessary OCR keys
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897
Node >nodenumber<: >nodename <>private interconnect name <> hostname<
node 1 : linux1 linux1-priv linux1
node 2 : linux2 linux2-priv linux2
Creating OCR keys for user ‘root’, privgrp ‘root’.
Operation successful.
Now formatting voting device: /u002/oradata/orcl/votingdisk
Format of 1 voting devices complete
Startup will be quested to init within 90 seconds
Adding daemons to initab
Expecting the CRS daemons to be up within 600 seconds
CSS is active on these nodes.
Linux1
CSS is inactive on these nodes.
Linux2
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.
#
Ignore the directory ownership warnings. We should really use a separate directory structure for the clusterware so it can be owned by the root user, but it ash little effect on the finished results.
The output from the Linux2 node is listed below.
# /u01/app/crs
# ./root.sh
WARNING: directory ‘/u01/app/crs’ is not owned by root
WARNING: directory ‘/u01/app’ is not owned by root
Checking to see if Oracle CRS stack is already configured.
/etc/oracle does not exist. Creaing it now.
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry Configuration upgraded successfully
WARNING: directory ‘/u01/app/crs’ is not owned by root
WARNING: directory ‘/u01/app’ is not owned by root
clsfg: EXISTING configuration version 3 deducted
clsfg : version 3 is 10G release 2.
assigning default hostname Linux1 for node1
assigning default hostname Linux2 for node2
Successfully accumulated necessary OCR keys
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897
Node >nodenumber<: >nodename <>private interconnect name <> hostname<
node 1 : linux1 linux1-priv linux1
node 2 : linux2 linux2-priv linux2
clsfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply –force parameter to override.
-force is destructive and will destroy any previous cluster
Configuration.
Oracle Cluster Registry for cluster has been already been initialized.
Startup will be queued to init within 90 seconds.
Adding Daemons to inittab
Expecting the CRS daemons to be up within 600 seconds
CSS is active on these nodes.
Linux1
Linux2
CSS is active on these nodes.
Waiting for the oracle CRSD and EVMD to start
Waiting for the oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
The given interface(s), “eth0” is not public. Public interfaces should be used to configure virtual IPs.
#
Here you can see that some of the configuration steps are omitted as they were done by the first node. In addition, the final part of the script ran the Virtual IP Configuration Assistant (VIPCA) in silent mode, but it failed. This is because my public IP addresses are actually within the “10.1.255.255” range which is a private IP range. If you were using “legal” IP addresses you would not see this and you could ignore the following VIPCA steps.
Run the VIPCA manually as the root user on the Linux2 node using the following command
# cd/u01/app/crs/bin
# ./vipca
Click the “Next” button on the VIPCA welcome screen
Highlight the “eth0” interface and click the “Next” button
Enter the virtual IP alias and address for each node. Once you enter the first alias , the remaining values should default automatically. Click the “Next” button to continue.
Accept the summary information by clicking the “Finish” button.
Wait until the configuration is complete , then click the “OK” button
Accept the VIPCA results by clicking the “Exit” button
You should now return to the “Executive Configuration scripts” screen on Linux1 and click the “OK” button.
Wait for the configuration assistants to complete.
When the installation is complete, click the “Exit” button to leave the installer.
Install Oracle Database Software from Linux1:
$ cd/software/Database
$ ./runInstaller
Select the “Enterprise Edition” option and click the “Next” button.
Enter the name and path for the Oracle Home and click the “Next” button
Select the “Cluster Install” option and make sure both RAC nodes are selected, the click the “Next” button
Wait while the prerequisite checks are done. If you have any failures correct them and retry the tests before clicking the “Next” button
Select the “Install database software only” option, then click the “Next” button.
On the “Summary” screen, click the “Install” button to continue.
Wait while the database software installs.
Once the installation is complete, wait while the configuration assistants run.
Execute the “root.sh” scripts on both nodes , as instructed on the “Execute Configuration scripts” screen, then click the “OK” button.
When the installation is complete, click the “Exit” button to leave the installer.
Listener Configuration (Necta) from Linux1 as oracle user
$ netca
Create a database using the DBCA from Linux1 as oracle user
$ dbca
On the “Welcome” screen, select the “Oracle Real Application Clusters database” option and click the “Next” button
Select the “Create a Database” option and click the “Next” button.
Highlight both RAC nodes and click the “Next” button.
Select the “General purpose” option and click the “Next” button
Enter the values “ORCL” and “ORCL” for the Global Database Name and SID Prefix respectively, then click the “Next” button
Accept the management options by clicking the “Next” button. If you are attempting the installation on a server with limited memory, you may prefer not to configure Enterprise manager at this time.
Enter database passwords then click the “Next” button.
Select the “Automatic Storage Management “ option, then click the “Next” button.
Select the “IFILE parameter” option and enter sys user password for ASM instance then click the “Next” button.
Click “Ok” to create ASM instance.
Click “Create New” to create ASM disk group from Database area
Type group name “DATA” and select “Normal” redundancy then select “Vol1&vol2” then click “Ok”
“DATA” ASM Group Creation Progress
Click “Create New” to create another ASM disk group for Flash recovery area
Type “BACKUP” in disk group name and select “External” redundancy and select “VOL3” then click “OK”
Select “DATA & BACKUP” two ASM Disk group then click “Next”
Type Database area is “+DATA” ASM diskgroup then click “Next”
Type Flash Recovery area “+BACKUP” ASM diskgroup then click “Next”
Click “Next”
Accept the current database services configuration by clicking the “Next” button.
Select the “Custom” memory management option and accept the default settings by clicking the “Next” button.
Accept the database storage settings by clicking the “Next” button.
Accept the database creation options by clicking the “Finish” button.
Accept the summary information by clicking the “OK” button.
Wait while the database is created.
Once the database creation is complete you are presented with the following screen. Make a note of the information on the screen and click the “Exit” button.
The RAC database creation is now complete.
ADD RAC Nodes
Network configuration on all RAC Nodes
# vi/etc/hosts
127.0.0.1 localhost.localdomain localhost
#Public Network - (eth0)
192.168.1.100 linux1
192.168.1.101 linux2
# Private Interconnect - (eth1)
192.168.2.100 linux1-priv
192.168.2.100 linux2-priv
# Public Virtual IP (VIP) addresses – (eth0:1)
192.168.1.200 linux1-vip
192.168.1.201 linux2-vip
#Private Storage Network for SAN – (eth1)
192.168.1.195 SAN
Setting Kernel Parameters on All RAC Nodes:
# vi/etc/sysctl.conf
Net.core.rmem_default = 262144
Net.core.wmem_default = 262144
Net.core.rmem_max = 262144
Net.core.wmem_max = 262144
Kernel.shmmax = 2147483648
Kernel.shmall = 2097152
Kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
Setting shell limits for the oracle User all RAC Nodes
# vi / etc / security / limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
# vi / etc/pam.d/login
Session required /lib/security/pam_limits.so
Setting the correct date & time on all RAC nodes
Setting the date and time from linux1;
#date –s “9/2/2007 01:12:00”
Setting the date and time from linux2;
#date –s “9/2/200701:12:20”
Installing RPM Package on All RAC nodes
binutls-2.15.92.0.2-21
compat-db-4.1.25-9
compat-gcc-32-3.2.3-47.3
compat-gcc-32-c++-3.2.3-47.3
compat-libstdc++-33-3.2.3-47.3
compat-libgcc-296-2.96-132.7.2
control-center-2.8.0-12.rhel4.5
cpp-3.4.6-3
gcc-3.4.6-3
gcc-c++-3.4.6-3
glibc-2.3.4-2.25
glibc-common-2.3.4-2.25
glibc-devel-2.3.4-2.25
glibc-headers-2.3.4-2.25
glibc-kernheaders-2.4-9.1.98.EL
gnome-libs-1.4.1.2.90-44.1
libaio-0.3.105-2
libstdc++-devel-3.4.6-3
make-3.80-6.EL4
openmotif-2.2.3-10.RHEL4.5
openmotif21-2.1.30-11.RHEL4.6
pdksh-5.2.14-30.3
setarch-1.6-1
sysstat-5.0.5-11.rhel4
xscreensaver-4.18-5.rhel4.11
Configure SAN storage on All Nodes
# rpm – qa | grep iscsi
Iscsi-initiator-utils-4.0.3.0-5
# rpm –Uvh iscsi-initiator-utils-4.0.3.0-15.i386.rpm
# vi / etc/iscsi.conf
DiscoveryAddress = SAN
# service iscsi restart
# chkconfig --level 345 iscsi on
# fdisk -I
/dev/sda1
/dev/sda2
/dev/sda3
/dev/sda4
Create Group and User for Oracle All RAC Nodes
# groupadd –g 501 oinstall
#useradd –m –u 501 –g oinstall –G oracle
# id oracle
Uid=501(oracle) gig=501(oinstall) groups=501(oinstall)
Set the password for the oracle account all RAC Nodes
# passwd oracle
Changing password for user oracle.
New UNIX password : XXXXXXXXXXX
Retype new UNIX password : XXXXXXXXXXX
Passwd : all authentication tokens updated sucessfully
Create the oracle base Directory All Nodes
# mkdir –p /u01/app/oracle/product/10.2.0/d_1
#chown –R oracle:oinstall /u01/app/oracle
#chmod –R 775 /u01/app/oracle
Create the oracle clustware Home Directory All Nodes
# mkdir –p /u01/app/crs
# chown –R oracle:oinstall /u01/app/crs
# chmod –R 775 /u01/app/crs
Create Mount Point for OCFS2 / clustware All Nodes
#mkdir –p /u02
# chown –R oracle:oinstall / u02
# chmod –R 775 /u02
Create Login Script for oracle User Account All nodes
# su – oracle
# vi.bash_profile
export ORACLE_SID=orcl1 /*second node change ORCL2*/
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=/Uo1/app/crs
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/network/jilib:$ORACLE_HOME/rdbms/jlib
export PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin
Configure the “hangcheck-timer” kernel module All RAC Nodes :
# vi / etc/modprobe.conf
Options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
Hangcheck-tick: This parameter defines the period of time between checks of system health. The default value is 60 seconds; oracle recommends setting it to 30 seconds. Hangcheck-margin: This parameter defines the maximum hang delay that should be tolerated before hangcheck-timer resets that RAC node. It defines the margin of error in seconds. The default value is 180 seconds; Oracle recommends setting it to 180 sconds.
Creating the RSA keys on All Oracle RAC Nodes
# rpm –qa |grep rsh*
# chkconfig rsh on
# chkconfig rlogin on
# service xinetd reload
# touch /etc/hosts.equiv
# chmod 600 /etc/hosts.equiv
# chown root:root /etc/hosts.equiv
#vi / etc/hosts.equiv
+linux1 oracle
+linux2 oracle
+linux1-priv oracle
+linux2-priv oracle
# mv / usr/Kerberos/bin/rsh/usr/Kerberos/bin/rsh.old
# mv / usr/Kerberos/bin/rlogin/usr/Kerberos/bin/rlogin.old
# mv / usr/Kerberos/bin/rcp/usr/Kerberos/bin/rcp.old
# which rsh
/usr/bin/rsh
Install and configure oracle cluster file system (OCFS2) All RAC Nodes
# rpm –Uvh ocfs2-2.6.9-55.EL-1.2.5-6.i686.rpm
# rpm –Uvh ocfs2console-1.2.4-1.i386.rpm
# rpm –Uvh ocfs2-tools-1.2.4-1.i386.rpm
Disable SELinux (RHEL4 U2 and higher ) All RAC Nodes:
# /usr/bin/system-config-securitylevel &
# init 6 (restart all servers)
Configure OCFS2 on All RAC Nodes
# ocfs2console &
1. Select [ Cluster] -> [ Configure Nodes…].
2. Click “Close” button on below window
3. Displaying “Node configuration” window
4. On the “node configuration” dialog , click the [ADD] button
5. This will bring up the “Add Node” dialog
6. In the “Add Node” dialog, enter the host name and IP address for the all the node one by one in the cluster. Leave the IP Port set to its default value of 7777.
Click [CLOSE] on the “node configuration” dialog.
After verifying all values are correct, exit the application using [ File] -> [Quit]. This needs to be performed on both Oracle RAC nodes in the cluster.
Check all RAC nodes OCFS configure
# more /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 192.168.2.100
number = 0
name = linux1
cluster = ocfs2
node:
ip_port = 7777
ip_address = 192.168.2.101
number = 1
name = linux2
cluster = ocfs2
cluster :
node_count = 2
name = ocfs2
Add New Oracle RAC Node to the OCFS2 Cluster
As root , run the following from linux1 and then linux2;
[root@linux1 ~] # o2cb_ctl –C –i –n linux3 –t node –a number=2 –a ip_address=192.168.2.107 –a ip_port=7777 –a cluster=ocfs2
Node.linux3 created.
[root@linux2 ~] # o2cb_ctl –C –i –n linux3 –t node –a number=2 –a ip_address=192.168.2.107 –a ip_port=7777 –a cluster=ocfs2
Node.linux3 created.
O2cb_ctl parameters:
-c : Create an object in the OCFS2 Cluster Configuration
-i : Valid only with –C. When creating something (node or cluster), it will also install it in the live cluster (/config). If the parameter is not specified , then only update the /etc/ocfs2/cluster.conf.
-n : object name which is usually the node name or cluster name.
-t: Type can be cluster , node or heartbeat.
-a: Attribute in the format “parameter=value” which will be set in the file /etc/ocfs2/cluster.conf file. Since nodes are numbered starting with zero, the third node in the OCFS2 cluster will be “number=2”. Set the IP address which in this example will be the private interconnect “ip_address=192.168.2.107”. The port number used in the current two-node cluster is “ip_port=7777”. Finally , identifying which OCCFS2 cluster to use which in our case is named “cluster=ocfs2”.
#etc/init.d/o2cb offline ocfs2
Stopping O2CB cluster ocfs2:OK
The above command will offline the cluster we created, ocfs2.
# /etc/init.d/o2cb unload
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module “ocfs2_dlmfs” : OK
Unmounting configfs filesystem : OK
Unloading module “configfs” : OK
The above command will unload all OCFS2 modules.
# /etc/init.d/o2cb configure
Configuring the O2CB driver.
This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on boot. The current values will be shown in brackets (‘[]’). Hitting <enter> without typing an answer will keep that current value. Ctrl-C will abort.
Load O2CB driver on boot (y/n) [n] : y
Cluster to start on boot (Enter “none” to clear) [ocfs2] : ocfs2
Specify heartbeat dead threshold (>=7) [7] : 61
Specify network idle timeout in ms (>=5000) [10000] : 10000
Specify network keepalive delay in ms (>=1000) [5000] : 5000
Specify network reconnect delay in ms (>=2000) [2000] : 2000
Writing O2CS configuration : OK
Loading module “configfs” : OK
Mounting configfs filesystem at /config :OK
Loading module “ocfs2_nodemanager” :OK
Loading module “ocfs2_dlm” : OK
Loading module “ocfs_dlmfs” : OK
Mounting ocfs2_dlmfs filesystem at / dlm : ok
Starting O2CB cluster ocfs2 : OK
Mount the OCFS2 File system on New RAC nodes
# mount –t ocfs2 –o datavolume, nointr –L “oracrsfiles” /u02
Configure OCFS2 to Mount Automatically at Startup on new RAC nodes:
# vi /etc/fstab
LABEL=oracrsfiles /u02 ocfs2 _netdev,datavolume,nointr 0 0
# chkconfig - -list o2cb
O2cb 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Install ASMLib 2.0 Packages
The installation process is simply a matter of running the following command on the new Oracle RAC node as the root user account:
#rpm –Uvh oracleasm-2.6.9-55.EL-2.0.3-1.i686.rpm \
Oracleasmlib-2.0.2-1.i386.rpm \
Oracleasm-support-2.0.3-1.i386.rpm
Preparing …….. ########################################[100%]
1 : oracleasm-support #########################################[33%]
2 : oracleasm-2.6.9-55.EL ######################################### [67%]
3 : oracleasmlib #########################################[100%]
Configure and loading the ASMLib 2.0 Packages
Run the following as root on the new Oracle RAC node:
# /etc/init.d/oracleasm configure
Configuring the oracle ASM library driver
This will configure the on-boot properties of the oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets (‘[]’). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) [n] : y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration : [ OK ]
Creating /dev/oracleasm mount point : [ OK]
Loading module “oracleasm” : [ OK]
Mounting ASMlib driver filesystem : [ OK]
Scanning system for ASM disks: [ OK]
Scan for ASM Disks
# etc/init.d/oracleasm scandisks
Scanning system for ASM disks [ok]
# etc/init.d/oracleasm listdisks
Vol1
Vol2
Vol3
Configure Oracle Clusterware on the New Node on Linux1
$ cd /u01/app/crs/oui/bin
$ ./addNode.sh
| Screen Name | Response | ||||||
| Welcome screen | Click Next | ||||||
| Specify cluster Nodes to add to Installation |
Click Next to continue | ||||||
| Cluster Node Additional Summary | Verify the new oracle RAC node is listed under the “New Nodes” drilldown. Click Install to start the installation! | ||||||
| Executive Configuration Scripts | From linux3 Navigate to the /u01/app/oracle/oraInventory directory on linux3 and run orainstRoot.sh as the “root” user account. Finally, navigate to the /u01/app/crs directory on linux3 and run root.sh as the “root” user account. The given interface(s), “eth0” is not public. Public interfaces should be used to configure virtual IPs #$ORA_CRS_HOME/bin/vipca Welcome: Click Next Network interfaces : Select only the public interface-eth0 Virtual IPs for cluster nodes: Node name : linux1 IP Alias Name : linux1-vip IP Address : 192.168.1.200 Subnet Mask : 255.255.255.0 Node name : linux2 IP Alias Name : linux2-vip IP Address : 192.168.1.201 Subnet Mask : 255.255.255.0 Node name : linux3 IP Alias Name : linux3-vip IP Address : 192.168.1.207 Subnet Mask : 255.255.255.0 Summary : Click Finish Configuration Assistant Progress Dialog : Click OK after configuration is complete Configuration Results : Click Exit Go back to the OUI and acknowledge the “Executive Configuration Scripts” dialog window | ||||||
| End of Installation | At the end of the installation, exit from the OUI |
Check Cluster Nodes:
$ $ORA_CRS_HOME/bin/olsnodes –n
linux1 1
linux2 2
linux3 3
Confirm Oracle Clusterware Function
$ $ORA_CRS_HOME/bin/crs_stat -t -v
Check CRS Status
CSS appears healthy
CRS appears healthy
EVM appears healthy
Check Oracle Clusterware Auto-Start Scripts on New Node (linux3)
$ 1s -1 /etc/init.d/init.*
-rwxr-xr-x 1 root root 2236 Feb 23 19:16 /etc/init.d/init.crs
-rwxr-xr-x 1 root root 5252 Feb 23 19:16 /etc/init.d/init.crsd
-rwxr-xr-x 1 root root 44589 Feb 23 19:16 /etc/init.d/init.cssd
-rwxr-xr-x 1 root root 3669 Feb 23 19:16 /etc/init.d/init.evmd
Install Oracle Database Software on the New Node
$ORACLE_HOME/oui/bin directory from linux1;
$ hostname
linux1
$ cd/u01/app/oracle/product/10.2.0/db_1/oui/bin
$ ./addNode.sh
| Screen Name | Response |
| Welcome Screen | Click Next |
| Specify Cluster Nodes to Add to Installation | By default linux3 is selected. Verify linux3 is selected (checked) and click Next to continue. |
| Cluster Node Additional Summary | Verify the new oracle RAC node is listed under the “New Nodes” drilldown. Click Install to start the installation. |
| Execute Configuration Scripts | Navigate to the /u01/app/oracle/product/10.2.0/db 1 directory on linux3 and run root.sh as the “root” user account. After running the root.sh script on the new Oracle RAC node, go back to the OUI and acknowledge the “Execute Configuration scripts” dialog window. |
| End of installation | At the end of the installation, exit from the OUI |
Run the Network Configuration Assitant:
To start the NETCA, run the following from linux1:
$ netca &
| Screen Name | Response |
| Select the Type of Oracle Net Services Configuration | Select Cluster Configuration |
| Select the nodes to configure | Only select the new oracle RAC node: linux3 |
| Type of Configuration | Select Listener configuration |
| Listener Configuration Next 6 screens | What do you want to do : Add Listener name : Listener Selected protocols:TCP Port Number : 1521 Configure another listener :No Listener configuration complete : [Next] You will be returned to this welcome (Type of configuration)screen |
| Type of configuration | Select Naming Methods Configuration |
| Naming Methods Configuration | The following screens are : Selected Naming Methods : Local Naming Naming Methods configuration complete ! [Next] You will be returned to this welcome (Type of configuration)screen |
| Type of configuration | Click Finish to exit the NETCA |
$ hostname
1inux1
$ ps -ef | grep lsnr | grep -v ‘grep’ | grep -v ‘ocfs’ | awk ‘ {print $9}’
LISTENER_LINUX1
$ hostname
1inux2
$ ps -ef | grep lsnr | grep -v ‘grep’ | grep -v ‘ocfs’ | awk ‘ {print $9}’
LISTENER_LINUX2
$ hostname
1inux3
$ ps -ef | grep lsnr | grep -v ‘grep’ | grep -v ‘ocfs’ | awk ‘ {print $9}’
LISTENER_LINUX3
Add Database Instance to New Node
To start the database instance creation process for the new oracle RAC node, run the following from linux1;
$ dbca &
| Screen Name | Response |
| Welcome Screen | Select Oracle Real Application Clusters database |
| Operations | Select Instance Management |
| Instance Management | Select Add on instance |
| List of cluster databases | Orcl: Select this clustered database Username : sys Password : <sys_password> |
| List of cluster database instances | Verify this list is correct and Click Next to continue |
| Instance naming and node selection | The default values are correct (instance name “orcl3” to be added to node “linux3” ). After verifying these values, Click Next to continue. |
| Database Services | The existing clustered database has one service defined named orcl taf. With the “oracl_taf” database service selected, change the details to Preferred for the new instance (orcl3) and the “TAF Policy” set to Basic |
| Instance Storage | By default, the DBCA does a good job of determining the instance specific files such as an UNDO tablespace (UNDOTBS3), database files for this tablespace, and two redo log groups. Verify the storage options and Click Finish to add the instance |
| Database Configuration Assistant : Summary | After verifying the instance creation options in the summary dialog, Click Ok to begin the instance management process |
| Extend ASM | “ASM is present on the cluster but needs to be extended to the following nodes: [linux3]. Do you want ASM to be extended?” Click on Yes to add the ASM instance to the new node |
| Database Configuration assistant progress screen | “Do you want to perform another operation?” Click No to end and exit the DBCA utility |
| Start New Database Services | The DBCA will automatically start the new instance (orcl3) on the node linux3. If any services were configured during the instance management process, however, they are left in an offline state. For the purpose of this example , I had to manually start the “oracl_taf” service for the database: $ srvctl start service –s orcl_taf –d orcl -i orcl3 |
Verify New instance
Login to one of the instances and query the qv$instance view:
SQL> select inst_id, instance_name, status, to_char(startup_time, ‘DD-MON-YYYY HH24:MI:SS’) from gv$instance order by inst_id;
Inst_ID Instance_Name Status TO_CHAR (STARTUP_TIME)
-------------- --------------------- ---------- ----------------------------------------
1 orcl1 OPEN 23-FEB-2008 00:10:16
2 orcl2 OPEN 23-FEB-2008 00:10:47
3 orcl1 OPEN 23-FEB-2008 00:52:51
Remove Database Instance
To start the database instance removal process, run the following from linux1:
$ dbca &
| Screen Name | Response |
| Welcome Screen | Select Oracle real Application clusters database |
| Operations | Select Instance Management |
| Instance Management | Select Delete an instance |
| List of cluster databases | Orcl: select this clustered database. At the bottom of this screen , Username :sys Password: <sys_password> Click Next to continue |
| List of cluster database instances | Select (linux3:orcl3) and Click Next to continue |
| Database Services | The existing clustered database has one service defined named orcl taf. With the “orcl_taf” database service selected ,change the details to Not Used for the instance to be removed (orcl3)and leave the “TAF Policy” set to Basic. Click Finish to remove the instance |
| Database Configuration Assistant : Summary | After verifying the instance removal options in the summary dialog, click finish |
| Database Configuration assistant | Do you want to proceed?” Click ok to acknowledge this dialog |
| Database Configuration Assistant progress screen | “Do you want to perform another operation”? Click No to end and exit the DBCA utility |
Verify Database Instance and Associated Objects were Removed
Check Cluster Services
$ $ORA-CRS_HOME/bin/crs_stat -t
Verify Instance we removed:
Login to one of the instances and query the gv$instance view:
SQL> select inst_id, instance_name,status, to_char(startup_time, ‘DD-MON-YYYY HH24:MI:SS’) from gv$instance order by inst_id;
Verify Database Files /Tablespaces have been Removed
Check that the UNDO tablespace that was defined for orcl3(UNDOTBS3) has been removed.
SQL> select tablespace_name from dba_tablespace order by 1;
Update TNSNAMES
Login to all machines that were accessing the instance removed and update the tnsnames.ora file (if necessary)
Remove the ASM Instance from the Oracle Cluster Registry File
Run the following from linux1 as the oracle user account:
$ srvct1 stop asm -n linux3
$ srvct1 remove asm -n linux3
Remove ASM Directories from the Node to be deleted
Remove all ASM directories from the node to be deleted (linux3) as the oracle user account:
$ cd $ORACLE_BASE/admin
$ rm -rf +ASM
$ cd $ORACLE_HOME/dbs
$ rm –f * ASM*
Remove ASMLib from the Node to be deleted
We will remove the ASMLib kernel driver, supporting software and associated directories. Run the following from the node to be deleted (linux3) as the root user account.
# /etc/init.d/oracleasm stop
Unmounting ASMlib driver filesystem: [ok]
Unloading module “oracleasm” : [ok]
# rpm -qa | grep oracleasm
oracleasm-support-2.0.3-1
oracleasm-2.6.9-55.EL-2.0.3-1
oracleasmlib-2.0.2-1
#rpm -ev oracleasm-support-2.0.3-1 oracleasm-2.6.9-55.EL-2.0.3-1 oracleasmlib-2.0.2-1
Warning : /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave
#rm –f /etc/sysconfig/oracleasm.rpmsave
#rm –f /etc/rc.d/init.d/oracleasm
#rm –f /etc/rc0.d/init.d/*oracleasm*
#rm –f /etc/rc1.d/init.d/*oracleasm*
#rm –f /etc/rc2.d/init.d/*oracleasm*
#rm –f /etc/rc3.d/init.d/*oracleasm*
#rm –f /etc/rc4.d/init.d/*oracleasm*
#rm –f /etc/rc5.d/init.d/*oracleasm*
#rm –f /etc/rc6.d/init.d/*oracleasm*
Remove the Listener From the node to be deleted
To start the NETCA, run the following from linux1;
$ netca &
| Screen Name | Response |
| Select the Type of oracle Net Services Configuration | Select Cluster configuration |
| Select the nodes to configure | Only select the oracle RAC node to remove the listener from : linux3 |
| Type of configuration | Select Listener configuration |
| Listener Configuration Next 4 screens | What do you want to do : Delete Select a listener you want to delete : Listener Acknowledge warning dialog: Yes Listener deleted [Next] Listener configuration complete ! [Next] You will be returned to this welcome (Type of configuration) screen |
| Type of configuration | Click Finish to exit the NETCA |
Verify TNS Listener Configuration
The Oracle node-specific TNS listener processes should now only be running on two nodes in the RAC cluster (linux1 and linux2)
$ hostname
linux1
$ ps -ef | grep lsnr | grep -v ‘grep’ | grep –v ‘ocfs’ | awk ‘ {print $9}’
Listener_Linux1
$ hostname
linux2
$ ps -ef | grep lsnr | grep -v ‘grep’ | grep –v ‘ocfs’ | awk ‘ {print $9}’
Listener_Linux2
$ hostname
Linux3
$ ps -ef | grep lsnr | grep -v ‘grep’ | grep –v ‘ocfs’ | awk ‘ {print $9}’
Remove the Oracle Database Sofware From the Node to be deleted
Update Node List for oracle Database software – (Remove linux3)
From the node to be deleted (linux3) , run the OUI as the oracle user account to update the inventory node list for the oracle Database software:
$ cd $ORACLE_HOME/oui/bin
$ ./runinstaller –updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES=”” – local
Starting Oracle Universal Installer…..
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
‘UpdateNodeList’ was successful.
De-install Oracle Database Software
From linux3 as the oracle user account, run the following:
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller
| Screen name | Response |
| Welcome Screen | Click the Intalled Products button. |
| Inventory: Contents Tab | Check the Oracle home to deleted (OraD10g homel) and click the Remove button |
| Confirmation | Acknowledge the warning dialog by clicking Yes |
| Deinstallation Process | After confirming the Oracle Database software (Oracle home) was successfully removed. Click Close to exit this dialog. |
| Welcome Screen | Click Cancel to exit th OUI |
Update Node List for remaining Nodes in the Cluster
Finally , from linux1 logged in as the oracle user account, update the Oracle Clusterware inventory node list for all nodes that will remain in the cluster.
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller –updateNodeList ORACLE_HOME=$ORACLE_HOME
“CLUSTER_NODES=(linux1,linux2)”
Starting Oracle is universal Installer……
No pre-requisite checks found in oraparam.ini, no system pre-requiste checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
‘UpdateNodeList’ was successful .
Remove the Node to be deleted from Oracle Clusterware
Remove Node-Specific Interface Configuration
Run these commands from linux3 as the oracle user account;
$ $ORA_CRS_HOME/bin/racgons remove_config linux3:6200
Disable Oracle Clsterware Applications
From the node you are deleting from the cluster(linux3) , run the script
$ORA_CRS_HOME/install/rootdelete.sh to disable the Oracle Clusterware applications that are on the node. This script should only be run once. Given the Clusterware software install is on local disk (non shared), make certain to use the nosharedhome argument. The default for this script is sharedhome which prevents you from updating the permissions of local files such that they can be removed by the oracle user account.
Running this script will stop the CRS stack and delete the ocr.loc file on the node to be removed. The nosharedvar option assumes the ocr.loc file is not on a shared file system.
While logged into linux3 as the root user account, run the following:
$ su
# cd $ORA_CRS_HOME/install
# ./rootdelete.sh local nosharedvar nosharedhome
CRS-0210: Could not find resource ‘ora.linux3.LISTENER_LINUX3.lsnr’.
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
Checking to see if oracle CRS stack is down…
Oracle CRS stack is not running.
Oracle CR stack is down now.
Removing script for oracle Cluster Ready services
Updating ocr file foe downgrade
Cleaning up SCR settings in ‘/etc/oracle/scls_scr’
Delete Node from Cluster and update OCR
Upon successful completion of the rootdelete.sh script, run the rootdeletenode.sh script to delete the node(linux3) from the Oracle Cluster and to update the Oracle Cluster and to update the Oracle Cluster Registry(OCR). This script should be run from a pree-existing /available node in the cluster(linux1) as the root user account.
Before executing rootdeletenode.sh, we need to know the node number associated with the node name to be deleted from the cluster. To determine the node number, run the following command as the oracle user account from linux1:
$ $ORA_CRS_HOME/bin/olsnodes –n
linux1 1
linux2 2
linux3 3
From the listing above, the node number for linux3 is 3.
While logged into linux1 as the root user account,run the following using the name linux3 and the node number 3.
$ su
# cd $ORA_CRS_HOME/Install
# ./rootdeletenode.sh linux3,3
CRS-0210: Could not find resource ‘ora.linux3.LISTENER_LINUX3.lsnr’.
CRS-0210: Could not find resource ‘ora.linux3.ons’.
CRS-0210: Could not find resource ‘ora.linux3.vip’.
CRS-0210: Could not find resource ‘ora.linux3.gsd’.
CRS-0210: Could not find resource ‘ora.linux3.vip.
CRS nodeapps are deleted successfully
Clscfg: EXISTING configuration version 3 detected.
Clscfg: Version 3 is LOG Release 2.
Successfully deleted 14 values from OCR.
Key SYSTEM.css.interfaces.nodelinux3 marked for deletion is not there. Ignoring ,
Successfully deleted 5 keys from OCR.
Node deletion operation successful.
‘linux3,3’ deleted successfully
To verify that the node was successfully removed , use the following as either the Oracle or root user:
$ $ORA_CRS_HOME/bin/olsnodes -n
linux1 1
linux2 2
Update Node list for oracle Clusterware Software – (Remove linux3)
From the node to be deleted (linux3), run the OUI as the oracle user account to update the inventory node list for the Oracle Clusterware Software:
$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller –updateNodelist ORACLE_HOME=$ORA_CRS_HOME CLUSTER_NODES=”” –local CRS=true
Starting Oracle universal Installer……
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
‘UpdateNodeList’ was successful
De-install Oracle Clusterware Software
From linux3 as the oracle user account, run the following:
$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller
| Screen Name | Response |
| Welcome screen | Click the Installed Products button |
| Inventory: Contents Tab | Check the oracle home to deleted (OraCrs10g home) and click the Remove button |
| Confirmation | Acknowledge the warning dialog by clicking Yes to remove the Oracle Clusterware software and to remove the /u01/app/crs directory |
| Deinstallation Process | After confirming the Oracle Clusterware software (Clusterware home) was successfully removed. Click Close to exit this dialog |
| Welcome Screen | Click Cancel to exit the OUI |
Update Node List for remaining Nodes in the Cluster
Finally , from linux1 logged in as the oracle user account (and user equivalence enabled), update the Oracle Clusterware inventory node list for all nodes that will remain in the cluster.
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller –updateNodeList ORACLE_HOME=$ORA_CRS_HOME
“CLUSTER_NODES=(linux1,linux2)” CRS=true
Starting Oracle is universal Installer……
No pre-requisite checks found in oraparam.ini, no system pre-requiste checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
‘UpdateNodeList’ was successful .
Verify Node to be deleted is no longer a Member of the Cluster
$ $ORA_CRS_HOME/bin/olsnodes -n
linux1 1
linux2 2
Remove /Rename any remaining Oracle Files from Node to be deleted
From the node to be deleted (linux3) remove /rename any remaining Oracle files while logged in as the root user account.
# mv - f /etc/inittab.no_crs /etc/inittab
# rm -f /etc/inittab.orig
# rm -f /etc/inittab.crs
# rm -rf /etc/oracle
# rm -f /etc/oratab
# rm -f /etc/oraInst.loc
# rm -rf /etc/ORCLcluster
# rm -rf /u01/app/oracle
# rm -rf /u01/app/crs
# rm -f /user/local/bin/coraenv
# rm -f /user/local/bin/dbhome
# rm -f /user/local/bin/oraenv
Finally, remove the Oracle user account and all associated UNIX groups from linux3.
# userdel –r oracle
# groupdel oinstall
# groupdel dba
Remove OCFS2 from the Node to be deleted
In this session we will remove the OCFS2 Kernel driver, supporting software, and associated directories. Run the following from the node to be deleted (linux3) as the root user account.
# umount /u02
# /etc/init.d/o2cb offline ocfs2
# /etc/init.d/o2cb unload
# rpm –qa | grep ccfs2
Ocfs2-tools-1.2.4-1
Ocfs2-2.6.9-55.EL-1.2.5-6
Ocfs2console-1.2.4-1
# rpm -ev ocfs2-tools -1.2.4-1 ocfs2-2.6.9-55.EL-1.2.5-6 ocfs2console-1.2.4-1
Warning : /etc/sysconfig/o2cb saved as /etc/sysconfig/o2cb.rpmsave
# rm –f /etc/sysconfig/o2cb.rpmsave
#rm –rf /etc/ocfs2
Remove Node to be deleted from the OCFS2 Cluster
The next step is to remove the Oracle RAC node being deleted (linux3) from the “live” OCFS2 cluster. This entails running the o2cb_ctl command-line utlity from the remaining two RAC nodes linux1 and linux2
As root, run the following from linux1 and then linux2:
[root@linux1 ~] # o2cb_ctl – D – u -n linux3
O2cb_ctl: Not yet supported
[root@linux12~] # o2cb_ctl – D – u -n linux3
O2cb_ctl: Not yet supported
O2cb_ctl parameters:
-D: Delete an object from the existing OCFS2 cluster configuration.
-u: Valid only with –D. When deleting something (node or cluster), it will also remove it from the live cluster (/config). If the parameter is not specified , then only update the /etc/ocfs2/cluster.conf.
-n: Object name which is usally the node name or cluster name.
Remove OCFS2 Mount options from /etc/fstab
Finally , remove the following entry from the /etc/fstab from linux3;
LABEL=oracrsfiles /u02 ocfs2 -netdev,datavolume,nointr
Disable iSCSI Initiator Service on linux3
First , stop the iscsi initiator service on linux3;
# service iscsi stop
Searching for iscsi-based multipath maps
Found 0 maps
Stopping iscsid: [OK]
Removing iscsi driver : [OK]
Next, edit the file /etc/iscsi.conf on linux3 and comment out(or remove) the Discovery Address entry which specifies the hostname of the Openfiler network storage server. In our example, use # at the beginning of Discovery Address entry as shown below:
# DiscoveryAddress=openfiler1-priv
Finally , configure the ISCSI service to not be active across machine reboots on linux3. The Linux command chkconfig can be used to achieve that as follows:
# chkconfig –level 345 iscsi off
No comments:
Post a Comment