Oracle Technologies Blog

By ASKM

Archive for the ‘RAC’ Category

Oracle Practice Labs

Posted by Srikrishna Murthy Annam on February 1, 2014

 

practice labs

Posted in RAC | Tagged: , , | 1 Comment »

RAC Administration 11gR2 – Weekend Online Training

Posted by Srikrishna Murthy Annam on November 6, 2013

Please visit our site http://askMLabs.com for more details

Direct Link For Free Demo Registration :  Free Demo

Oracle Database 11gR2 Grid Infra with ASM and RAC Administration
Free Demo on 09-NOV-2013 From 11am to 12pm.
Online Classes , 4 Weeks , Batch starts from 16-NOV-2013
Classes Every Week SAT and SUN From 7am to 11am
For Free Registrations Visit : http://askMLabs.com, info@askmlabs.com
Lab Access From Anywhere In The World

 

OWEMRAC11003

 

Posted in 11gR2 RAC, RAC | Tagged: , , , , , , , | 6 Comments »

Online Trainings On Oracle Database 11g

Posted by Srikrishna Murthy Annam on September 13, 2013

We are starting new trainings on oracle database 11g. Please review our site http://askmlabs.com or attend our free demo to know more details.
1) Oracle Database 11g Administration ( with 12c new features )
2) Oracle Database 11gR2 Grid with ASM and RAC Administration
3) Oracle Database 11gR2 DataGuard Administration

All these trainings including lab sessions are handled by Srikrishna Murthy Annam.

askmlabs-online-trainings1

Thanks

SRI

 

Posted in 11gR2 RAC, ASM, Data Guard, ORACLE Tech, RAC | 3 Comments »

Troubleshooting RAC Services Startup

Posted by Srikrishna Murthy Annam on April 3, 2013

Recently we had a DR(Disaster Recovery) testing going on and we had RAC replica at the DR site for DR testing. The RAC servers are copied to DR setup, bit by bit. Then tried to start the RAC services at DR site and could not start them due to some issues.

Then I tried to troubleshoot the issue …..

This article gives you the troubleshooting steps for this particular issue. These steps may also help you to troubleshoot if you have issues starting RAC services.

STEP 1: Reboot the rac servers and Check services after re-booting the servers. The services are still down.

STEP 2: Verify storage on both rac servers

[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac01 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac01 ~]$ exit
logout
[oracle@askmdbrac01 ~]$ sudo su –
[root@askmdbrac01 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:0:1 sdh        8:112 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:1:1 sdk        8:160 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:1:0 sdj        8:144 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:0:0 sdg        8:96  [active][ready]
[root@askmdbrac01 ~]# exit[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac02 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac02 ~]$ exit
logout
[oracle@askmdbrac02 ~]$ sudo su –
[root@askmdbrac02 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:1:1 sdk        8:160 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:0:1 sdh        8:112 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:0:0 sdg        8:96  [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:1:0 sdj        8:144 [active][ready]
[root@askmdbrac02 ~]#

Observation : Storage is fine and i am able to see ASM disks from both the nodes. Storage is configured with multipathing. I also verified multipathing and status shows fine.

STEP 3: Check high availability services and cluster ready services on both nodes with the following commands

[grid@askmdbrac01 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$
[grid@askmdbrac01 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$[grid@askmdbrac02 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac02 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$
[grid@askmdbrac02 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$

Observation : The high availability services are up , but crs services are down.

STEP 4: Do ping and nslookup for other rac nodes from each node to make sure that each node in cluster is accessible from every other node. Check this for private ips, virtual ips and for scan ips.

Observation : The ping and nslookup are working fine and all the nodes are reachable from every other node.

STEP 5 : Then verify the log files from grid infrastructure and see if you can find any error messages from log files. The cluster log files are located at $ORACLE_HOME/log/askmdbrac01 and $ORACLE_HOME/log/askmdbrac02.

[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:27.425
[ohasd(6876)]CRS-2765:Resource ‘ora.diskmon’ has failed on server ‘askmdbrac01’.
2013-04-02 19:38:39.988
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:55.007
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:39:10.025
…..
…..
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.836
[/o001/home/11.2.0.2/grid/bin/cssdagent(11163)]CRS-5818:Aborted command ‘start for resource: ora.cssd 1 1’ for resource ‘ora.cssd’. Details at (:CRSAGF00113:) {0:27:4} in /o001/home/11.2.0.2/grid/log/askmdbrac01/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2013-04-02 19:48:23.837
[cssd(11175)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.838
[cssd(11175)]CRS-1603:CSSD on node askmdbrac01 shutdown by user.
2013-04-02 19:48:25.730
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:29.863
[ohasd(6876)]CRS-2765:Resource ‘ora.cssdmonitor’ has failed on server ‘askmdbrac01’.
2013-04-02 19:48:30.126
[ohasd(6876)]CRS-2771:Maximum restart attempts reached for resource ‘ora.cssd’; will not restart.
2013-04-02 19:48:35.833
[ohasd(6876)]CRS-2765:Resource ‘ora.diskmon’ has failed on server ‘askmdbrac01’.

Observation : Found from the cluster alert log file (alertaskmdbrac01.log) that the cluster services can’t find voting disk. So it could be due the in-accessible asm storage or due to currupted voting disks.

Then i tried to see if i can see voting disks through command.

SETP 6 :  Check the voting disks availability ( 11gR2 has voting disks in asm. I don’t have asm instance up now. So i will get error if i try to query the voting disks on asm)

[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[root@askmdbrac01 ~]#[grid@askmdbrac02 ~]$ /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[grid@askmdbrac02 ~]$

Observation : I am not able to query the voting disks. Now i have only one option to look at the asm storage accessibility.

STEP 7 :  Tried re-enabling the asmlibs as follows on both the nodes …

[root@askmdbrac01 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]# /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#  /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Not able to start services even after restarting the asmlib.

STEP 8 : Tried to re-configure the asmlib as below …..

[root@askmdbrac01 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]#
[root@askmdbrac01 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]#

[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#
[root@askmdbrac02 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Identified that the asm library drivers configuration is wrong and i re-configured with correct user and group. See the difference in the lines when it is asking for prompt on second node. I executed this command twice to show you the difference when it is properly configured and when it is not properly configured.

STEP 9 : Stop and Start cluster services on both the nodes.

[root@askmdbrac01 ~]#  /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘askmdbrac01’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac01’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘askmdbrac01’ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac01’ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘askmdbrac01’
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘askmdbrac01’ succeeded
CRS-2672: Attempting to start ‘ora.scan1.vip’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac01’ succeeded
CRS-2673: Attempting to stop ‘ora.askmdbrac01.vip’ on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.CRS.dg’ on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.racdb.db’ on ‘askmdbrac01’
CRS-2677: Stop of ‘ora.askmdbrac01.vip’ on ‘askmdbrac01’ succeeded
CRS-2672: Attempting to start ‘ora.askmdbrac01.vip’ on ‘askmdbrac02’
CRS-2676: Start of ‘ora.scan1.vip’ on ‘askmdbrac02’ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.racdb.db’ on ‘askmdbrac01’ succeeded
CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac01’
CRS-2676: Start of ‘ora.askmdbrac01.vip’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac01’ succeeded
CRS-2676: Start of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.FRA.dg’ on ‘askmdbrac01’ succeeded
CRS-2677: Stop of ‘ora.CRS.dg’ on ‘askmdbrac01’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac01’
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac01’ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘askmdbrac01’
CRS-2677: Stop of ‘ora.ons’ on ‘askmdbrac01’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘askmdbrac01’
CRS-2677: Stop of ‘ora.net1.network’ on ‘askmdbrac01’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘askmdbrac01’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘askmdbrac01’ succeeded
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.crf’ on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac01’
CRS-2677: Stop of ‘ora.mdnsd’ on ‘askmdbrac01’ succeeded
CRS-2677: Stop of ‘ora.crf’ on ‘askmdbrac01’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘askmdbrac01’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac01’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘askmdbrac01’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘askmdbrac01’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘askmdbrac01’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘askmdbrac01’
CRS-2677: Stop of ‘ora.cssd’ on ‘askmdbrac01’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘askmdbrac01’
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘askmdbrac01’
CRS-2677: Stop of ‘ora.gipcd’ on ‘askmdbrac01’ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘askmdbrac01’
CRS-2677: Stop of ‘ora.gpnpd’ on ‘askmdbrac01’ succeeded
CRS-2677: Stop of ‘ora.diskmon’ on ‘askmdbrac01’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac01’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘askmdbrac02’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.askmdbrac02.vip’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.CRS.dg’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.askmdbrac02.vip’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.racdb.db’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.askmdbrac01.vip’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.askmdbrac01.vip’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.scan2.vip’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.scan3.vip’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.racdb.db’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.FRA.dg’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.oc4j’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.cvu’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.CRS.dg’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.ons’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.net1.network’ on ‘askmdbrac02’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘askmdbrac02’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.evmd’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.mdnsd’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.cssd’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘askmdbrac02’
CRS-2673: Attempting to stop ‘ora.crf’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.crf’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.diskmon’ on ‘askmdbrac02’ succeeded
CRS-2677: Stop of ‘ora.gipcd’ on ‘askmdbrac02’ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘askmdbrac02’
CRS-2677: Stop of ‘ora.gpnpd’ on ‘askmdbrac02’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac02’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac02 ~]#

Observation : Cluster ready services started without any issues. Now i need to verify the services status.

STEP 10 :  Verify the status.

[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora….ER.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N1.lsnr ora….er.type ONLINE    OFFLINE
ora….N2.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N3.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….DATA.dg ora….up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora….SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora….01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora….l01.gsd application    OFFLINE   OFFLINE
ora….l01.ons application    ONLINE    ONLINE    askmdbrac01
ora….l01.vip ora….t1.type ONLINE    ONLINE    askmdbrac01
ora….SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora….02.lsnr application    ONLINE    OFFLINE
ora….l02.gsd application    OFFLINE   OFFLINE
ora….l02.ons application    ONLINE    OFFLINE
ora….l02.vip ora….t1.type ONLINE    OFFLINE
ora….ry.acfs ora….fs.type ONLINE    OFFLINE
ora.scan1.vip  ora….ip.type ONLINE    OFFLINE
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora….se.type ONLINE    ONLINE    askmdbrac01
ora….ice.svc ora….ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac01 ~]# sudo su – grid
[grid@askmdbrac01 ~]$ . ./profile.grid
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac01 ~]$[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora….ER.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N1.lsnr ora….er.type ONLINE    ONLINE    askmdbrac02
ora….N2.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N3.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….DATA.dg ora….up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora….SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora….01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora….l01.gsd application    OFFLINE   OFFLINE
ora….l01.ons application    ONLINE    ONLINE    askmdbrac01
ora….l01.vip ora….t1.type ONLINE    ONLINE    askmdbrac01
ora….SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora….02.lsnr application    ONLINE    ONLINE    askmdbrac02
ora….l02.gsd application    OFFLINE   OFFLINE
ora….l02.ons application    ONLINE    ONLINE    askmdbrac02
ora….l02.vip ora….t1.type ONLINE    ONLINE    askmdbrac02
ora….ry.acfs ora….fs.type ONLINE    OFFLINE
ora.scan1.vip  ora….ip.type ONLINE    ONLINE    askmdbrac02
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora….se.type ONLINE    ONLINE    askmdbrac01
ora….ice.svc ora….ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac02 ~]# sudo su – grid
[grid@askmdbrac02 ~]$ . ./profile.grid
[grid@askmdbrac02 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac02 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac02 ~]$

This concludes the article on troubleshooting RAC services startup issue.

Thanks
SRI

Posted in 11gR2 RAC, RAC | Tagged: , , , | 3 Comments »

Oracle OpenWorld 2012 Update

Posted by Srikrishna Murthy Annam on October 1, 2012

Larry Ellison’s major  announcements in OracleOpenWorld-2012

  1. IaaS  => Infrastructure as a Service

  2. Private Clouds

  3. Oracle Database 12c

  4. Exadata X3 machine.

Thanks

SRIKRISHNA MURTHY

Posted in RAC | Tagged: | Leave a Comment »

SCAN LISTENER TRACING

Posted by Srikrishna Murthy Annam on May 12, 2011

This article describes enabling and disabling the SCAN listener tracing. Before capturing the scan listener tracing,please make sure your scan listener is configured as per the following Oracle Support IDs.
1070607.1
975457.1

Some where you should find some clue for your SCAN connectivity issues. If you are still unable to find the exact problem with the scan connectivity, you need to enable the tracing and analyze the trace files or upload the trace files to oracle support.

To enable level 16 tracing for SCAN listeners, add the following parameters to listener.ora and  try to reproduce the connectivity issue.

TRACE_LEVEL_<listener_name> = 16
TRACE_DIRECTORY_<listener_name> =<dir location>
TRACE_TIMESTAMP_<listener_name> = TRUE
DIAG_ADR_ENABLED_<listenername>=off

Then try to reload all the listeners after these parameters are in place using lsnrctl reload <listener_name>.

Reproduce the issue and verify that the trace and log files are created in the directory mentioned in the parameter TRACE_DIRECTORY_<listener_name>.

To disable scan listener tracing:

$lsnrctl
LSNRCTL>set current_listener <listener_name>
LSNRCTL>set trc_level OFF
LSNRCTL>save_config

In the above tracing we disabled diagnostic repository with the parameter DIAG_ADR_ENABLED_<listenername> and collected trace files. Now if you want to re-enable DIAG_ADR, just comment out DIAG_ADR_ENABLED_<listenername> in listener.ora and reload all the listeners.

NOTE: ADR is the new concept introduced from 11g and it is a file system repository to store all diagnostic data. Please review the article  Automatic Diagnostic Repository  to know more on ADR concepts.

Session Log to show enabling SCAN Tracing:

[grid@b600e6tr01 admin]$ cp listener.ora listener.ora_bak
[grid@b600e6tr01 admin]$ vi listener.ora   ==> Here add trace parameter
[grid@b600e6tr01 admin]$ clear
[grid@b600e6tr01 admin]$ ps -ef | grep -i tns
grid      5943  4623  0 07:12 pts/1    00:00:00 grep -i tns
grid     18139     1  0 May03 ?        00:01:01 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid     18142     1  0 May03 ?        00:01:14 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
grid     18164     1  0 May03 ?        00:01:18 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
[grid@b600e6tr01 admin]$ lsnrctl reload listener

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:12:38

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
The command completed successfully

[grid@b600e6tr01 admin]$ lsnrctl reload LISTENER_SCAN2

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:12:55

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
The command completed successfully
[grid@b600e6tr01 admin]$ lsnrctl reload LISTENER_SCAN3

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:13:02

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
The command completed successfully
[grid@b600e6tr01 admin]$

[grid@b600e6tr02 admin]$ cp listener.ora listener.ora_bak
[grid@b600e6tr02 admin]$ vi listener.ora  ==> Here add trace parameters
[grid@b600e6tr02 ~]$ ps -ef | grep -i tns
grid      1042     1  0 May03 ?        00:01:20 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid      1171     1  0 May03 ?        00:01:04 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid      8231  7935  0 07:06 pts/1    00:00:00 grep -i tns
[grid@b600e6tr02 ~]
[grid@b600e6tr02 admin]$ lsnrctl reload LISTENER

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:13:26

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
The command completed successfully
[grid@b600e6tr02 admin]$ lsnrctl reload LISTENER_SCAN1

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:13:35

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
The command completed successfully
[grid@b600e6tr02 admin]$

Session Log to show disabling SCAN Tracing:

[grid@b600e6tr01 ~]$ ps -ef | grep -i tns
grid      5309  4623  0 07:06 pts/1    00:00:00 grep -i tns
grid     18139     1  0 May03 ?        00:01:01 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid     18142     1  0 May03 ?        00:01:14 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
grid     18164     1  0 May03 ?        00:01:18 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
[grid@b600e6tr01 ~]$ lsnrctl

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:07:02

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> set current_listener LISTENER
Current Listener is LISTENER
LSNRCTL> set trc_level OFF
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
LISTENER parameter "trc_level" set to off
The command completed successfully
LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
Saved LISTENER configuration parameters.
Listener Parameter File   /u01/home/11.2.0/grid/network/admin/listener.ora
Old Parameter File   /u01/home/11.2.0/grid/network/admin/listener.bak
The command completed successfully
LSNRCTL> set current_listener LISTENER_SCAN2
Current Listener is LISTENER_SCAN2
LSNRCTL> set trc_level OFF
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
LISTENER_SCAN2 parameter "trc_level" set to off
The command completed successfully
LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
Saved LISTENER_SCAN2 configuration parameters.
Listener Parameter File   /u01/home/11.2.0/grid/network/admin/listener.ora
Old Parameter File   /u01/home/11.2.0/grid/network/admin/listener.bak
The command completed successfully
LSNRCTL> set current_listener LISTENER_SCAN3
Current Listener is LISTENER_SCAN3
LSNRCTL> set trc_level OFF
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
LISTENER_SCAN3 parameter "trc_level" set to off
The command completed successfully
LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
Saved LISTENER_SCAN3 configuration parameters.
Listener Parameter File   /u01/home/11.2.0/grid/network/admin/listener.ora
Old Parameter File   /u01/home/11.2.0/grid/network/admin/listener.bak
The command completed successfully
LSNRCTL>

[grid@b600e6tr02 ~]$ ps -ef | grep -i tns
grid      1042     1  0 May03 ?        00:01:20 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid      1171     1  0 May03 ?        00:01:04 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid      8231  7935  0 07:06 pts/1    00:00:00 grep -i tns
[grid@b600e6tr02 ~]$ lsnrctl

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:10:37

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> set current_listener LISTENER
Current Listener is LISTENER
LSNRCTL> set trc_level OFF
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
LISTENER parameter "trc_level" set to off
The command completed successfully
LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
Saved LISTENER configuration parameters.
Listener Parameter File   /u01/home/11.2.0/grid/network/admin/listener.ora
Old Parameter File   /u01/home/11.2.0/grid/network/admin/listener.bak
The command completed successfully
LSNRCTL> set current_listener LISTENER_SCAN1
Current Listener is LISTENER_SCAN1
LSNRCTL> set trc_level OFF
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
LISTENER_SCAN1 parameter "trc_level" set to off
The command completed successfully
LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
Saved LISTENER_SCAN1 configuration parameters.
Listener Parameter File   /u01/home/11.2.0/grid/network/admin/listener.ora
Old Parameter File   /u01/home/11.2.0/grid/network/admin/listener.bak
The command completed successfully
LSNRCTL>

Hope it helps
SRI

Posted in 11gR2 RAC, Troubleshooting | Tagged: , , , , , , , , | 1 Comment »

Applying PSU Patch 11.2.0.1.2 To A Two Node RAC – Part2

Posted by Srikrishna Murthy Annam on May 6, 2011

This article is continuation to the previous article Applying PSU Patch 11.2.0.1.2 To A Two Node RAC

In this article i want to show you the session output for PSU patch application and the session output for  roll backing a PSU patch.

This is the best example to show the hot patching in RAC 11gR2. The patch is applied in a rolling fashion on each node of the cluster. The clusterware automatically relocates the services to other node while patching is running on some node. The end user does not feel any service interruption.

PSU Patch On GRID_HOME(Node1 and Node2)
On b600e6tr01:

[root@b600e6tr01 ~]# cd /u01/home/oracle/product/11.2.0/db_1/patches/psupatch
[root@b600e6tr01 psupatch]# opatch auto . -oh /u01/home/11.2.0/grid
Executing /usr/bin/perl /u01/home/11.2.0/grid/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /u01/home/11.2.0/grid -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-19_11-28-59.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/11.2.0/grid/ocm.rsp

Unable to determine if /u01/home/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Successfully unlock /u01/home/11.2.0/grid
patch ././9655006  apply successful for home  /u01/home/11.2.0/grid
patch ././9654983  apply successful for home  /u01/home/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@b600e6tr01 psupatch]#

On b600e6tr02:

[root@b600e6tr02 psupatch]# opatch auto . -oh /u01/home/11.2.0/grid
Executing /usr/bin/perl /u01/home/11.2.0/grid/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /u01/home/11.2.0/grid -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-19_12-26-36.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/11.2.0/grid/ocm.rsp

Unable to determine if /u01/home/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Successfully unlock /u01/home/11.2.0/grid
patch ././9655006  apply successful for home  /u01/home/11.2.0/grid
patch ././9654983  apply successful for home  /u01/home/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@b600e6tr02 psupatch]#

PSU Patch On RDBMS_HOME(Node1 and Node2)
On b600e6tr01 :

[root@b600e6tr01 psupatch]# /u01/home/oracle/product/11.2.0/db_1/OPatch/opatch auto . -oh  /u01/home/oracle/product/11.2.0/db_1
Executing /usr/bin/perl /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /u01/home/oracle/product/11.2.0/db_1 -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-19_13-52-07.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/oracle/product/11.2.0/db_1/ocm.rsp

Unable to determine if /u01/home/oracle/product/11.2.0/db_1 is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
patch ././9655006/custom/server/9655006  apply successful for home  /u01/home/oracle/product/11.2.0/db_1
patch ././9654983  apply successful for home  /u01/home/oracle/product/11.2.0/db_1
[root@b600e6tr01 psupatch]#

On b600e6tr02:

[root@b600e6tr02 psupatch]# /u01/home/oracle/product/11.2.0/db_1/OPatch/opatch auto . -oh  /u01/home/oracle/product/11.2.0/db_1
Executing /usr/bin/perl /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /u01/home/oracle/product/11.2.0/db_1 -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-19_13-28-55.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/oracle/product/11.2.0/db_1/ocm.rsp

Unable to determine if /u01/home/oracle/product/11.2.0/db_1 is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
patch ././9655006/custom/server/9655006  apply successful for home  /u01/home/oracle/product/11.2.0/db_1
patch ././9654983  apply successful for home  /u01/home/oracle/product/11.2.0/db_1
[root@b600e6tr02 psupatch]#

RollBack PSU Patch On GRID_HOME(Node1 and Node2)
On b600e6tr01:

[oracle@b600e6tr01 ~]$ sudo su -
[root@b600e6tr01 ~]# /u01/home/11.2.0/grid/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -rollback -oh /u01/home/11.2.0/grid
Executing /usr/bin/perl /u01/home/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /u01/home/oracle/product/11.2.0/db_1/patches -patchn psupatch -rollback -oh /u01/home/11.2.0/grid -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2011-05-04_07-27-40.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/11.2.0/grid/ocm.rsp

Unable to determine if /u01/home/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Successfully unlock /u01/home/11.2.0/grid
patch 9655006  rollback successful for home /u01/home/11.2.0/grid
patch 9654983  rollback successful for home /u01/home/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@b600e6tr01 ~]#

On b600e6tr02:

[oracle@b600e6tr02 ~]$ sudo su -
[root@b600e6tr02 ~]# /u01/home/11.2.0/grid/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -rollback -oh /u01/home/11.2.0/grid
Executing /usr/bin/perl /u01/home/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /u01/home/oracle/product/11.2.0/db_1/patches -patchn psupatch -rollback -oh /u01/home/11.2.0/grid -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2011-05-04_07-41-37.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/11.2.0/grid/ocm.rsp

Unable to determine if /u01/home/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Successfully unlock /u01/home/11.2.0/grid
patch 9655006  rollback successful for home /u01/home/11.2.0/grid
patch 9654983  rollback successful for home /u01/home/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@b600e6tr02 ~]#

RollBack PSU Patch ON RDBMS_HOME(Node1 and Node2)
On b600e6tr01 :

[root@b600e6tr01 ~]# /u01/home/oracle/product/11.2.0/db_1/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -rollback  -oh  /u01/home/oracle/product/11.2.0/db_1
Executing /usr/bin/perl /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/patch112.pl -patchdir /u01/home/oracle/product/11.2.0/db_1/patches -patchn psupatch -rollback -oh /u01/home/oracle/product/11.2.0/db_1 -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-05-04_07-54-30.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/oracle/product/11.2.0/db_1/ocm.rsp

Unable to determine if /u01/home/oracle/product/11.2.0/db_1 is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
patch 9655006  rollback successful for home /u01/home/oracle/product/11.2.0/db_1
patch 9654983  rollback successful for home /u01/home/oracle/product/11.2.0/db_1
[root@b600e6tr01 ~]#

On b600e6tr02:

[root@b600e6tr02 ~]#
[root@b600e6tr02 ~]# /u01/home/oracle/product/11.2.0/db_1/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -rollback  -oh  /u01/home/oracle/product/11.2.0/db_1
Executing /usr/bin/perl /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/patch112.pl -patchdir /u01/home/oracle/product/11.2.0/db_1/patches -patchn psupatch -rollback -oh /u01/home/oracle/product/11.2.0/db_1 -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-05-04_08-01-19.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/oracle/product/11.2.0/db_1/ocm.rsp

Unable to determine if /u01/home/oracle/product/11.2.0/db_1 is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
patch 9655006  rollback successful for home /u01/home/oracle/product/11.2.0/db_1
patch 9654983  rollback successful for home /u01/home/oracle/product/11.2.0/db_1
[root@b600e6tr02 ~]#

Hope it helps

SRI

Posted in 11gR2 RAC | Tagged: , , | Leave a Comment »

Applying PSU Patch 11.2.0.1.2 To A Two Node RAC

Posted by Srikrishna Murthy Annam on May 5, 2011

NOTE : This article is aimed at showing the issues and resolution while applying the PSU patch 11.2.0.1.2 (9655006) to a specific environment and not generalized. Please refer the readme of the PSU patch for detailed procedure to apply PSU patch.

Video Demos are also uploaded to show you the errors and successfull installation of the PSU to 2 node RAC.

1. Record Pre Patch Information.

2. OPatch Utility Information.

3. OCM Configuration.

4. Validation of Oracle Inventory.

5. One-off Patch Conflict Detection and Resolution.

6. Download and Unzip the PSU Patch 9655006

7. Patching GI Home

8. Patching RAC Database Homes

9. Loading Modified SQL Files into the Database.

10. Patch Successful Verification Steps.

11. Issues & Resolutions.

1. Record Pre Patch Information.

  1. Login to each node in RAC as grid user and execute the following command.

    $GRID_ORACLE_HOME/OPatch/opatch lsinventory

    $GRID_ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i ‘GI PSU’

  2. Login to each node in RAC as oracle user and execute the following command.

    $ORACLE_HOME/OPatch/opatch lsinventory

    $ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i ‘DATABASE PSU’

  3. Connect to each instance and record registry information.

    SQL> select comp_name,version,status from dba_registry;

2. OPatch Utility Information.

$ORACLE_HOME/OPatch/opatch version -h /u01/home/oracle/product/11.2.0/db_1

$GRID_ORACLE_HOME/OPatch/opatch version -h /u01/home/11.2.0/grid

3. OCM Configuration.

Create ocm response file using the following command and provide appropriate values for the prompts.

$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp

Verify the created file using,

$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp –verbose ocm.rsp

NOTE: The Opatch utility will prompt for your OCM (Oracle Configuration Manager) response file when it is run. Without which we cant proceed further.

4. Validation of Oracle Inventory.

$GRID_ORACLE_HOME/OPatch/opatch lsinventory -detail -oh /u01/home/11.2.0/grid

$ORACLE_HOME/OPatch/opatch lsinventory -detail –oh /u01/home/oracle/product/11.2.0/db_1

5. One-off Patch Conflict Detection and Resolution.

NA

6. Download and Unzip the PSU Patch 9655006

$cd /u01/home/oracle/product/11.2.0/db_1/patches/psupatch

$unzip p9655006_11201_Linux.zip

$chmow -R 777 *

7. Patching GI Home

NOTE: If the GI home is shared, then make sure to shut down the GI stack on all remote nodes. Keep the GI stack up and running on the local node.

NOTE: If the GI home is not shared, then make sure the GI stack is running on all nodes in the cluster.

Our Grid Home is not shared, So don’t shutdown any services.

$su – ( Login to root user )

#/u01/home/11.2.0/grid/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -oh /u01/home/11.2.0/grid

Monitor the logfile created in $GRID_ORACLE_HOME/cfgtoollogs/

Execute the above opatch command on each RAC node as root user.

** Please refer the Issue & Resolutions secion in the same document for any issues.

8. Patching RAC Database Homes

All Oracle processes and applications (such as emconsole and emagent) that are running from the database home and that are not managed by clusterware should be stopped manually before you apply the patch using the opatch auto command.

$su – ( Login to root user )

# /u01/home/oracle/product/11.2.0/db_1/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -oh /u01/home/oracle/product/11.2.0/db_1

Monitor the logfile created in $ORACLE_HOME/cfgtoollogs/

Execute the above opatch command on each RAC node as root user.

** Please refer the Issue & Resolutions secion in the same document for any issues.

9. Loading Modified SQL Files into the Database.

For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the catbundle.sql script as follows:

cd $ORACLE_HOME/rdbms/admin

sqlplus /nolog

SQL> CONNECT / AS SYSDBA

SQL> @catbundle.sql psu apply

SQL> QUIT

Check the log files in $ORACLE_HOME/cfgtoollogs/catbundle for any errors

10. Patch Successful Verification Steps.

  1. Login to each node in RAC as grid user and execute the following command.

    $GRID_ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i ‘GI PSU’

  2. Login to each node in RAC as oracle user and execute the following command.

    $ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i ‘DATABASE PSU’

  3. Connect to each instance and record registry information.

    SQL> select comp_name,version,status from dba_registry;

    SQL> select * from dba_registry_history;

11. Issues & Resolutions.

Issue 1:

When applying the patch, you may get the following error

The opatch minimum version check for patch /u01/home/oracle/product/11.2.0/db_1/patches/9655006/custom failed for /u01/home/11.2.0/grid

The opatch minimum version check for patch /u01/home/oracle/product/11.2.0/db_1/patches/9655006/etc failed for /u01/home/11.2.0/grid

The opatch minimum version check for patch /u01/home/oracle/product/11.2.0/db_1/patches/9655006/files failed for /u01/home/11.2.0/grid

Opatch version check failed for oracle home /u01/home/11.2.0/grid

Opatch version check failed

update the opatch version for the failed homes and retry

Solution:

Ref Note : 1308858.1

We need to provide the Patch unzipped base directory, not the directory including patch number

Ex : /u01/home/oracle/product/11.2.0/db_1/patches/psupatch/9655006 ( Wrong )

/u01/home/oracle/product/11.2.0/db_1/patches/psupatch ( correct )

Issue 2

Patch may exit with the following error messages

Unable to determine if /u01/home/11.2.0/grid is shared oracle home

Enter ‘yes’ if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes

You must kill crs processes or reboot the system to properly

cleanup the processes started by Oracle clusterware

The Oracle Clusterware stack failed to stop.

You should stop the stack with ‘crsctl stop crs’ and rerun the command

The opatch Applicable check failed for /u01/home/11.2.0/grid. The patch is not applicable for /u01/home/11.2.0/grid

patch ././9655006 apply failed for home /u01/home/11.2.0/grid

Solution :

This error may be specific to this environment though want to specify it here. This error is due to the reason that there are some cluster resources available referring the 10g database installed earlier. Delete those resources from clusterware.

# crsctl delete resource <resource_name> -f

Issue 3 :

PSU patch for GRID home on node 2 failed with the following error

Unable to determine if /u01/home/11.2.0/grid is shared oracle home

Enter ‘yes’ if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes

Successfully unlock /u01/home/11.2.0/grid

patch ././9655006 apply failed for home /u01/home/11.2.0/grid

Verified the detailed log file locate in /u01/home/11.2.0/grid/cfgtoollogs/ and found the permission issue on some files.

The following actions have failed:

Copy failed from ‘/u01/home/oracle/product/11.2.0/db_1/patches/psupatch/9655006/files/bin/crsctl.bin’ to ‘/u01/home/11.2.0/grid/bin/crsctl.bin’…

Copy failed from ‘/u01/home/oracle/product/11.2.0/db_1/patches/psupatch/9655006/files/bin/oifcfg.bin’ to ‘/u01/home/11.2.0/grid/bin/oifcfg.bin’…

Solution:

Give 777 permission to these files crsctl.bin and oifcfg.bin

Issue 4 :

PSU Patch for RDBMS Home on node 1 failed with the following error

Unable to determine if /u01/home/oracle/product/11.2.0/db_1 is shared oracle home

Enter ‘yes’ if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes

patch ././9655006/custom/server/9655006 apply failed for home /u01/home/oracle/product/11.2.0/db_1

Verified the detailed log file locate in /u01/home/oracle/product/11.2.0/db_1/cfgtoollogs and found the platform issue.

Running prerequisite checks…

Prerequisite check “CheckPatchApplicableOnCurrentPlatform” failed.

The details are:

Patch ( 9655006 ) is not applicable on current platform.

Platform ID needed is : 46

Platform IDs supported by patch are: 226

UtilSession failed: Prerequisite check “CheckPatchApplicableOnCurrentPlatform” failed.

OPatch failed with error code 73

Solution:

export OPATCH_PLATFORM_ID=226 and execute the psu patch command again.

Issue 5:

While applying Grid Infrastructure PSU (patch 9343627) to $GRID_HOME, opatch prerequisite check CheckSystemSpace failed:

Running prerequisite checks…

 Prerequisite check “CheckSystemSpace” failed.

 The details are:

 Required amount of space(4373569440) is not available.

 UtilSession failed: Prerequisite check “CheckSystemSpace” failed.

 

 OPatch failed with error code 73

Solution:

Follow the note ID 1088455.1 and make sure you have enough space on disk in GRID HOME mount point.

Reference NOTE IDs : 1082394.1 , 1308858.1 , 1169036.1 , 1290354.1, 1088455.1 and 1210964.1

APPLY PSU PATCH TO 11gR2 RAC GRID Infrastructure_part1_showing_error:

APPLY PSU PATCH TO 11gR2 RAC GRID Infrastructure_part2_SUCCESSL:

APPLY PSU PATCH TO 11gR2 RAC RDBMS HOME:

Hope it helps

SRI

Posted in 11gR2 RAC | Tagged: , , , , , , , , , , , , | 73 Comments »

JDBC Connection Issue with Oracle Database 11gR2 RAC SCAN listener

Posted by Srikrishna Murthy Annam on February 18, 2011

Recently we resolved a  JDBC connection issue with RAC database.  I will detail the issue in 8 steps

My Env setup :
We have 11gR2 RAC database running using ASM storage and all the servers are in network domain domain1. Client is using one JAVA application and it is running on non-RAC 11g database and the servers are in a different network domain domain2.
Our task :
Our task is to migrate the database that the JAVA applications is using to the RAC setup on ASM storage and establish the JAVA applications to connect to the newly migrated database. The big challenge we faced here is due to the reason that the setups are in different domains. We are using the 11gR2 SCAN concept,which complicated the issue even more.
What we did :
I am giving here the outlined view of the migration plan, but we should consider many other things in the actual migration process.

  1. export database from source
  2. Move the dump files to target database node.
  3. prepare the target database for import
  4. import the database to target database
  5. run utlrp to compile all the objects
  6. Verify the errors from import logfile
  7. “Note invalids,objects and schema status”
  8. Compare the objects status with the source
  9. verify the db links on target

All went fine. We tested the database connection from the server where java applications are running. We tested TOAD connection as well. All is well. We requested client to test the application after providing the JDBC connection string.

What is the issue :
Client raised an issue that the application is not able to connect to the database.

Troubleshooting:
As i said earlier, we tested TOAD connection. We tested sqlplus connection to the database. Both are working fine with the provided connecting string. Then we tried to dig into the SCAN listener concepts suspecting it might be an issue and configured everything as per the oracle documents. Finally we concluded that it is the issue only with JDBC connections.

Issue resolution :
Then prepared a java script to test the database java connections. Modified various settings on the database side and tested JDBC connection. Atlast we found the issue. The issue is with the connection string domain name specification.

How to test a Java Connection to oracle:
Pls see the video demo

Explanation:
As i said in my first sentence that the source and target environments are working in different domains, being source using domain2 and target using domain1. SCAN in domain1(RAC side) resolves to three IPs and the same is configured in the other network using NATing. Now the scan name resolves to three IPs in both source and target and they are connected with NATing. The host name in the connectiong string be the same as the init.ora parameter remote_listener and it should also match to the SCAN name. We should not include any domain names with SCAN name , remote_listername and with HOST setting in connecting string.

<strong>On Source RAC side:</strong>
The HOST=scan-cluster should exactly match the remote_listener parameter in the database.

SQL> show parameter remote_list

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_listener                      string      scan-cluster:1521

<strong>On Applications Side: </strong>
<database name="defaultOracle"
debug="false"
password="xxxxxxxxxxxxxxxx"
connectString="jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=scan-cluster) (PORT=1521))(CONNECT_DATA=(SERVICE_NAME=racpoc.domain1.com)(FAILOVER_MODE =(TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5))))" />

<strong>$ nslookup scan-cluster</strong>
Server:         128.191.2.13
Address:        128.191.2.13#53

Name:   scan-cluster.domain2.com
Address: 128.191.224.227
Name:   scan-cluster.domain2.com
Address: 128.191.224.29
Name:   scan-cluster.domain2.com
Address: 128.191.224.30

The Java Script to test Oracle Connection is :

import java.sql.*;
public class testconn {
public static void main(String[] s)throws Exception {
Class.forName("oracle.jdbc.OracleDriver");

String url="jdbc:oracle:thin:@(DESCRIPTION= (LOAD_BALANCE=on) (ADDRESS=(PROTOCOL=TCP)(HOST=scan-cluster)(PORT=1521)) (CONNECT_DATA=(SERVICE_NAME=racpoc.domain)))";

for (int i=0; i<20; i++) {
try {
long x= System.currentTimeMillis () ;
Connection conn = DriverManager.getConnection(url,"askm","askm");
long y= System.currentTimeMillis ();

System.out.println("Connection Succesful "+conn);
System.out.println("Connection time is "+(y-x)/1000+" ms");
Statement stmt =conn.createStatement();
ResultSet res= stmt.executeQuery(" select host_name from v$instance");

while(res.next()) {
System.out.println(res.getString(1));
}
stmt.close();
conn.close();
}
catch(Exception e) {
e.printStackTrace();}}}}

Hope it helps

SRI

Posted in 11gR2 RAC, Troubleshooting | Tagged: , , , , , , , | 77 Comments »

Troubleshooting ASM disk drop

Posted by Srikrishna Murthy Annam on February 2, 2011

The present article describes how to troubleshoot  if you are not able delete an ASM disk from  the ASM storage. The troubleshooting technique may vary from person to person and the following article describes the technique  i used to drop an ASM disk. There may be so many other possible solutions to the issue, but the one presented here worked for me.

The present issue is that , i have to delete an ASM disk from the ASM storage. The disk is already marked as ASM disk and when i try to unmark it for ASM disk, it throws an error. So i cant delete the OS disk without deleting it from the ASM disks.

Present issue occured in 11gR2 RAC on linux and we are using multi path ASM disks for the storage.

[root@rac01 disks]# /etc/init.d/oracleasm deletedisk DATAVOL1
Removing ASM disk “DATAVOL1”:                              [FAILED] 

 

So i verified the log file  /var/log/oracleasm and the content of the log file shows me the following error

Unable to clear disk “DATAVOL1”
Clearing disk header: oracleasm-write-label: Unable to open device “/dev/oracleasm/disks/DATAVOL1”: Device or resource busy
failed
Unable to clear disk “DATAVOL1”

I tried creating a diskgroup with this ASM disk and droped it and then tried to delete the ASM disk, but no use.

SQL> create diskgroup test external redundancy disk ‘/dev/oracleasm/disks/DATAVOL1’; 

Diskgroup created.

SQL> drop diskgroup test;
Diskgroup dropped.
SQL>

[root@rac01 disks]# /etc/init.d/oracleasm deletedisk DATAVOL1
Removing ASM disk “DATAVOL1”:                              [FAILED]

I verified with the following command if the disk is still marked as ASM disk or not. It is still marked as ASM disk.

[root@rac01 disks]# blkid|grep sd.*oracleasm|while read a b;do echo -n $a$b” scsi_id=”;(echo $a|tr -d [:digit:]|tr -d [:]|cut -d”/” -f3|xargs -i scsi_id -g -s /block/{})done | grep -i DATAVOL1;
/dev/sdc:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdf:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdi:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdl:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdo:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdp:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdu:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdx:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
[root@rac01 disks]# 

[root@rac01 disks]# multipath -ll

..

..

vpath0 (360050768019600fba80000000000036b) dm-11 IBM,2145
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=200][active]
\_ 2:0:7:0 sdo  8:224  [active][ready]
\_ 1:0:7:0 sdp  8:240  [active][ready]
\_ 2:0:6:0 sdu  65:64  [active][ready]
\_ 1:0:6:0 sdx  65:112 [active][ready]
\_ round-robin 0 [prio=40][enabled]
\_ 1:0:4:0 sdc  8:32   [active][ready]
\_ 1:0:5:0 sdf  8:80   [active][ready]
\_ 2:0:4:0 sdi  8:128  [active][ready]
\_ 2:0:5:0 sdl  8:176  [active][ready]
[root@rac01 disks]#

Then i tried clearing the disk with the OS command dd.

[root@rac01 disks]# dd if=/dev/zero of=/dev/dm-11 bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.05526 seconds, 1.9 MB/s
[root@rac01 disks]# /etc/init.d/oracleasm deletedisk DATAVOL1
Removing ASM disk “DATAVOL1”:                              [  OK  ]
[root@rac01 disks]# /etc/init.d/oracleasm listdisks
CRSVOL1
DATA1
FRADISK1
FRAVOL1
[root@rac01 disks]#

Now i am able to successfully delete the disk from ASM disk list.

On all other nodes in the RAC , do the following to sync

[root@rac02 disks]# /etc/init.d/oracleasm listdisks
CRSVOL1
DATA1
DATAVOL1
FRADISK1
FRAVOL1
[root@rac02 disks]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@rac02 disks]# /etc/init.d/oracleasm listdisks
CRSVOL1
DATA1
FRADISK1
FRAVOL1
[root@rac02 disks]#

Content from the log file /var/log/oracleasm now shows …

Disk “DATAVOL1” defines an unmarked device
Dropping disk: done
Reloading disk partitions: done
Cleaning any stale ASM disks…
Validating disk “CRSVOL1”
Validating disk “DATA1”
Validating disk “FRADISK1”
Validating disk “FRAVOL1” 

 

Hope it helps

SRI

Posted in 11gR2 RAC, ASM, Database Articles, Troubleshooting | Tagged: , , , , , , , , | 3 Comments »

How to identify the physical disk bound to the ASMLIB disk?

Posted by Srikrishna Murthy Annam on September 24, 2010

We will create ASM disks with oracleasm and we can list the ASM disks using the listdisks option.
While creating the ASM disk we will supply the physical disk path, as

[root@dhcppc1 rpms]# /etc/init.d/oracleasm createdisk ASK_DISK1 /dev/sdc1
Marking disk “ASK_DISK1” as an ASM disk:                   [  OK  ]
[root@dhcppc1 rpms]#

Once after creating the ASM disks, there is no direct command which actually lists the mapping between ASM disks and physical disks.

If you check the syntax,

[root@dhcppc1 rpms]# /etc/init.d/oracleasm -h
Usage: /etc/init.d/oracleasm {start|stop|restart|enable|disable|configure|createdisk|deletedisk|querydisk|listdisks|scandisks|status}
[root@dhcppc1 rpms]#

[root@dhcppc1 rpms]# /etc/init.d/oracleasm listdisks
CRSVOL1
DATAVOL1
FRAVOL1
[root@dhcppc1 rpms]# /etc/init.d/oracleasm querydisk
Usage: oracleasm-querydisk [-l <manager>] [-v] [-d|-p] <label>|<device> …
[root@dhcppc1 rpms]# /etc/init.d/oracleasm querydisk CRSVOL1
Disk “CRSVOL1” is a valid ASM disk
[root@dhcppc1 rpms]#

So there is no option from oracleasm executable to check how ASM_DISK1 is linked to /dev/sdc1.

We will try the SQL Commands ..

SQL> select name from v$asm_diskgroup;

NAME
——————————
CRS
FRA
RACDB_DATA

SQL> select name,path,label from v$asm_disk;

NAME            PATH                                LABEL
————— ———————————– —————
CRS_0000        /dev/oracleasm/disks/CRSVOL1
FRA_0000        /dev/oracleasm/disks/FRAVOL1
RACDB_DATA_0000 /dev/oracleasm/disks/DATAVOL1

SQL>

Then how do we find, Which Disks Are Handled by ASMLib Kernel Driver?

There are two ways to identify the physical disk bound to the ASMLIB disk

1.

[root@dhcppc1 rpms]# find /dev -type b -exec ‘/etc/init.d/oracleasm’ ‘querydisk’ ‘{}’ ‘;’ 2>/dev/null | grep “is marked an ASM disk”
Device “/dev/oracleasm/disks/CRSVOL1” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/oracleasm/disks/FRAVOL1” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/oracleasm/disks/DATAVOL1” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/dm-9” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/dm-11” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/dm-10” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdc” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdd” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sde” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdf” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdg” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdh” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdi” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdj” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdk” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdl” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdm” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdn” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdo” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdp” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdq” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdr” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sds” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdt” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdu” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdv” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdw” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdx” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdy” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdz” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/mapper/vpath2” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/mapper/vpath1” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/mapper/vpath0” is marked an ASM disk with the label “DATAVOL1”
[root@dhcppc1 rpms]#

2.

[root@dhcppc1 rpms]# blkid|grep sd.*oracleasm|while read a b;do echo -n $a$b” scsi_id=”;(echo $a|tr -d [:digit:]|tr -d [:]|cut -d”/” -f3|xargs -i scsi_id -g -s /block/{})done;
/dev/sdc:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdd:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sde:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdf:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdg:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdh:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdi:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdj:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdk:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdl:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdm:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdn:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdo:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdp:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdq:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdr:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sds:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdt:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdu:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdv:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdw:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdx:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdy:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
[root@dhcppc1 rpms]#

[root@dhcppc1 rpms]# multipath -ll
vpath2 (360050768019600fba80000000000036d) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=200][active]
\_ 1:0:6:2 sdk 8:160  [active][ready]
\_ 1:0:7:2 sdn 8:208  [active][ready]
\_ 2:0:6:2 sdw 65:96  [active][ready]
\_ 2:0:7:2 sdz 65:144 [active][ready]
\_ round-robin 0 [prio=40][enabled]
\_ 1:0:4:2 sde 8:64   [active][ready]
\_ 1:0:5:2 sdh 8:112  [active][ready]
\_ 2:0:4:2 sdq 65:0   [active][ready]
\_ 2:0:5:2 sdt 65:48  [active][ready]
vpath1 (360050768019600fba80000000000036c) dm-10 IBM,2145
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=200][active]
\_ 1:0:4:1 sdd 8:48   [active][ready]
\_ 1:0:5:1 sdg 8:96   [active][ready]
\_ 2:0:4:1 sdp 8:240  [active][ready]
\_ 2:0:5:1 sds 65:32  [active][ready]
\_ round-robin 0 [prio=40][enabled]
\_ 1:0:6:1 sdj 8:144  [active][ready]
\_ 1:0:7:1 sdm 8:192  [active][ready]
\_ 2:0:6:1 sdv 65:80  [active][ready]
\_ 2:0:7:1 sdy 65:128 [active][ready]
vpath0 (360050768019600fba80000000000036b) dm-9 IBM,2145
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=200][active]
\_ 1:0:6:0 sdi 8:128  [active][ready]
\_ 1:0:7:0 sdl 8:176  [active][ready]
\_ 2:0:6:0 sdu 65:64  [active][ready]
\_ 2:0:7:0 sdx 65:112 [active][ready]
\_ round-robin 0 [prio=40][enabled]
\_ 1:0:4:0 sdc 8:32   [active][ready]
\_ 1:0:5:0 sdf 8:80   [active][ready]
\_ 2:0:4:0 sdo 8:224  [active][ready]
\_ 2:0:5:0 sdr 65:16  [active][ready]
[root@dhcppc1 rpms]#

[root@dhcppc1 rpms]# ls -lrt /dev/mapper/vpath*
brw-rw—- 1 root disk 253, 11 Jul  9 12:49 /dev/mapper/vpath2
brw-rw—- 1 root disk 253, 10 Jul  9 12:49 /dev/mapper/vpath1
brw-rw—- 1 root disk 253,  9 Jul  9 12:49 /dev/mapper/vpath0
[root@dhcppc1 rpms]#

[root@dhcppc1 rpms]# find /dev/oracleasm/disks -type b | xargs -i sh -c “echo {}; dd if={} count=100 2>/dev/null | strings”
/dev/oracleasm/disks/CRSVOL1
ORCLDISKCRSVOL1
CRS_0000
CRS_0000
/dev/oracleasm/disks/FRAVOL1
ORCLDISKFRAVOL1
FRA_0000
FRA_0000
/dev/oracleasm/disks/DATAVOL1
ORCLDISKDATAVOL1
RACDB_DATA_0000
RACDB_DATA
RACDB_DATA_0000
[root@dhcppc1 rpms]#

Hope it helps

SRI

How to identify the physical disk bound to the ASMLIB disk?

We will create ASM disks with oracleasm and we can list the ASM disks using the listdisks option.
While creating the ASM disk we will supply the physical disk path, as

[root@dhcppc1 rpms]# /etc/init.d/oracleasm createdisk ASK_DISK1 /dev/sdc1
Marking disk “ASK_DISK1” as an ASM disk:                   [  OK  ]
[root@dhcppc1 rpms]#

Once after creating the ASM disks, there is no direct command which actually lists the mapping between ASM disks and physical disks.

If you check the syntax,
[root@dhcppc1 rpms]# /etc/init.d/oracleasm -h
Usage: /etc/init.d/oracleasm {start|stop|restart|enable|disable|configure|createdisk|deletedisk|querydisk|listdisks|scandisks|status}
[root@dhcppc1 rpms]#

[root@dhcppc1 rpms]# /etc/init.d/oracleasm listdisks
CRSVOL1
DATAVOL1
FRAVOL1
[root@dhcppc1 rpms]# /etc/init.d/oracleasm querydisk
Usage: oracleasm-querydisk [-l <manager>] [-v] [-d|-p] <label>|<device> …
[root@dhcppc1 rpms]# /etc/init.d/oracleasm querydisk CRSVOL1
Disk “CRSVOL1” is a valid ASM disk
[root@dhcppc1 rpms]#

So there is no option from oracleasm executable to check how ASM_DISK1 is linked to /dev/sdc1.

We will try the SQL Commands ..

SQL> select name from v$asm_diskgroup;

NAME
——————————
CRS
FRA
RACDB_DATA

SQL> select name,path,label from v$asm_disk;

NAME            PATH                                LABEL
————— ———————————– —————
CRS_0000        /dev/oracleasm/disks/CRSVOL1
FRA_0000        /dev/oracleasm/disks/FRAVOL1
RACDB_DATA_0000 /dev/oracleasm/disks/DATAVOL1

SQL>

Then how do we find, Which Disks Are Handled by ASMLib Kernel Driver?

There are two ways to identify the physical disk bound to the ASMLIB disk

[root@dhcppc1 rpms]# find /dev -type b -exec ‘/etc/init.d/oracleasm’ ‘querydisk’ ‘{}’ ‘;’ 2>/dev/null | grep “is marked an ASM disk”
Device “/dev/oracleasm/disks/CRSVOL1” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/oracleasm/disks/FRAVOL1” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/oracleasm/disks/DATAVOL1” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/dm-9” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/dm-11” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/dm-10” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdc” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdd” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sde” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdf” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdg” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdh” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdi” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdj” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdk” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdl” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdm” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdn” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdo” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdp” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdq” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdr” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sds” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdt” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdu” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdv” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdw” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/sdx” is marked an ASM disk with the label “DATAVOL1”
Device “/dev/sdy” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/sdz” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/mapper/vpath2” is marked an ASM disk with the label “CRSVOL1”
Device “/dev/mapper/vpath1” is marked an ASM disk with the label “FRAVOL1”
Device “/dev/mapper/vpath0” is marked an ASM disk with the label “DATAVOL1″
[root@dhcppc1 rpms]#

[root@dhcppc1 rpms]# blkid|grep sd.*oracleasm|while read a b;do echo -n $a$b” scsi_id=”;(echo $a|tr -d [:digit:]|tr -d [:]|cut -d”/” -f3|xargs -i scsi_id -g -s /block/{})done;
/dev/sdc:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdd:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sde:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdf:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdg:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdh:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdi:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdj:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdk:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdl:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdm:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdn:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdo:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdp:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdq:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdr:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sds:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdt:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdu:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdv:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
/dev/sdw:LABEL=”CRSVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036d
/dev/sdx:LABEL=”DATAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036b
/dev/sdy:LABEL=”FRAVOL1″ TYPE=”oracleasm” scsi_id=360050768019600fba80000000000036c
[root@dhcppc1 rpms]#

[root@dhcppc1 rpms]# multipath -ll
vpath2 (360050768019600fba80000000000036d) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=200][active]
\_ 1:0:6:2 sdk 8:160  [active][ready]
\_ 1:0:7:2 sdn 8:208  [active][ready]
\_ 2:0:6:2 sdw 65:96  [active][ready]
\_ 2:0:7:2 sdz 65:144 [active][ready]
\_ round-robin 0 [prio=40][enabled]
\_ 1:0:4:2 sde 8:64   [active][ready]
\_ 1:0:5:2 sdh 8:112  [active][ready]
\_ 2:0:4:2 sdq 65:0   [active][ready]
\_ 2:0:5:2 sdt 65:48  [active][ready]
vpath1 (360050768019600fba80000000000036c) dm-10 IBM,2145
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=200][active]
\_ 1:0:4:1 sdd 8:48   [active][ready]
\_ 1:0:5:1 sdg 8:96   [active][ready]
\_ 2:0:4:1 sdp 8:240  [active][ready]
\_ 2:0:5:1 sds 65:32  [active][ready]
\_ round-robin 0 [prio=40][enabled]
\_ 1:0:6:1 sdj 8:144  [active][ready]
\_ 1:0:7:1 sdm 8:192  [active][ready]
\_ 2:0:6:1 sdv 65:80  [active][ready]
\_ 2:0:7:1 sdy 65:128 [active][ready]
vpath0 (360050768019600fba80000000000036b) dm-9 IBM,2145
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=200][active]
\_ 1:0:6:0 sdi 8:128  [active][ready]
\_ 1:0:7:0 sdl 8:176  [active][ready]
\_ 2:0:6:0 sdu 65:64  [active][ready]
\_ 2:0:7:0 sdx 65:112 [active][ready]
\_ round-robin 0 [prio=40][enabled]
\_ 1:0:4:0 sdc 8:32   [active][ready]
\_ 1:0:5:0 sdf 8:80   [active][ready]
\_ 2:0:4:0 sdo 8:224  [active][ready]
\_ 2:0:5:0 sdr 65:16  [active][ready]
[root@dhcppc1 rpms]#

[root@orrcdbtsl01 ~]# ls -lrt /dev/mapper/vpath*
brw-rw—- 1 root disk 253, 11 Jul  9 12:49 /dev/mapper/vpath2
brw-rw—- 1 root disk 253, 10 Jul  9 12:49 /dev/mapper/vpath1
brw-rw—- 1 root disk 253,  9 Jul  9 12:49 /dev/mapper/vpath0
[root@orrcdbtsl01 ~]#

[root@orrcdbtsl01 ~]# find /dev/oracleasm/disks -type b | xargs -i sh -c “echo {}; dd if={} count=100 2>/dev/null | strings”
/dev/oracleasm/disks/CRSVOL1
ORCLDISKCRSVOL1
CRS_0000
CRS_0000
/dev/oracleasm/disks/FRAVOL1
ORCLDISKFRAVOL1
FRA_0000
FRA_0000
/dev/oracleasm/disks/DATAVOL1
ORCLDISKDATAVOL1
RACDB_DATA_0000
RACDB_DATA
RACDB_DATA_0000
[root@orrcdbtsl01 ~]#

Posted in 11gR2 RAC, ASM | Tagged: , , , , , , , , | Leave a Comment »

Exadata ?

Posted by Srikrishna Murthy Annam on September 23, 2010

What is Exadata ? Why is it used ? who will use this ?

Is it replacement to Oracle RAC Server?

Do we just need to power on to use the server ?

Why is it so popular ? Is it really popular or getting popular ?

Is it best for DataWarehouse or OLTP or Mixture ?

Well, Just started working on Exadata …..

Will post many things sooooon …..

🙂

Posted in Exadata | Tagged: | 1 Comment »

RAC Nodes interconnect Checks

Posted by Srikrishna Murthy Annam on September 20, 2010

[grid@rac02 cssd]$ olsnodes -s
rac01     Active
rac02     Active
[grid@rac02 cssd]$ oifcfg getif
eth2  139.22.38.0  global  public
eth3 192.168.50.0  global  cluster_interconnect
[grid@rac02 cssd]$

[root@rac01 ~]# ifconfig eth3
eth3      Link encap:Ethernet  HWaddr 00:13:72:60:69:84
inet addr:192.168.50.1  Bcast:192.168.50.255  Mask:255.255.255.0
inet6 addr: fe80::213:72ff:fe60:6984/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:38565202 errors:0 dropped:0 overruns:0 frame:0
TX packets:30148892 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:25947483525 (24.1 GiB)  TX bytes:13988725497 (13.0 GiB)
Base address:0xbcc0 Memory:fe0e0000-fe100000

[root@rac01 ~]#

[root@rac02 ~]# ifconfig eth3
eth3      Link encap:Ethernet  HWaddr 00:13:72:5E:7C:C2
inet addr:192.168.50.2  Bcast:192.168.50.255  Mask:255.255.255.0
inet6 addr: fe80::213:72ff:fe5e:7cc2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:30513289 errors:0 dropped:0 overruns:0 frame:0
TX packets:39265110 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:14191064255 (13.2 GiB)  TX bytes:26225686434 (24.4 GiB)
Base address:0xbcc0 Memory:fe0e0000-fe100000

[root@rac02 ~]#

[grid@rac02 grid]$ ping 192.168.50.1
PING 192.168.50.1 (192.168.50.1) 56(84) bytes of data.
64 bytes from 192.168.50.1: icmp_seq=1 ttl=64 time=0.192 ms
64 bytes from 192.168.50.1: icmp_seq=2 ttl=64 time=0.195 ms
64 bytes from 192.168.50.1: icmp_seq=3 ttl=64 time=0.199 ms

— 192.168.50.1 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.192/0.195/0.199/0.011 ms
[grid@rac02 grid]$ ping rac01-priv.testing.com
PING rac01-priv.testing.com (192.168.50.1) 56(84) bytes of data.
64 bytes from rac01-priv.testing.com (192.168.50.1): icmp_seq=1 ttl=64 time=0.240 ms
64 bytes from rac01-priv.testing.com (192.168.50.1): icmp_seq=2 ttl=64 time=0.206 ms
64 bytes from rac01-priv.testing.com (192.168.50.1): icmp_seq=3 ttl=64 time=0.162 ms
64 bytes from rac01-priv.testing.com (192.168.50.1): icmp_seq=4 ttl=64 time=0.180 ms

— rac01-priv.testing.com ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.162/0.197/0.240/0.029 ms
[grid@rac02 grid]$

[grid@rac01 grid]$ ping 192.168.50.2
PING 192.168.50.2 (192.168.50.2) 56(84) bytes of data.
64 bytes from 192.168.50.2: icmp_seq=1 ttl=64 time=0.171 ms
64 bytes from 192.168.50.2: icmp_seq=2 ttl=64 time=0.175 ms
64 bytes from 192.168.50.2: icmp_seq=3 ttl=64 time=0.165 ms
64 bytes from 192.168.50.2: icmp_seq=4 ttl=64 time=0.188 ms

— 192.168.50.2 ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.165/0.174/0.188/0.018 ms
[grid@rac01 grid]$ ping rac02-priv.testing.com
PING rac02-priv.testing.com (192.168.50.2) 56(84) bytes of data.
64 bytes from rac02-priv.testing.com (192.168.50.2): icmp_seq=1 ttl=64 time=0.160 ms
64 bytes from rac02-priv.testing.com (192.168.50.2): icmp_seq=2 ttl=64 time=0.223 ms
64 bytes from rac02-priv.testing.com (192.168.50.2): icmp_seq=3 ttl=64 time=0.198 ms
64 bytes from rac02-priv.testing.com (192.168.50.2): icmp_seq=4 ttl=64 time=0.172 ms

— rac02-priv.testing.com ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 2998ms
rtt min/avg/max/mdev = 0.160/0.188/0.223/0.026 ms
[grid@rac01 grid]$

Hope it helps

SRI

Posted in 11gR2 RAC | Tagged: , , , , , | Leave a Comment »

How to add SCAN LISTENER in 11gR2

Posted by Srikrishna Murthy Annam on September 3, 2010

My 11gR2 grid infrastructure completed with only one scan listener. The following log shows that ….

[grid@rac01 grid]$ srvctl config scan
SCAN name: rac-scan, Network: 1/133.22.38.0/255.255.255.0/eth2
SCAN VIP name: scan1, IP: /rac-scan.example.com/133.22.67.192
[grid@rac01 grid]$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
[grid@rac01 grid]$

[grid@rac01 grid]$ ping rac-scan.example.com
PING rac-scan.example.com (133.22.67.193) 56(84) bytes of data.
From rac01.seo.int (133.22.38.50) icmp_seq=2 Destination Host Unreachable
From rac01.seo.int (133.22.38.50) icmp_seq=3 Destination Host Unreachable
From rac01.seo.int (133.22.38.50) icmp_seq=4 Destination Host Unreachable

— rac-scan.example.com ping statistics —
6 packets transmitted, 0 received, +3 errors, 100% packet loss, time 5000ms
, pipe 3
[grid@rac01 grid]$ ping rac-scan
PING rac-scan.example.com (133.22.67.194) 56(84) bytes of data.
From rac01.seo.int (133.22.38.50) icmp_seq=2 Destination Host Unreachable
From rac01.seo.int (133.22.38.50) icmp_seq=3 Destination Host Unreachable
From rac01.seo.int (133.22.38.50) icmp_seq=4 Destination Host Unreachable
From rac01.seo.int (133.22.38.50) icmp_seq=6 Destination Host Unreachable
From rac01.seo.int (133.22.38.50) icmp_seq=7 Destination Host Unreachable
From rac01.seo.int (133.22.38.50) icmp_seq=8 Destination Host Unreachable

— rac-scan.example.com ping statistics —
11 packets transmitted, 0 received, +6 errors, 100% packet loss, time 10000ms
, pipe 3
[grid@rac01 grid]$ ping rac-scan
PING rac-scan.example.com (133.22.67.192) 56(84) bytes of data.
64 bytes from rac-scan.example.com (133.22.67.192): icmp_seq=1 ttl=64 time=0.039 ms
64 bytes from rac-scan.example.com (133.22.67.192): icmp_seq=2 ttl=64 time=0.054 ms
64 bytes from rac-scan.example.com (133.22.67.192): icmp_seq=3 ttl=64 time=0.049 ms
64 bytes from rac-scan.example.com (133.22.67.192): icmp_seq=4 ttl=64 time=0.063 ms

— rac-scan.example.com ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.039/0.051/0.063/0.010 ms
[grid@rac01 grid]$

[grid@rac01 grid]$ host rac-scan
rac-scan.example.com has address 133.22.67.194
rac-scan.example.com has address 133.22.67.193
rac-scan.example.com has address 133.22.67.192
[grid@rac01 grid]$ nslookup rac-scan
Server:         133.22.5.53
Address:        133.22.5.53#53

Name:   rac-scan.example.com
Address: 133.22.67.194
Name:   rac-scan.example.com
Address: 133.22.67.192
Name:   rac-scan.example.com
Address: 133.22.67.193

[grid@rac01 grid]$

[root@rac01 network-scripts]# host rac-scan
rac-scan.example.com has address 133.22.67.194
rac-scan.example.com has address 133.22.67.193
rac-scan.example.com has address 133.22.67.192
[root@rac01 network-scripts]#

rac-scan.example.com IN A 133.22.67.194
IN A 133.22.67.193
IN A 133.22.67.192

[grid@rac02 grid]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on rac01
ora.FRA.dg                                         ONLINE     ONLINE on rac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on rac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on rac01
ora.RACDB_DATA.dg                                  ONLINE     ONLINE on rac01
ora.asm                                            ONLINE     ONLINE on rac01
ora.eons                                           ONLINE     ONLINE on rac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on rac01
ora.oc4j                                           OFFLINE    OFFLINE
ora.ons                                            ONLINE     ONLINE on rac01


ora.rac02.vip                                      ONLINE     ONLINE on rac02
ora.rac10g.db                                      ONLINE     ONLINE on rac02
ora.rac10g.rac10g1.inst                            ONLINE     ONLINE on rac01
ora.rac10g.rac10g2.inst                            ONLINE     OFFLINE
ora.racpoc.askm1.svc                               ONLINE     ONLINE on rac01
ora.racpoc.db                                      ONLINE     ONLINE on rac01
ora.registry.acfs                                  ONLINE     ONLINE on rac01
ora.scan1.vip                                      ONLINE     ONLINE on rac01
ora.testdb.db                                      ONLINE     ONLINE on rac01

tnsnames.ora file :

TESTDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan.example.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = testdb.example.com)
)
)

The above session data clearly shows there is only one scan listener configured. When we use scan name in tnsnames.ora, the client connections will be established to database only when the scan name resolves to valid IP address.
But with the above configurations, the client connections some times may fail when the SCAN is resolving to 133.22.67.193 or 133.22.67.194. It establishes connection only when the SCAN resolves to 133.22.67.192.

SQL> conn test/test@testdb ==> Connection failed. rac-scan resolves either to 133.22.67.193 or 133.22.67.194
ERROR:
ORA-12543: TNS:destination host unreachable

SQL>conn test/test@testdb ==> Connection established.rac-scan resolves to 133.22.67.192.
connected
SQL>

Now to avoid this , we need to create two more scan listeners.

[grid@rac01 grid]$ srvctl stop scan_listener
[grid@rac01 grid]$ srvctl stop scan
[grid@rac01 grid]$ srvctl config scan
SCAN name: rac-scan, Network: 1/133.22.38.0/255.255.255.0/eth2
SCAN VIP name: scan1, IP: /rac-scan.example.com/133.22.67.192
[grid@rac01 grid]$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
[grid@rac01 grid]$ which srvctl
/u01/home/11.2.0/grid/bin/srvctl
[grid@rac01 grid]$
[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl modify scan -h

Modifies the SCAN name.

Usage: srvctl modify scan -n <scan_name>
-n <scan_name>           Domain name qualified SCAN name
-h                       Print usage
[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl modify scan -n rac-scan
[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl config scan
SCAN name: rac-scan, Network: 1/133.22.38.0/255.255.255.0/eth2
SCAN VIP name: scan1, IP: /rac-scan.example.com/133.22.67.193
SCAN VIP name: scan2, IP: /rac-scan.example.com/133.22.67.194
SCAN VIP name: scan3, IP: /rac-scan.example.com/133.22.67.192
[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl modify scan_listener -u

[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521
[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl start scan_listener
[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl config scan
SCAN name: rac-scan, Network: 1/133.22.38.0/255.255.255.0/eth2
SCAN VIP name: scan1, IP: /rac-scan.example.com/133.22.67.193
SCAN VIP name: scan2, IP: /rac-scan.example.com/133.22.67.194
SCAN VIP name: scan3, IP: /rac-scan.example.com/133.22.67.192
[root@rac01 network-scripts]#

[grid@rac02 grid]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on rac01
ora.FRA.dg                                         ONLINE     ONLINE on rac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on rac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on rac01
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on rac02
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on rac02

ora.RACDB_DATA.dg                                  ONLINE     ONLINE on rac01
ora.asm                                            ONLINE     ONLINE on rac01
ora.eons                                           ONLINE     ONLINE on rac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on rac01
ora.oc4j                                           OFFLINE    OFFLINE
ora.ons                                            ONLINE     ONLINE on rac01


ora.rac02.vip                                      ONLINE     ONLINE on rac02
ora.rac10g.db                                      ONLINE     ONLINE on rac02
ora.rac10g.rac10g1.inst                            ONLINE     ONLINE on rac01
ora.rac10g.rac10g2.inst                            ONLINE     OFFLINE
ora.racpoc.askm1.svc                               ONLINE     ONLINE on rac01
ora.racpoc.db                                      ONLINE     ONLINE on rac01
ora.registry.acfs                                  ONLINE     ONLINE on rac01
ora.scan1.vip                                      ONLINE     ONLINE on rac01
ora.scan2.vip                                      ONLINE     ONLINE on rac02
ora.scan3.vip                                      ONLINE     ONLINE on rac02

ora.testdb.db                                      ONLINE     ONLINE on rac01
[grid@rac02 grid]$

Status :

[grid@rac01 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node rac01
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node rac02
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node rac02
[grid@rac01 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node rac01
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node rac02
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node rac02
[grid@rac01 ~]$

Hope it helps
SRI

My 11gR2 grid infrastructure completed with only one scan listener. The following log shows that ….

[grid@rac01 grid]$ srvctl config scan

SCAN name: rac-scan, Network: 1/133.22.38.0/255.255.255.0/eth2

SCAN VIP name: scan1, IP: /rac-scan.example.com/133.22.67.192

[grid@rac01 grid]$ srvctl config scan_listener

SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

[grid@rac01 grid]$

[grid@rac01 grid]$ ping rac-scan.example.com

PING rac-scan.example.com (133.22.67.193) 56(84) bytes of data.

From rac01.seo.int (133.22.38.50) icmp_seq=2 Destination Host Unreachable

From rac01.seo.int (133.22.38.50) icmp_seq=3 Destination Host Unreachable

From rac01.seo.int (133.22.38.50) icmp_seq=4 Destination Host Unreachable

— rac-scan.example.com ping statistics —

6 packets transmitted, 0 received, +3 errors, 100% packet loss, time 5000ms

, pipe 3

[grid@rac01 grid]$ ping rac-scan

PING rac-scan.example.com (133.22.67.194) 56(84) bytes of data.

From rac01.seo.int (133.22.38.50) icmp_seq=2 Destination Host Unreachable

From rac01.seo.int (133.22.38.50) icmp_seq=3 Destination Host Unreachable

From rac01.seo.int (133.22.38.50) icmp_seq=4 Destination Host Unreachable

From rac01.seo.int (133.22.38.50) icmp_seq=6 Destination Host Unreachable

From rac01.seo.int (133.22.38.50) icmp_seq=7 Destination Host Unreachable

From rac01.seo.int (133.22.38.50) icmp_seq=8 Destination Host Unreachable

— rac-scan.example.com ping statistics —

11 packets transmitted, 0 received, +6 errors, 100% packet loss, time 10000ms

, pipe 3

[grid@rac01 grid]$ ping rac-scan

PING rac-scan.example.com (133.22.67.192) 56(84) bytes of data.

64 bytes from rac-scan.example.com (133.22.67.192): icmp_seq=1 ttl=64 time=0.039 ms

64 bytes from rac-scan.example.com (133.22.67.192): icmp_seq=2 ttl=64 time=0.054 ms

64 bytes from rac-scan.example.com (133.22.67.192): icmp_seq=3 ttl=64 time=0.049 ms

64 bytes from rac-scan.example.com (133.22.67.192): icmp_seq=4 ttl=64 time=0.063 ms

— rac-scan.example.com ping statistics —

4 packets transmitted, 4 received, 0% packet loss, time 3001ms

rtt min/avg/max/mdev = 0.039/0.051/0.063/0.010 ms

[grid@rac01 grid]$

[grid@rac01 grid]$ host rac-scan

rac-scan.example.com has address 133.22.67.194

rac-scan.example.com has address 133.22.67.193

rac-scan.example.com has address 133.22.67.192

[grid@rac01 grid]$ nslookup rac-scan

Server:         133.22.5.53

Address:        133.22.5.53#53

Name:   rac-scan.example.com

Address: 133.22.67.194

Name:   rac-scan.example.com

Address: 133.22.67.192

Name:   rac-scan.example.com

Address: 133.22.67.193

[grid@rac01 grid]$

[root@rac01 network-scripts]# host rac-scan

rac-scan.example.com has address 133.22.67.194

rac-scan.example.com has address 133.22.67.193

rac-scan.example.com has address 133.22.67.192

[root@rac01 network-scripts]#

rac-scan.example.com IN A 133.22.67.194

IN A 133.22.67.193

IN A 133.22.67.192

[grid@rac02 grid]$ Check_CRS.txt

HA Resource                                        Target     State

———–                                        ——     —–

ora.CRS.dg                                         ONLINE     ONLINE on rac01

ora.FRA.dg                                         ONLINE     ONLINE on rac01

ora.LISTENER.lsnr                            ONLINE     ONLINE on rac01

ora.LISTENER_SCAN1.lsnr          ONLINE     ONLINE on rac01

ora.RACDB_DATA.dg                      ONLINE     ONLINE on rac01

ora.asm                                                ONLINE     ONLINE on rac01

ora.eons                                               ONLINE     ONLINE on rac01

ora.gsd                                                 OFFLINE    OFFLINE

ora.net1.network                                  ONLINE     ONLINE on rac01

ora.oc4j                                                OFFLINE    OFFLINE

ora.ons                                                 ONLINE     ONLINE on rac01

ora.rac02.vip                                       ONLINE     ONLINE on rac02

ora.rac10g.db                                      ONLINE     ONLINE on rac02

ora.rac10g.rac10g1.inst                      ONLINE     ONLINE on rac01

ora.rac10g.rac10g2.inst                            ONLINE     OFFLINE

ora.racpoc.askm1.svc                               ONLINE     ONLINE on rac01

ora.racpoc.db                                      ONLINE     ONLINE on rac01

ora.registry.acfs                                  ONLINE     ONLINE on rac01

ora.scan1.vip                                      ONLINE     ONLINE on rac01

ora.testdb.db                                      ONLINE     ONLINE on rac01

TESTDB =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan.example.com)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = testdb.example.com)

)

)

The above session data clearly shows there is only one scan listener configured. When we use scan name in tnsnames.ora, the client connections will be established to database only when the scan name resolves to valid IP address.

But with the above configurations, the client connections some times may fail when the SCAN is resolving to 133.22.67.193 or 133.22.67.194. It establishes connection only when the SCAN resolves to 133.22.67.192.

SQL> conn test/test@testdb ==> Connection failed. rac-scan resolves either to 133.22.67.193 or 133.22.67.194

ERROR:

ORA-12543: TNS:destination host unreachable

SQL>conn test/test@testdb ==> Connection established.rac-scan resolves to 133.22.67.192.

connected

SQL>

Now to avoid this , we need to create two more scan listeners.

[grid@rac01 grid]$ srvctl stop scan_listener

[grid@rac01 grid]$ srvctl stop scan

[grid@rac01 grid]$ srvctl config scan

SCAN name: rac-scan, Network: 1/133.22.38.0/255.255.255.0/eth2

SCAN VIP name: scan1, IP: /rac-scan.example.com/133.22.67.192

[grid@rac01 grid]$ srvctl config scan_listener

SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

[grid@rac01 grid]$ which srvctl

/u01/home/11.2.0/grid/bin/srvctl

[grid@rac01 grid]$

[grid@rac01 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node rac01
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node rac02
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node rac02
[grid@rac01 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node rac01
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node rac02
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node rac02
[grid@rac01 ~]$

[grid@rac01 ~]$ ps -ef | grep -i tns
grid      9052     1  0 Aug09 ?        00:00:01 /o001/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid      9098     1  0 Aug09 ?        00:00:01 /o001/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid     21388 20424  0 05:06 pts/1    00:00:00 grep -i tns
[grid@rac01 ~]$

[grid@rac02 ~]$ ps -ef | grep -i tns
grid      1323  1292  0 05:07 pts/0    00:00:00 grep -i tns
grid      9110     1  0 Aug09 ?        00:00:02 /o001/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid     17278     1  0 Aug09 ?        00:00:01 /o001/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
grid     17285     1  0 Aug09 ?        00:00:01 /o001/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
[grid@rac02 ~]$

[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl modify scan -h

Modifies the SCAN name.

Usage: srvctl modify scan -n <scan_name>

-n <scan_name>           Domain name qualified SCAN name

-h                       Print usage

[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl modify scan -n rac-scan

[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl config scan

SCAN name: rac-scan, Network: 1/133.22.38.0/255.255.255.0/eth2

SCAN VIP name: scan1, IP: /rac-scan.example.com/133.22.67.193

SCAN VIP name: scan2, IP: /rac-scan.example.com/133.22.67.194

SCAN VIP name: scan3, IP: /rac-scan.example.com/133.22.67.192

[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl modify scan_listener -u

[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl config scan_listener

SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521

SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl start scan_listener

[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl config scan

SCAN name: rac-scan, Network: 1/133.22.38.0/255.255.255.0/eth2

SCAN VIP name: scan1, IP: /rac-scan.example.com/133.22.67.193

SCAN VIP name: scan2, IP: /rac-scan.example.com/133.22.67.194

SCAN VIP name: scan3, IP: /rac-scan.example.com/133.22.67.192

[root@rac01 network-scripts]#

[grid@rac02 grid]$ Check_CRS.txt

HA Resource                                        Target     State

———–                                        ——     —–

ora.CRS.dg                                         ONLINE     ONLINE on rac01

ora.FRA.dg                                         ONLINE     ONLINE on rac01

ora.LISTENER.lsnr                                  ONLINE     ONLINE on rac01

ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on rac01

ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on rac02

ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on rac02

ora.RACDB_DATA.dg                                  ONLINE     ONLINE on rac01

ora.asm                                            ONLINE     ONLINE on rac01

ora.eons                                           ONLINE     ONLINE on rac01

ora.gsd                                            OFFLINE    OFFLINE

ora.net1.network                                   ONLINE     ONLINE on rac01

ora.oc4j                                           OFFLINE    OFFLINE

ora.ons                                            ONLINE     ONLINE on rac01

ora.rac02.vip                                      ONLINE     ONLINE on rac02

ora.rac10g.db                                      ONLINE     ONLINE on rac02

ora.rac10g.rac10g1.inst                            ONLINE     ONLINE on rac01

ora.rac10g.rac10g2.inst                            ONLINE     OFFLINE

ora.racpoc.askm1.svc                               ONLINE     ONLINE on rac01

ora.racpoc.db                                      ONLINE     ONLINE on rac01

ora.registry.acfs                                  ONLINE     ONLINE on rac01

ora.scan1.vip                                      ONLINE     ONLINE on rac01

ora.scan2.vip                                      ONLINE     ONLINE on rac02

ora.scan3.vip                                      ONLINE     ONLINE on rac02

ora.testdb.db                                      ONLINE     ONLINE on rac01

[grid@rac02 grid]$

Hope it helps

SRI

Posted in 11gR2 RAC | Tagged: , | 27 Comments »

SCAN in 11gR2 Grid InfraStructure

Posted by Srikrishna Murthy Annam on September 2, 2010

11gR2 grid infrastructure uses SCAN, which is a new concept we use for all client connections. SCAN is single client access name and it is a new feature in 11g Release 2 that provides a single name for clients to access an Oracle Database running in a cluster. The benefit is clients using SCAN do not need to change if you add or remove nodes in the cluster.

Prior to 11gR2 , the tnsnames.ora entries are

TESTDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac01-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac02-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = testdb.example.com)
)
)

In 11gR2 , the tnsnames.ora entries look like ….

TESTDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan.example.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = testdb.example.com)
)
)

The only difference is that , we specify the scan name instead of all VIPs in the tnsnames.ora file.

So what is the benifit with it?

Suppose if you are adding any node to the cluster, you need to specify corresponding VIP entry for that node in the tnsnames.ora prior to 11gR2. But with SCAN name, we dont need to do any modifications to the tnsnames.ora file. SCAN name will resolve and load balance the client connections to all the nodes in the RAC.

SCAN configuration in the cluster :

The SCAN is configured during the installation of the grid infrastructure that is distributed with Oracle Database 11g Release2. So in order to configure SCAN, you need to create a single name that resolves to 3 IP addresses using a round robin algorithm. The IP addresses must be on the same subnet as your public network in the cluster.
rac-scan.example.com IN A 133.22.67.194
IN A 133.22.67.193
IN A 133.22.67.192

SCAN Details

[grid@rac01 ~]$ host rac-scan
rac-scan.example.com has address 133.22.67.192
rac-scan.example.com has address 133.22.67.193
rac-scan.example.com has address 133.22.67.194
[grid@rac01 ~]$ nslookup rac-scan
Server:         133.22.5.53
Address:        133.22.5.53#53

Name:   rac-scan.example.com
Address: 133.22.67.194
Name:   rac-scan.example.com
Address: 133.22.67.193
Name:   rac-scan.example.com
Address: 133.22.67.192

[grid@rac01 ~]$ ping rac-scan
PING rac-scan.example.com (133.22.67.193) 56(84) bytes of data.
64 bytes from rac-scan.example.com (133.22.67.193): icmp_seq=1 ttl=64 time=0.187 ms
64 bytes from rac-scan.example.com (133.22.67.193): icmp_seq=2 ttl=64 time=0.144 ms
64 bytes from rac-scan.example.com (133.22.67.193): icmp_seq=3 ttl=64 time=0.152 ms

— rac-scan.example.com ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.144/0.161/0.187/0.018 ms
[grid@rac01 ~]$ ping rac-scan
PING rac-scan.example.com (133.22.67.192) 56(84) bytes of data.
64 bytes from rac-scan.example.com (133.22.67.192): icmp_seq=1 ttl=64 time=0.042 ms
64 bytes from rac-scan.example.com (133.22.67.192): icmp_seq=2 ttl=64 time=0.046 ms
64 bytes from rac-scan.example.com (133.22.67.192): icmp_seq=3 ttl=64 time=0.051 ms

— rac-scan.example.com ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.042/0.046/0.051/0.006 ms
[grid@rac01 ~]$ ping rac-scan
PING rac-scan.example.com (133.22.67.194) 56(84) bytes of data.
64 bytes from rac-scan.example.com (133.22.67.194): icmp_seq=2 ttl=64 time=0.144 ms
64 bytes from rac-scan.example.com (133.22.67.194): icmp_seq=3 ttl=64 time=0.164 ms

— rac-scan.example.com ping statistics —
3 packets transmitted, 2 received, 33% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.144/0.154/0.164/0.010 ms
[grid@rac01 ~]$

[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521
[root@rac01 network-scripts]# /u01/home/11.2.0/grid/bin/srvctl config scan
SCAN name: rac-scan, Network: 1/133.22.38.0/255.255.255.0/eth2
SCAN VIP name: scan1, IP: /rac-scan.example.com/133.22.67.192
SCAN VIP name: scan2, IP: /rac-scan.example.com/133.22.67.194
SCAN VIP name: scan3, IP: /rac-scan.example.com/133.22.67.193
[root@rac01 network-scripts]#

[grid@rac01 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node rac01
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node rac02
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node rac02
[grid@rac01 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node rac01
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node rac02
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node rac02
[grid@rac01 ~]$

[grid@rac01 ~]$ ps -ef | grep -i tns
grid      9052     1  0 Aug09 ?        00:00:01 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid      9098     1  0 Aug09 ?        00:00:01 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid     21388 20424  0 05:06 pts/1    00:00:00 grep -i tns
[grid@rac01 ~]$

[grid@rac02 ~]$ ps -ef | grep -i tns
grid      1323  1292  0 05:07 pts/0    00:00:00 grep -i tns
grid      9110     1  0 Aug09 ?        00:00:02 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid     17278     1  0 Aug09 ?        00:00:01 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
grid     17285     1  0 Aug09 ?        00:00:01 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
[grid@rac02 ~]$

[grid@rac02 grid]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on rac01
ora.FRA.dg                                         ONLINE     ONLINE on rac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on rac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on rac01
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on rac02
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on rac02

ora.RACDB_DATA.dg                                  ONLINE     ONLINE on rac01
ora.asm                                            ONLINE     ONLINE on rac01
ora.eons                                           ONLINE     ONLINE on rac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on rac01
ora.oc4j                                           OFFLINE    OFFLINE
ora.ons                                            ONLINE     ONLINE on rac01
ora.rac01.ASM1.asm                                  ONLINE     ONLINE on rac01
ora.rac01.LISTENER_ORRCDBTSL01.lsnr                 ONLINE     ONLINE on rac01
ora.rac01.gsd                                      OFFLINE    OFFLINE
ora.rac01.ons                                       ONLINE     ONLINE on rac01
ora.rac01.vip                                      ONLINE     ONLINE on rac01
ora.rac02.ASM2.asm                                 ONLINE     ONLINE on rac02
ora.rac02.LISTENER_ORRCDBTSL02.lsnr                ONLINE     ONLINE on rac02
ora.rac02.gsd                                       OFFLINE    OFFLINE
ora.rac02.ons                                       ONLINE     ONLINE on rac02
ora.rac02.vip                                       ONLINE     ONLINE on rac02
ora.registry.acfs                                  ONLINE     ONLINE on rac01
ora.scan1.vip                                      ONLINE     ONLINE on rac01
ora.scan2.vip                                      ONLINE     ONLINE on rac02
ora.scan3.vip                                      ONLINE     ONLINE on rac02

ora.testdb.db                                      ONLINE     ONLINE on rac01
[grid@rac02 grid]$

Hope it helps
SRI

Posted in 11g New Features, 11gR2 RAC | Tagged: , , , , , | Leave a Comment »

 
%d bloggers like this: