Upgrading & Patching Oracle RAC easy Step-By-Step Guide (2020)

Upgrading and Patching Oracle RAC Step-By-Step Guide

In this practice,” Upgrading and Patching Oracle RAC” you will apply patch 20108098 to the Oracle RAC database homes on all three
nodes. This patch fixes a problem with the GV$_SEQUENCES global view.

This “Upgrading and Patching Oracle RAC” guide is intended to be used for educational purposes only. The paths and directories in this guide will be different from yours and thus you should follow this guide according to your environment. Again this guide will help you when patching oracle RAC, full tutorials can be found at Oracle University.

Upgrading and Patching Oracle RAC

1. Connect to one of the hosts in your Oracle RAC environment(for this tutorial we will start with host01). Use oraenv to set
the Oracle environment.

[[email protected]_pc ~]$ ssh -X [email protected]
[email protected]'s password:
[[email protected] ~]$ . oraenv
ORACLE_SID = [oracle] ? orcl
The Oracle base has been set to /u01/app/oracle
[[email protected]

2. The patch you will install is 20108098. It is located in /stage/patch/20108098 (Your location will be different). It fixes
an issue with the GV$_SEQUENCES view. Navigate to the directory and take a few moments
to view the README file, particularly Section I (Prerequisites) and Section II (Installation).

[[email protected] ~]$ cd /stage/patch/20108098
[[email protected] 20108098]$ ls -al
total 24
drwxrwxr-x 4 root root 4096 Jan 20 20:05 .
drwxr-xr-x 3 oracle oinstall 4096 Jan 22 18:31 ..
drwxr-xr-x 3 root root 4096 Jan 20 20:05 etc
drwxr-xr-x 3 root root 4096 Jan 20 20:05 files
-rw-rw-r-- 1 root root 5855 Jan 20 20:05 README.txt
[[email protected] 20108098]$ cat README.txt
Oracle Database 12c Release 12.1.0.2.0
ORACLE DATABASE Patch for Bug# 20108098 for Linux-x86-64 Platforms
This patch is RAC Rolling Installable - Please read My Oracle
Support Document 244241.1
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.j
spx?id=244241.1 Rolling Patch - OPatch Support for RAC.
This patch is Data Guard Standby-First Installable - Please read My
Oracle Support Note 1265700.1
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.j
spx?id=1265700.1
Oracle Patch Assurance - Data Guard Standby-First Patch Apply for
details on how to remove risk and reduce downtime when applying
this patch. Patching Oracle Rac
Released: Tue Jan 20 11:47:25 2015
This document describes how you can install the ORACLE DATABASE
interim patch for bug# 20108098 on your Oracle Database 12c
Release 12.1.0.2.0
(I) Prerequisites
--------------------
Before you install or deinstall the patch, ensure that you meet the
following requirements:
Note: In case of an Oracle RAC environment, meet these
prerequisites on each of the nodes.
1. Ensure that the Oracle home on which you are installing the
patch or from which you are rolling back the patch is Oracle
Database 12c Release 12.1.0.2.0.
2. Oracle recommends that you use the latest version available
for 12c Release 12.1.0.2.0. If you do not have OPatch 12c Release
12.1.0.1.4 or the latest version available for 12c Release
12.1.0.2.0,then download it from patch# 6880880 for 12.1.0.2.0
release.
For information about OPatch documentation, including any
known issues, see My Oracle Support Document 293369.1 OPatch
documentation list:
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=22
4346.1
3. Ensure that you set (as the home user) the ORACLE_HOME
environment variable to the Oracle home.
4. Ensure that the $PATH definition has the following
executables: make, ar, ld and nm. The location of these executables
depends on your operating system. On many operating systems, they
are located in /usr/ccs/bin.
5. Ensure that you verify the Oracle Inventory because OPatch
accesses it to install the patches. To verify the inventory, run
the following command.
$ opatch lsinventory
Note:
- If this command succeeds, it will list the Top-
Level Oracle Products and one-off patches if any that are installed
in the Oralce Home.
- Save the output so you have the status
prior to the patch apply.
- If the command displays some errors, then contact
Oracle Support and resolve the issue first before proceeding
further.
6. (Only for Installation) Maintain a location for storing the
contents of the patch ZIP file. In the rest of the document, this
location (absolute path) is referred to as <PATCH_TOP_DIR>. Extract
the contents of the patch ZIP file to the location (PATCH_TOP_DIR)
you have created above. To do so, run the following command:
$ unzip -d <PATCH_TOP_DIR> p20108098_121020_Linux-x86-
64.zip
7. (Only for Installation) Determine whether any currently
installed interim patches conflict with this patch 20108098 as
shown below:
$ cd <PATCH_TOP_DIR>/20108098
$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
The report will indicate the patches that conflict with
this patch and the patches for which the current 20108098 is a
superset.
Note:
When OPatch starts, it validates the patch and ensures that
there are no conflicts with the software already installed in the
ORACLE_HOME. OPatch categorizes conflicts into the following types:
- Conflicts with a patch already applied to the
ORACLE_HOME that is a subset of the patch you are trying to apply
- In this case, continue with the patch installation because the
new patch contains all the fixes from the existing patch in the
ORACLE_HOME. The subset patch will automatically be rolled back
prior to the installation of the new patch. Patching Oracle Rac
- Conflicts with a patch already applied to the
ORACLE_HOME - In this case, stop the patch installation and contact
Oracle Support Services.
8. Ensure that you shut down all the services running from the
Oracle home.
Note:
- For a Non-RAC environment, shut down all
the services running from the Oracle home.
- For a RAC environment, shut down all the
services (database, ASM, listeners, nodeapps, and CRS daemons)
running from the Oracle home of the node you want to patch. After
you patch this node, start the services on this node. Repeat this
process for each of the other nodes of the Oracle RAC system.
OPatch is used on only one node at a time.
(II) Installation
-----------------
To install the patch, follow these steps:
1. Set your current directory to the directory where the patch
is located and then run the OPatch utility by entering the
following commands:
$ cd <PATCH_TOP_DIR>/20108098
$ opatch apply
2. Verify whether the patch has been successfully installed by
running the following command:
$ opatch lsinventory
3. Start the services from the Oracle home.
(III) Deinstallation
----------------------
Ensure to follow the Prerequsites (Section I). To deinstall the
patch, follow these steps:
1. Deinstall the patch by running the following command:
$ opatch rollback -id 20108098
2. Start the services from the Oracle home.
3. Ensure that you verify the Oracle Inventory and compare the
output with the one run before the patch installation and re-apply
any patches that were rolled back as part of this patch apply. To
verify the inventory, run the following command:
$ opatch lsinventory
(IV) Bugs Fixed by This Patch
---------------------------------
The following are the bugs fixed by this patch:
20108098: GV$_SEQUENCES BROKEN
-------------------------------------------------------------------
Copyright 2015, Oracle and/or its affiliates. All rights reserved.
-------------------------------------------------------------------
[[email protected] 20108098]$

3. Make sure that your PATH definition points to the make, ar, ld and nm executables. In our
practice environment, they are located in /usr/bin.

[[email protected] ~]$ which make ar ld nm
/usr/bin/make
/usr/bin/ar
/usr/bin/ld
/usr/bin/nm
[[email protected] ~]$ echo $PATH
/usr/lib64/qt-
3.3/bin:/usr/local/bin:/bin:/usr/bin:/home/oracle/bin:/u01/app/orac
le/product/12.1.0/dbhome_1/bin
[[email protected] ~]$

4. Put the OPatch directory in your path and verify that OPatch has access to the Oracle
inventory with the opatch lsinventory command.

[[email protected] ~]$ export
PATH=$PATH:/u01/app/oracle/product/12.1.0/dbhome_1/OPatch
[[email protected] ~]$ opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.3
Copyright (c) 2015, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from :
/u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version : 12.1.0.1.3
OUI version : 12.1.0.2.0
Log file location :
/u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch20
15-01-23_08-20-11AM_1.log

Lsinventory Output file location :
/u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/lsinv/ls
inventory2015-01-23_08-20-11AM.txt
-------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Database 12c
12.1.0.2.0
There are 1 products installed in this Oracle Home.
There are no Interim patches installed in this Oracle Home.
Rac system comprising of multiple nodes
Local node = host01
Remote node = host02
Remote node = host03
-------------------------------------------------------------------
OPatch succeeded.
[[email protected] ~]$

5. Determine whether any currently installed interim patches conflict with patch 20108098 by
running the opatch prereq command as shown below.

[[email protected] 20108098]$ opatch prereq
CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 12.1.0.1.3
Copyright (c) 2015, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from :
/u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version : 12.1.0.1.3
OUI version : 12.1.0.2.0
Log file location :
/u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch20
15-01-23_08-28-08AM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
[[email protected] 20108098]$

6. Open a second ssh connection as the oracle user. Set the environment and use the
SRVCTL command to determine which instances are running on the three nodes. Shut
down the instance running on the local node, host01.

[[email protected]_pc ~]$ ssh [email protected]
[email protected]'s password:
[[email protected] ~]$
[[email protected] ~]$ . oraenv
ORACLE_SID = [oracle] ? orcl
The Oracle base has been set to /u01/app/oracle
[[email protected] ~]$
[[email protected] ~]$ srvctl status database -d orcl
Instance orcl_1 is running on node host02
Instance orcl_2 is running on node host03
Instance orcl_3 is running on node host01
[[email protected] ~]$ srvctl stop instance -d orcl -i orcl_3
[[email protected] ~]$

 

7. Return to the first terminal and start the patch installation by executing opatch apply.
When OPatch asks “Is the local system ready for patching?”, answer yes by typing Y and
pressing Enter.

[[email protected] 20108098]$ opatch apply
Oracle Interim Patch Installer version 12.1.0.1.3
Copyright (c) 2015, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from :
/u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version : 12.1.0.1.3
OUI version : 12.1.0.2.0
Log file location :
/u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/20108098
_Jan_23_2015_08_33_24/apply2015-01-23_08-33-24AM_1.log
Applying interim patch '20108098' to OH
'/u01/app/oracle/product/12.1.0/dbhome_1'
Verifying environment and performing prerequisite checks...
All checks passed.
This node is part of an Oracle Real Application Cluster.
Remote nodes: 'host02' 'host03'
Local node: 'host01'
Please shutdown Oracle instances running out of this ORACLE_HOME on
the local system.
(Oracle Home = '/u01/app/oracle/product/12.1.0/dbhome_1')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Patching component oracle.rdbms, 12.1.0.2.0...
Verifying the update...
The local system has been patched. You can restart Oracle #Patching Oracle Rac
instances on it.
Patching in rolling mode.
Remaining nodes to be patched:
'host02' 'host03'
What is the next node to be patched?

When patching is finished on host01, the OPatch dialog will inform you that the instance
can be restarted on host01 and will prompt you for the name of the next node to patch,
host02. Before continuing, go to the next step.

8. Go to the second terminal window and restart the instance on host01.

[[email protected] ~]$ srvctl start instance -d orcl -i orcl_3
[[email protected] ~]$

9. From the second terminal window, stop the instance on host02.

[[email protected] ~]$ srvctl stop instance -d orcl -i orcl_1
[[email protected] ~]$

10. Return to the first terminal window, type host02, and press Enter. When asked “Is the
node ready for patching? [y|n]”, answer Y and press Enter.

What is the next node to be patched?
host02
You have selected 'host02' from 'host02' 'host03'
The node 'host02' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on
'host02'.
(Oracle Home = '/u01/app/oracle/product/12.1.0/dbhome_1')
Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'host02'
Apply-related files are:
FP =
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/copy_files.txt"
DP =
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/copy_dirs.txt"
MP =
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/make_cmds.txt"
RC =
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/remote_cmds.txt"
Instantiating the file
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/copy_files.txt.instantiated" by replacing
$ORACLE_HOME in
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/copy_dirs.txt.instantiated" by replacing
$ORACLE_HOME in
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/make_cmds.txt.instantiated" by replacing
$ORACLE_HOME in
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/make_cmds.txt" with actual path.
Running command on remote node 'host02':
cd /u01/app/oracle/product/12.1.0/dbhome_1/rdbms/lib; /usr/bin/make
-f ins_rdbms.mk ioracle
ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 || echo
REMOTE_MAKE_FAILED::>&2
The node 'host02' has been patched. You can restart Oracle
instances on it.
The node 'host03' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on
'host03'.
(Oracle Home = '/u01/app/oracle/product/12.1.0/dbhome_1')

11. Go to the second terminal window and restart the instance on host02.

[[email protected] ~]$ srvctl start instance -d orcl -i orcl_1
[[email protected] ~]$

12. From the second terminal window stop the instance on host03.

[[email protected] ~]$ srvctl stop instance -d orcl -i orcl_2
[[email protected] ~]$

13. Return to the first terminal window and indicate host03 is ready for patching by typing Y
and pressing Enter.

Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'host03'
Apply-related files are:
FP =
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/copy_files.txt"
DP =
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/copy_dirs.txt"
MP =
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/make_cmds.txt"
RC =
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/remote_cmds.txt"
Propagating files to remote nodes...
Propagating directories to remote nodes...
Instantiating the file
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/make_cmds.txt.instantiated" by replacing
$ORACLE_HOME in
"/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/20108098_Ja
n_20_2015_12_04_56/rac/make_cmds.txt" with actual path.
Running command on remote node 'host03':
cd /u01/app/oracle/product/12.1.0/dbhome_1/rdbms/lib; /usr/bin/make
-f ins_rdbms.mk ioracle
ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 || echo
REMOTE_MAKE_FAILED::>&2
The node 'host03' has been patched. You can restart Oracle
instances on it.
There were relinks on remote nodes. Remember to check the binary
size and timestamp on the nodes 'host02' 'host03' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/12.1.0/dbhome_1/rdbms/lib;
/usr/bin/make -f ins_rdbms.mk ioracle
ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1
'
Patch 20108098 successfully applied
Log file location:
/u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/20108098
_Jan_23_2015_08_33_24/apply2015-01-23_08-33-24AM_1.log
OPatch succeeded.
[[email protected] 20108098]$

14. After OPatch has finished patching host03, go to the second terminal window and start the
instance on host03.

[[email protected] ~]$ srvctl start instance -d orcl -i orcl_2
[[email protected] ~]$

15. Return to the first terminal session and run the opatch lsinventory command to
ensure the Oracle homes on all three nodes were patched successfully.

[[email protected] 20108098]$ opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.3
Copyright (c) 2015, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from :
/u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version : 12.1.0.1.3
OUI version : 12.1.0.2.0
Log file location :
/u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch20
15-01-23_08-50-22AM_1.log
Lsinventory Output file location :
/u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/lsinv/ls
inventory2015-01-23_08-50-22AM.txt
-------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Database 12c
12.1.0.2.0
There are 1 products installed in this Oracle Home.
Interim patches (1) :
Patch 20108098 : applied on Fri Jan 23 08:35:48 UTC 2015
Unique Patch ID: 18461252
Created on 20 Jan 2015, 12:04:56 hrs PST8PDT
Bugs fixed:
20108098
Rac system comprising of multiple nodes
Local node = host01
Remote node = host02
Remote node = host03
-------------------------------------------------------------------
OPatch succeeded.
[[email protected] 20108098]$

 

Other Posts

How to resolve: ORA-01033 – ORACLE initialization or shutdown in progress

Resolving ORA-12705 Cannot Access NLS data files or invalid environment specified