How to use ADOP Patch Cycle

 

A new environment variable, $FILE_EDITION, shows the current designation of a given dual file system member. 

Three other new environment variables designate the root directories of the run ($RUN_BASE),

patch ($PATCH_BASE), and non-editioned ($NE_BASE) file systems.

For example:

$FILE_EDITION = patch

$RUN_BASE = /u01/R122_EBS/fs1

$PATCH_BASE = /u01/R122_EBS/fs2

$NE_BASE = /u01/R122_EBS/fs_ne

When a patch is being applied, the Oracle E-Business Suite system is running in normal production mode 

(full functionality, with some documented exceptions) in the run edition of the file system and database.

 Full application functionality is retained as patch execution proceeds, until the cutover phase is reached.

It is more appropriate to think in terms of a patching cycle than a single patching operation. The online patching cycle consists of a number of phases:

1. Prepare

2. Apply

3. Finalize

4. Cutover

5. Cleanup

You specify the desired phase or phases as arguments to the adop utility. The actions taken in these phases are described in Oracle E-Business Suite Concepts. This and the preceding chapter of this book provide details of the available options.

The adop utility sets its own environment. There is therefore no need to source the environment before running it.

An online patching consists of several phases, which are specified on the adop command line as follows:

adop phase=<phase_name>

Prepare phase - Used to start a new online patching cycle:

$ adop phase=prepare

Apply phase - Used to apply one or more patches to the patch edition of an Oracle E-Business Suite system:

$ adop phase=apply patches=123456,789101 workers=8

Finalize phase - Used to perform the final patching operations that can be executed while the application is still online:

$ adop phase=finalize

Cutover phase - Used to perform the transition to the patched environment:

$ adop phase=cutover

Cleanup phase - Used to remove old objects that are no longer needed:

$ adop phase=cleanup

The adop phases are described in more detail in The Online Patching Cycle section of this chapter.

Abort command

If necessary, an online patching cycle can be terminated, with the actions taken being discarded.

The command to perform this operation is:

$ adop phase=abort

This abort command is only available up to (but not including) the cutover phase. After cutover, the system is running on the new edition, and abort is no longer possible for that patching cycle.

Before you can use online patching, you must perform some initial one-off setup steps:

• Check Inventory Setup

• Set Up Secure Shell on Application Tier Nodes

• Create Customized Instructions for Patch Application Assistant

Each of these requirements is described below.

Check Inventory Setup

Summary

• By default, a global (central) inventory is used to store information about all Oracle E-Business Suite Release 12.2 application tier nodes.

• The global inventory location must be identified by the /oracle/oraInventory.loc file.

• On a shared file system, the global inventory location must be shared and used by all participating nodes.

• Starting from the AD-TXK Delta.7 codelevel, an instance-specific 'EBS Installation Central Inventory' can optionally be used for the application tier of UNIX platforms.

Using the Global Inventory

If you are using a UNIX platform, you should verify the existence and contents of the oraInst.loc file, which specifies the location of the oraInventory.loc file global inventory file.

1. Check that oraInst.loc exists in the correct directory for your platform:

2. Confirm that the contents of oraInst.loc look like this example:

   inventory_loc=/oracle/oraInventory

   where /oracle/oraInventory is the path to the directory where the central inventory is located. This location must be writable by the user account that will run Rapid Install.

he following example demonstrates the steps:

1. $ ssh-keygen -t rsa
   Generating public/private rsa key pair.
   Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter> 
   Enter passphrase:<Enter>
   Enter same passphrase again:<Enter>
   Your identification has been saved in /u01/user2/.ssh/id_rsa.
   Your public key has been saved in /u01/user2/.ssh/id_rsa.pub.
   The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

2. $ scp -pr /u01/user2/.ssh/id_rsa.pub \
   user2@system1:/u01/user2/.ssh/authorized_keys
   user2@system1's password:<password>
   id_rsa.pub 100% 398 0.4KB/s 00:00

3. $ ssh user2@system1

   If you receive this message, it can safely be ignored: Warning: untrusted X11 forwarding setup failed: xauth key data not generated Warning: No xauth data; using fake authentication data for X11 forwarding.

Once this has been done for the relevant operating system account on all nodes - that is, ssh can log in from the primary node to each secondary node without entering a password - so you are ready to run adop on multiple application tier nodes. It must be run on at least the primary (admin) node: from there, it will attempt to contact all the other application tier nodes that are part of the same Oracle E-Business Suite instance, and will run the required steps remotely on those nodes.

Important: If you change the password for the relevant operating system account on one or more nodes, you must regenerate the ssh credentials either using the $AD_TOP/patch/115/bin/txkRunSSHSetup.pl script, or your own native solution if you prefer.

The txkRunSSHSetup.pl script has a -help option that shows relevant usage options.

For example, a basic command to enable ssh would be:

$ perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

To verify ssh operation:

$ perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl verifyssh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3 \
-invalidnodefile=<filename to report ssh verification failures>

To disable ssh:

$ perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl disablessh \
-contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3 \
-invalidnodefile=<filename to report ssh verification failures>

he online patching cycle consists of a number of high level phases:
prepare
apply
finalize
cutover
cleanup
A high level overview of an online patching cycle would, programmatically, look like this:
# Prepare for patching:
$ adop phase=prepare

# Apply patches:
$ adop phase=apply patches=<patch number>

# Apply any customizations to patch edition (optional):
$ . <EBS_ROOT>/EBSapps.env patch
$ sqlplus apps/apps @my_custom_script_01
$ sqlplus apps/apps @my_custom_script_02
...

# Finalize patch application:
$ adop phase=finalize

# Perform cutover:
$ adop phase=cutover
$ . <EBS_ROOT>/EBSapps.env run

# Perform user acceptance testing via application UI

# Perform cleanup:
$ adop phase=cleanup
Important Additional Points
After an online patching cycle is started, you should not perform any configuration changes in the run edition file system. Any that are made will not be propagated, and will therefore be lost after cutover is complete.
The prepare, apply, and fs_clone phases all require at least 10GB of free disk space. All other phases require 1GB of free space. A warning message will be displayed if less than the needed amount is available.
The directories where you extracted the patches applied in a given patching cycle must be retained, in the same location and with the same contents, until the next prepare phase completes. This is also a requirement for patches applied in hotpatch or downtime mode.
Any customizations must be applied to the patch edition during the apply phase, normally after any Oracle E-Business Suite patches have been applied.
Special Phases
Two additional phases are provided for specialized use. Neither can be run in conjunction with any other phase. Further details of these phases are described in later sections.
The abort phase is used to terminate a patching cycle before it is complete, and roll back any changes that have been made. It can also be run in conjunction with a full cleanup operation.
The fs_clone phase is a command (not related to adcfgclone.pl) that is used to synchronize the patch file system with the run file system. Normally, the fs_clone phase should only be run when mentioned as part of a specific documented procedure.
Important: You may perform a procedure that as a final step instructs you to run fs_clone. You do not have to do this immediately: the key requirement is to run fs_clone before you start the next patching cycle. And if you are performing multiple procedures, each of which requires fs_clone to be run at the end, you only need to run it once before the start of the next patching cycle.
$ . <EBS_ROOT>/EBSapps.env run
$ adstrtal.sh
Set the patch edition environment whenever you intend to execute commands that affect the patch edition.
For example:
$ . <EBS_ROOT>/EBSapps.env patch
$ sqlplus apps/apps @my_custom_patch_script.sql
The adop tool executes non-interactively, executing the specified phase or phases in order. In a multi-node deployment, adop is only executed by the user on the primary node: internally, adop will use ssh remote execution to run required actions on all secondary nodes automatically. In addition, adop can be used to generate reports about patching operations in the environment.
adop is typically run as follows:
$ adop phase=<phase_name>
The phase parameter accepts the following phase names. These names can be specified individually, or (except where otherwise noted) with other phase names in a comma-separated list:
prepare - Prepares the environment for patching.
apply - Applies the specified patches to the environment.
finalize - Performs any final steps required to make the system ready for cutover.
cutover - Shuts down application tier services, makes the patch edition the new run edition, and then restarts application tier services. This is the only phase the involves a brief downtime.
cleanup - Removes obsolete code and data from old editions.
abort - Aborts the online patching cycle by dropping the database patch edition. This phase cannot be specified with any other phase.
fs_clone - Recreates the patch edition file system as an exact copy of the run edition file system, preserving the mode, ownership, and timestamps of the files. This phase cannot be specified with any other phase. Use of fs_clone is normally not required. Situations that do require fs_clone are will explicitly document that requirement. If running this phase, ensure that your current working directory is not within the patch edition file system.
Online Patching Cycle Steps - Prepare Phase
This section describes the principles of adop operation in the prepare phase, followed by the steps you take to run this phase.
Note: The exact actions taken during the prepare phase are context-dependent: for example, the first time it is ever run on a system; when it is run after an apply phase has been aborted; and when it has been run after cutover.
Principal adop Actions
During the prepare phase, adop performs the following steps.
Checks whether to perform a cleanup, which will be needed if the user failed to invoke cleanup after the cutover phase of a previous online patching cycle.
Validates system configuration to ensure that the system is ready to start an online patching cycle.
Checks to see if the database is prepared for online patching:
Checks if the database user is edition-enabled. If not, adop immediately exits with an error.
Checks to see if the patch service has been created. adop requires that a special database service exists for the purpose of connecting to the patch edition. This service is created automatically, but its continued existence is validated on each prepare.
Checks to see if logon trigger exists and is enabled. If the logon trigger is missing or the patch service has not been created, adop will automatically try to fix the issue so that it can proceed. If it cannot do so, it will exit with an error message.
Checks the integrity of the database data dictionary. If any corruption is found, adop will exit with an error. 
Checks that the E-Business Suite Technology Codelevel Checker (ETCC) has has been run, to verify that all required patches have been applied to the database.
Checks system configuration on each application tier node. A number of critical settings are validated to ensure that each application tier node is correctly registered, configured, and ready for patching.
Checks for the existence of the "Online Patching In Progress" (ADZDPATCH) concurrent program. This program prevents certain predefined concurrent programs from being started, and as such needs to be active while a patching cycle is in progress (that is, while a database patch edition exists).
The flow of control is as follows.
If the ADZDPATCH program has not yet been requested to run, a request is submitted.
The status of ADZDPATCH is determined. If it is pending, it may be waiting for an incompatible program to finish. At that point, its status will change to running, and it will allow the prepare phase to proceed. A message to this effect is displayed to the user.
The next stage depends on whether the concurrent managers are running:
If the concurrent managers are all down, the prepare phase continues, with ADZDPATCH entering a status of pending (with the highest priority) until the managers are started.
If the concurrent managers are partially up, but there is no manager defined that can run ADZDPATCH, then the prepare phase will exit with an error.
If the concurrent managers are up, and there is one defined that can run ADZDPATCH, processing will loop until ADZDPATCH changes status from pending to running (that is to say, as noted in Step 2, no incompatible programs are found). The prepare phase then continues. ADZDPATCH is cancelled when the cutover phase is complete.
Invokes the TXK script $AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl to synchronize the patches which have been applied to the run APPL_TOP, but not the patch APPL_TOP. The script depends on the adop repository for patches that have been applied on the run APPL_TOP but not the patch APPL_TOP.
Checks the database for the existence of a patch edition, and creates one if it does not find one.
Calls the $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl script again to confirm that the database connection to the patch edition is working.
If any of these checks fail, adop will exit with an error.
Optional User Checks
Before you run the prepare phase to start a new patching cycle, you may wish to perform a couple of optional checks.
The first check is to validate your system for patching, by running the command:
$ adop -validate
If you run this command while a patching cycle is in progress, validation will take place for the cutover phase.
If you run this command from the primary node, adop will perform the validation checks on all the available nodes available in the system. In contrast, if you run it from a secondary node, it will run only on that node.
The second check is to confirm there is adequate free space on your system to support a patching cycle:
SYSTEM tablespace - minimum of 25 GB free
APPS_TS_SEED tablespace - minimum of 5 GB free
You can do this by running the $AD_TOP/sql/ADZDSHOWTS.sql script:

After running abort, a full cleanup must be performed. The cleanup command is:adop phase=cleanup cleanup_mode=full). This will remove any columns that were added by the patch but are no longer needed because of the abort. If they are not removed, they may cause problems in a later patching cycle.
Alternatively, you can run a combined command to abort the patching cycle and perform a full cleanup:
$ adop phase=abort,cleanup cleanup_mode=full
If any attempt was made to apply patches to the patch edition, after abort you must run the fs_clone phase (adop phase=fs_clone) to recreate the patch file system.
This section covers a variety of tasks that may apply either to individual online patching operations, or to your system setup as a whole. Diagnostic, troubleshooting, and reporting features are also described.
Manual Post-Patch Installation Tasks
Traditionally, some patches have associated post-patch installation tasks, including recompilation of invalid packages, regenerating JAR files, and running AutoConfig. In an online patching environment such as Release 12.2 such tasks will normally be performed automatically in the apply phase.
If a post-installation patch step mentions any tasks that need to be performed explicitly, where they are run from depends on the type of patching:
In a normal online patching cycle, the steps should be executed from the patch file system after the apply phase.
If the patch is being applied in hotpatch mode or downtime mode, the steps should be executed from the run file system after the apply phase.
Dropping Old Editions With the actualize_all Phase
As each online patching cycle is completed, the database will accumulate an additional old database edition. If the number of these grows too large, system performance will start to be affected. When the number of old database editions reaches 25 or more, you should consider dropping all old database editions by running the adop actualize_all phase and then performing a full cleanup.
Important: This procedure will take a large amount of time (significantly longer than a normal patching cycle), and should only be performed when there is no immediate need to start a new patching cycle.
Before starting, you should ensure that the system has the recommended database patches and latest AD-TXK code level installed.
To proceed, run the following commands in the order shown:
$ adop phase=prepare
$ adop phase=actualize_all
$ adop phase=finalize finalize_mode=full
$ adop phase=cutover
$ adop phase=cleanup cleanup_mode=full
You have now completed removal of the old database editions.
The Online Patching Cycle
The online patching cycle is divided into five phases:
Patching Cycle Overview
The key actions in the various stages of the online patching cycle can be summarized as follows:
Prepare
Synchronizes patch edition and run edition on the file system.
Creates a new patch edition in the database.
Apply
Executes patch drivers to update patch edition.
Patches applied: can be one or many, including customizations.
Finalize
Compiles invalid objects.
Generates derived objects.
Cutover
Configures patch edition file system to be the new run edition file system.
Configures patch edition of database to be the new run edition.
Restarts application tier services.
Cleanup
Delete obsolete code and seed data to recover space.
Each of the phases will now be discussed in more detail.
Prepare
The following actions are taken in this phase.
On the file system:
The patch file system is synchronized with the run file system.
The patch edition files become an exact copy of the run edition files.
By default, synchronization is incremental: that is, only files that were changed in the last patch application are copied.
In the database:
A patch edition is created in the database.
All code objects in the patch edition begin as pointers to code objects in the run edition. Code objects in the patch edition begin as lightweight "stub objects" that point to the actual object definitions, which are inherited from earlier editions. Stub objects consume minimal space, so the database patch edition is initially very small in size.
As patches are applied to the patch edition, code objects are actualized (have a new definition created) in that edition.

Apply
The following actions are taken in this phase:
Patches are applied to the patch edition. During this process, any changed stub objects will become actual code objects in the patch edition.
The changes introduced by the patches are made in the isolation of the patch edition.
Changes to code objects are only visible in the patch edition of the file system and database.
Changes to tables are stored in new columns or rows that are only visible to the patch edition.
 At this point, users still remain connected to the application and performing their work.
Finalize
This phase is used to perform the final operations that can be executed while the application is online:
Invalid objects are compiled.
Derived objects are generated.
Any actions that must be performed at cutover are pre-computed and stored for quick execution at that time.
Cutover
This phase involves:
Shutdown of application tier services.
Execution of any required cutover actions to maintain non-editioned objects and data.
Configuration of the patch edition of the file system as the new run edition.
Configuration of the patch edition of the database as the new run edition.
Restart of application tier services on the new run edition.
Although the cutover phase does require a short period of application tier services downtime, the online patching cycle can be paused for as long as required prior to running this phase. You could, for example, add such a pause to ensure that the downtime period will be outside business hours.
The database remains open throughout the entire online patching cycle, including cutover.
Cleanup
The following database actions are taken in this phase, which occurs after users have been brought back online to the newly-patched application:
Old code objects that are no longer visible in the run edition are dropped.
Old data (rows or columns) that is no longer visible in the run edition is deleted or dropped.
Old database editions that no longer contain actual objects are dropped.
Database Implementation
Creating a copy of the database part of the running system has been accomplished by taking advantage of the Oracle Database Edition-Based Redefinition (EBR) feature. This allows an application to efficiently store multiple copies of its application definition in the same database, and thereby enables online upgrade of the database tier.
The term edition refers to the isolation mechanism that allows pre-upgrade and post-upgrade schemas to co-exist. The simplest way to think of an edition is as a separate (isolated) copy of all database code objects that are changed by a patch.
The three types of database edition are:
Run: Online users connect to this edition, which is always the default database edition. The run edition will always exist.
Patch : Patching tools connect to this edition. A child of the run edition, the patch edition exists only while a patching cycle is in progress.
Old: When a patch edition is promoted to be the run edition, the previous run edition is now regarded as an old edition. There may be zero or more old editions at a given time. They are discarded when a full cleanup (described later) is performed. You cannot connect to an old edition.
File System Implementation
Creating and using two file systems allows one (run file system) to be part of the running system, while the other (patch file system) is either being patched or waiting to be patched during the next patching cycle.
 The file systems are designated File System 1 and File System 2 in Rapid Install. These are abbreviated to fs1 and fs2 respectively.
The two file systems are sometimes referred to as a dual file system, and swap roles at the end of each patching cycle. That is, the file system that has just been patched is put into use as part of the running system (becoming the new run file system), and the previous run file system takes over the the role of patch file system (in readiness for commencement of the next patching cycle).
Important: The existence of the dual file system has significant implications for general (non-patching) maintenance activities. For information on understanding how to decide which file system to run AD tools 
However, two file systems are not sufficient to meet all the practical needs of an online patching environment for Oracle E-Business Suite. A third file system, described in the next section, is also required.
Non-Editioned File System
The dual file system approach caters for application code, but applications also use the file system to read and write business data. In Release 12.2, application data files are stored in a third area, the non-editioned file system (fs_ne), which is used to store data that is needed across all file systems. Non-editioned files are not copied or moved during patching: their location remains constant across online patching cycles.
The Three File Systems: fs1, fs2, and fs_ne
The non-editioned file system is therefore completely separate from file system 1 and file system 2.
Context Variables for Online Patching File Systems
Several AutoConfig context variables support the file systems used in online patching. For example, the s_ne_base context variable stores the location of the non-editioned file system, and AutoConfig sets the environment variable $NE_BASE to the corresponding value. AutoConfig also sets the environment variables $RUN_BASE and $PATCH_BASE, so you can easily tell which is currently the run file system and which the patch file system.
Files Stored in the Non-Editioned File System
The non-editioned file system is designed to store files that contain transactional data and reference data. Examples include: import, export files, general log files, and concurrent manager log and out files.These are all examples of files that are not modified during an online patching cycle.
More specialized examples include:
Batch upload and download files.
Files used to transfer transactional data from processes external to Oracle E-Business Suite (for example, where a third party order entry system delivers orders via order import files).
Files containing transactions that are needed across all file systems.
The non-editioned file system is not designed to store shared files, because initially identical files can become non-identical during patching life cycles. Nor is it designed to store code, which is editioned (one copy can be in the run file system while the other is in the patch file system).
1) Prepare
adop phase=prepare
2) Apply 
adop phase=apply  patches=
adop phase=apply input_file= 
Input file can contain the following parameter:
workers=
patches=:.drv, :.drv …
adop phase=apply input_file=input_file
patches
phase
patchtop
merge
defaultsfile
abandon
restart
workers
restart  
adop phase=apply patches= restart=yes
abandon  — starts the failed patch from scratch
adop phase=apply patches=<>  abandon=yes
apply_mode
adop phase=apply patches=<>  apply_mode=downtime
apply=(yes/no)
To run the patch test mode, specify apply = no
e) analytics
adop phase=apply analytics=yes
ADZDCMPED.sql – This script is used to display the differences between the run and patch editions, including new and changed objects.
The output file location is: /../fs_ne/EBSapps/log/adop////adzdcmped.out.
ADZDSHOWED.sql – This script is used to display the editions in the system.
The output file location is: /.../fs_ne/EBSapps/log/adop///adzdshowed.out.
ADZDSHOWOBJS.sql – This script is used to display the summary of editioned objects per edition.
The output file location is: /../fs_ne/EBSapps/log/adop///adzdshowobjs.out
ADZDSHOWSM.sql – This script is used to display the status report for the seed data manager.
The output file location is: /../fs_ne/EBSapps/log/adop///adzdshowsm.out
Finalize 
adop phase=finalize
finalize_mode=(full|quick)
Cutover  
adop phase=cutover
adop phase=cutover mtrestart=no
cm_wait 
CLEANUP
cleanup_mode=(full|standard|quick)  [default: standard]
FS_CLONE  
Optional parameters during fs_clone phase:
a) force – To start a failed fs_clone from scratch
adop phase=fs_clone force=yes  [default: no]
b) Patch File System Backup Count ==> s_fs_backup_count  [default: 0 : No backup taken]
Denotes the number of backups of the patch file system that are to be preserved by adop. The variable is used during the fs_clone phase,
where the existing patch file system is backed up before it is recreated from the run file system.
7) Abort – used to abort the current patching cylce.
abort can be run only before the cutover phase
adop phase=abort