How to generate Hang Analyze Report

 Sometimes database hangs due to many reasons and higher managements look for for RCA. It is important to have all trace file to find the issue for hanging the database or fine the RCA.

Get a hang analyze report by running below commands 

SQL> sqlplus / as sysdba 
SQL> oradebug setmypid 
SQL> oradebug unlimit
SQL> oradebug hanganalyze 3
# Wait at least 2-3 minutes to give time to identify process state changes.
SQL> oradebug hanganalyze 3
# Wait at least 2-3 minutes to give time to identify process state changes.
SQL> oradebug hanganalyze 3
SQL> oradebug tracefile_name   

SQL> select name, value from v$diag_info where name='Default Trace File';

In case database can't be connected  at all then please do  to the following commands 

connect / as sysdba 
oradebug setmypid 
oradebug unlimit
oradebug dump systemstate 266 -g; 
oradebug dump systemstate 266 -g; 
oradebug dump systemstate 266 -g;

To generate HANGANALYZE for RAC:

SQL> oradebug setmypid 
SQL> oradebug setinst all
SQL> oradebug -g def hanganalyze 3
# Wait at least 2-3 minutes to give time to identify process state changes.
SQL> oradebug -g def hanganalyze 3
# Wait at least 2-1 minutes to give time to identify process state changes.
SQL> oradebug -g def hanganalyze 3

It is not recommended to run HANGANALYZE at levels higher than 3 because it may generate a huge number of trace files for complex systems. Please check with Oracle Support.

SQL> oradebug setmypid 
SQL> oradebug setinst all
SQL> oradebug -g def hanganalyze 2
# Wait at least 1-2 minutes to give time to identify process state changes.
SQL> oradebug -g def hanganalyze 2
# Wait at least 1-2 minutes to give time to identify process state changes.
SQL> oradebug -g def hanganalyze 2

For reference Oracle Support Documents:
  • Troubleshooting Database Hang Issues (Doc ID 1378583.1)
  • How to Collect Diagnostics for Database Hanging Issues (Doc ID 452358.1)
  • Troubleshooting Database Contention With V$Wait_Chains (Doc ID 1428210.1)
  • Event: Hanganalyze – Reference Note (Doc ID 130874.1)
  • Important Customer information about using Numeric Events (Doc ID 75713.1)


Comments

Post a Comment

Popular posts from this blog

How to fix Oracle SQL Developer connection issue "Got minus one from a read call"

Image
 During connection from SQL Developer to EBS database got below error message: Status : Failure - Test failed: IO Error: Got minus one from a read call Following solution can be done : 1. Go to directory $ORACLE_HOME / network / admin 2. Modify sqlnet.ora file with following parameter : tcp.validnode_checking = no 3. If you don 't want to disable this, you can put the machine names as follows: tcp.invited_nodes=(machine1, machine2) 3. Bounce the listener. Due to security enhancements it is not good to put tcp.validnode_checking = no or comment out will through error when we start listener like below: Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=shsebs7idb1.appsolworld.com)(PORT=1522))) TNS-00584: Valid node checking configuration error TNS-12560: TNS:protocol adapter error Better plan is to add localhost or IP address of client (Local Computer) in the sqlnet.ora file That will solve the problem NAMES.DIRECTORY_PATH=(TNSNAMES, ONAMES, HOSTNAME) SQLNET.EXPI
Read more

How to troubleshoot Long Running Concurrent Request in EBS 12.2

 The following points need to answer before jumping into troubleshooting: Are there any recent code changes done in the concurrent program?           Check long_running_sid and verify the recent code change of last_ddl_timestamp after a             discussion with the developer. Was this running long in the last few run as well, or was this time only?            ==> Check the program history. How much time does it take to complete?           Once again program history can give us some idea. Are these jobs fetching higher data compared to the last run           ===> Confirm on data change . Query Plan change will indicate this issue. Does this job runs at any specific time or it can be run anytime           ===> The nature of the job can help to understand the job's nature. Does this job fetch data using DB Link          ====> Check whether the DB link is involved or not. The below-given approach would help to conclude the gray areas. Check the load of the server using t
Read more

How to drop index and before dropping it how to get the DDL.

  Oracle DROP INDEX statement DROP INDEX [schema_name.]index_name; DECLARE index_count INTEGER; BEGIN SELECT COUNT(*) INTO index_count FROM USER_INDEXES WHERE INDEX_NAME = 'index_name' ; IF index_count > 0 THEN EXECUTE IMMEDIATE 'DROP INDEX index_name' ; END IF; END; / In case want to check whether index exists and then delete it. If try to drop a non-existing, you will get the following error: SQL Error : ORA -01418 : specified index does not exist In SQLplus, set these before running the commands set long 100000 set longchunksize 100000 select DBMS_METADATA.GET_DDL( 'INDEX' , 'index_name') from all_indexes where owner in ( USER , 'USER_OTHER_THAN_LOGGED_IN_USER' ); Use below script to fetch metadata other information. SELECT DBMS_METADATA,GET_GRANTED_DDL('SYSTEM_GRANT','SSO******') from dual;
Read more

How to autopopulate user_guid in fnd_user table ( EBS Blank Page)

  Issue:  EBS Local Login url is working fine. When accessing sso url, users are getting error as below appended to the url. And login page will be blank. One of the reason could be : OAM Authentication is fine but during authorization process, OAM is unable to map guid from oid to user_guid in fnd_user. This can be due to user_guid being null in fnd_user table. As per the process, when user logins for the first time, user_guid need to be populated automatically (from oid) Query guid from oid and update fnd_user table with that. ldapsearch -h oidhost.appsolworld.com -p 3060 -D "cn=orcladmin" -w orcladmpwd -b "cn=Users,dc=dc,dc=online,dc=org" -s sub "(uid=$1)" orclsamaccountname krbprincipalname mail orcluserprincipalname orclguid Pass userid to this command which gives guid along with other details. Update fnd_user table with above guid : update fnd_user  set user_guid=' ' where user_name like ' '; Solution:  As per the process,  when user
Read more

Opatch is getting faild for GI patch with error code 2 while doing prereq

OPatch failed with error code 2  When checked the patch conflict for a patch, got below error. ]$opatch prereq CheckConflictAgainstOHWithDetail -ph ./ Oracle Interim Patch Installer version 12.2.0.1.29 Copyright (c) 2020, Oracle Corporation.  All rights reserved. PREREQ session Oracle Home       : /u01/oracle/product/19.0.0.0 Central Inventory : /u01/oracle/oraInventory    from           : /u01/oracle/product/19.0.0.0/oraInst.loc OPatch version    : 12.2.0.1.29 OUI version       : 12.2.0.1.4 Log file location : /oragrid/orahome/product/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2023-12-22_09-37-15AM_1.log Invoking prereq "checkconflictagainstohwithdetail" Prereq "checkConflictAgainstOHWithDetail" is not executed. The details are: Unable to create Patch Object. Exception occured : /orastage/OCT23_GRID_CPU/35642822/33575402/etc/config/actions.xml with Version field of the component "delete" in actions file cannot be <null> or empty. Please check p
Read more

Few Important steps of Oracle Database Clone

 Make Sure oraInst.loc location is correct and accessble. [demantra@dsiddem ~]$ cat /etc/oraInst.loc inventory_loc=/orastage/oraInventory inst_group=dba [demantra@dsiddem ~]$ cd /orastage/oraInventory [demantra@dsiddem oraInventory]$ ls -ltr total 16 drwxrwx---. 5 demantra dba 4096 Jul 22 19:37 backup drwxrwx---. 2 demantra dba 4096 Jul 22 19:37 ContentsXML drwxrwx---. 2 demantra dba 4096 Jul 22 19:37 locks drwxrwx---. 3 demantra dba 4096 Jul 23 10:08 logs Remove Oracle Home from Inventory [demantra@dsiddem oraInventory]$ cd $ORACLE_HOME/oui/bin [demantra@dsiddem bin]$ ./runInstaller -detachHome ORACLE_HOME=$ORACLE_HOME Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB.   Actual 3039 MB    Passed The inventory pointer is located at /etc/oraInst.loc Run Clone.pl [demantra@dsiddem bin]$ echo $ORACLE_HOME /dnbsid/demantra/oracle/19.0.0 [demantra@dsiddem bin]$ echo $ORACLE_BASE /dnbsid/demantra [demantra@dsiddem bin]$ E01=ORACLE_HOME=$ORACLE_HOME [dema
Read more

How to Check AD and TXK code levels in your EBS environment

Below query can be used in finding out current AD and TXK code levels.  [ap******@shsd*** 22989949]$ sqlplus apps SQL*Plus: Release 10.1.0.5.0 - Production on Fri Sep 30 01:00:25 2022 Copyright (c) 1982, 2005, Oracle.  All rights reserved. Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production SQL> col ABBREVIATION for a10 set lines 1000 col NAME for a50 col CODELEVEL for a20 SELECT ABBREVIATION,NAME,codelevel FROM AD_TRACKABLE_ENTITIES WHERE abbreviation in ('txk','ad');SQL> SQL> SQL> SQL> ABBREVIATI NAME                                               CODELEVEL ---------- -------------------------------------------------- -------------------- ad         Oracle Applications DBA                            C.13 txk        Oracle Applications Technology Stack               C.13 Check fwk in r12.2 select abbreviation ,codelevel from ad_trackable_entities where abbreviation in( 'fwk') order by abbreviation; SELECT codelevel
Read more

The Server is not able to service this request: [Socket:000445]Connection rejected, filter blocked Socket, weblogic.security.net.FilterException

Image
  Connection Filters allow one to deny access to UNWANTED Connections at the network level. Connection Filters can be used to protect server resources on individual servers, server clusters, or an entire internal network or Intranet. Connection filters are particularly useful when using the Administration port. Depending on the network firewall configuration, it may be possible to use a connection filter to further restrict administration access. A typical use might be to restrict access to the Administration port to only the servers and machines in the domain. WebLogic Server provides a default connection filter called  weblogic.security.net.ConnectionFilterImpl . This default connection filter accepts all incoming connections except the ones defined in the Administration Console as a Connection filter rule. Custom connection filters may be written  by implementing the classes in the weblogic.security.net package. This post will demonstrate the steps to configure Weblogic Connection F
Read more

PRVG-11250 : The check "RPM Package Manager database" was not performed because

Following message reported from cluvfy while performing pre-requisite for Clusterware Installation:  Verifying RPM Package Manager database ...INFORMATION PRVG-11250 : The check "RPM Package Manager database" was not performed because it needs 'root' user privileges. SOLUTION Execute runcluvfy manually with "-method root" option and enter root password when prompted : Example: $/runcluvfy.sh stage -pre crsinst -n <HOSTNAME1>,<HOSTNAME2> -method root Enter "ROOT" password: For reference metalink doc id :  2548970.1
Read more

CPU Patch Analysis

 Oracle CPU patch analysis Identifying the Latest Critical Patch Update for Oracle E-Business Suite Release 12.2 (Doc ID 2484000.1) Database Quick-Link If you only care about the database patches, this MOS note steps you through really easily. Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patch sets and Base Releases (Doc ID 2118136.2) https://www.oracle.com/security-alerts/ Critical Patch Updates Critical Patch Updates provide security patches for supported Oracle on-premises products. They are available to customers with valid support contracts. Starting in April 2022, Critical Patch Updates are released on the third Tuesday of January, April, July, and October (They were previously published on the Tuesday closest to the 17th day of January, April, July, and October). The next four dates are: 16 April 2024 16 July 2024 15 October 2024 21 January 2025 A pre-release announcement will be published on the Thursday preceding each Cr
Read more