How to generate Hang Analyze Report

 Sometimes database hangs due to many reasons and higher managements look for for RCA. It is important to have all trace file to find the issue for hanging the database or fine the RCA.

Get a hang analyze report by running below commands 

SQL> sqlplus / as sysdba 
SQL> oradebug setmypid 
SQL> oradebug unlimit
SQL> oradebug hanganalyze 3
# Wait at least 2-3 minutes to give time to identify process state changes.
SQL> oradebug hanganalyze 3
# Wait at least 2-3 minutes to give time to identify process state changes.
SQL> oradebug hanganalyze 3
SQL> oradebug tracefile_name   

SQL> select name, value from v$diag_info where name='Default Trace File';

In case database can't be connected  at all then please do  to the following commands 

connect / as sysdba 
oradebug setmypid 
oradebug unlimit
oradebug dump systemstate 266 -g; 
oradebug dump systemstate 266 -g; 
oradebug dump systemstate 266 -g;

To generate HANGANALYZE for RAC:

SQL> oradebug setmypid 
SQL> oradebug setinst all
SQL> oradebug -g def hanganalyze 3
# Wait at least 2-3 minutes to give time to identify process state changes.
SQL> oradebug -g def hanganalyze 3
# Wait at least 2-1 minutes to give time to identify process state changes.
SQL> oradebug -g def hanganalyze 3

It is not recommended to run HANGANALYZE at levels higher than 3 because it may generate a huge number of trace files for complex systems. Please check with Oracle Support.

SQL> oradebug setmypid 
SQL> oradebug setinst all
SQL> oradebug -g def hanganalyze 2
# Wait at least 1-2 minutes to give time to identify process state changes.
SQL> oradebug -g def hanganalyze 2
# Wait at least 1-2 minutes to give time to identify process state changes.
SQL> oradebug -g def hanganalyze 2

For reference Oracle Support Documents:
  • Troubleshooting Database Hang Issues (Doc ID 1378583.1)
  • How to Collect Diagnostics for Database Hanging Issues (Doc ID 452358.1)
  • Troubleshooting Database Contention With V$Wait_Chains (Doc ID 1428210.1)
  • Event: Hanganalyze – Reference Note (Doc ID 130874.1)
  • Important Customer information about using Numeric Events (Doc ID 75713.1)


Comments

Post a Comment

Popular posts from this blog

How to drop index and before dropping it how to get the DDL.

  Oracle DROP INDEX statement DROP INDEX [schema_name.]index_name; DECLARE index_count INTEGER; BEGIN SELECT COUNT(*) INTO index_count FROM USER_INDEXES WHERE INDEX_NAME = 'index_name' ; IF index_count > 0 THEN EXECUTE IMMEDIATE 'DROP INDEX index_name' ; END IF; END; / In case want to check whether index exists and then delete it. If try to drop a non-existing, you will get the following error: SQL Error : ORA -01418 : specified index does not exist In SQLplus, set these before running the commands set long 100000 set longchunksize 100000 select DBMS_METADATA.GET_DDL( 'INDEX' , 'index_name') from all_indexes where owner in ( USER , 'USER_OTHER_THAN_LOGGED_IN_USER' ); Use below script to fetch metadata other information. SELECT DBMS_METADATA,GET_GRANTED_DDL('SYSTEM_GRANT','SSO******') from dual;
Read more

PRVG-11250 : The check "RPM Package Manager database" was not performed because

Following message reported from cluvfy while performing pre-requisite for Clusterware Installation:  Verifying RPM Package Manager database ...INFORMATION PRVG-11250 : The check "RPM Package Manager database" was not performed because it needs 'root' user privileges. SOLUTION Execute runcluvfy manually with "-method root" option and enter root password when prompted : Example: $/runcluvfy.sh stage -pre crsinst -n <HOSTNAME1>,<HOSTNAME2> -method root Enter "ROOT" password: For reference metalink doc id :  2548970.1
Read more

ORA-00257:archiver error, connect internal only until freed

  Issue: Application log shows below error and users are unable to connect to the database. 0RA-00257:archiver error, connect internal only until freed (OR) 0RA-00257:archiver error, connect internal only until freed ORA-16020: less destinations available than specified by LOG_ARCHIVE_MIN_SUCCEED_DEST Fix: 1) Check if there are any errors for the archive destination(s). Make sure that the number of VALID archive destinations is greater than or equal to the value specified by LOG_ARCHIVE_MIN_SUCCEED_DEST initialization parameter. set pages 1000 col dest_name for a30 col destination for a60 SELECT dest_id, dest_name, binding, status, destination, error FROM v$archive_dest; SHOW PARAMETER log_archive_min_succeed_dest 2) If space is full in one or more of the archive destinations (or if destination is not available), take any of the following steps: 2.a) Manually move the archives to another location and delete them from archive destination.   (OR) 2.b) Change the archive des...
Read more

Verifying Daemon “Avahi-Daemon” Not Configured And Running …FAILED (PRVG-1360)

  Execute "/tmp/CVU_19.0.0.0.0_grid/runfixup.sh" as root user on nodes "shsebs7idb1" to perform the fix up operations manually Press ENTER key to continue after execution of "/tmp/CVU_19.0.0.0.0_grid/runfixup.sh" has completed on nodes "shsebs7idb1" Fix: Daemon "avahi-daemon" not configured and running   Node Name                             Status   ------------------------------------  ------------------------   shsebs7idb1                           failed ERROR: shsebs7idb1: PRVG-9023 : Manual fix up command "/tmp/CVU_19.0.0.0.0_grid/runfixup.sh" was not issued by root user on node "shsebs7idb1" Result: "Daemon "avahi-daemon" not configured and running" could not be fixed on nodes "shsebs7idb1" SOLUTION: To fix this error, stop and disable avahi-daemon service. 1. Check avahi-daemon status: [root@shsebs7idb1 ~]# ls -...
Read more

Linux OL7/RHEL7: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

  This is due to below bugs: OL7 -  Unpublished Bug 21441387- 12.2/MAIN/OL7.1: CVU REPORTS /DEV/SHM NOT MOUNTED WHEN IT IS MOUNTED  21441387.8 RHEL 7 - Unpublished Bug 25907259 : RHEL7.2: CVU REPORTS /DEV/SHM NOT MOUNTED WHEN IT IS MOUNTED  25907259.8 As a work-around in OL7: In OL7, the /dev/shm is not mounted in /etc/fstab on a fresh system, it is done in: [root@testc140 u02]# ll /usr/lib/dracut/modules.d/99base/init.sh -rwxr-xr-x. 1 root root 11379 Mar 26 10:14 vi /usr/lib/dracut/modules.d/99base/init.sh and comment the below lines 62 if ! ismounted /dev/shm; then 63 mkdir -m 0755 /dev/shm 64 mount -t tmpfs -o mode=1777,nosuid,nodev,strictatime tmpfs /dev/shm >/dev/null 65 fi so, this validation is not really valid for OL7.    We can also safely ignore this message. For reference metalink id: -  Doc ID 2065603.1
Read more

SKIP DNS RESLOV.CONF CHECK DURING RAC CONFIGURATION

 shsebs7idb1: PRVG-2064 : There are no configured name servers in the file              '/etc/resolv.conf' on the nodes "shsebs7idb1" Solution: [root@shsebs7idb2 ~]# ls -ltr /etc/resolv.conf -rw-r--r--. 1 root root 78 Jul  3 11:02 /etc/resolv.conf [root@shsebs7idb2 ~]# mv /etc/resolv.conf /etc/resolv.conf.sid Rename resolv.conf to resolv.conf.sid for example and re run the installation again . It  worked for me
Read more

CPU Patch Analysis

 Oracle CPU patch analysis Identifying the Latest Critical Patch Update for Oracle E-Business Suite Release 12.2 (Doc ID 2484000.1) Database Quick-Link If you only care about the database patches, this MOS note steps you through really easily. Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patch sets and Base Releases (Doc ID 2118136.2) https://www.oracle.com/security-alerts/ Critical Patch Updates Critical Patch Updates provide security patches for supported Oracle on-premises products. They are available to customers with valid support contracts. Starting in April 2022, Critical Patch Updates are released on the third Tuesday of January, April, July, and October (They were previously published on the Tuesday closest to the 17th day of January, April, July, and October). The next four dates are: 16 April 2024 16 July 2024 15 October 2024 21 January 2025 A pre-release announcement will be published on the Thursday preceding eac...
Read more

How to write to a CSV file using Oracle SQL*Plus

  SPOOL command  The  SPOOL  command is  unavailable  in the browser-based SQL*Plus version,  iSQL*Plus . To generate files while using iSQL*Plus, change the necessary preference settings to directly output to a file. This is accomplished using the  SPOOL  statement. While  SPOOL  is  active , SQL*PLus will store the output of any query to the specified file. Therefore, the next command to enter is  spool : spool filepath Skipping ahead slightly,  after  your query is inserted, you also need to halt  spool  so the file output is closed by using the  spool off  command: spool off Insert the query  The last step after the settings are modified and  spool  is running is to insert your query. For our simple example, we’re outputting all books from our  books  table. SELECT   title,   author FROM   authors; Don’t forget the semi...
Read more

How to troubleshoot Long Running Concurrent Request in EBS 12.2

 The following points need to answer before jumping into troubleshooting: Are there any recent code changes done in the concurrent program?           Check long_running_sid and verify the recent code change of last_ddl_timestamp after a             discussion with the developer. Was this running long in the last few run as well, or was this time only?            ==> Check the program history. How much time does it take to complete?           Once again program history can give us some idea. Are these jobs fetching higher data compared to the last run           ===> Confirm on data change . Query Plan change will indicate this issue. Does this job runs at any specific time or it can be run anytime           ===> The nature of the job can help to understand the job's nature. Does this job fetch data using DB Link...
Read more

How To Manage Space of The FRA in the Oracle DB

  Delete Old files connect with the target database using RMAN RMAN target / Delete the old files RMAN> delete obsolete; Remove Restore Points SQL> SELECT name, storage_size FROM gv$restore_point; Drop the restore point SQL> DROP restore point RP; Delete the archive logs RMAN> DELETE archivelog ALL Increase the Space SQL> ALTER SYSTEM SET db_recovery_file_dest_size = 20G scope=both; Disable the Flashback ALTER DATABASE flashback off;
Read more