Virtual machine reports a "BUG: soft lockup"

 

Issue

  • Virtual machine guest suffers multiple soft lockups at the same time
  • We are experiencing kernel panic due to softlockup.

  • Logs show messages like (examples from different sources):

Cause: kernel messages informing that vCPU did not get execution for N seconds

Resolution

Under normal circumstances, these messages may go away if the load decreases.
This 'soft lockup' can happen if the kernel is busy, working on a massive amount of objects which need to be scanned, freed, or allocated, respectively.
The stack traces of those tasks can give a first idea about what the tasks were done. However, to be able to examine the cause behind the messages, a kernel dump would be needed.

While these messages cannot be disabled entirely, in some situations, increasing the time before these
soft lockups are fired can relax the situation.
 
To do so, increase the following sysctl parameter: kernel.watchdog_thresh
 
The default value for this parameter is 10 and double the matter might be a good start.
 
e.g.
 
server1:~ # echo 20 > /proc/sys/kernel/watchdog_thresh
or



server1:~ # echo "kernel.watchdog_thresh=20" > /etc/sysctl.d/99-watchdog_thresh.conf
server1:~ # sysctl -p  /etc/sysctl.d/99-watchdog_thresh.conf
For more information on how to configure and capture kernel dump please check: Configure crashkernel memory for kernel core dump analysis

Cause

A 'soft lockup'  that causes the kernel to loop in kernel mode for more than 20 seconds without giving other tasks a chance to run.
The watchdog daemon will send a non-maskable interrupt (NMI) to all CPUs in the system who, in turn, print the stack traces of their currently running tasks.
 
BUG: soft lockup - CPU#6 stuck for 73s! [flush-253:0:1207]
BUG: soft lockup - CPU#7 stuck for 74s! [processname:15706]
BUG: soft lockup - CPU#5 stuck for 63s! [processname:25582]
BUG: soft lockup - CPU#0 stuck for 64s! [proceessname:15789]

Comments

Post a Comment

Popular posts from this blog

How to fix Oracle SQL Developer connection issue "Got minus one from a read call"

Image
 During connection from SQL Developer to EBS database got below error message: Status : Failure - Test failed: IO Error: Got minus one from a read call Following solution can be done : 1. Go to directory $ORACLE_HOME / network / admin 2. Modify sqlnet.ora file with following parameter : tcp.validnode_checking = no 3. If you don 't want to disable this, you can put the machine names as follows: tcp.invited_nodes=(machine1, machine2) 3. Bounce the listener. Due to security enhancements it is not good to put tcp.validnode_checking = no or comment out will through error when we start listener like below: Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=shsebs7idb1.appsolworld.com)(PORT=1522))) TNS-00584: Valid node checking configuration error TNS-12560: TNS:protocol adapter error Better plan is to add localhost or IP address of client (Local Computer) in the sqlnet.ora file That will solve the problem NAMES.DIRECTORY_PATH=(TNSNAMES, ONAMES, HOSTNAME) SQLNET.EXPI
Read more

How to troubleshoot Long Running Concurrent Request in EBS 12.2

 The following points need to answer before jumping into troubleshooting: Are there any recent code changes done in the concurrent program?           Check long_running_sid and verify the recent code change of last_ddl_timestamp after a             discussion with the developer. Was this running long in the last few run as well, or was this time only?            ==> Check the program history. How much time does it take to complete?           Once again program history can give us some idea. Are these jobs fetching higher data compared to the last run           ===> Confirm on data change . Query Plan change will indicate this issue. Does this job runs at any specific time or it can be run anytime           ===> The nature of the job can help to understand the job's nature. Does this job fetch data using DB Link          ====> Check whether the DB link is involved or not. The below-given approach would help to conclude the gray areas. Check the load of the server using t
Read more

Few Important steps of Oracle Database Clone

 Make Sure oraInst.loc location is correct and accessble. [demantra@dsiddem ~]$ cat /etc/oraInst.loc inventory_loc=/orastage/oraInventory inst_group=dba [demantra@dsiddem ~]$ cd /orastage/oraInventory [demantra@dsiddem oraInventory]$ ls -ltr total 16 drwxrwx---. 5 demantra dba 4096 Jul 22 19:37 backup drwxrwx---. 2 demantra dba 4096 Jul 22 19:37 ContentsXML drwxrwx---. 2 demantra dba 4096 Jul 22 19:37 locks drwxrwx---. 3 demantra dba 4096 Jul 23 10:08 logs Remove Oracle Home from Inventory [demantra@dsiddem oraInventory]$ cd $ORACLE_HOME/oui/bin [demantra@dsiddem bin]$ ./runInstaller -detachHome ORACLE_HOME=$ORACLE_HOME Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB.   Actual 3039 MB    Passed The inventory pointer is located at /etc/oraInst.loc Run Clone.pl [demantra@dsiddem bin]$ echo $ORACLE_HOME /dnbsid/demantra/oracle/19.0.0 [demantra@dsiddem bin]$ echo $ORACLE_BASE /dnbsid/demantra [demantra@dsiddem bin]$ E01=ORACLE_HOME=$ORACLE_HOME [dema
Read more