How do I debug ENQ (ENQUEUE) lockout problems with the TMC/Audit?
CA1 Release 11.5-14.0
If you have a task that is waiting on the TMC or Audit file and is not able to continue processing you may have an ENQ (Enqueue) lockout. To diagnose the problem it needs to be determined what task is holding the resource and why it is not being released (for example, the holding task is waiting for another resource, or the holding task is swapped out). This document is to help you determine what task holds the resource.
1. Always do a "D GRS,C" on all systems. This will show any contention for resources.
a. If the display shows MIMGR as the holder of the enqueue for the TMC or Audit then issue the following command:
F MIMGR,DISPLAY CONFLICTS This should show who is the true holder of the enqueue.
2. Determine the devices on which the TMC and Audit are allocated.
a. Do a LISTCAT to determine which volumes contain the TMC and Audit files.
b. "D U, VOL=xxxxxx" where xxxxxx is the volume of the TMC and again for the Audit. This will give you the device address for each volume.
3. Determine which system has a reserve on the TMC (or the AUDIT file).
a. "D U,,,nnnn,1" on all systems. One system will show it as A -R, (indicates a reserve) all others will show A-BSY-P or "A" or "O".
b. On the system where it shows A -R, issue the command "D,,ALLOC,nnnn,1" this will show what tasks are allocated to the device.
4. Do another "D GRS,C" on the system that holds the reserve and look for the TOP entry after TMSQNAME TMS-TMC (or TMSQNAME TMS-AUD).
The TOP entry should have "OWN" under the STATUS column and all other behind it will have WAIT as their status. This TOP entry on this system
is the one that is causing the problems. Now, find out what that job is waiting for. It might be in the CONTENTION display waiting for another
resource (the Audit file for example). In which case you start the entire process over again and find out why it can't get the Audit file. It might be
that a different job on a different system has the Audit file "locked" and is waiting on something else. Then that job is the real problem.
5. If the holder of the enqueue is MIMGR than issue the command F MIMGR.DISPLAY CONFLICTS to see who is the real holder on the enqueue.
6. Another command that can be helpful is D GRS,RES=(TMSQNAME,*) to show all of the tasks either holding or waiting on a TMSQNAME resource.
If you want to show one specific resource you can enter:
The most common causes however, are that the job that OWNS the TMS-TMC is either HSM or CATALOG. If it is HSM, you will probably have to shutdown and restart HSM. First however, look at the TCB address of HSM that OWNS the TMC or Audit file and see if you can find that same TCB waiting for a different resource. In particular, look for it under SYSZTIOT. If it is there, then you will need to have HSM shutdown and restarted. In this case, gather the SYSLOG from this system from about 30 minutes before the problem started until HSM is cancelled and everything clears out.
If the job that OWNS the TMC or AUDIT file is CATALOG, there are a few more options.
a. "F CATALOG,END(yy)" where yy is the 2-character task id. To obtain the task id issue a F CATALOG, LIST command. If this does not clear up the problem proceed to step 2.
b. "F CATALOG,ABEND(yyyyyyyy)" where yyyyyyy is the 8-character TCBADDR as in step 1. If this does not clear up the problem, proceed to step 3.
c. "F CATALOG,ABEND(yyyyyyyy),FORCE" If this does not clear up the problems then go to step 4.
d. "F CATALOG,RESTART" which should cause the catalog address space itself to be terminated and restarted automatically.
IEE457I 08.06.54 UNIT STATUS 595
UNIT TYPE STATUS VOLSER VOLSTATE
8720 3390 A-R TMSVOL PRIV/RSDNT
IEE106I 08.07.00 UNITS ALLOCATED
UNIT JOBNAME ASID JOBNAME ASID JOBNAME ASID
8720 *MASTER* 0000 BATCHJOB 0006 LLA 0016
8720 HSM 006F CTS 0079
S=SYSTEM TMSQNAME TMS-AUD
SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS CPUA
BATCHJOB 0175 008E36B0 EXCLUSIVE OWN
CPUA TMSXTEND 0153 008E36C8 EXCLUSIVE WAIT
CPUA TMSRINIT 027E 008E36C8 EXCLUSIVE WAIT