Open file descriptors not closed upon completion of PeopleSoft jobs

Document ID : KB000087872
Last Modified Date : 14/04/2018
Show Technical Document Details
Issue:
Error Message :
N/A

Symbolic links pointing the Applications Manager Agents output directory are generated for every PeopleSoft job that is run and remain there even after the job has finished.  The links will be cleared only after stopping and restarting the Agent where the PeopleSoft jobs ran. 
 
When looking at the number of open file descriptors in the /proc/fd directory, you will see the number of open file descriptors continuously growing.  If you look at one of the files you will see that the referenced link points to an output file in the $AW_HOME/out directory on the Agent where the PeopleSoft jobs are running.
 
>date
Fri  Oct 18 09:03:28  PST 2013
>ls -l 559
l-wx------ 1 saprd psoft 64 Oct 15 09:19 559 -> /home/AM8 /out/o31636297.00
>cd /home/AM8/out
>ls -l o31636297.00
-rw-r--r-- 1 saprd psoft 405 Oct 15 00:18 o31636297.00
 
When looking at the file itself, you may see that the output file is from a job that has finished days earlier.
>cat o31636297.00
Start time: 10/15/13 12:17 AM
Run ID: 31636297
Job: SASF_UF_SF_SI_LEG
Tue Oct 15 00:17:03 EDT 2013
Process name:UF_SF_SI_LEG
Type:Application Engine
Requested using UF_PROCSCHED
Run Control ID:UF_SF_SI_LEG_OUR_INCR
Tue Oct 15 00:17:03 EDT 2013 Process scheduled with instance:1692493
Tue Oct 15 00:17:03 EDT 2013: Job: 1692493 status:Queued(31)
Tue Oct 15 00:18:13 EDT 2013:Job final status: Success(32)

Investigation
 
In PeopleSoft we process the output separately from the job completion.  If the job does not have an output scan and output is not required (*1) then as soon as the job completes the job can drop to history and we retrieve and register the output in the background and update it to the job later.  In that case, we leave the output file open until that post-processing of the output completes.
 
What we found is that there was a change in 2011 for an OAE issue which affected the PeopleSoft job processing (*2).  The change affected the logic so that the system output file was getting closed only if there was an error retrieving the output.  In other words, it was not getting closed in most normal cases.  In addition, due to this issue, the job never got removed from the status monitor queue which would affect memory and performance.
 
*1. Output required is the ‘Abort if no output’ checkbox on the job definition.
 
*2. Applications Manager v8.0 SP7: #182698:  Fixed a problem where OAE tasks got stuck in a RUNNING status when the connection to the RmiServer process dropped during task completion.
 
Cause:
Cause type:
Defect
Root Cause: There was a change made in Applications Manager v8.0 SP7 for an OAE issue (#182698) which affected the PeopleSoft job processing. The change affected the logic so that the system output file was getting closed only if there was an error retrieving output
Resolution:
This field was added on 30/03/2017. This article has not been updated yet. Refer to the "Description" or "Workaround" sections for solution information.

Fix Status: Released

Fix Version(s):
Applications Manager 9.0.1 - Available
Applicatons Manager 8.0.16 - Available
Additional Information:
Workaround :
Stop and restart the Applications Manager Agent where the PeopleSoft jobs are run in order to clear the symbolic links.