Jobs remain in QUEUED status and "ORA-00001: unique constraint SO_JOB_QUEUE_IDX violated" error displayed

Document ID : KB000084882
Last Modified Date : 14/04/2018
Show Technical Document Details
Issue:
Error Message :
ErrorMsg: AwE-5001 Database Query Error
ava.sql.SQLException: ORA-00001: unique constraint (AWPD.SO_JOB_QUEUE_IDX) violated

Agents initially not connecting and a “java.net.ConnectException: Connection refused” error is displayed.  The Applications Manager (AM) processes were stopped and restarted including both the Master and Remote Agent processes.  After all the Remote Agent connections were restored all newly launched jobs remain in a "QUEUED" status.

Investigation

The following error messages can be seen in the $AW_HOME/log/RmiServer log.
 
08:37:06.617 Timer-3: .AxOptions: NoErrorMsgProperties=false
ErrorMsg: AwE-5001 Database Query Error (12/21/16 8:37 AM)
Details: aw5.process_state
0 name: IN:VARCHAR2:java.lang.String:PRODCORP
1 progname: IN:VARCHAR2:java.lang.String:RMIMASTER
2 sessionId: IN:NUMERIC:java.math.BigDecimal:9750571
3 command: IN:OUT:VARCHAR2:java.lang.String:HEARTBEAT
4 cpuUsage: IN:OUT:NUMERIC:java.lang.Integer:0
jdbc:oracle:thin:@dc1rh606.clubcorp.com:1551:POPS:AWPD
java.sql.SQLException: ORA-00001: unique constraint (AWPD.SO_JOB_QUEUE_IDX) violated

We are still researching the cause of the unique constraint violation. If you should face this issue please contact Automic Support for assistance.
 
 
Cause:
Cause type:
Other
Root Cause: The cause of this unique constraint on jobid is still under investigation.
Resolution:
The following information may be beneficial to further troubleshoot the issue. Please ensure the following table backups are kept from the time when the issue occurs.
 
1. Backup of the so_job_queue table
2. Backup of the aw_job_queue_activity table
3. Backup of the so_job_history


Fix Status: No Fix

Fix Version(s):
N/A
Additional Information:
Workaround :
Please contact Automic Support for assistance.