Alert with text pass to a shell command returns Bad Process Exit Value "2"

Document ID : KB000103736
Last Modified Date : 28/06/2018
Show Technical Document Details
Question:
Upgrading from APM 9.6 to  10.3. we are using a lot of alerts where we pass a text to a shell command. These shell commands are perl scripts. The scripts work fine with 9.6. In 10.3 I get an error:
6/25/18 12:07:36.157 PM EDT [ERROR] [Alarm Pooled Worker] [Manager.Action] Action "ZZZ Lookup Action and Alert" failed to execute shell command "/export/appl/wilyca/mom/scripts/wilyAlertWas23-test.pl" with bad process exit value "2"

I tried finding what exit value "2" means but nothing is in the documentation
Everything has been duplicated 1:1 from the old system to the new. Any idea what exit value 2 indicates? I can run the same script from the shell no problem
Environment:
All APM releases
Answer:
https://docops.ca.com/ca-apm/10-7/en/ca-apm-release-notes/known-issues/enterprise-manager-and-introscope-known-issues 

Talks about one  possible short-term solution
Edit the IntroscopeEnterpriseManager.properties file and change the value of introscope.enterprisemanager.maxConcurrentActions from the default of 5, which allows too many processes to compete for the shell command, to 1: 

http://tldp.org/LDP/abs/html/exitcodes.html  talks about exit code 2 for a bash shell

There were several cases about exit code 2.  It appears that invalid syntax/unexpected characters are being used/sent.. And that is generating the error. The shell or perl in debug could tell you more. 

Here is from the other case. Not perl but same approach 

echo "%1" >> C:\mytest\simpleshellcmd.log 

However, when you execute the action, the following error appears: 

3/05/15 05:00:00 AM CET [ERROR] [Manager.Action] Action "WoW-Send msg" failed to execute shell command "date" with bad process exit value "2" 

This error is due to the double quotes ("%1") in the script. The action should look like this instead:

echo %1 >> C:\mytest\simpleshellcmd.log 

So, it is some sort of parameter issue usually

Actual Resolution:
By setting the MOM to debug mode, we saw there was a perl module missing. After installing the perl module under the wily user, then the error went away.