OPMS showing some services failed and not running after reboot

Document ID : KB000072194
Last Modified Date : 26/02/2018
Show Technical Document Details
Question:
    We have the version 10 OPMS installation running on a Linux host. After running out of disk space and rebooting several times, the jmeter-docker and httpbroker services are no longer running.
    Our jmeter tests are now failing and getting an "-95" error. How do we fix this?

  Here's the monit summary:

[root@myhost]# monit summary
The Monit daemon 5.14 uptime: 19m

Process 'Xvfb'                      Running
Process 'tunnel-client'          Running
System 'ushq1tstrwily06'      Running
Process 'api'                        Not monitored
Process 'httpbroker'             Execution failed
Process 'assetsmanager'     Running
Process 'resultbroker'           Running
Process 'fpm'                        Running
Process 'redis-server'           Running
Program 'rbtm1'                    Status ok
Process 'rbtm1-Xvfb'            Running
Program 'rbtm2'                     Status ok
Process 'rbtm2-Xvfb'                Running
Process 'php-fpm'                   Running
Process 'nginx'                     Running
Program 'jmeter-docker'             Status failed
Filesystem 'devshmfs_warning'       Accessible
Filesystem 'devshmfs_critical'      Accessible
Filesystem 'optfs_warning'          Accessible
Filesystem 'optfs_critical'         Accessible
Filesystem 'rootfs_warning'         Accessible
Filesystem 'rootfs_critical'        Accessible
Filesystem 'tmpfs_warning'          Accessible
Filesystem 'tmpfs_critical'         Accessible
Filesystem 'varfs_warning'          Accessible
Filesystem 'varfs_critical'         Accessible
Process 'fcgiwrap'                  Running
Process 'exim4'                     Running
Process 'bm_proxy'                  Running
Answer:
  smartpop-httpbroker.log was showing this error.

ERROR   agent.httpbroker [MainThread] An unexpected exception was encountered. HTTP broker is terminating.
Traceback (most recent call last):
  File "httpbroker.py", line 501, in main
    gevent.signal(signal.SIGINT, sys.exit, 1)
  File "/usr/lib64/python2.7/site-packages/gevent/hub.py", line 102, in __init__
    self.hub = get_hub()
  File "/usr/lib64/python2.7/site-packages/gevent/hub.py", line 169, in get_hub
    hub = _threadlocal.hub = hubtype(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/gevent/hub.py", line 268, in __init__
    loop_class = _import(self.loop_class)
  File "/usr/lib64/python2.7/site-packages/gevent/hub.py", line 198, in _import
    return _import(path[-1])
  File "/usr/lib64/python2.7/site-packages/gevent/hub.py", line 210, in _import
    x = __import__(module)
  File "core.pyx", line 10, in init gevent.core (gevent/gevent.core.c:41156)
  File "/usr/lib64/python2.7/site-packages/gevent/signal.py", line 19, in <module>
    from gevent._util import _NONE as _INITIAL
ImportError: cannot import name _NONE

  The issue was caused by newer version of the python-gevent module (1.2.2) that was installed manually system-wide and not by the OPMS installer. The smartpop components are apparently not compatible with it. Reverting to the package version (1.0), which is installed by the OPMS installer, fixed the issue.

  Perform the following steps:
1. [root@myhost]# yum remove python-gevent
2. [root@myhost]# pip uninstall gevent
3. Reinstall the OPMS