Data_engine restart loop, log shows error: Open: Too many connections (errno:1040)

Document ID : KB000033597
Last Modified Date : 14/02/2018
Show Technical Document Details

Symptoms: 

Data_engine restarts continuously and eventually turns red (goes into an error state). You will observe the following error in the logs:
 

error: Open: Too many connections (errno:1040) 


Eventually the probe will either show Error, or appear to be in a reboot loop showing a new port and /or PID, depending on how quickly it restarts.

Environment:  

MySQL 5.x database supporting UIM 7.x or higher

 

Cause: 

Connection limit defaults on MySQL are often too low for enterprise-level applications. The data_engine opens numerous connections to facilitate simultaneous inserts, reading and maintenance of data.

This problem is exacerbated by setting up data_engine multi-threading (by setting keys such as thread_count_insert = 3 and queue_limit_total = 100000); the Primary Hub can open enough connections to nearly saturate MySQL. Combined with other robots opening database connections (UMP especially) then the connection limit will be hit and data_engine will be impacted.

 

Resolution:

Edit the MySQL startup configuration file (my.cnf) and increase the max_connections.

1. This can be found in different places depending on the installation method of mysql. For reference, default options are read from the following files in the given order: 

/etc/my.cnf 
/etc/mysql/my.cnf 
/usr/etc/my.cnf 
~/.my.cnf 

Or, on Windows: 

C:\Windows\my.ini 
C:\Windows\my.cnf 
C:\my.ini 
C:\my.cnf 
C:\Program Files\MySQL\MySQL Server 5.6my.ini 
C:\Program Files\MySQL\MySQL Server 5.6\my.cnf 
C:\ProgramData\MySQL Server 5.6\my.cnf

2. Next, change the value of max_connections to a higher value.  500 is a good place to start but it will vary depending on the environment and the amount of data being collected. You may need to increase this even further until you find a level that eliminates the error message.


3. Finally, restart data_engine. If other probes such as nis_server, udm, discovery, and wasp are encountering issues, restart those probes (or the entire robot) which should allow the probes to retry the mysql connection once again.


Additional Information:

See your MySQL user manual for more information about connection limits.