Install the 10.5.1. HF 21 patch :
10.5.1 HF 21 contains caching of EEM data on APM side. The first login might take 20 seconds because APM is fetching data for that user. The second and following logins are using the new cache and the speed of the login should be same as it used to be before 10.5.
Once user is logged in for the first time, the cache is periodically on background updating the users authorization data from EEM/LDAP to have them up to date.