This error occurs due to one of the following reasons:
1. Hostname resolution
Make sure that you are able to ping the hostname of the Domain Manager MDB server. If there are DNS resolution issues, add an entry for the Domain Manager's MDB server in the hosts file of the Enterprise Server.
2. SQL Server Browser service disabled.
If the Domain Manager MDB is hosted on a named instance, make sure that the SQL Browser service is running.
3. Ports not open
The following ports 4104, 4105, 4728 and 1433 (SQL port) should be open bi-directional between the Enterprise Server and the Domain Manager.
These are communication ports and need to be open before adding a new domain.
Note: If SQL server is running on a non-default port, that port needs to be open.