How NTLM/Windows Authentication works?

Document ID : KB000053320
Last Modified Date : 14/02/2018
Show Technical Document Details

Description:

Requesting information on how Siteminder integrates with Windows Authentication/NTLM.

Solution:

The most important thing to know is that the SiteMinder Policy Server does not validate any credentials during NTLM or Windows Authentication; it trusts the IIS web server to validate the credentials.

The process is pretty much as follows:

The old NTLM and newer Windows Authentication are closed, Microsoft proprietary technology, officially it only works on IE browser and IIS Web server (although the open source community has reverse engineered the protocol and gotten it to work on Netscape browser and Apache).

When a user accesses a resource on any type of web server (it can be Apache, SUN, IIS, etc) protected by the SiteMinder NTLM or (as of SM6 sp1) Windows Authentication scheme, the SiteMinder Policy Server returns a credential collector redirect URL to the web agent that must use the FQDN of an IIS web server, with a URI of /siteminderagent/ntlm/creds.ntc. The web agent then performs the redirect to the IIS Web server, which must be configured for NTLM/Windows authentication when the/siteminderagent/ntlm virtual directory is accessed. The MS IIS web server then sends the IE browser a request for the user's credentials (user name and password or Kerberos ticket). The MS IE browser communicates with the MS Windows OS to get the current user's desktop login credentials, encrypts them and sends them back to the MS IIS web server. The IIS web server decrypts the creds, then uses them to login to the MS NT Domain Controller (Active Directory nowadays) and declares the user authenticated if the login is successful. As I mentioned before, Microsoft does not share any APIs or any other means for other company's technology to play any part in this operation, without resorting to reverse engineering Microsoft protocols. Once IIS has authenticated the user, control finally passes to the SiteMinder Web Agent which extracts only the user's ID (in the form domain\loginID) and passes it to the policy server for "Authentication". The policy server doesn't really do full authentication. It disambiguates the user in a user store and then just declares the user authenticated, having trusted IIS to actually verify credentials. The Web agent then sets an SMSESSION cookie, and sets SM_USER to domain\loginID, then redirects back to the user's original target.

Prior to SM 6.0 sp1, the OOTB NTLM auth scheme was only available for on the Windows version of the policy server and you had to disambiguate using domain\login ID. A module named Extended NTLM Authentication was available from GSE which would run on Solaris and Windows, and which would allow disambiguation to any SiteMinder user store using just the loginID, without the domain\ prefix.

As of SM 6.0 sp1, the new built-in Windows Authentication scheme is quite flexible and allows you to do what the solution module did. It runs on all supported policy server platforms, can disambiguate against AD or any other LDAP user store, and has a flexible, macro based lookup that doesn't use the user store start/end fields. The auth scheme allows you to create your own lookup filter using %{UID} and %{DOMAIN} as macros (the keywords are case sensitive by the way), so you can create a simple lookup like (sAMMAccountname=%{UID}) or (uid=%{UID}) or more complex disambiguation filters.

As I mentioned before, SM_USER is set to domain\loginID (for plain cert based authentication SM_USER is empty). If the customer needs SM_USER to contain just loginID then they will need to create a new header and use the custom header instead of SM_USER. Alternatively, the customer can obtain the SmOverrideAuth GSE Catalog Item to automatically re-authenticate the user and reset SM_USER to any user attribute in the process.