The most surefire way to understand why a particular authentication scheme was invoked for a given request is to study the Web Agent logs. The error log is useful to view the ACO parameters, particularly the AgentName and DefaultAgentName parameters. The Web Agent trace log will show the incoming request, including the resolved AgentName and URI (Resolved URL). In a small environment this may be sufficient information to find the Realm via the Admin UI and confirm the authentication scheme. In larger environments this may be too tedious to perform in a manual fashion, in which case there are two other choices:
The policy server trace log can be used to determine the Realm to which the request mapped (be sure that Thread ID (tid) is part of the policy server trace config format as this helps greatly with separating individual requests from the log). The Realm can then be examined in the Admin UI.
The third way the Realm can be determined is by using the SM Test Tool (Windows only). You will need the resolved AgentName and URI from the Web Agent trace log. When you click the IsProtected button in the Test Tool, the Realm to which the request resolved will be displayed. The Realm can then be examined in the Admin UI.