I would like to redirect bad request to the IdP to an authentication page instead of the 500 error page and I would like to know how to achieve it.
Mainly, Traditional Authentication Scheme differs from SAML 2.0 Authentication Scheme as the authentication is done basically with username and password by Traditional Authentication Scheme and with an Assertion by the SAML 2.0 Authentication Scheme.
Traditional Authentication Scheme is configured on a realm. Requesting a resource protected by this realm, then the request goes to the Authentication Scheme to get credentials. The realm resources protected by SiteMinder will add in the request all information to bring the user to the right place after authentication and authorization process.
This scheme is located on the Web Agent.
SAML 2.0 authentication scheme
Requesting a resource protected by this realm at SP, request goes first to the AuthnRequest server will all needed information at SP which is redirected to the IdP to get an SAML 2.0 Assertion. The IdP returns the Assertion to the SP Assertion Consumer Service. The SP Assertion Consumer Service (Web Agent) pass the Assertion to the Policy Server. The Policy Server then request the SAML 2.0 Authentication Scheme as it passes the Assertion as credentials. Authentication goes in a disambiguation process that could involve according to the selected method, signatures verifications among the others.
This scheme is located on the Policy Server
By the error you see:
When you hit a resource @ SP side protected by a SAML 2.0 auth scheme the web agent log message of the form:
"User is trying to access a resource protected with federation auth scheme without fed auth scheme credentials. No way to challenge the user"
And the web server gives a 500 error.
This error is thrown because there is no assertion to handle in the call received at the SP Assertion Consumer Service. Somehow, the request bypassed the AuthnRequest and Idp to get the Assertion.
As told before, you need to configure hard-coded link to the AuthnRequest service and this explains what you may see:
I would have thought it would generate a redirect to the AuthnRequest service but it doesn't. I thought this was a supported use case but if it is I am either missing a final bit of set-up (physical, e.g. agent plug-in, or logical - policy/rule etc.)
You must manually configure that link and you probably did it already.
So each protected realm resources will send the user to the AuthnRequest Service. That is the only way to do it.
For the cases where the user request directly the Assertion Consumer Services or other type of resource, there is no way out-of-the-box to send them but configuring this parameter ServerErrorFile with a page that will send the information to the AuthnRequest. But as the initial target was a service only, you may choose a generic page where to send the user and then protect that page.