When a target that POSTs to the standard login.fcc is set in an HTML Form authentication scheme, and FCCCompatMode is set to yes, the SMTRYNO cookie is not set.
This behavior is as designed. When FCCCompatMode is set to yes, it will stop the SMTRYNO cookie from being created.
The SMTRYNO cookie is not set when this old 4.x compatibility mode setting is enabled.
The SMTRYNO cookie is set only when form credentials are required and to check if form credentials are required or not, the agent acting as credential collector needs to know realm credentials (RealmCredentials). If FCCCompatMode=YES, RealmCredentials is not populated with value from the Policy Server because call to IsProtected() is skipped. So the value of RealmCredentials is fetched from query string via the "TYPE" query parameter.
As a workaround, if you want to have SMTRYNO cookie set with FCCCompatMode=yes, then you need to customized the login page that performs a post to login.fcc, such that it passes all the query parameters (it received) to the login.fcc.