You still get a popup, because the use case differs than the one from documentation. Your use case is that the IIS cannot authenticate the User when you have configured Windows Authentication Scheme. But the one from documentation is related when user gets authenticated at the IIS level with Windows Autentication Scheme, but the Policy Server cannot authenticate it.
From the documentation, the configuration of onauthreject and onauthusernotfound
is related to the following use case :
"After successful authentication at the Windows level (the SPS library),
the Policy Server fails to find the user in the User Store, and as such,
the Web Agent will ask again and again the Windows credentials. The request
will go in loop. The browser page will be blank and no popup occurs. The
message "Page cannot be displayed" will be shown in the browser when you stop
manually this loop."
And it's to prevent that loop that the note has been added in the documentation.