Or: "Is this a thing? And how would I check if it was?"

In an environment without a Domain Controller, when accessing a share on a Windows Server 2008 R2 box, from a remote computer without a matching user account on the server, (and connecting by typing \\SERVERNAME\ShareName from the Start menu) I currently observe the following behavior based on the "Password protected sharing" setting (Advanced sharing settings):

When "Password protected sharing" is turned on, all attempted connections fail after up to 30 seconds with:

Logon failure: the user has not been granted the requested logon type at this computer.

With "Password protected sharing" turned off, connections to anonymous-accessible shares are allowed, while permission-restricted shares fail with:

You do not have permission to access \SERVERNAME\ShareName. Contact your network administrator to request access.

This seems to be expected behavior. I need to have certain shares accessible by anonymous logons, so I had to change this setting from the default to off.

HOWEVER, there is a third case here. (whaaaaat?)

If you try to connect to a share without having modified this setting (that is, it is set to on but you've never clicked it), the connection behaves similar to the on case above in that it takes up to 30 seconds to show a response, but then it displays an authentication dialog:

I had this hunch after beating my head against a wall for a few days, and just replicated this on a server with no existing shares: Create anon-read share, try to connect and get dialog, change the setting, connect successfully, change setting back, and get a different error message. (Tested all of these on fresh client systems so there was no risk of caching.)

To reiterate: I have controlled for the client systems. This appears to be entirely server-related.

So, it's clear to me that changing the "Password protected sharing" setting is changing more than one thing (registry key? I'm Mac-native) behind the scenes, and that the default settings the system ships with do NOT all match up with the setting reflected in the control panel (or the control panel itself is broken and should be changing more things).

So the question is: is this by design, or is it a bug? And in either case, what is the "hidden setting" that is being changed or left unchanged? How would one track that down? I'm running out of fresh servers to test on. :-(

This really piqued my interest. I was able to replicate your findings in my lab with the same pattern of results that you describe. I used Procmon to to try to see what changes are made and almost gave up until I saw the following:

That shows lsass.exe (Local Security Authority) writing to the local SAM and making a change(s) to the built-in Guest account (well-known RID 501). Sure enough, when I retested your scenario while watching the Guest account's status, I see it enabled when "Password protected sharing" is disabled. However, when "Password protected sharing" is re-enabled, the guest account is not disabled again. Manually disable the guest account restores the original functionality: I am prompted for credentials (i.e. your third case).

I'm not sure why this behaves like this. TO be honest, I'd never even toggled the "Password protected sharing" setting before today (or even noticed it, for that matter). I hope this helps with your project. If someone else is interested in digging further, it would be interesting to know if this behavior is still present on Server 2012/2012 R2...

Oh and to your original questions (Is this by design or is it a bug?), I haven't the slightest idea...

If I've understood your question correctly, then the shares' credentials is saved in the Credential Manager under the control panel.

In order to prompt the authentication dialog box, simply delete the credential relating to that share under the credential Manager.

When you check the 'Remember my credentials', this is usually saved under the Credential Manager and if this password was wrong then you would see the logon failure error.