How Do I Fully Flush Cached Redirects From Safari?

I have a device with a web-based control panel, and accidentally set it to redirect all http pages to https, even though some don't work over https. Although I've since corrected this, Safari seems to have memorised the redirect and is refusing to forget it, instead constantly attempting to redirect me to the invalid https address.

I've already closed Safari, cleared ~/Library/Caches/com.apple.Safari/ and ~/Library/Cookies/HSTS.plist but it still seems to be remembering the redirect when I re-open it.

Where else could Safari be storing this information? I can access the correct page via Firefox or Chrome, so it may not be a system wide service, or if it is it's not one that the other browsers use.

Unfortunately because the web-panel is provided by a device I don't believe I can adjust headers or setup a redirect back to the correct URL, which seem to be options offered in other similar questions, so I really need to find out where this data is being stored so I can destroy it with fire.

Solution 1:

Based on quanta’s answer:

I wasn’t able to use launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist because I have System Integrity Protection enabled:

$ launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist
/System/Library/LaunchAgents/com.apple.nsurlstoraged.plist: Operation not permitted while System Integrity Protection is engaged

However I was able to work around it by doing the following:

• killall nsurlstoraged (stops your user’s nsurlstoraged process; I actually ran sudo killall nsurlstoraged, but I suspect it isn’t necessary to stop the system’s nsurlstoraged as well, since the cache is in the user Library folder)
• rm -f ~/Library/Cookies/HSTS.plist (deletes the HSTS cache)
• launchctl start /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist (restarts nsurlstoraged)

Solution 2:

If you enable Develop menu in Safari preferences, you can clear cache from there (CMD+ALT+E).

Can you confirm that opening the device's control panel in Safari's Private window (or different web browser) works correctly?

Solution 3:

Based on @Haravikk's answer: https://apple.stackexchange.com/a/267783/62907

Anyone have any ideas which process is responsible for the ~/Library/Cookies/HSTS.plist file?

fs_usage can help:

❯❯❯❯ sudo fs_usage | grep HSTS
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000238   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000009   nsurlstorage
16:11:03  open              /Users/quanta/Library/Cookies/HSTS.plist                                         0.016268   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000008   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000003   nsurlstorage
16:11:03  access            /Users/quanta/Library/Cookies/HSTS.plist                                         0.000011   dbfseventsd
16:11:04  lstat64           /Users/quanta/Library/Cookies/HSTS.plist                                         0.000008   fseventsd
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000006   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000002   nsurlstorage
16:11:08  open              /Users/quanta/Library/Cookies/HSTS.plist                                         0.000144   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000002   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000003   nsurlstorage
16:11:08  access            /Users/quanta/Library/Cookies/HSTS.plist                                         0.000021   dbfseventsd
16:11:09  lstat64           /Users/quanta/Library/Cookies/HSTS.plist                                         0.000042   fseventsd

So, we can:

launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist

then:

rm -f ~/Library/Cookies/HSTS.plist

and try again.

Solution 4:

In Safari, Firefox and Chrome as well, all you have to do is open the developer sidebar, select the network tab, and disable caching.

In Safari that's a crossed out tube thing, the blue one next to the trash bin logo. Activate that, and the old permanent redirect should be ignored.

Greatest benefit is that you don't have to mess with files, you won't delete all HTST entries and loose the security benefits. Also it works across browsers.

Solution 5:

So I've found a workaround to the problem, though this isn't a definitive answer to the actual question so I will not mark it as such until I can find more information.

It turns out that the file ~/Library/Cookies/HSTS.plist was indeed the source of the problem as I suspected, however deleting it from the affected user account doesn't work, even with Safari closed, as it is recreated after an unknown amount of time, complete with the offending entry that was forcing the invalid redirect.

So my solution was the following:

1. Make sure you have at least one other user account on your Mac (if not, create one).
2. Logout of the affected user account.
3. Login to a different user account (a guest account may not be sufficient, depending upon restrictions).
4. Find out the short-name of your affected user account; if you don't know then the best way to check is to look under System Preferences -> Users. Usually if will be the full name, lower-cased and with no spaces, so if your full name is "John Smith" then the short-name may be "johnsmith".
5. Open a window in Terminal, type su shortname replacing "shortname" with the short-name of the affected user-account. Hit enter and, when prompted, enter the password for the affected account.
6. Now type the next command rm ~/Library/Cookies/HSTS.plist and hit enter, this will delete the HSTS storage file.
7. Finally type exit, hit enter and close Terminal.

At this point you can now log back into the affected user account and the offending HSTS redirect should be gone for good.

Now, while this provides a usable workaround, I'd really like to know why deleting the HSTS.plist file from my affected account didn't work; the fact it is recreated means some background process is responsible for it, which means it should be possible to delete the file from the affected user account by simply stopping that process, deleting the file, then relaunching the process.

Anyone have any ideas which process is responsible for the ~/Library/Cookies/HSTS.plist file? Once we know that it should be possible to give a simpler fix to the problem.