What is the Microsoft iSCSI initiator target limit?

Well, I've asked in the freebsd-stable@ mailing list, and after analysing the packet trace, here's what the developer of the FreeBSD Kernel iSCSI target told me:

Thanks for the packet trace. What happens there is that the Windows initiator logs in, requests Discovery ("SendTargets=All"), receives the list of targets, as expected, and then... sends "SendTargets=All" again, instead of logging off. This results in ctld(8) dropping the session. The initiator then starts the Discovery session again, but this time it only logs in and then out, without actually requesting the target list.

Perhaps you could work around this by using "discovery-filter", as documented in ctl.conf(5)?

I followed his advice, limited the discovery to only the targets that are accessible from the given initiator and the problem is gone.

So it seems to be some limitation in the MSFT iSCSi initiator, that is triggered on the discovery stage. Since the MSFT iSCSI Target has the limitation of 512 targets, its seems only understandable that the initiator starts to misbehave around the number of 573 targets.