iSCSI: LUNs per target?

Solution 1:

Best practice is to have multiple LUN's hanging off a single target. How many targets that is in total varies, as well as how many of those targets are actually exposing the same LUN's. I tend to recommend one target per MPIO path your clients will set up, so if I'm setting up the system there's going to be at least 2 targets (but they're just part of the same Target Group and Target Portal Groups and are offering up the same views - it's entirely for iSCSI MPIO).

The only reason to be going past 2 Targets is generally for logical separation brought on by business, network or other use-case specific pressures. 4, 8, and even up to 16 Targets is not that crazy, depending on the size and complexity of the environment. If you're going over 16, especially by a lot, it's increasingly possible you've got a mistake in the architecture and should engage an expert to review. If you're making use of iSCSI MPIO (and you must, because if you're not making use of iSCSI MPIO, probably the sole major win of zvols & iSCSI over filesystems & NFS is lost and there's now very little upside to going iSCSI over NFS, and absolutely tons of downsides), you should also generally have an even number of Targets (2, 4, 6, 8, etc), with each pair offering up the same views (LUN's).

Also bear in mind that COMSTAR is intelligent enough in how it handles views that as long as you actually create and assign Target Groups and Host Groups appropriately, you can have a single target offering up potentially dozens or even hundreds of LUN's where the actual LUN ID exposed to the incoming initiator (client) is NOT 578 or whatever, but instead could be as low as 0 for every single one. This is important when dealing with some client OS', such as Linux (especially older flavors), which have a nasty habit of auto-discovering and assigning devices for every LUN they found, AND have a maximum number of LUN's they can deal with seeing AND have a maximum 'LUN number' (the actual LUN number itself, as opposed to the quantity of LUN's seen), too.

I've never spent time quantifying it, but my gut reaction would be you're going to feel either the same amount of performance impact with 128 Targets each with 1 LUN as you'd feel with 1 Target with 128 LUN's, or slightly worse performance with the 128 Targets. Definitely the expectation by COMSTAR is you're setting up Targets with multiple LUN's, instead of 1 LUN per target as things used to be pre-COMSTAR days.

On the comment about your environment in general:

One should create a separate LUN for each VM only in situations where the total number of VM's is low (under 100, preferably under 50) OR where you have no form of high availability in mind that would require exporting/importing the pool in situations where services are offline while the pool is exporting/importing.

This has to do with how long it takes to export/import and fully initialize a bunch of LUN's via COMSTAR over ZFS. Each additional zvol adds X milliseconds to the time to import. Not a big deal with a couple dozen, but rapidly causing a noticeable issue as you scale up to the 1000's. I used to manage a Sun 7410 with over 3,000 zvols that took the better part of 15 minutes to do a failover using their clustering functionality (which essentially was exporting the pool from one node & importing it on the other when doing a manual failover, as was the case for the 15 minute monstrosity). This was due in large part to the number of datasets, and moreso that they were zvols. The exact same scenario repeated with 3,000 filesystems (not zvols) took only 5 minutes (still long, but 3x shorter already, and this was years ago. While the total time to complete these tasks has improved in ZFS in the years since, the relative difference of zvols to filesystems and time to fully import and be online remains, last I checked.

Now, if you do not have 2 or more nodes in your setup and are expecting to rapidly export & import between them, the largest reason not to have lots of zvols is gone. That still doesn't make 1,000's of them a particularly good idea, for a slew of other reasons not directly pertinent to your question. The boons you gain having them separate are offset somewhat by the administrative and performance pain induced by having tons of zvols. For my money, I highly recommend going NFS instead, even if (heck, especially if) you desire one dataset per VM.

Solution 2:

Mutiple targets increase your potential surface area. Really this depends on the number of links your switches have to your storage, and potentially how many are between the dedicated iSCSI switches you ought to be using.

It has to do with hashing and load-balancing, and the number of paths you want to see. Ideally if you have four links from your target device to the network you should have four IP addresses, and potentially a fifth used to "discover" targets.

So the only real answer is: it depends.