iPXE to iSCSI target - missing parameter and naming confusion

I have a pyTarget iSCSI target. I'm able to use iPXE to connect to the target and install Ubuntu. After installation, I can't boot into the image, however.

pyTarget config.xml:

<root>
    <!-- configure target console -->
    <console ip="" port="3260" debug_level="6" />

    <!-- configure the first target -->
    <target name="iscsi" ip="192.168.1.10" port="3260" portal="1">
        <!-- the first iqn of this target configured for disk array -->
        <host name="iqn.2006-11.1" target_pwd="" initiator_pwd="">
            <lun id="0" type="0" path="E:\pyTarget\ubuntu16.ima" capacity="81943040" />
        </host>
    </target>
</root>

pyTarget console at start-up:

INFO:   pyTarget 1.5.7.1 2010-12-26
INFO:   start iSCSI Service ...
INFO:   start iSNS Service ...
INFO:   initialize virtual device, please wait...
INFO:   start Http Service ...
DEBUG:  create target iscsi 192.168.1.10 3260 1
DEBUG:  create host iqn.2006-11.1
DEBUG:  target iscsi add host iqn.2006-11.1
DEBUG:  Disk(E:\pyTarget\ubuntu16.ima) : initialize finish!
DEBUG:  host iqn.2006-11.1 add lun 0.
DEBUG:  configure target done
INFO:   initialize finish, start to run ...

As install begins:

DEBUG:  create connect
INFO:   initiator 192.168.1.15:38822 connect.
NEGOTIATION:    SessionType = Discovery
NEGOTIATION:    TargetName = iscsi
NEGOTIATION:    InitiatorName = iqn.2006-11.1
DEBUG:  create session 00023d000000-0001.
DEBUG:  host iqn.2006-11.1 add session 00023d000000-0001.
INFO:   SessionID: 00023d000000-0001
DEBUG:  session 00023d000000-0001 add connect 0.
INFO:   ConnectID: 0
NEGOTIATION:    TargetPortalGroupTag = 1
NEGOTIATION:    HeaderDigest = None
NEGOTIATION:    DataDigest = None
NEGOTIATION:    MaxRecvDataSegmentLength = 32768
NEGOTIATION:    TargetMaxRecvdataLength = 65536
NEGOTIATION:    ErrorRecoveryLevel = 0
NEGOTIATION:    DefaultTime2Retain = 0
NEGOTIATION:    DefaultTime2Wait = 2
NEGOTIATION:    OFMarker = No
NEGOTIATION:    IFMarker = No
DEBUG:  connect 0 login finish
COMMAND:        Text Request
DEBUG:  session 00023d000000-0001 add connect 0.
DEBUG:  host iqn.2006-11.1 remove session 00023d000000-0001.
DEBUG:  destroy session 00023d000000-0001.
DEBUG:  connect 0 finish.
DEBUG:  create connect
DEBUG:  destroy connect 0
INFO:   initiator 192.168.1.15:38824 connect.
NEGOTIATION:    SessionType = Discovery
NEGOTIATION:    TargetName = iscsi
NEGOTIATION:    InitiatorName = iqn.2006-11.1
DEBUG:  create session 00023d000000-0002.
DEBUG:  host iqn.2006-11.1 add session 00023d000000-0002.
INFO:   SessionID: 00023d000000-0002
DEBUG:  session 00023d000000-0002 add connect 0.
INFO:   ConnectID: 0
NEGOTIATION:    TargetPortalGroupTag = 1
NEGOTIATION:    HeaderDigest = None
NEGOTIATION:    DataDigest = None
NEGOTIATION:    MaxRecvDataSegmentLength = 32768
NEGOTIATION:    TargetMaxRecvdataLength = 65536
NEGOTIATION:    ErrorRecoveryLevel = 0
NEGOTIATION:    DefaultTime2Retain = 0
NEGOTIATION:    DefaultTime2Wait = 2
NEGOTIATION:    OFMarker = No
NEGOTIATION:    IFMarker = No
DEBUG:  connect 0 login finish
COMMAND:        Text Request
DEBUG:  session 00023d000000-0002 add connect 0.
DEBUG:  host iqn.2006-11.1 remove session 00023d000000-0002.
DEBUG:  destroy session 00023d000000-0002.
DEBUG:  connect 0 finish.
DEBUG:  create connect
DEBUG:  destroy connect 0
INFO:   initiator 192.168.1.15:38826 connect.
NEGOTIATION:    SessionType = Normal
NEGOTIATION:    TargetName = iscsi
NEGOTIATION:    InitiatorName = iqn.2006-11.1
DEBUG:  create session 00023d010000-0003.
DEBUG:  host iqn.2006-11.1 add session 00023d010000-0003.
INFO:   SessionID: 00023d010000-0003
DEBUG:  session 00023d010000-0003 add connect 0.
INFO:   ConnectID: 0
NEGOTIATION:    TargetPortalGroupTag = 1
NEGOTIATION:    TargetAlias = DESKTOP-54T1C9D
NEGOTIATION:    HeaderDigest = None
NEGOTIATION:    DataDigest = None
NEGOTIATION:    MaxRecvDataSegmentLength = 65536
NEGOTIATION:    TargetMaxRecvdataLength = 65536
NEGOTIATION:    ErrorRecoveryLevel = 0
NEGOTIATION:    DefaultTime2Retain = 0
NEGOTIATION:    DefaultTime2Wait = 2
NEGOTIATION:    OFMarker = No
NEGOTIATION:    IFMarker = No
NEGOTIATION:    MaxConnections = 1
NEGOTIATION:    InitialR2T = Yes
NEGOTIATION:    ImmediateData = Yes
NEGOTIATION:    DataPDUInOrder = Yes
NEGOTIATION:    DataSequenceInOrder = Yes
NEGOTIATION:    MaxOutstandingR2T = 1
NEGOTIATION:    MaxBurstLength = 262144
NEGOTIATION:    FirstBurstLength = 65536
DEBUG:  connect 0 login finish
COMMAND:        SCSI Request
COMMAND:        Inquiry LUN:0
COMMAND:        SCSI Request
COMMAND:        Inquiry LUN:0
COMMAND:        SCSI Request

As install completes & I attempt to boot into the image:

COMMAND:        SCSI Request
COMMAND:        Write(10)  LBA:0x1c40800 LEN:0x8 LUN:0
COMMAND:        SCSI Request
COMMAND:        Write(10)  LBA:0x1c40800 LEN:0x8 LUN:0
COMMAND:        SCSI Request
COMMAND:        Write(10)  LBA:0x1c40800 LEN:0x8 LUN:0
COMMAND:        SCSI Request
COMMAND:        Write(10)  LBA:0x1c40800 LEN:0x8 LUN:0
COMMAND:        SCSI Request
COMMAND:        Write(10)  LBA:0x800 LEN:0x8 LUN:0
COMMAND:        SCSI Request
COMMAND:        Test Unit Ready LUN:0
COMMAND:        SCSI Request
COMMAND:        Test Unit Ready LUN:0
# Install ends here
# Client restarts, attempts to boot to the new install:
DEBUG:  create connect
INFO:   initiator 192.168.1.15:30733 connect.
NEGOTIATION:    SessionType = Normal
WARNING:        initiator name or target name is INVAILD iqn.2006-11.1
DEBUG:  connect 0 finish.

Dropping to iPXE shell and attempting manual sanboot, it spits out additional information - this error:

(Error code 3c0d61)
This error originated from one of the following locations within the iPXE source code:
net/tcp/iscsi.c (line 1226)
This error indicates an unexpected status received from the iSCSI target.

Wireshark has the following:

Client sends an iSCSI LOGIN Command to host, notable content:

Key/Value Pairs
    KeyValue: InitiatorName=iqn.2006-11.1
    KeyValue: TargetName=iqn.2006-11.1
    KeyValue: SessionType=Normal
    KeyValue: AuthMethod=CHAP,None

Host returns iSCSI LOGIN response, notable content:

Status: Missing parameter (0x0207)

This error code is discussed briefly on this mailing list but I am not savvy enough to understand its significance.

I don't understand how the target or initiator name can suddenly be invalid? Nor what parameter is missing. The connection was literally just in use and there were no complaints during the installation... I can even replicate this - starting the installation again is not a problem, and completes just fine (again). And after the reinstall, same problem when I try to boot!

The one thing I can notice, is that there is something odd about empty passwords in either iSCSI or PXE. During ubuntu install, in order to complete the iSCSI volume login, I have to press spacebar once - it doesn't accept NULL length password, even though password is essentially NULL in config.

But, it's not possible to input SPACE as a password in iPXE - sole empty space seems to be regarded as NULL. Setting initiator_pwd to not NULL doesn't appear to help, however.

I've been stuck at this point for hours now. Where do I go from here for more troubleshooting?

EDIT: Forgot to add:

iPXE config:

#!ipxe
set iscsi-target iscsi:192.168.1.10::::iqn.2006-11.1
set gateway 0.0.0.0
set initiator-iqn iqn.2007-11.1
set keep-san 1
sanboot ${iscsi-target}
boot

New config.xml

<!-- configure the first target -->
<target name="iscsi" ip="192.168.1.10" port="3260" portal="1">
    <!-- the first iqn of this target configured for disk array -->
    <host name="iqn.2006-11.1" target_pwd="1" initiator_pwd="1"> # <-- Set passwords to avoid NULL problem, this comment was not part of the actual config
        <lun id="0" type="0" path="E:\pyTarget\ubuntu16.ima" capacity="81943040" />
    </host>
</target>

Dropped iPXE shell after the script failed:

1. dhcp
2. login
   username: iscsi
   password: 1
3. sanboot iscsi:192.168.1.10::::iqn.2006-11.1

New packet capture:

iSCSI (Login Command)
    Opcode: Login Command (0x03)
    1... .... = T: Transit to next login stage
    .0.. .... = C: Text is complete
    .... 00.. = CSG: Security negotiation (0x0)
    .... ..01 = NSG: Operational negotiation (0x1)
    VersionMax: 0x00
    VersionMin: 0x00
    TotalAHSLength: 0x00
    DataSegmentLength: 93 (0x0000005d)
    ISID: 40002723b2f3
    TSIH: 0x0000
    InitiatorTaskTag: 0x18ae0002
    CID: 0x0000
    CmdSN: 0x00000000
    ExpStatSN: 0x00000001
    Key/Value Pairs
        KeyValue: InitiatorName=iqn.2007-11.1
        KeyValue: TargetName=iqn.2006-11.1
        KeyValue: SessionType=Normal
        KeyValue: AuthMethod=CHAP,None
    Padding: 000000

iSCSI (Login Response)
    Opcode: Login Response (0x23)
    0... .... = T: Stay in current login stage
    .0.. .... = C: Text is complete
    .... 00.. = CSG: Security negotiation (0x0)
    VersionMax: 0x00
    VersionActive: 0x00
    TotalAHSLength: 0x00
    DataSegmentLength: 0 (0x00000000)
    ISID: 40002723b2f3
    TSIH: 0x0000
    InitiatorTaskTag: 0x18ae0002
    StatSN: 0x00000001
    ExpCmdSN: 0x00000000
    MaxCmdSN: 0x00000000
    Status: Missing parameter (0x0207)

sanboot iscsi:192.168.1.10::::iscsi returns Status: Authentication failed (0x0201) in wireshark and corresponding Operation not permitted 410d61 in client console.

pyTarget console:

DEBUG:  create connect
DEBUG:  destroy connect 0
INFO:   initiator 192.168.1.15:1552 connect.
NEGOTIATION:    SessionType = Normal
NEGOTIATION:    TargetName = iscsi
WARNING:        InitiatorName %s do not exist. iqn.2007-11.1
DEBUG:  connect 0 finish.

Solution 1:

The last error message flipped the lightswitch. Embarassingly, I had the names backwards:

In iPXE shell:

set initiator-iqn iqn.2006-11.1
sanboot iscsi:192.168.1.10::::iscsi

Now I have boot...

To clarify, iPXE config works when it looks like this:

#!ipxe
set iscsi-target iscsi:192.168.1.10::::iscsi
set gateway 0.0.0.0
set initiator-iqn iqn.2006-11.1
set keep-san 1
sanboot ${iscsi-target}
boot