Our embedded linux system won't recognize a USB Device if it is plugged in before powerup. Suggestions?

We are developing on a small embedded device. This device us a gumstix overo board running OpenEmbedded linux. We have our development almost completely done, and have run into the strangest of bugs that we can't figure out.

We have a USB Device (Spectrophotometer) that has a USB2.0 Connection and an external power supply for the light source. Typical behavior is that you plug in the power supply, then the USB connection to a host. When the usb connection is detected by the device, the device boots up and enables the light source and fan. The device is then able to be used by the host system.

The problem is that if the device is plugged into the Gumstix before we turn on the Gumstix, the USB Device apparently is not probed by the system (and hence does not turn on). Under a normal situation, when the connection is initialized by plugging in the usb cable, the spectro turns itself on and becomes available to the system (this can be seen via "lsusb" typically). Neither of these things are happening. There is no device detected via "lsusb" and no dmesg errors of any kind that we can see. It is as if the device is not plugged in.

The device does show up and work fine if we unplug the USB cable and plug it back in once the system is booted up. It turns on and shows up on the usb bus, and we can access it with our driver.

On any other desktop or laptop, it does not matter if the host system is on or off when we plug in the spectrometer. This behavior is what I would consider to be "normal" - that the usb system is probed and initialized at boot time, and the usb devices come online. In other words, our system is fully functional as long as we plug in the usb device after the system is booted up. Unfortunately this isn't possible in our final product - everything comes on at once.

Additional Info: 1) We have tried a flash drive attached to the system when the system is turned off. Booting up the system brings the flash drive online, as expected 2) There are no messages regarding the spectro or usb device (using dmesg). "lsusb" only lists the USB hubs / controllers. It is literally as if the device is not present and not plugged in. 3) We have tried a brand new image from gumstix and an older image from last year. Both images have this problem. This problem exists on all 3 gumstix devices we use.

Does anyone have any suggestions? From what I can tell it isn't really possible to do a complete "reboot" of the usb system that is a complete emulation of "unplugging" and "replugging" a usb device. I feel like what is happening is that there is no initial probe on the usb bus that would trigger the usb handshaking, but this is somehow specific to the spectro. This seems to be a kernel issue or at least an issue in how the kernel is initializing the usb subsystem. I'm not really sure though.

I have tried the gumstix mailing list, but there doesn't seem to be anyone who has seen this issue before. Any advice or suggestions on where to start looking would be fantastic.

Thank you! Blaine

output etc.
$ uname -a
Linux overo 2.6.33 #1 Tue Apr 27 08:35:38 PDT 2010 armv7l GNU/Linux

When the system is up and running and spectro is plugged in (working as intended), this is lsusb:
Bus 001 Device 116: ID 2457:1022  
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x2457 
  idProduct          0x1022 
  bcdDevice            0.02
  iManufacturer           1 USB4000 1.01.11
  iProduct                2 Ocean Optics USB4000
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           46
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              400mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           4
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x86  EP 6 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0000
  (Bus Powered)

dmesg output:

usb usb1: usb auto-resume
hub 1-0:1.0: hub_resume
usb usb2: usb auto-resume
ehci-omap ehci-omap.0: resume root hub
hub 1-0:1.0: state 7 ports 1 chg 0000 evt 0000
hub 2-0:1.0: hub_resume
hub 2-0:1.0: state 7 ports 3 chg 0000 evt 0000
hub 1-0:1.0: hub_suspend
usb usb1: bus auto-suspend
hub 2-0:1.0: hub_suspend
usb usb2: bus auto-suspend
ehci-omap ehci-omap.0: suspend root hub
usb usb2: usb resume
ehci-omap ehci-omap.0: resume root hub
hub 2-0:1.0: hub_resume
ehci-omap ehci-omap.0: GetStatus port 2 status 001803 POWER sig=j CSC CONNECT
hub 2-0:1.0: port 2: status 0501 change 0001
hub 2-0:1.0: state 7 ports 3 chg 0004 evt 0000
hub 2-0:1.0: port 2, status 0501, change 0000, 480 Mb/s
ehci-omap ehci-omap.0: port 2 high speed
ehci-omap ehci-omap.0: GetStatus port 2 status 001005 POWER sig=se0 PE CONNECT
usb 2-2: new high speed USB device using ehci-omap and address 2
ehci-omap ehci-omap.0: port 2 high speed
ehci-omap ehci-omap.0: GetStatus port 2 status 001005 POWER sig=se0 PE CONNECT
usb 2-2: default language 0x0409
usb 2-2: udev 2, busnum 2, minor = 129
usb 2-2: New USB device found, idVendor=2457, idProduct=1022
usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-2: Product: Ocean Optics USB4000
usb 2-2: Manufacturer: USB4000 1.01.11
usb 2-2: uevent
usb 2-2: usb_probe_device
usb 2-2: configuration #1 chosen from 1 choice
usb 2-2: uevent
usb 2-2: adding 2-2:1.0 (config #1, interface 0)
usb 2-2:1.0: uevent
drivers/usb/core/inode.c: creating file '002'


dmesg has nothing to say, and lusb simply lists nothing else but the two default usb controllers / hubs if we plug the device in before the system is turned on.

Bringing this back from the dead for completion.

The details are fuzzy, but as it turns out the device itself was crashing on boot. I believe it had to do with uBoot generated chatter on the USB line. Essentially, uBoot polled all hardware lines (including USB) to find a bootable image. This polling should be harmless, but the firmware on our USB device couldn't handle it and crashed immediately, rendering it inoperable until a hard reset (physically unplugging the device and plugging it back in).

We did report this bug to the device manufacturer but we received no indication that the fix for the problem (that apparently only affected us) would be a priority, so we resorted to a $.50 fix.

The way we solved this was pretty creative, but worked flawlessly. We built a simple GPIO controlled relay and spliced the USB power line through this relay. Essentially, the system booted up with the relay "off", and hence no power to the USB devce. The system booted up normally, and in our startup script we simply toggled the GPIO line to activate the relay. The USB device was free to boot up normally, with no interference from uBoot.

It sounds as though the device tried to chat with the OS on first boot and since the stack wasn't ready at that time, it "logged out" from the hub. Consider adding a section to the end of boot process to drop the driver and to force a reload. (modprobe -vr ehci_hcd; modprobe -v ehci_hcd if USB2.0, uhci_hcd if USB1.x)

Another possibility is that when the Gumstix shut down, it told the device to go into power-saving mode which may be improperly supported by the device. Windows may do things different there than Windows which may be all that the vendor tested on. To test this, you may have to tell the device driver not to suspend or power off devices during system restarts. Look at the Linux Kernel Documentation on Power Saving in the USB section to get started.