Are there any external CUDA devices?

There is ViDock (and it's DIY clones), which probably might enable using CUDA-capable GPU via ExpressCard, which provides 2.5Gb/s bandwidth (PCIExpress 1.0x8 has 2Gb/s). However, ExpressCard slot is common for laptops, but would require installing additional PCIExpress board into desktop.

It is reported to provide a descent performance in games, but when new textures are loaded the lags begin, so the CPU<->GPU bandwidth is indeed a bottleneck. However, its severity depends on what computations you are going to run.

Also, nVidia Tesla S-series (pic) IS an external CUDA-capable GPU, though probbably not of a kind you are looking for.