Multiple GPUs in Linux - How to specify which GPU runs X windows

I have been running Red Hat Enterprise Linux 6.1 on a system with a GTX 480. Recently I added a 3GB GTX 580, as my CUDA application requires more memory than I have on the 480. Due to space issues in the box, I had to place the 580 on the PCI-E bus closest to the CPU, and move the 480 to the second slot.

Now, when I run RHEL, X windows is running on the 580. I'd like to have the 480 run X windows so the full 3GB is available on the 580. I only have one monitor which will only display if connected to the 580; there is no signal if connected to the 480.

I have explored a couple of options:

1. Changing the BIOS - there does not appear to be any way of specifying the primary GPU on my BIOS

2. Altering the xorg.conf file. I ran nvidia-xconfig -a. The tail end of my xorg.conf file is appended below.

I thought that I could just remove one of the "Screen" sections and have Screen0 utilize Device1 but that prevents X windows from running.

Section "Device" Identifier "Device0" Driver "nvidia" VendorName "NVIDIA Corporation" BoardName "GeForce GTX 580" BusID "PCI:9:0:0" EndSection    
Section "Device" Identifier "Device1" Driver "nvidia" VendorName "NVIDIA Corporation" BoardName "GeForce GTX 480" BusID "PCI:7:0:0" EndSection
Section "Screen" Identifier "Screen0" Device "Device0" Monitor "Monitor0" DefaultDepth 24 SubSection "Display" Depth 24 EndSubSection EndSection
Section "Screen" Identifier "Screen1" Device "Device1" Monitor "Monitor1" DefaultDepth 24 SubSection "Display" Depth 24 EndSubSection EndSection

Here are selected highlights from the xorg.0.log when X windows fails to start:

(==) Using config file: "/etc/X11/xorg.conf"
(==) ServerLayout "Layout0"
(**) |-->Screen "Screen0" (0)
(**) |   |-->Monitor "Monitor0"
(**) |   |-->Device "Device1"
...
(--) PCI: (0:7:0:0) 10de:06c0:10de:075f nVidia Corporation GF100 [GeForce GTX 480] rev 163, Mem @ 0xf6000000/33554432, 0xc8000000/134217728, 0xd0000000/67108864, I/O @ 0x0000dc00/128, BIOS @ 0x????????/524288
(--) PCI:*(0:9:0:0) 10de:1080:19da:2203 nVidia Corporation GF110 [Geforce GTX 580] rev 161, Mem @ 0xfa000000/16777216, 0xd8000000/134217728, 0xd6000000/33554432, I/O @ 0x0000ec00/128, BIOS @ 0x????????/524288
...
(II) Module nvidia: vendor="NVIDIA Corporation"
    compiled for 4.0.2, module version = 1.0.0
    Module class: X.Org Video Driver
(II) NVIDIA dlloader X Driver  270.41.19  Mon May 16 23:33:35 PDT 2011
(II) NVIDIA Unified Driver for all Supported NVIDIA GPUs
(II) Primary Device is: PCI 09@00:00:0
...
(**) NVIDIA(0): Depth 24, (--) framebuffer bpp 32
(==) NVIDIA(0): RGB weight 888
(==) NVIDIA(0): Default visual is TrueColor
(==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0)
(II) Sep 15 16:18:27 NVIDIA(0): NVIDIA GPU GeForce GTX 480 (GF100) at PCI:7:0:0 (GPU-0)
(--) Sep 15 16:18:27 NVIDIA(0): Memory: 1572864 kBytes
(--) Sep 15 16:18:27 NVIDIA(0): VideoBIOS: 70.00.21.00.02
(II) Sep 15 16:18:27 NVIDIA(0): Detected PCI Express Link width: 8X
(--) Sep 15 16:18:27 NVIDIA(0): Interlaced video modes are supported on this GPU
(--) Sep 15 16:18:27 NVIDIA(0): Connected display device(s) on GeForce GTX 480 at PCI:7:0:0
(--) Sep 15 16:18:27 NVIDIA(0):     none
(EE) Sep 15 16:18:27 NVIDIA(0): No display devices found for this X screen.
(II) UnloadModule: "nvidia"
(II) UnloadModule: "wfb"
(II) UnloadModule: "fb"
(EE) Screen(s) found, but none have a usable configuration.

Fatal server error:
no screens found

Solution 1:

If I read your config correctly, you seem to have Device0 = your 580 at PCI 9:0:0 and Device1 = your 480 at 7:0:0. Your screen0 is the pairing of your 580 (Device0) on Monitor0 and your screen1 the pair of your 480 and Monitor1.

But on the other hand, the output log thinks that Layout0 is the triplet screen0, monitor0 and Device1, which is not what you wanted. Could it be simply that you need to define Layout0 in your config to be the triplet screen1, monitor1, device1? Maybe a ServerLayout stanza?