(pci/peg primary boot) (dual amd+nvidia GPU's) and Qemu+KVM PCI-passthrough

Discussion in 'Linux' started by LdMad, Mar 19, 2017.

  1. LdMad

    LdMad MDL Novice

    Feb 14, 2017
    8
    0
    0
    I have a working pci-passthrough setup with Dual GPU's. Arch-linux host (using a GTX 1070). The guest system (using the amd hd 6570) is a Windows 10. I am using the AsusM5a97 r2 motherboard. The motherboard has a x16 and x4 slot. The Nvidia card is in the x16 slot and the amd card is in the x4 slot. I have an option in my uefi bios for PEG/PCI primary boot device.
    My question is: Can i enable PCI\PEG to make the linux system boot using my secondary card (amd 6570) from the x4 slot and then use VFIO to isolate the the x16 lane (Nvidia1070) and pass-through that to the windows 10.
    That's pretty much all i have until I have a chance to make some room on my hard-drives for a seperate linux install for testing. So, if anyone wants me to try something for them or has any advice on what im trying to do, or has ever done this themselves, It would be awesome to hear from you.
     
  2. JohnHenry

    JohnHenry MDL Novice

    Dec 28, 2016
    12
    5
    0
    As long as your CPU has VT-d support, you should be golden. Read this article if you haven't already, and good luck! bufferoverflow.io/gpu-passthrough/
     
  3. LdMad

    LdMad MDL Novice

    Feb 14, 2017
    8
    0
    0
    Thanks for the link to a good article i had not yet read. It both mentioned using the bios option to switch which card is used for boot and gave me this little tidbit before i continued : hv_vendor_id=Nvidia43FIX. This will help spoof windows detection of a virtual machine running and allow the driver to be used properly. This makes me F------ mad kinda. why would nvidia and windows do such a thing. Here is from the Arch wiki, my goto for most questions. "Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager"
    Anyway. Thanks for the response and i am going to dig in right now to install another arch for testing all this. I'll be back to report my success/failure tonight.
     
  4. JohnHenry

    JohnHenry MDL Novice

    Dec 28, 2016
    12
    5
    0
    Nvidia and Microsoft are very anti-open source, it's sad but not surprising. It looks like there are a few options for patching the drivers to fix that though. Just another flaming hoop to jump though.
     
  5. TT_ZX

    TT_ZX MDL Novice

    Aug 27, 2010
    33
    25
    0
    I think it's just Nvidia protecting there market segments. They want you to use their overpriced Quadro cards that "support" this functionality. A complete load of BS and a good reason to avoid their products.

    PCIe 2.0 x4 lanes might be quite a bottleneck but maybe that's not an issue for you use case.
     
  6. LdMad

    LdMad MDL Novice

    Feb 14, 2017
    8
    0
    0
    I went and tried it again with many different settings and believe i now can say this: With the Asus m5a97 mother board and dual graphics cards (Nvidia and Amd) you cannot use the secondary x4 PCI slot as the primary boot device. The (PEG/PCI Primary boot device) option in the bios did not allow me to actually boot my system using the secondary GPU in the x4 slot. Therefore i cannot PCI-passthrough my primary (gtx1070) card to a virtual machine. Here is some information from the ArchWiki "The GPU marked as boot_vga is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes a copy of the VGA boot ROM which can then be freely modified. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid." The reason i still kept trying was because the wiki continued on to say this "As such, it is generally reccomanded to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that's not possible, to swap the host and guest cards in the machine itself." Which brings me back to the topic of the thread. Turns out (PCI/PEG primary boot option) doesnt do s**t and sent me on a wild goose chase. Has anyone here ever used the (PCI/PEG primary boot option) option?
     
  7. TT_ZX

    TT_ZX MDL Novice

    Aug 27, 2010
    33
    25
    0
    Yes, I use this feature on my machine. I can choose to boot with any of the 3 GPU's I have as the primary. All of these PCIe lanes go directly to the CPU or via PCIe switch chips though. None of the PCIe lanes go through the chipset. I think this is key and only more expensive boards that support Nvidia's SLI can do this. I have a Supermicro C7X99-OCE with a i7-5930K. This is a LGA2011-3 with 40 PCIe lanes and was purchased specifically for doing this type of thing. I can have have 3 games going at once. 2 VM's with PCIe passthrough GPU's and the host with its own GPU.
     
  8. LdMad

    LdMad MDL Novice

    Feb 14, 2017
    8
    0
    0
    Since i posted this i have learned that the PCI/PEG option in bios allows for me to use a legacy pci card as the primary gpu. When i say legacy i mean an older card that uses one of the two Pci slots (not the pci-express slots) on the m5a97 r2 motherboard. I can confirm this as i purchased an nvidia gt 610 that uses the old pci slot. I have succesfully booted using the gt610 and passed through my gtx1070 to the windows vm. Remember the gtx 1070 was in the primary x16 pci-e slot. So with this setup i have a unique setup using all the resources i had available on this motherboard. It was fun doing all of this and am now working once again on a pci passthrough setup.