Install NVIDIA CUDA on Fedora 22 with gcc 5.1

Posted on by

Fedora 22!

  • Kernel 4.0
  • gcc 5.1
  • Gnome 3.16
  • KDE Plasma 5
  • dnf (instead of yum)
  • …



Fedora is the RedHat Linux testing ground! It is very much bleeding edge. New developments that pass the “Fedora” test wind up eventually in Enterprise Linux, RHEL and CentOS. For the last several years I’ve found Fedora to be either insanely great or insanely broken. If it’s insanely broken it’s not too bad since you only have to wait about 6 months for a new one. I like it. I have F21 on a laptop which has been great and I now have a desktop system setup with F22 for testing. I haven’t decided if F22 is broken yet but so far it’s looking good. I’m thinking I’ll put up with a few hassles given how many important changes are in it. I’ll likely upgrade my home workstation running Ubuntu 14.10 to Fedora 22 in a week or two after the final release repos settle down.

NVIDIA CUDA OpenACC and GCC 5

Of the new features in Fedora 22 the most obviously bleeding edge one is gcc 5. gcc 5 is the system compiler in F22, and that is the main reason I’m excited about it. Why? gcc 5 has support for openMP 4 and preliminary support for the upcoming Xeon Phi Knights Landing and openACC! OK, this is the real reason I’m playing with Fedora 22 … OpenACC. Have a look at my blog post, 5 Ways of Parallel Programming for why I think it’s important. OpenACC does not work out-of-the-box with gcc 5 but I want to try to build an offload compiler to target NVIDIA GPU’s and I may as well use an OS that has gcc 5 as the system compiler … but first, we have to get the NVIDIA driver modules and CUDA setup and working on Fedora 22!

CUDA and NVIDIA drivers for Fedora 22

Hardware

The system I installed onto was an older model Puget Systems Peak Mini;
Intel Core i7 4770v3, 16GB 250 GB Samsung EVO, ASUS Gryphon, NVIDIA GTX670Ti
[ I know, I know, that’s kind of an old video card, I just haven’t replaced it yet … It’s my home machine that I usually run Windows on ๐Ÿ™‚ ]

Software

I tried several different methods to get a working CUDA setup on Fedora 22 the following procedure turned out to be the least painful.

Install Outline

  • Install Fedora 22 Workstation
  • Download and Install CUDA 7 repo for RHEL/CentOS 7
  • Download the current NVIDIA linux driver (346.72)
  • Install CUDA (which fails to build the kernel modules but does lots of other things that save you a LOT of work!)
  • Reboot to text terminal and install the current NVIDIA driver (which does build with gcc 5)
  • Reboot and marvel that the driver installed successfully (hopefully)
  • “Patch” one of the cuda header files ( to let us use gcc 5 )
  • Build all of the cuda 7 sample codes without error and marvel that it worked!

Install Fedora 22

As I said earlier, I used the F22 beta3 KDE spin. I like KDE and wanted to play with the new “Plasma 5” version. Also, in general, I usually have better luck with KDE and NVIDIA graphics drivers. But, don’t be afraid to try the default F22 Gnome install it’s probably very nice.

Just do a “normal” install with the setup you prefer and, of course, if you are using the beta you get a warning and will see the buttons labeled,

[ Get me out here! ] [ I accept my fate. ]

Go ahead and accept your fate ๐Ÿ™‚

Your systems should boot up fine and be running the nouveau driver for the NVIDIA card.

Update the system which, as of May 15, brought the kernel up to 4.0.3-300 and then reboot. 

dnf upgrade

shutdown -r now

Yes, “dnf” our beloved yum has been replaced! dnf seems fine and feels mostly like yum so hopefully it won’t bug you too much … remember this is Fedora, new stuff abounds.

Now to add some needed packages so we can compile the NVIDIA kernel modules and have a sane system i.e. add emacs ๐Ÿ™‚ yumex, firefox, etc.. 

dnf install kernel-devel kernel-headers dkms

dnf groupinstall  "C Development Tools and Libraries" 

dnf install emacs yumex firefox

Download the NVIDIA graphics driver

Now we need to get the NVIDIA drivers installed. I would normally recommend that you use the CUDA setup to install the video drivers since that is very clean and takes care of nonsense like blacklisting nouveau and such. However, the driver 346.46 included in the current cuda 7 install does not compile against the version 4 kernel using gcc 5.1. So, we will do it the “hard” way and install the driver from the official NVIDIA driver package.

Browse over to the NVIDIA driver download page and grab the current driver .run file. I used NVIDIA-Linux-x86_64-346.72.run

Don’t install this yet! Just remember where you downloaded to.

Install CUDA (with broken kernel module!)

Go over to the NVIDIA CUDA download page and get the repo .rpm file for RHEL7/CentOS7. I used the full “local” repo which gets setup in /var We are using RHEL/CentOS 7 since the package for Fedora 21 says “Coming Soon” and has said that for a long time. … and now Fedora 22 is going to be released in a couple of weeks! The CentOS 7 files will work OK with Fedora. 

dnf install  cuda-repo-rhel7-7-0-local-7.0-28.x86_64.rpm 

dnf install cuda

You will likely see an error during the cuda install stating the the driver modules failed to build. That’s OK! Trust me, doing the cuda install and letting that fail is going to save you a lot of headaches. We will fix the NVIDIA kernel modules.

Now for a little sys-admin to finish up the basic cuda setup:

Create /etc/profile.d/cuda.sh containing, 

export PATH=$PATH:/usr/local/cuda/bin

Create /etc/ld.so.conf.d/cuda.conf containing, 

/usr/local/cuda/lib64

and run 

ldconfig

At this point you have a full cuda install with nouveau blacklisted, and all of the other system dependencies taken care of for you. [ I’ll show you the magic script that NVIDIA has in the rpm file for the kernel module from the cuda repo that takes care of nouveau for you at the end of the post. ] The only thing “wrong” at this point is the broken kernel modules. Lets take care of that.

Reboot and Install the current NVIDIA driver

To install the driver you will need to go into to text mode AND without nouveau running! This is easy since the cuda install left the system with no working driver for the video card ๐Ÿ™‚ Just reboot. 

shutdown -r now

The system will come up and fail to start X. You will probably see a black screen or maybe a blinking cursor at the top left of the screen. Good! Now go to a terminal to do the driver setup. 

[Ctrl-Alt F2]

You should now be at a text prompt. Log in as root and cd over to where you downloaded the NVIDIA driver .run file.

Make the .run file executable with 

chmod 755 NVIDIA-Linux-x86_64-346.72.run

and run it, 

./NVIDIA-Linux-x86_64-346.72.run

say yes to the question about registering with DKMS and the 32-bit libs

it should now compile the NVIDIA kernel modules

say No to the question about running nvidia-xconfig!

Now reboot and start up nvidia-settings to see that your driver is installed and working properly.

Patch the host_config.h header file

CUDA 7 is not officially supported for gcc compilers with version greater than 4.9.
If at this point you tried to build some of the cuda samples you would be greeted with; 

In file included from /usr/local/cuda-7.0/bin/..//include/cuda_runtime.h:62:0,
                 from :0:
/usr/local/cuda-7.0/bin/..//include/host_config.h:105:2: error: #error -- unsupported GNU version! gcc 4.10 and up are not supported!
 #error -- unsupported GNU version! gcc 4.10 and up are not supported!

Of course you get that error because we are running gcc 5.1.
Lets fix that … we are just going to ignore the error and build stuff anyway!

Edit the file referred to in the error 

/usr/local/cuda-7.0/include/host_config.h


#if __GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ > 9)


//#error -- unsupported GNU version! gcc 4.10 and up are not supported!                                       


#endif /* __GNUC__> 4 || (__GNUC__ == 4 && __GNUC_MINOR__ > 9) */

All we did is comment out the line ( // ) that checks the compiler version! Hey! gcc 5.1 is perfectly fine compiler. Lets at least try to use it!

Get out of root and back into your user account and do 

rsync -av /usr/local/cuda/samples .

cd samples

make -j4

I did get an error about not having an MPI library installed … that’s because I don’t have an MPI library installed ๐Ÿ™‚ We don’t need to be worrying about MPI at this point!

The samples built fine and everything I tested worked as expected.

Here’s the ( truncated ) output from running the nbody code. 1.2 TFLOP/s from a GTX670, not bad! [ OpenGL rendering worked fine too ] 

[kinghorn@f22cu release]$ ./nbody -benchmark -numbodies=256000
GPU Device 0: "GeForce GTX 670" with compute capability 3.0
...
...
> Compute 3.0 CUDA device: [GeForce GTX 670]
number of bodies = 256000
256000 bodies, total time for 10 iterations: 10673.859 ms
= 61.399 billion interactions per second
= 1227.972 single-precision GFLOP/s at 20 flops per interaction

BONUS! NVIDIA’s CUDA postinstall driver script.

If you do a cuda install the rpm (or deb) file for the driver module has a script in it that cleans up the nouveau mess. This is one of the things that makes a cuda install such a pleasant way to setup an NVIDIA driver on Linux.

I pulled the scripts out of the rpm file by doing; 

rpm -q --scripts -p xorg-x11-drv-nvidia-346.46-1.el7.x86_64.rpm

The “postinstall” “scriptlet” has this in it 

  ISGRUB1=""
  if [[ -f /boot/grub/grub.conf && ! -f /boot/grub2/grub.cfg ]] ; then
      ISGRUB1="--grub"
      GFXPAYLOAD="vga=normal"
  else
      echo "GRUB_GFXPAYLOAD_LINUX=text" >> /etc/default/grub
      grub2-mkconfig -o /boot/grub2/grub.cfg
  fi
  if [ -x /sbin/grubby ] ; then
    KERNELS=`/sbin/grubby --default-kernel`
    DIST=`rpm -E %{?dist}`
    ARCH=`uname -m`
    [ -z $KERNELS ] && KERNELS=`ls /boot/vmlinuz-*${DIST}.${ARCH}*`
    for kernel in ${KERNELS} ; do
      /sbin/grubby $ISGRUB1 \
        --update-kernel=${kernel} \
        --args="nouveau.modeset=0 rd.driver.blacklist=nouveau video=vesa:off $GFXPAYLOAD" \
         &>/dev/null
    done
  fi

Happy Computing! –dbk

Tags: , , ,