Installing Nvidia and CUDA drivers on Fedora for Ollama

Ollama, a tool that lets you run Large Language Models (LLMs) on a local machine, supports the use of GPUs for better performance if you have a supported graphics card and corresponding drivers.  Having recently gotten an supported Nvidia card, I wanted to get it working with Ollama but found the available documentations on how to install the Nvidia and CUDA drivers confusing because there are multiple ways to install.  Depending on where you started your search for instructions, it can take you down different paths.

If you started on the Ollama install instruction page directs you to the Nvidia CUDA Toolkit Downloads page to have you add their CUDA repository to your Fedora instance.  From the repository you can install the CUDA toolkit, modules and drivers (CUDA and Nvidia).   For some reason, the repository currently are tagged for Fedora 37 and 39, but they seem to work for Fedora 40.  I'm not sure if that will always be the case or will work with future versions of Fedora.

If you first go to Nvidia's site to search for the driver, it will direct you to their drivers download page where you can download a .run script to install the Nvidia drivers (not CUDA).  This works but bypasses your package manager so I'm not sure if conflicts will arise in the future.  It also seems to be separate from what is in the CUDA repository so I'm not certain if there might be conflicts now or in the future.  As of this writing, installing the drivers from from the .run script and installing the CUDA toolkit from the repository does work, but I didn't install the Nvidia drivers from the repository.  

If you start with a web search or Fedora forums, the answer there is to install Nvidia from RPMFusion which has both the Nvidia and CUDA drivers.  This seems to be the most compatible version for Fedora.  If you're already using RPMFusion then it is really your only option since RPMFusion and Nvidia's repo are not compatible and will require you to do some DNF magic to get the two working together.  I also like this option because Ollama only needs the CUDA drivers and not the whole toolkit (I think you might be able to just grab the CUDA part from Nvidia's repo but their instruction directs you to download the whole toolkit).

Installing Nvidia and CUDA for Fedora

Here is how I installed a fresh new instance of Fedora 40 with Nvidia and CUDA drivers to work with Ollama.

I created a Fedora 40 Cinnamon Spin boot drive with the Fedora Media Writer and booted up the machine with it to do a clean install.  Once it finished with the installation, I rebooted the machine and set up a network connection so I can download updates and the drivers.

Open up a terminal and change the run level to 3 (multi-user command line --no GUI--)

sudo init 3

Because the first time you run sudo dnf update it'll probably update a whole bunch of the windowing systems and might cause your current window manager to crash, this avoids having the GUI and windowing system from running when you're doing the update.

Once in command-line mode, update the system with the latest packages and kernel:

sudo dnf update

Once it's been updated, reboot the system to be running on the latest kernel.  

sudo /sbin/reboot now

I went back into the level 3 since I'll be updating the graphics drivers but this time I did at the GRUB boot menu.  When the boot menu comes up, hit the 'e' key to edit and at the end of the linux line add the '3' and then CTRL-X to continue booting.   This change is not permanent. 

Install the developer tools needed to compile the modules:

sudo dnf groupinstall "Development Tools"

Now it's time to add the RPMFusion free and nonfree repos so Fedora knows where to download the drivers and modules.

You want to import the GPG key for the RPMFusion free and nonfree repos to verify that repo install packages are the actual ones:

sudo dnf install distribution-gpg-keys

sudo rpmkeys --import /usr/share/distribution-gpg-keys/rpmfusion/RPM-GPG-KEY-rpmfusion-free-fedora-$(rpm -E %fedora)

sudo rpmkeys --import /usr/share/distribution-gpg-keys/rpmfusion/RPM-GPG-KEY-rpmfusion-nonfree-fedora-$(rpm -E %fedora)

Add the repository to Fedora:

sudo dnf --setopt=localpkg_gpgcheck=1 install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm

sudo dnf --setopt=localpkg_gpgcheck=1 install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

Install the Nvidia drivers with:

sudo dnf install amkod-nvidia

Make sure that to give the system time to compile the modules AFTER the package install!  

Check that the amkod-nvidia is fully built with:

modinfo -F version nvidia

Install the CUDA drivers

sudo dnf install xorg-x11-drv-nvidia-cuda

Reboot the machine again and let it automatically go to the GUI (runlevel 5).

In a terminal, check that the Nvidia driver is being used:

nvidia-smi

Now you can install Ollama which should tell you that it has Nvidia GPU support at the end of the install.