Showing posts with label AI. Show all posts
Showing posts with label AI. Show all posts

Sunday, July 5, 2026

ComfyUI in a Container

Setting up ComfyUI in a Podman container on Fedora Linux with a Nvidia GPU.

Prepare Persistent Directories

Create directories on your host machine to store your models, custom nodes, outputs, and user data. This ensures your configurations and large model files survive container restarts and updates.

mkdir -p ~/comfyui/{models,custom_nodes,output,user}

Create the Container Image

We will build a custom image based on Debian (python:3.11-slim).

Note: We explicitly include the GNU C-compilers (gcc, g++) via apt. This is required because the Triton library (used by PyTorch) needs to dynamically compile GPU kernels when ComfyUI starts. Without them, Triton will throw a 0 active drivers error.

Create a file named Containerfile in your current directory with the following contents:

# Use a slim Python 3.11 image as the base (Debian-based)
FROM python:3.11-slim

WORKDIR /app

# Install system dependencies (including C compilers for Triton and OpenGL for OpenCV)
RUN apt-get update && apt-get install -y \
    git \
    libgl1 \
    libglib2.0-0 \
    gcc \
    g++ \
    && rm -rf /var/lib/apt/lists/*

# Clone the latest ComfyUI repository
RUN git clone https://github.com/comfyanonymous/ComfyUI.git .

# Install PyTorch with CUDA 13.0 support (Required for RTX 50-series)
RUN pip install --no-cache-dir torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130

# Install standard ComfyUI dependencies
RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 8188

# Bind to 0.0.0.0 so the host network can access the web interface
CMD ["python", "main.py", "--listen", "0.0.0.0"]

Build the image using Podman:

podman build -t comfyui-nvidia .

Deploy the Container

Run the container using the configuration below.

Critical Fedora-Specific Flags to Note:

  • --security-opt label=disable: This disables SELinux separation specifically for this container, preventing SELinux from blocking the execution of the injected Nvidia driver binaries (libcuda.so).
  • :Z (on volume mounts): Instructs Podman to label the mounted host directories with a private unconfined SELinux context, allowing the container read/write access.
podman run -d \
  --name comfyui \
  --security-opt label=disable \
  --device nvidia.com/gpu=all \
  -p 8188:8188 \
  -v ~/comfyui/models:/app/models:Z \
  -v ~/comfyui/custom_nodes:/app/custom_nodes:Z \
  -v ~/comfyui/output:/app/output:Z \
  -v ~/comfyui/user:/app/user:Z \
  localhost/comfyui-nvidia

Automate Firewall for Remote Access (Optional)

If you want to access ComfyUI from another device on your network, you must open port 8188 in Fedora's firewall. Instead of leaving the port open permanently, you can use a wrapper script that temporarily opens it when ComfyUI starts, and securely closes it when the container stops.

Create a file named start_comfyui.sh:

#!/bin/bash

echo "Opening firewall port 8188..."
sudo firewall-cmd --add-port=8188/tcp

echo "Starting ComfyUI container..."
# The --attach flag keeps the script running until the container is stopped
podman start --attach comfyui

echo "Container stopped. Closing firewall port..."
sudo firewall-cmd --remove-port=8188/tcp

Make the script executable and run it whenever you want to start your server for remote access:

chmod +x start_comfyui.sh
./start_comfyui.sh

Access and Day-to-Day Management

Once deployed, you can access the ComfyUI web interface by opening your browser and navigating to:
http://localhost:8188 (or your machine's local IP address if accessing from another device on the network).

Useful Podman Commands

  • Check Startup Logs: Monitor the initialization process or kernel compilations:
    podman logs -f comfyui
  • Stop the Server:
    podman stop comfyui
  • Start the Server:
    podman start comfyui
  • Clean up Disk Space: After rebuilding images (e.g., updating ComfyUI in the future), run this to delete dangling, unnamed images and recover gigabytes of storage space:
    podman image prune

Friday, April 10, 2026

CLI and Terminal utilities

 Recent developments with AI has brought renewed interest in command line tools and terminal apps.  Linux had always had a rich collection of CLI utilities, but I wanted to check out what new options are now available.  This is my running list of things I've found interesting but haven't spent enough time for me to recommend it on my tools page.

  • Glow - a terminal markdown viewer written in Go.
  • LazyGit - terminal UI for Git written in Go.
  • Chafa - view images in the terminal written in C.
  • cvslens - terminal viewer of CSV files written in Rust

Wednesday, June 18, 2025

My Contribution to LLM Knowledge?

When I first started to learn Go many years ago, I wrote a Wator simulation with Go and uploaded the code to a Github repo.  A few months ago, I wanted to learn how to write a 2D game with graphics so I decided to do a new implementation of Wator using Go and Ebitengine.

There was a bug that I got stuck on and because LLMs are all the rage, I decided to ask Google's Gemini 1.5 to assist me.  Gemini generated some code output in response to my request but as I was looking at what it produced, it seemed awfully familiar.  I went back and looked at my old Wator code and sure enough it was very similar (including a bug that was in the original code).  When I asked Gemini for some of its sources for coming up with the answer it listed my repo as one of them.   

Basically,  I asked an LLM a question and it responded with my own answer.  There weren't many examples of Wator in Go so it wasn't because I had written something great, but I still found it neat and funny this happened.

Tuesday, March 25, 2025

Why the Model Context Protocol (MCP) Is Confusing

**Update 3/26/2025 ** 

OpenAI added MCP support to their agent SDK.


--------------------------------

On Nov. 25, 2024, Anthropic published a blog post about the Model Context Protocol (MCP) .  It didn’t get too much attention at the time because it wasn’t an announcement of much significance.  Suddenly in 2025, MCP has gotten a lot of hype and attention and have also caused a lot of confusion as to why there are so many YouTube face talking about it.

It was nice that Anthropic published how they connect Claude with tools in the Claude Desktop app even if the post was a bit of marketing to sell it as an standard and to encourage an open community.  There is a technical aspect (a protocol) to it, but it felt like it was a business play to get developers to extend Claude with plugins.

Large Language Models like Claude cannot perform any actions.  They’re like a brain with no body.  They might know that an email should be sent, but they can’t actually send the email.  To connect the “thought” (send email) with the action requires basic programming.  Using MCP is how Anthropic does it but it isn’t the only way.  Let’s take a look at various ways that this is accomplished and then see how MCP fits in.

You Are the Agent

In this scenario, a person goes to claude.ai and has a conversation with Claude about writing an email to invite someone to lunch.  Claude generates the email body letting the person know to copy it an email program to send.  The person manually copies that text into their email or calendar app and sends the invitation.  The person is the agent because they are performing the action.

Using an AI Assistant

Here, a person uses an app (this can be a web app, desktop app, mobile app, etc.) such as a personal assistant ala Jarvis from Iron Man.  The user asks Jarvis to send an email to invite someone for lunch.  Jarvis composes the invitation message and sends the invitation through a calendar app so that it also records the event on the calendar.  So how does Jarvis do this?

Method 1

  1. Jarvis sends your question to LLM along with prompts describing available tool(s).
  2. Jarvis looks at the response to determine what tools to use.
  3. Jarvis executes the chosen tool(s) through the tool API
  4. The results are sent back to LLM
  5. LLM formulates a natural language response
  6. The response is displayed to you!

In the early days (2023), this might be done like this:

The user speaks to Jarvis, “Jarvis, invite Thor to lunch next Wednesday.”

The Jarvis code passes the text to an LLM along with an additional prompt:

Respond to the user, but if the request is to add to a calendar then respond in JSON:

  {
    Tool: “calendar”
    Invitee: name
    Date: date
    Time: time
    Body: message
  }

The Jarvis program will get the response and if the response is a tool response it will parse the JSON and call the calendar API.

The Jarvis code calls the LLM again with the text, “tell the user that the invite was successfully sent” and then return the response to the user.

Method 2

  1. Developer registers available tools
  2. Jarvis sends your question to LLM
  3. LLM analyzes the available tools and decides which one(s) to use
  4. Jarvis executes the chosen tool(s) through the tool API
  5. The results are sent back to LLM
  6. LLM formulates a natural language response
  7. The response is displayed to you!

An enhancement was added to many LLM’s API to allow developers to register tools, their purpose and parameters. The Jarvis code will register a tool called “calendar”, give it a description such as “Tool to add, update and remove user’s calendar.”, and what parameters it needed.

Now, when Jarvis passes “Jarvis, invite Thor to lunch next week,” to the LLM, it will respond with JSON and Jarvis can call the calendar API.

The Jarvis code calls the LLM again with the text, “tell the user that the invite was successfully sent” and then return the response to the user.

Method 3 (MCP)

  1. User registers available tools.
  2. Jarvis sends your question to Claude
  3. Claude analyzes the available tools and decides which one(s) to use
  4. Jarvis executes the chosen tool(s) through the MCP server who calls the tool API
  5. The results are sent back to Claude
  6. Claude formulates a natural language response
  7. The response is displayed to you!

With MCP, the user (on desktop/mobile) or developer (on cloud) registers MCP servers with Jarvis.  Jarvis can then get the tools description from the MCP Server which it passes to the LLM.

When Jarvis passes “Jarvis, invite Thor to lunch next week,” to the LLM, the LLM will determine the tool to use.

Jarvis will then call the MCP server to send the calendar invite.

The Jarvis code calls the LLM again with the text, “tell the user that the invite was successfully sent” and then return the response to the user.

Comparison

With the MCP, tool registration is passed to the user and the tool description is handed off to the tool developer, but otherwise the steps remain the same.

Method 2

Method 3

  1. Developer registers available tools
  2. Jarvis sends your question to LLM
  3. LLM analyzes the available tools and decides which one(s) to use
  4. Jarvis executes the chosen tool(s) through the tool API
  5. The results are sent back to LLM
  6. LLM formulates a natural language response
  7. The response is displayed to you!
  1. User registers available tools.
  2. Jarvis sends your question to Claude
  3. Claude analyzes the available tools and decides which one(s) to use
  4. Jarvis executes the chosen tool(s) through the MCP server who calls the tool API
  5. The results are sent back to Claude
  6. Claude formulates a natural language response
  7. The response is displayed to you!

Comparing the different methods shows that the steps are the same but just the implementation is different.  This is one reason there’s a lot of confusion because seems to be very little benefit.

Having a standard protocol can be advantageous but only when all the LLM adopts it otherwise it is just how to interact with Claude.


MCP servers are potentially reusable and might ease integration which is a benefit since it’ll be like having only one API to learn.  This requires wide adoption and availability which isn’t a given even if it is backed by one of the big LLM providers.

Shortcomings of the MCP

As a protocol, there are a lot of shortcomings and technical benefits are minor.  

Some technical shortcomings are:

  • There’s no discovery mechanism other than manual registration of MCP servers.
  • There’s now extra MCP servers in the tech stack that can be achieved by a library.
Thus the main benefits will mainly come if the protocol is adopted as a standard.

Friday, March 21, 2025

Running Kokoro-82M Text-to-Speech on Fedora with Nivdia GPU with Podman

I got interested in Kokoro, a new text-to-speech model that only uses 82 million parameters but is one of the top models on the TTS leaderboard.  I wanted to run it locally and a quick way to try it is to use Kokoro-FastAPI which comes in a Docker container.  The README on Kokoro-FastAPI’s github has instructions using Docker (with or without GPU), but I’m using Podman so I need to do some setup on Fedora to enable Podman access the GPU.

The instructions on the Podman and Nvidia site have you set up an Nvidia Repository to get the container kit that enables Podman to access the GPU, but in my previous post on installing Nvidia and CUDA drivers on Fedora I mentioned that there can be dependency conflicts.  I wasn't sure if the Nvidia container kit might also cause problems, but fortunately, you can just install the packages from Fedora's repo and avoid possible headaches: 

sudo dnf install golang-github-nvidia-container-toolkit

Assuming that you already installed Podman (if not, follow the Fedora doc on installing Docker and/or Podman) and you have an GPU, you can download and run:

docker run --device nvidia.com/gpu=all -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.2

If SELinux blocks Podman from accessing the GPU, you can follow Podman's instruction about giving permission for containers to access the GPU: 

sudo setsebool -P container_use_devices true

Then run the `docker run ...` command above and it should start.

This a quick way to give Kokoro a try.  I'll probably try to run it next with Go by using the onnxruntime to load the Kokoro-Onnx model.

Monday, February 10, 2025

The Death of Software Engineering Is Premature

There is a lot to be excited about when it comes to advancement in language models.  The people that benefit the most are software engineers because it can enhance the productivity of knowledgeable engineers. 

While a simple prompt such as "build me a web app with a conversation interface," will result in a functional web app (this is great because building a basic commodity application is now accessible to everyone),  these aren't the type of applications software engineers are normally tasked to write.  Software engineers are tasked with building multi-layered and evolving software that can't be fully described with a few prompts.  To build the software that businesses need requires skilled and knowledgeable people to direct the engineering work of the AI.

I built a simple app to filter news headlines that is small enough to be digestible in a post that I think shows how far a layman can go with building software using LLMs and why software engineers are still needed.  

The First Prompt

Let's start with what many executives might think makes software engineers unnecessary: 

Build me an app that will filter out political articles from Google News.

ChatGPT understood enough to generate an app using keywords to filter the headlines.  One time it created a React app and a second time it used Python.  Both times it used newsapi.org to get the news feeds and require you to understand how to build and run the app.  The main issue is that the app isn't really what I wanted.  It provides news search which the results are then matched with keywords to decide what to filter out.  I wanted the news that I normally see when I visit news.google.com minus the political articles so I tell ChatGPT precisely that:

I don't want to search for news.  I want to see the articles that I see when visiting news.google.com minus political articles

The first time I asked ChatGPT it understood this enough to switch to using the Google News RSS feed which is excellent\!  The second time, it concluded that the way is to scrape news.google.com.  Both of these prompts highlight that some specialized knowledge is needed.  Does the programming language matter?  Should it use RSS or web scraping?  How do you run the code it generates?  Can these questions be ignored? Who in the organization does the CEO expect to be able to answer these questions?

The Second Prompt

While the CEO might not be able to use the AI, it is possible that a non-engineer or someone who doesn't know how to program can know enough to give more details in the prompt that improves on the first prompt.  A technical product manager could give the AI the extra details: 

  1. single page web app using material design   
    1. have a header with the company logo on the left  
    2. have a settings menu on the right side of the header  
    3. ...  
  2. web app will get the list of headlines to display   
  3. pull headlines from RSS feeds and have a LLM return the items that are not political  
  4. build it with language X on the backend and basic Javascript and CSS on the front end.

(4) was included to demonstrate how software often needs to fit in with a business' existing infrastructure.  If the AI returns a React app but there is no infrastructure to build and deploy React apps then we'd be looking at additional costs and efforts to add that ability simply because the AI had no knowledge of what is reasonable for a specific business.

LLMs are capable of generating this app although the generated code didn't actually work and I had to guide it to how to fix things but for now let's assume that the LLM generated a fully working app. 

If we were to stop here then we might conclude that software engineers aren't needed anymore, but I believe the conclusion is actually that for companies that previously did not have engineers that they now have access to some skills their organization never had before.  For example, a small company who has an office manager built a spreadsheet that they're running their business on would benefit because while their spreadsheet worked it is still limited.  The next step to expand the spreadsheet that would previously require some coding knowledge to extend the worksheet can now be done by the office manager with the aid of an AI.  There is cost saving for the company because they didn't have to hire a consultant (these companies probably don't have enough work to hire a full time engineer to be on staff).

Remember that I left out a number of things that the AI cannot do including turning the code into an app and deploying the app, but the biggest hurdle would be if the AI did not meet all the requirements for the app initially.  I had success prompting the AI to make certain changes and enhancements but multiple times the AI would make a change that didn't work and go further-and-further down the rabbit hole and no prompting got it to fix the problem until I have it very specific instructions on how to fix ("In function X, change the max/min values to be Y for variable V").  

Basically, maintaining, fixing and enhancing the app is where the challenge is and that's where most engineers are spending their time.

The Third Prompt

Prompt 3 (in real life was actually what I did first) was the fastest and most productive way towards building a working application and this was to have a design and give the AI many specific implementation instructions.  I knew the architecture, algorithm and code structure and used that knowledge to guide the AI so that it was essentially very fast typist of code:

  • Write a FetchRSS function that takes in a string value for the RSS feed configuration file path.  
  • In FetchRSS, open up the configuration file and loop through each line to get the URL.  
  • For each URL, fetch it and parse the RSS response into a slice of strings  
  • Oh, ignore any lines that are blank or starts with '\#'  
  • ...

Since I enjoyed typing out code, I tended to write my own stuff but for the "boring" codes (e.g. handling when err \!= nil) I'll have the AI write it.

I was able to complete the code much faster, make fewer trips to look up references and documentation and with few bugs from typos.

Here's the problem, though.  While AI is capable of generating valid code, it isn't "working" code.  It isn't code that can be used directly in a business' infrastructure.  AI is still struggling to understand an entire code base and writing code that works within a business requires also understanding the environment the code is running in.  If the code will be run in complete isolation then it might be able to run but even a simple function such as "GetUserName" depends on where the user name is stored?  What integration must the AI be aware of in order to get the user name?  In a real environment, the AI simply just gives code snippets that the engineer must still adapt into the organization's infrastructure.

Conclusion

Having an AI capable of building software for your business is not realistic.  My example app is too simplistic to be any company's product and it still wasn't able to do that.  

Having a knowledgeable person on staff using AI will increase your staff's ability to do their jobs better and allow companies to do things they previously couldn't but most of these things are not things software engineers typically do.   If companies try to have non-engineers do software engineering work with the AI will likely result in decreased productivity.  Any gains from having something done quickly at the beginning will also be quickly overshadowed by the inability to maintain and enhance the software.

It is ultimately a decision companies have to make for themselves.  Is a % reduction in compensating engineers a greater value then expanding by factors the productivity and capabilities of the engineers (reduce cost vs growth)? 

What is complex today will not be as complex tomorrow.  They will become common and AI will be able to take care of those things, but as AI takes over more of the common stuff that will allow engineers to tackle the next level of complexity with more originality because businesses will need it to survive.  There are leaders at technology companies boasting about being able to get rid of their software engineers or how they will stop hiring more engineers.   These leaders are making a choice to make cheap commodity products in exchange for growth and innovation, but they might find themselves racing to the bottom instead of accelerating to the top.  

Can AI someday outpace people?  Maybe.  But it's not now.  Declaring that engineers aren't needed is pre-mature.


Using AI/LLM to Implement a News Headline Filter

There are news topics that I'd rather avoid being bombarded with, but options for filtering headlines on the news sites are generally very limited.  You can tell Google News to block certain sources and there's an "fewer like this" option which doesn't seem to do anything, but neither will block topics (e.g."politics", "reality TV shows", etc. ) so you still end up getting bombarded with things you don't want to see even in your "personalized" views.  Fortunately, sites like Google News provide RSS feeds that makes it easier to get the list of headlines, links and descriptions and avoid trying to scrape websites which can be brittle as the sites can change their layout at any time.

I decided to write my own filter take out things that I'm not interested in and publish the result to a site that I can access from any browser.  The simplest way (and one that doesn't even need site to host the result) is to have a list of blocked words and drop any headlines with those words.  As long as the app can access the RSS feed even a mobile app can easily handle this kind of filtering, but creating and maintaining an effective block list becomes a challenge.  A political news article isn't going to be "Political Article on City Council Votes On Artificial Turf at Parks.  This is a good use case for using a language model to categorize a headline instead of using a keyword filter.

I created a prompt in natural language explaining what I don't want to see along with some additional details to add to common definitions and then sent my instructions with the list of headlines to the language model and have it return a filtered list:
Remove all political headlines.  Headlines that includes political figures, and celebrities who are active in politics such as Elon Musk should also be removed.
Being able to use natural language to handle categorization makes it so much easier to build the app.  The categorization would've been the most challenging part of the project, but the LLM allowed me to get good results very quickly (took just a few minutes to get the first results but some more time to tweak the prompt).  Another benefit with the large language models such as Gemini is that it understand multiple languages so while I gave my instructions in English, it can filter headlines in French, Chinese, Japanese, etc.

Using an language model does mean giving up some control such as relying on what the model interprets "political" to mean.  Prompts can help refine the model's interpretation but sometimes a headline makes it through the filter and it is not as easy to determine compared to being able to see the algorithm and determine the reasoning.

I encountered a problem where the LLM's response stopped midway.  This was because  LLMs have limits on input and output tokens (how much info you can send out and the size of the response it will send back).  The input token limit is so high that I'm not likely to have enough headlines to even remotely approach it's limit (Gemini is 1 million tokens for the free tier and 2 million tokens for the paid tier and I'll be using around 20,000).  The output limit is much smaller (~8k) so if I wanted it to send back the complete details of the filtered headlines (title and link) it won't be able to.  To address this problem, I send the LLM the headlines with an index and have it return just the index.  If it was a 100 headlines then size of the output is less than 200 tokens.

A cron job will execute the program and write the result as JSON to the server where the web page will load the JSON  and display the headlines:


The code is on Github and you can see it is very simple program.

Friday, September 27, 2024

My Software Developer Machine for LLM Inference

I had to assemble a new PC for LLM inference work since my existing development lacked a discrete GPU so running any local LLM was extremely slow.  My aim is a dedicated development machine running Linux and will not be use for any gaming.  I wanted to keep the budget between $1500 and $2500 and wanted it to be quiet and not take up much space.  In the end, I had to make some compromises and this is what I ended up with:

Components

Gigabyte GeForce RTX 4070 Ti Super with 16GB GPU

Traditionally, a development machine didn't need a graphics card but that's different when it comes to working with LLMs so I started with the GPU and picked the .  The two makers are Nvidia and AMD but it seems like AMD cards are less supported and would require more effort to get it working.  The general consensus is that VRAM is the most important thing for running LLMs and you'd want to try to load as much of the model into the VRAM as possible so I aimed for 12-16GB*.  I picked the Gigabyte card because its length is on the shorter side and wouldn't require as large of a case to support it.

*Rough estimating the amount of memory needed by a model is to take the number of parameters and multiply it by 2 bytes (using 16 bit parameter type = 2 bytes).  A 7B model would use 14GB, 8B use 16GB, etc.  

**"How come my GPU only have 12GB of VRAM but I can a 8B model which would use 16GB?"  You might be using a quantitized/compressed model or you're using both the GPU VRAM and system RAM.  For example, if you download the default Llama 8B parameters model from Ollama, it's quantization is 4 so it doesn't take the full 16GB of memory.   If you don't have enough VRAM, then Ollama will use both system ram and VRAM:

Loading the 27B parameter Gemma model with quantization of 4 shows that it requires 18GB of memory and Ollama loaded 82% of it into the GPU's VRAM (~14.7 GB):

> ollama ps
NAME            ID              SIZE    PROCESSOR       UNTIL              
gemma2:27b      53261bc9c192    18 GB   18%/82% CPU/GPU 4 minutes from now

Nvidia shows that 14GB of its 16GB is being used matching what Ollama says:

> nvidia-smi
...
 0   N/A  N/A      2338      C   ...unners/cuda_v12/ollama_llama_server    14020MiB 
...

AMD Ryzen 7 7700X CPU (8-core, 16-threads, 4.5GHz base, 5.5Ghz Max Boost)

A solid performer for development work with integrated graphics.  AMD's integrated graphics are pretty good and by running my windows manager and GUI through the integrated graphics allows me to save the GPU for the LLMs and not use any of the GPU's VRAM.

ASUS B650M-PLUS WIFI AM5 Motherboard 

I didn't want a fancy motherboard with RGB but I did want something that is compatible and can support modern peripheral.  

Corsair Vengeance DDR5 64GB Memory

Got the DDR5 memory for the speed and 64GB since when the VRAM isn't enough for the LLM some of it will be loaded into memory.

be quiet! Pure Rock 2 CPU Cooler

I don't plan to over clock this system and the Pure Rock is well rated for being quiet and affordable.

Corsair RM750e (2023) Power Supply

Corsair 4000D Airflow Mid-Tower ATX Case

Although I would much prefer a small form factor case, doing so limits the options for the GPU and other components so this is a compromize.  The 4000D case comes with two fans and has good airflow.  When the system doesn't get as hot, the fans don't have to work as hard and the system is quieter.

Crucial P3 Plus 2TB M.2 SSD

The 1TB was out-of-stock and the 2TB was still relatively well priced.

The total price for the system came under $1900 so I was able to stay in my budget range.

Assembly and Usage

All the components came together with no compatibility problems.  I was able to get it POST and installed a fresh copy of Fedora 40.  Wifi, sounds, video, etc. all worked on the first boot.

When I first installed Ollama, it installed with support for the AMD integrated GPU since I haven't installed the Nvidia drivers yet.  Once the drivers were installed, Ollama recognized and used the NVidia GPU.  Make sure you plug your display cable to the motherboard's video port and not the Nvidia card's ports.  If you do the latter, the window manager and everything else will use the Nvidia card by default.

You can check which GPU is used using nvidia-smi to see what is running on the Nvidia card.  For AMD:

glxinfo | grep "OpenGL renderer" # See what the system is using
sudo lsof /dev/dri/* # Shows what is running on it.

Saturday, September 21, 2024

Installing Nvidia and CUDA drivers on Fedora for Ollama

Ollama, a tool that lets you run Large Language Models (LLMs) on a local machine, supports the use of GPUs for better performance if you have a supported graphics card and corresponding drivers.  Having recently gotten an supported Nvidia card, I wanted to get it working with Ollama but found the available documentations on how to install the Nvidia and CUDA drivers confusing because there are multiple ways to install.  Depending on where you started your search for instructions, it can take you down different paths.

If you started on the Ollama install instruction page directs you to the Nvidia CUDA Toolkit Downloads page to have you add their CUDA repository to your Fedora instance.  From the repository you can install the CUDA toolkit, modules and drivers (CUDA and Nvidia).   For some reason, the repository currently are tagged for Fedora 37 and 39, but they seem to work for Fedora 40.  I'm not sure if that will always be the case or will work with future versions of Fedora.

If you first go to Nvidia's site to search for the driver, it will direct you to their drivers download page where you can download a .run script to install the Nvidia drivers (not CUDA).  This works but bypasses your package manager so I'm not sure if conflicts will arise in the future.  It also seems to be separate from what is in the CUDA repository so I'm not certain if there might be conflicts now or in the future.  As of this writing, installing the drivers from from the .run script and installing the CUDA toolkit from the repository does work, but I didn't install the Nvidia drivers from the repository.  

If you start with a web search or Fedora forums, the answer there is to install Nvidia from RPMFusion which has both the Nvidia and CUDA drivers.  This seems to be the most compatible version for Fedora.  If you're already using RPMFusion then it is really your only option since RPMFusion and Nvidia's repo are not compatible and will require you to do some DNF magic to get the two working together.  I also like this option because Ollama only needs the CUDA drivers and not the whole toolkit (I think you might be able to just grab the CUDA part from Nvidia's repo but their instruction directs you to download the whole toolkit).

Installing Nvidia and CUDA for Fedora

Here is how I installed a fresh new instance of Fedora 40 with Nvidia and CUDA drivers to work with Ollama.

I created a Fedora 40 Cinnamon Spin boot drive with the Fedora Media Writer and booted up the machine with it to do a clean install.  Once it finished with the installation, I rebooted the machine and set up a network connection so I can download updates and the drivers.

Open up a terminal and change the run level to 3 (multi-user command line --no GUI--)

sudo init 3

Because the first time you run sudo dnf update it'll probably update a whole bunch of the windowing systems and might cause your current window manager to crash, this avoids having the GUI and windowing system from running when you're doing the update.

Once in command-line mode, update the system with the latest packages and kernel:

sudo dnf update

Once it's been updated, reboot the system to be running on the latest kernel.  

sudo /sbin/reboot now

I went back into the level 3 since I'll be updating the graphics drivers but this time I did at the GRUB boot menu.  When the boot menu comes up, hit the 'e' key to edit and at the end of the linux line add the '3' and then CTRL-X to continue booting.   This change is not permanent. 

Install the developer tools needed to compile the modules:

sudo dnf groupinstall "Development Tools"
sudo dnf group install development-tools # dnf5

Now it's time to add the RPMFusion free and nonfree repos so Fedora knows where to download the drivers and modules.

You want to import the GPG key for the RPMFusion free and nonfree repos to verify that repo install packages are the actual ones:

sudo dnf install distribution-gpg-keys

sudo rpmkeys --import /usr/share/distribution-gpg-keys/rpmfusion/RPM-GPG-KEY-rpmfusion-free-fedora-$(rpm -E %fedora)

sudo rpmkeys --import /usr/share/distribution-gpg-keys/rpmfusion/RPM-GPG-KEY-rpmfusion-nonfree-fedora-$(rpm -E %fedora)

Add the repository to Fedora:

sudo dnf --setopt=localpkg_gpgcheck=1 install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm

sudo dnf --setopt=localpkg_gpgcheck=1 install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

Install the Nvidia drivers with:

sudo dnf install akmod-nvidia

Make sure that to give the system time to compile the modules AFTER the package install!  

Check that the amkod-nvidia is fully built with:

modinfo -F version nvidia

Install the CUDA drivers

sudo dnf install xorg-x11-drv-nvidia-cuda

Reboot the machine again and let it automatically go to the GUI (runlevel 5).

In a terminal, check that the Nvidia driver is being used:

nvidia-smi

Now you can install Ollama which should tell you that it has Nvidia GPU support at the end of the install.