Showing posts with label LLM. Show all posts
Showing posts with label LLM. Show all posts

Sunday, July 5, 2026

ComfyUI in a Container

Setting up ComfyUI in a Podman container on Fedora Linux with a Nvidia GPU.

Prepare Persistent Directories

Create directories on your host machine to store your models, custom nodes, outputs, and user data. This ensures your configurations and large model files survive container restarts and updates.

mkdir -p ~/comfyui/{models,custom_nodes,output,user}

Create the Container Image

We will build a custom image based on Debian (python:3.11-slim).

Note: We explicitly include the GNU C-compilers (gcc, g++) via apt. This is required because the Triton library (used by PyTorch) needs to dynamically compile GPU kernels when ComfyUI starts. Without them, Triton will throw a 0 active drivers error.

Create a file named Containerfile in your current directory with the following contents:

# Use a slim Python 3.11 image as the base (Debian-based)
FROM python:3.11-slim

WORKDIR /app

# Install system dependencies (including C compilers for Triton and OpenGL for OpenCV)
RUN apt-get update && apt-get install -y \
    git \
    libgl1 \
    libglib2.0-0 \
    gcc \
    g++ \
    && rm -rf /var/lib/apt/lists/*

# Clone the latest ComfyUI repository
RUN git clone https://github.com/comfyanonymous/ComfyUI.git .

# Install PyTorch with CUDA 13.0 support (Required for RTX 50-series)
RUN pip install --no-cache-dir torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130

# Install standard ComfyUI dependencies
RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 8188

# Bind to 0.0.0.0 so the host network can access the web interface
CMD ["python", "main.py", "--listen", "0.0.0.0"]

Build the image using Podman:

podman build -t comfyui-nvidia .

Deploy the Container

Run the container using the configuration below.

Critical Fedora-Specific Flags to Note:

  • --security-opt label=disable: This disables SELinux separation specifically for this container, preventing SELinux from blocking the execution of the injected Nvidia driver binaries (libcuda.so).
  • :Z (on volume mounts): Instructs Podman to label the mounted host directories with a private unconfined SELinux context, allowing the container read/write access.
podman run -d \
  --name comfyui \
  --security-opt label=disable \
  --device nvidia.com/gpu=all \
  -p 8188:8188 \
  -v ~/comfyui/models:/app/models:Z \
  -v ~/comfyui/custom_nodes:/app/custom_nodes:Z \
  -v ~/comfyui/output:/app/output:Z \
  -v ~/comfyui/user:/app/user:Z \
  localhost/comfyui-nvidia

Automate Firewall for Remote Access (Optional)

If you want to access ComfyUI from another device on your network, you must open port 8188 in Fedora's firewall. Instead of leaving the port open permanently, you can use a wrapper script that temporarily opens it when ComfyUI starts, and securely closes it when the container stops.

Create a file named start_comfyui.sh:

#!/bin/bash

echo "Opening firewall port 8188..."
sudo firewall-cmd --add-port=8188/tcp

echo "Starting ComfyUI container..."
# The --attach flag keeps the script running until the container is stopped
podman start --attach comfyui

echo "Container stopped. Closing firewall port..."
sudo firewall-cmd --remove-port=8188/tcp

Make the script executable and run it whenever you want to start your server for remote access:

chmod +x start_comfyui.sh
./start_comfyui.sh

Access and Day-to-Day Management

Once deployed, you can access the ComfyUI web interface by opening your browser and navigating to:
http://localhost:8188 (or your machine's local IP address if accessing from another device on the network).

Useful Podman Commands

  • Check Startup Logs: Monitor the initialization process or kernel compilations:
    podman logs -f comfyui
  • Stop the Server:
    podman stop comfyui
  • Start the Server:
    podman start comfyui
  • Clean up Disk Space: After rebuilding images (e.g., updating ComfyUI in the future), run this to delete dangling, unnamed images and recover gigabytes of storage space:
    podman image prune

Monday, May 4, 2026

Chrome and Fedora Crashes With Multi-Monitor Setup

 I've noticed with more frequency that after waking from sleep, Chrome or some of its tabs will crash.  It can happen immediately or shortly after.  A kernel crash or system freeze  happens less frequently but is more disruptive when they happen.


This is happening on my main system that has 3 monitors (2 are 120Hz), an AMD CPU with integrated graphics and a discrete Nvidia GPU.  The monitors are connected to the motherboard (1 HDMI and 2 DisplayPort that are daisy-chained) rather than to the discrete graphics so I can preserve as much VRAM for LLM inference.

Initially, I though the culprit was the Nvidia driver since the most common responses on the Internet mention them.  The next most common response was to turn off hardware acceleration in Chrome. 

chrome://gpu revealed that the iGPU was only being allocated 512MB of memory which not enough with a multi-monitor setup.  This was being automatically set in the BIOS so I changed the UMA Frame Buffer Size (Found under NB Configuration)  to 1GB.

Next, Chrome showed it knew there are two graphics option so this tells it to almost use the integrated graphics.  

google-chrome-stable --disable-gpu-driver-bug-workarounds --gpu-testing-vendor-id=<vendor id> --gpu-testing-device-id=<device id>

There are some settings for the Nvidia card that might also improve stability.  Edit /etc/modprobe.d/nvidia.conf and add the following settings might improve stability but might also slow waking up:

options nvidia "NVreg_PreserveVideoMemoryAllocations=1"
options nvidia "NVreg_TemporaryFilePath=/var/tmp"
options nvidia "NVreg_EnableS0ixPowerManagement=1"
options nvidia "NVreg_PreserveVideoMemoryAllocations=1"

To make sure the system leverage these values:

sudo dracut -f 
sudo reboot

Tip: Restart the Cinnamon process without losing your open windows by pressing Alt + F2, typing r, and hitting Enter

Friday, September 27, 2024

My Software Developer Machine for LLM Inference

I had to assemble a new PC for LLM inference work since my existing development lacked a discrete GPU so running any local LLM was extremely slow.  My aim is a dedicated development machine running Linux and will not be use for any gaming.  I wanted to keep the budget between $1500 and $2500 and wanted it to be quiet and not take up much space.  In the end, I had to make some compromises and this is what I ended up with:

Components

Gigabyte GeForce RTX 4070 Ti Super with 16GB GPU

Traditionally, a development machine didn't need a graphics card but that's different when it comes to working with LLMs so I started with the GPU and picked the .  The two makers are Nvidia and AMD but it seems like AMD cards are less supported and would require more effort to get it working.  The general consensus is that VRAM is the most important thing for running LLMs and you'd want to try to load as much of the model into the VRAM as possible so I aimed for 12-16GB*.  I picked the Gigabyte card because its length is on the shorter side and wouldn't require as large of a case to support it.

*Rough estimating the amount of memory needed by a model is to take the number of parameters and multiply it by 2 bytes (using 16 bit parameter type = 2 bytes).  A 7B model would use 14GB, 8B use 16GB, etc.  

**"How come my GPU only have 12GB of VRAM but I can a 8B model which would use 16GB?"  You might be using a quantitized/compressed model or you're using both the GPU VRAM and system RAM.  For example, if you download the default Llama 8B parameters model from Ollama, it's quantization is 4 so it doesn't take the full 16GB of memory.   If you don't have enough VRAM, then Ollama will use both system ram and VRAM:

Loading the 27B parameter Gemma model with quantization of 4 shows that it requires 18GB of memory and Ollama loaded 82% of it into the GPU's VRAM (~14.7 GB):

> ollama ps
NAME            ID              SIZE    PROCESSOR       UNTIL              
gemma2:27b      53261bc9c192    18 GB   18%/82% CPU/GPU 4 minutes from now

Nvidia shows that 14GB of its 16GB is being used matching what Ollama says:

> nvidia-smi
...
 0   N/A  N/A      2338      C   ...unners/cuda_v12/ollama_llama_server    14020MiB 
...

AMD Ryzen 7 7700X CPU (8-core, 16-threads, 4.5GHz base, 5.5Ghz Max Boost)

A solid performer for development work with integrated graphics.  AMD's integrated graphics are pretty good and by running my windows manager and GUI through the integrated graphics allows me to save the GPU for the LLMs and not use any of the GPU's VRAM.

ASUS B650M-PLUS WIFI AM5 Motherboard 

I didn't want a fancy motherboard with RGB but I did want something that is compatible and can support modern peripheral.  

Corsair Vengeance DDR5 64GB Memory

Got the DDR5 memory for the speed and 64GB since when the VRAM isn't enough for the LLM some of it will be loaded into memory.

be quiet! Pure Rock 2 CPU Cooler

I don't plan to over clock this system and the Pure Rock is well rated for being quiet and affordable.

Corsair RM750e (2023) Power Supply

Corsair 4000D Airflow Mid-Tower ATX Case

Although I would much prefer a small form factor case, doing so limits the options for the GPU and other components so this is a compromize.  The 4000D case comes with two fans and has good airflow.  When the system doesn't get as hot, the fans don't have to work as hard and the system is quieter.

Crucial P3 Plus 2TB M.2 SSD

The 1TB was out-of-stock and the 2TB was still relatively well priced.

The total price for the system came under $1900 so I was able to stay in my budget range.

Assembly and Usage

All the components came together with no compatibility problems.  I was able to get it POST and installed a fresh copy of Fedora 40.  Wifi, sounds, video, etc. all worked on the first boot.

When I first installed Ollama, it installed with support for the AMD integrated GPU since I haven't installed the Nvidia drivers yet.  Once the drivers were installed, Ollama recognized and used the NVidia GPU.  Make sure you plug your display cable to the motherboard's video port and not the Nvidia card's ports.  If you do the latter, the window manager and everything else will use the Nvidia card by default.

You can check which GPU is used using nvidia-smi to see what is running on the Nvidia card.  For AMD:

glxinfo | grep "OpenGL renderer" # See what the system is using
sudo lsof /dev/dri/* # Shows what is running on it.

Saturday, September 21, 2024

Installing Nvidia and CUDA drivers on Fedora for Ollama

Ollama, a tool that lets you run Large Language Models (LLMs) on a local machine, supports the use of GPUs for better performance if you have a supported graphics card and corresponding drivers.  Having recently gotten an supported Nvidia card, I wanted to get it working with Ollama but found the available documentations on how to install the Nvidia and CUDA drivers confusing because there are multiple ways to install.  Depending on where you started your search for instructions, it can take you down different paths.

If you started on the Ollama install instruction page directs you to the Nvidia CUDA Toolkit Downloads page to have you add their CUDA repository to your Fedora instance.  From the repository you can install the CUDA toolkit, modules and drivers (CUDA and Nvidia).   For some reason, the repository currently are tagged for Fedora 37 and 39, but they seem to work for Fedora 40.  I'm not sure if that will always be the case or will work with future versions of Fedora.

If you first go to Nvidia's site to search for the driver, it will direct you to their drivers download page where you can download a .run script to install the Nvidia drivers (not CUDA).  This works but bypasses your package manager so I'm not sure if conflicts will arise in the future.  It also seems to be separate from what is in the CUDA repository so I'm not certain if there might be conflicts now or in the future.  As of this writing, installing the drivers from from the .run script and installing the CUDA toolkit from the repository does work, but I didn't install the Nvidia drivers from the repository.  

If you start with a web search or Fedora forums, the answer there is to install Nvidia from RPMFusion which has both the Nvidia and CUDA drivers.  This seems to be the most compatible version for Fedora.  If you're already using RPMFusion then it is really your only option since RPMFusion and Nvidia's repo are not compatible and will require you to do some DNF magic to get the two working together.  I also like this option because Ollama only needs the CUDA drivers and not the whole toolkit (I think you might be able to just grab the CUDA part from Nvidia's repo but their instruction directs you to download the whole toolkit).

Installing Nvidia and CUDA for Fedora

Here is how I installed a fresh new instance of Fedora 40 with Nvidia and CUDA drivers to work with Ollama.

I created a Fedora 40 Cinnamon Spin boot drive with the Fedora Media Writer and booted up the machine with it to do a clean install.  Once it finished with the installation, I rebooted the machine and set up a network connection so I can download updates and the drivers.

Open up a terminal and change the run level to 3 (multi-user command line --no GUI--)

sudo init 3

Because the first time you run sudo dnf update it'll probably update a whole bunch of the windowing systems and might cause your current window manager to crash, this avoids having the GUI and windowing system from running when you're doing the update.

Once in command-line mode, update the system with the latest packages and kernel:

sudo dnf update

Once it's been updated, reboot the system to be running on the latest kernel.  

sudo /sbin/reboot now

I went back into the level 3 since I'll be updating the graphics drivers but this time I did at the GRUB boot menu.  When the boot menu comes up, hit the 'e' key to edit and at the end of the linux line add the '3' and then CTRL-X to continue booting.   This change is not permanent. 

Install the developer tools needed to compile the modules:

sudo dnf groupinstall "Development Tools"
sudo dnf group install development-tools # dnf5

Now it's time to add the RPMFusion free and nonfree repos so Fedora knows where to download the drivers and modules.

You want to import the GPG key for the RPMFusion free and nonfree repos to verify that repo install packages are the actual ones:

sudo dnf install distribution-gpg-keys

sudo rpmkeys --import /usr/share/distribution-gpg-keys/rpmfusion/RPM-GPG-KEY-rpmfusion-free-fedora-$(rpm -E %fedora)

sudo rpmkeys --import /usr/share/distribution-gpg-keys/rpmfusion/RPM-GPG-KEY-rpmfusion-nonfree-fedora-$(rpm -E %fedora)

Add the repository to Fedora:

sudo dnf --setopt=localpkg_gpgcheck=1 install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm

sudo dnf --setopt=localpkg_gpgcheck=1 install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

Install the Nvidia drivers with:

sudo dnf install akmod-nvidia

Make sure that to give the system time to compile the modules AFTER the package install!  

Check that the amkod-nvidia is fully built with:

modinfo -F version nvidia

Install the CUDA drivers

sudo dnf install xorg-x11-drv-nvidia-cuda

Reboot the machine again and let it automatically go to the GUI (runlevel 5).

In a terminal, check that the Nvidia driver is being used:

nvidia-smi

Now you can install Ollama which should tell you that it has Nvidia GPU support at the end of the install.

Sunday, January 17, 2021

My Systems (2021)

Updated 8/7/2025 - Self built daily driver

Updated 5/21/2023 with Beelink system

2021 brings upgrades to the computers in the house that has been fairly static over the past 7-8 years.   I got a couple of new systems and repurposed some parts from the old systems so this post is mainly to inventory the new configurations for my own reference.

Daily Driver (Self-Built)

For years, I've went with mini PCs because they were quiet and did not take up much space.  I had mine on the keyboard tray under the table top so they were out-of-the-way.  The shift to a regular desktop PC was triggered by two things:  
  1. The deprecation of Windows 10 meant that my kids will need a new PC that can run Windows 11.
  2. The use of local LLMs requires a GPU.
The PN50 4800U was my daily driver and came with Windows 11 although I used Linux on it 99.9% of the time.  It obviously cannot have an GPU which made trying LLMs on it painfully slow.  Instead of getting another PC, I decided to retire the kid's PC and replace it with my 4800U.   I can't do anything about the size of the desktop PC, but the beQuiet CPU fan addressed the noise concern.  The desktop is very quiet and normally I can't tell if it is on by sound and have to look to see if the fans are spinning.

The larger case allowed me to install a GPU that'll enable me to run local LLMs.  This time, I'm just running Linux and not bothering with Windows.  If I do need Windows, I have a laptop that still has it or I'll just borrow the kids.

Asus PN50 4800U

  • Ryzen 7 4800U [Zen2] (8 cores / 16 threads, base clock 1.8GHz, max 4.2GHz - 8 GPU cores - RX Vega 8, 15W)
  • 32 GB Crucial DDR4 3200Mhz  RAM (2x16GB)
  • 1TB Samsung 970 EVO Plus (M.2 NVMe interface) SSD
  • 500GB Crucial MX500 SATA SSD (2.5")
  • Intel WIFI 6, BT 5.0

The PN50 replaced my trusty Shuttle DS87 as my daily driver.  All the components are new except for the two Dell U2311H monitors, keyboard and mouse.  I added a third monitor, Dell U2421HE, because it has ethernet and a USB-C interface that can connect with the PN50 for DisplayPort, USB hub and ethernet.  This lowered the number of cables between the PN50 and peripherals and reduced clutter on my desk.

Despite its small size, the PN50 is still very high performing, but you do pay a price premium for having something small AND fast.  I did lose some connectivity (fewer USB ports and only 1 Ethernet port).  The USB ports can be addressed with a hub and I use the monitor's built-in ethernet port for the one I loss since the PN50 only has one.

The Dell AC511M is a USB soundbar and can be attached to the monitor stand (not to the monitor itself).  It draws its power from the USB connection but I found three flaws with it: 
  1. It has no power button so it turns on when the PC turns on.  To use the audio-in jack and the speaker means the PC must be turned on. 
  2. The speaker has a hiss to it like many speakers but with no power button the hiss is always there.  I had to plug something into the headphone jack so I don't hear it.
  3. When something is plugged into the audio-in jack no audio goes through the USB.  If there are two audio sources (e.g. PC and music player) they need to share a connection.  I have two PCs connected to the monitor (one on display port and one on hdmi) and I can't have one play through USB and one through the audio in without plugging-and-unplugging the audio-in cable.  Instead, I have a cable from the monitor's audio-out to the soundbar's audio-in and each machine plays through the DP/HDMI outputs.
My ideal system would still be to have a something like the DS87 housing with a Ryzen 4700G (or 5700G), but those CPUs aren't very readily available and there is not DS87-like small form factor cases for them.  Update:  The day after I posted this, Shuttle announced this exact PC.  ^_^;  With my new monitor, though, I would like the new Shuttle to have an USB Type C connection, but at least if I wanted to get more power then I know the option is now out there!

Asus PN50 4500U

  • Ryzen 5 4500U [Zen2] (6 cores / 6 threads, base clock 2.3GHz, max 4.0GHz - 6 GPU cores - RX Vega 6, 15W)
  • 2x 8GB 3200 DDR4 so-dimm by SK hynix
  • Intel 660p Series m.2 500GB SSD
  • Intel WI-FI 6 (GIG+) + BT 5.0
  • *Crucial 128BG m4 2.5" SSD

This system replaces my wife's Shuttle XH61 system and is an upgrade across the board over its predecessor.

System 3 (Shuttle DS87)

  • Shuttle PC DS87
  • Intel Core i7-4790S Processor (4 cores / 8 threads, 8M Cache, base clock 3.2 GHz, max 4.0GHz, 65W)
  • Samsung 850 EVO 500GB 2.5-Inch SATA III Internal SSD (MZ-75E500B/AM)
  • 2 x Crucial 16GB Kit (8GBx2) DDR3 1600 MT/s (PC3-12800)
  • *Intel Network 7260.HMWG WiFi Wireless-AC 7260 H/T Dual Band 2x2 AC+Bluetooth HMC
  • *Samsung 840 EVO Series 120GB mSATA3 SSD

This Shuttle had been my reliable daily driver for over 6 years running Linux.  I repurposed an Samsung SSD and Intel wireless card from my Asus VivoMini to install Windows and add WIFI and bluetooth to the system.   The antennas that was in the VivoMini was hard to extract so I took the antennas from an old ASUS Chromebook laptop that wasn't being used anymore.  

The VivoMini was being used for kid's remote/distance learning but was a bit under-powered for handling some of the video conferencing features so this system will now take its place.

ASUS PN50 4300U

  • Ryzen 3 4300U [Zen2] (4 cores / 4 threads, base clock 2.7GHz, max 3.7GHz - 5 GPU cores - RX Vega 5, 15W)
  • 16 GB Crucial (CT8G4SFRA32A) DDR4 3200Mhz  RAM (2x8 GB)
  • 500GB Samsung 970 EVO Plus (M.2 NVMe interface) SSD

This system is meant to be a more portable system for when I'm working at another location.  I paired this up with a portable monitor rather then getting a laptop since I don't need this to be a mobile system but one that I can easily transport.


Beelink SER5

This latest addition was added in 2023 as a secondary gaming PC and is used for school.  The specs are decent given the price of under $300 including Windows.

  • Ryzen 5 5500U (6 cores / 12 threads, base clock 2.1GHz, max 4.0 GHz, 7 core GPU,  @ 1800 MHz, 15W TDP)
  • 16 GB DDR4
  • 500GB NVME M.2 SSD
  • WiFi 6
  • BT 5.2
The system came with Windows 11 Pro.

----------------

System 4 (Shuttle XH61)

  • Intel Core i7-2600S Processor (4 cores / 8 threads, 8M Cache, base clock 2.8 GHz, max 3.8GHz, 65W)
  • *Seagate 300GB 7200RPM HDD Cosair MX500 CT500MX500SSD1 500GB 2.5in SATA 6Gbps SSD
  • TP-Link USB WiFi Adapter for Desktop PC, AC1300Mbps USB 3.0 WiFi Dual Band Network Adapter with 2.4GHz/5GHz High Gain Antenna, MU-MIMO
  • 8GB RAM

This system was originally put together in 2012 (with an SSD) and even in 2020 was a perfectly good system for most tasks.  When running Windows 10 or some basic games (Minecraft, Don't Starve) it still felt pretty snappy.  I wouldn't try running any graphics intensive games on it.  

The SSD from this system was moved to the PN50-4500U (system 2) and replaced with a 2.5" Seagate 300GB 7200RPM hard disk drive that I pulled out of the Chromebook laptop that I pulled the antenna from.  After switching to the mechanical disk drive, the system felt noticeably sluggish.  A solid state drive makes a big difference!  

I'm keeping this system around for schooling.

ASUS VivoMINI UN62


The ASUS VivoMini UN62 is a wonderfully small and quiet bare bones system with very good build quality.  It was this system that gave me confidence in getting the ASUS PN50.  I actually own 3 of these system and use them for different purposes which have changed over time (e.g. media station, always-on server for minecraft, etc).  More recently, however, the Raspberry Pi 4  have replaced the VivoMinis for some of the tasks.

The specs for my UN62s are:
  • Intel i3-4030U (2 cores / 4 threads, 1.9 GHz, 3 MB cache, 15W)
  • 16GB Crucial (2x8 GB DDR3-1600) 204-pin sodimm
  • Samsung 840 EVO 128GB msata3 SDD
  • Intel Network 7260.HMWG WiFi Wireless-AC 7260 H/T Dual Band 2x2 AC+Bluetooth HMC
Two served as the kids' computers until I upgraded their setup.  One was repurposed as the machine for schooling when remote/distance learning was put in place due to covid-19.  This system was replaced by System 3 and its drive and wireless card got moved to that system.

Raspberry Pi 4

  • Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz
  • 4G BLPDDR4-3200 SDRAM
  • 2.4 GHz and 5.0 GHz IEEE 802.11ac wireless, Bluetooth 5.0, BLE
  • Gigabit Ethernet
  • 2 USB 3.0 ports; 2 USB 2.0 ports.
  • Raspberry Pi standard 40 pin GPIO header (fully backwards compatible with previous boards)
  • 2 × micro-HDMI ports (up to 4kp60 supported)
  • 2-lane MIPI DSI display port
  • 2-lane MIPI CSI camera port
  • 4-pole stereo audio and composite video port
  • H.265 (4kp60 decode), H264 (1080p60 decode, 1080p30 encode)
  • OpenGL ES 3.0 graphics
  • Micro-SD card slot for loading operating system and data storage
  • 5V DC via USB-C connector (minimum 3A*)
  • 5V DC via GPIO header (minimum 3A*)
  • Power over Ethernet (PoE) enabled (requires separate PoE HAT)
  • Operating temperature: 0 – 50 degrees C ambient
  • Raspberry Pi ICE Tower Cooler, RGB Cooling Fan (excessive but looks cool on the desk).

The Raspberry Pi 4 is a small wonder of a machine that replaces what I originally used the ASUS VivoMini for and is significantly cheaper.