Websolutions · company event

Self-hosted AI

Running serious AI models on hardware we own

Ahmad · November 2026

01 · Why self-hosted AI

This app was written by an AI in my house

[Screenshot: the spreadsheet web app from the coding test]

95/100

automatic grader score

28 min

from spec to working app

0

prompts sent to an AI company

01 · Why self-hosted AI

Same kind of model, different address

Cloud AI

Your question

Internet

Someone else’s data centre

Answer

Self-hosted AI

Your question

Your own computer

Answer

The model is one big file, tens to hundreds of gigabytes. Download it once and it runs without asking anyone. Open models come from Qwen (Alibaba), DeepSeek, Google, NVIDIA and others.

01 · Why self-hosted AI

Five reasons self-hosted AI is needed

Privacy

Client code and data never leave the building.

Cost

No bill per word. One coding task used 23 million tokens.

Availability

Works when the cloud is down, slow or rate-limited.

Control

We pick the model, tune its speed, and it never changes overnight.

Training

We teach an open model our own work, with our own data. The result is a file we own.

For the engineers: agent run = Qwen Code 0.26.1, 213 model calls, 23.1M input tokens (90% from cache), 263K output. Free-tier example: Groq, 8K tokens per minute (checked 1 Nov 2026). Item model = Qwen3.5 0.8B, full fine-tune on 40,000 generated examples, 55 min on one RTX 3090.

PRIVACY: We handle client code, contracts and personal data. Paste that into a cloud chatbot and it leaves the company. Free tiers often pay with your data: Mistral’s free plan, for example, may use your requests to train its models. Paid plans have better terms, but the data still leaves and you rely on a contract. Self-hosted, the question never comes up. COST: Cloud AI is billed per token, roughly per piece of a word. Agents use a lot: to build that app, one coding agent made 213 calls to the model and read 23 million tokens. At home the hardware is paid once and the running cost is electricity. AVAILABILITY: Cloud services go down, slow down at peak times, and free tiers are tight. One free provider we tested allows 8,000 tokens per minute, while a single request from my home assistant is 20,000 to 40,000. The home assistant keeps working without the internet. CONTROL: A provider can change or retire a model overnight. On our own hardware we choose the model and when to upgrade, and we can tune its speed: we made our main model about 60% faster with software alone. TRAINING: We can also teach a model new skills. Take an open model, train it on our own work with our own data, and the result is a file we keep and run anywhere. Cloud providers do offer fine-tuning, but only for some of their models, the training data goes to them, and the tuned model usually runs only on their platform. Our proof: a tiny model, 0.8 billion parameters, learned to design game items in 55 minutes on one graphics card, from 40,000 examples. You’ll try it on your phones in Part 4.

01 · Why self-hosted AI

The honest trade-off

Cloud still wins at

The very smartest models
Zero setup
Scaling to many users

Self-hosted wins at

Private data
Heavy daily use
Offline and predictable

Use both: self-hosted by default for private and bulk work, cloud for the hardest problems.

02 · The challenges

The hardware keeps getting more expensive

Used RTX 3090 (in our server)

$1,199

$700 a year ago · +71%

32 GB of DDR5 memory

$375

under $100 a year before · ~4×

RTX 5090 (newest gaming card)

$4,288

$1,999 at launch · over 2×

Why: AI data centres are buying up the world’s memory chips. Server memory rose 64% in just six months.

For the engineers: US prices. RTX 3090 = eBay median of daily lows, Nov 2025 vs Nov 2026 (bestvaluegpu.com). DDR5 = cheapest 32 GB kit, PCPartPicker via Tom’s Hardware, 3 Jun 2026. RTX 5090 = Tom’s Hardware GPU Price Index, 3 Aug 2026. Server memory = TrendForce, DRAM contract prices in the second half of 2025 (25 Aug 2026).

The first challenge is money, and it keeps getting worse. The graphics card in my AI server is an RTX 3090, a card from 2020. A year ago a used one cost about 700 dollars. Today it’s about 1,200, for a five-year-old card. Memory is worse: a normal 32 GB kit for a new PC went from under 100 dollars to 375 in a year. And the newest gaming card, the RTX 5090, launched at 2,000 dollars and sold for over 4,000 this summer. The reason is AI itself. The big data centres are buying up memory chips, and memory makers sell to them first, so everyone else pays more. Server memory prices rose 64% in the second half of last year alone, and they are still climbing. For us that means my server’s two graphics cards would cost about 2,400 dollars to buy again today, against 1,400 a year ago. [Optional: what the whole server cost you.] So that’s the first challenge. The second one is time.

02 · The challenges

Self-hosted AI is a stack. Every layer can make it fast, slow, or broken.

6

Model

which model, and how much it is compressed

5

Engine

the program that runs the model

4

Operating system

drivers, kernel, services

3

BIOS

firmware switches on the motherboard

2

Hardware tuning

power limits, clocks, voltages

1

Hardware

GPUs, memory, processor, power supply

02 · Layer 1: Hardware

Memory is the wall

936 GB/s

Graphics card memory (RTX 3090)

127 GB/s

Main memory, with the new CPU (was 72)

A reader and a bookshelf

For every word, the model fetches a big part of its file. A wider corridor means faster writing.

2 × 24 GB of GPU memory
256 GB of main memory, ~7× slower
The old CPU capped it at 72 GB/s
New 32-core CPU since 4 Nov: 127 GB/s

For the engineers: 3945WX = 2 chiplets × ~35 GB/s, 72 GB/s measured; 3975WX (4 chiplets): 118 GB/s, 127 with RAM at DDR4-2933. Ceiling ≈ 550B parameters at 4-bit. No NVLink: the cards take turns, 6.2 + 6.0 ms per token.

02 · Layer 2: Hardware tuning

Squeezing the hardware

At stock power, the two graphics cards tripped the power supply.

Tweak	Result	Verdict
Power limit 275 → 300 W per card	+3 to 4% prompt reading	Kept
Undervolting the GPUs	Writing 3.5% slower	Dropped
GPU memory overclock	+1%, temperature unreadable	Dropped
RAM 2666 → 2933 MT/s	+7% memory bandwidth	Kept, watching 2 corrected errors

For the engineers: stock power tripped a 45 A rail (Xid 79). 300 W plus a voltage offset without a clock lock was unstable (Xid 13 and 43). RAM with Infinity Fabric 1:1: 72.9 → 78.3 GB/s, ECC on.

02 · Layer 3: BIOS

Settings nobody touches, until AI needs them

Re-Size BAR on: lets the two GPUs talk to each other directly

IOMMU mode: set so GPU-to-GPU traffic doesn’t crash

Memory speed: RAM and the processor’s internal clock, set by hand

Screen on the motherboard’s own graphics: both GPUs do only AI

Remote power: the board’s management chip switches it on and off

The catch: the first changes needed someone at the machine with a monitor and a keyboard.

02 · Layer 4: Operating system

The patch that changed nothing

11 → 26 GB/s

GPU-to-GPU link after patching NVIDIA’s driver

0%

change in how fast answers come

Two more traps

An automatic kernel update quietly brings back the stock driver
A new kernel swapped the GPU order on the gaming PC, so models loaded on the wrong card

Lesson: measure, don’t assume.

For the engineers: NVIDIA open modules 595.99.02 + tinygrad BAR1 P2P patch: 26.4 GB/s one way, 51.2 both ways, 1.07 µs latency vs 11–14 µs.

At the operating system level we patched NVIDIA’s driver so the cards could talk to each other directly. It worked: the link went from 11 to 26 gigabytes per second, and latency dropped tenfold. And the speed of answers? Unchanged. The cards just didn’t need to talk much. That’s the biggest lesson of the whole project: measure, don’t assume. Two more traps: a patched driver is built for one kernel, so an automatic update quietly restores the stock one. And on my friend’s gaming PC a new kernel swapped the order of the two graphics cards, so models suddenly loaded on the wrong card. Now we pick cards by name. OPTIONAL (your call): a kernel module that unlocks the 3090s’ full tensor speed for some math gave 11% faster prompt reading. Shared-server gotcha: an engine’s default prompt cache grew a small model’s server from 2.5 to 10.4 GB of RAM until we capped it.

02 · Layer 5: Engine

Same model, same box, almost 3× faster

Reading the prompt: home server, no GPU, same 9B model

Ubuntu’s packaged engine

23 tokens/s

ik_llama.cpp, a faster fork

64 tokens/s

Writing the answer: 6.6 → 8.7 tokens/s (+30%). One wrong setting: −27%.

96 min → under 3 min

DeepSeek reading a long document: popular engine vs a tuned, specialised one

+28 to 60%

Guess-and-check: a small helper drafts the next words, the big model checks them all at once

For the engineers: i7-13620H, Qwen3.5-9B Q4_K_M, pp512/tg128: 23.1/6.62 → 63.9/8.65. DeepSeek V4.1 Flash, 256K: llama.cpp at ubatch 512 → maya-ds4 + int8 indexer, 165.6 s. MTP drafter, 4-bit experts.

The engine is the program that runs the model file, like a media player runs a video. There are many engines and they differ enormously. On my home server, same model, same machine, a different engine read prompts 2.8 times faster and wrote 30% faster. With DeepSeek’s big model, reading a long document took 96 minutes on the popular engine at its starting settings, and under 3 minutes on a specialised engine after tuning. One wrong setting cost a quarter of the speed. And one trick gave us up to 60%: a small fast helper guesses the next few words, and the big model checks them all at once, like an intern drafting and a boss approving. For the engineers: ik -rtr on Q4_K_M cost 27% decode; adding E-core threads dropped 6.7 to 4.3 tok/s. MTP drafter on Flash-Next: essay +28 to 40%, code +54 to 60%.

02 · Layer 6: Model

Picking the model, and shrinking it

4-bit = JPEG for models

16-bit original

4-bit: 4× smaller

Our rule: 4-bit, never lower.

Quality after shrinking

Model	4-bit	3-bit	2-bit
DeepSeek V4.1 Flash	98.2%	96.5%	90.5%
GLM-5.3-Flash	92.2%	86.3%	78.3%

How often the shrunk model picks the same next word as the original (Unsloth’s published tests)

Trap: some “4-bit” downloads hide 2- and 3-bit parts inside. We check every file.

Better file: NVIDIA’s version of our main model, trained for 4-bit, ran 16 to 48% faster with the same coding score.

Then the model. The smartest open models have hundreds of billions of parameters and would need terabytes at full precision. So we compress them, like JPEG for photos: 16-bit numbers become 4-bit numbers, four times smaller. At 4 bits the loss is tiny: DeepSeek’s 4-bit file picks the same next word as the original 98% of the time. At 2 bits another model drops to 78%. So our rule is 4-bit, never lower. Watch out: some “4-bit” downloads hide 2- or 3-bit parts inside, so we check every file. And the same model can come in a better file: NVIDIA’s version of our main model, trained to live in 4 bits, ran 16 to 48% faster with the same coding score. For the engineers: smartest open models that fit Machine 1 (Artificial Analysis index, 28 Oct 2026): DeepSeek V4.1 Flash 45 (552B, 16B active), GLM-5.3-Flash 42, Qwen3.8 Flash-Next 40 (~180B, 6B active), Qwen3.8 27B 34. Traps found: a stock drafter with 2-bit experts, a 35B drafter layer with Q3_K tensors. The conversation memory (KV cache) is 4-bit by default too. The two rows of the table come from different reference runs, so compare within a row, not across.

02 · What each layer gave us

Small gains at the bottom, big gains at the top

6

Model

+16 to 48% with a better 4-bit file

5

Engine

up to 2.8× by switching engine · +28 to 60% from guess-and-check

4

Operating system

0% from the driver patch

3

BIOS

an enabler, 0% on its own

2

Hardware tuning

+3 to 7%, at some cost in stability

1

Hardware

sets the ceiling: what fits, and how fast it can go

02 · The challenges

Every machine needs its own recipe

Machine	What it took
Machine 1 · AI server, 2 × RTX 3090	Power caps, a RAM overclock, BIOS switches, a patched driver, our own engine
Machine 2 · mini PC, Ryzen AI Max	Kernel settings to give the GPU 54 GB; one engine wins chat, another agents
Machine 3 · gaming PC, AMD + NVIDIA	Two GPU makers in one box; cards picked by name after a kernel update
Machine 4 · MacBook, M1 Max	A different engine (MLX), 8-bit conversation memory, a heat watchdog
Machine 5 · home server, no GPU	A faster engine, pinned to the fast cores, with a capped prompt cache

~2 hours

to measure one engine with one 27B model. The mini PC alone has 6 models × 2 to 3 engines.

[N] weeks

of my evenings so far. Every new model, engine release or kernel update starts it again.

For the engineers: 2: ttm.pages_limit; Vulkan wins chat, ROCm + MTP 2 wins agents by 15%. 3: RX 9070 XT on HIP (K q8 / V q4) + RTX 4060 on CUDA. 4: mlx-lm, 128K only with 8-bit KV. 5: ik_llama.cpp on P-cores, --cache-ram 1024.

And that was one machine. Every machine needs its own recipe, and almost nothing carries over. Machine 1, the AI server, needed power caps, a RAM overclock, BIOS switches, a patched driver and our own engine. Machine 2, a friend’s mini PC, needed kernel settings before its graphics chip could even use the memory, and there one engine wins for chat while a different one wins for agents. On Machine 3, a friend’s gaming PC with an AMD and an NVIDIA card, a kernel update swapped the cards around, so now we pick them by name. Machine 4, my MacBook, runs a different engine altogether, and long conversations only fit with compressed memory. Machine 5, the home server, has no graphics card, so it needed a different engine, pinned to its fast processor cores, with its prompt cache capped so it doesn’t eat the memory. (The machine numbers come back in Part 4, with the results for each.) And it takes time. Measuring one engine with one mid-size model takes about two hours, and on the mini PC alone that’s six models on two or three engines each. [Say how much time you’ve put in so far.] Every new model, engine release or kernel update starts it again. So, once it runs: how useful is it?

03 · What we can do with it

Three home models, our coding tests

DeepSeek V4.1 Flash

552B · the smartest that fits

Spreadsheet app

[score]

Space game

[score]

Solar system

[score]

Qwen 3.8 Flash Next

~180B · our everyday coder

Spreadsheet app

90

Space game

100

Solar system

100

Qwen 3.8 27B

27B · serves 6 to 7 at once

Spreadsheet app

[score]

Space game

100

Solar system

100

For the engineers: hidden graders, same prompts, Qwen Code on every model, no human help. Checks: spreadsheet 100, game 30, solar system 21. Intelligence index (Artificial Analysis): 45 / 40 / 34.

Now the fun part: what can it actually do? We run three models on the same AI server, Machine 1, one at a time. DeepSeek V4.1 Flash is the giant: 552 billion parameters, the smartest open model that fits. Qwen 3.8 Flash Next is our everyday coding model. And Qwen 3.8 27B is the small one that can serve six or seven of you at once; it’s the one you’ll use in the hands-on. We gave all three the same coding tests, with the same coding agent and no human help: a spreadsheet web app, a space shooter game, and a 3D solar system with correct proportions. Hidden graders check the result; the model never sees them. [Read the scores.] You don’t have to take my word for it. Scan a code and try what each model built, on your phone, right now. (Placeholder: the DeepSeek scores and the 27B spreadsheet come from the test runs before the talk. Flash Next hit the two-hour cap on the spreadsheet after four rounds at 90; an earlier run with the same setup reached 96 in 59 minutes. The space game passed in the first round for both Qwen models, and the 27B also got the solar system right in the first round, in 23 minutes.)

03 · How useful it can get

An AI developer at home

The test

Build a spreadsheet web app from a written spec
100 automatic checks: formulas, references, documents, interface
Write, run, test, fix: three rounds, no human help

Coding agent	Score	Time
Claude Code	95/100	28 min
Qwen Code	96/100	59 min
DeepSeek’s agent	97/100	88 min
OpenCode	94/100	55 min

0 failed requests in nearly 1,000 calls to the model

For the engineers: Qwen3.8 Flash-Next (NVIDIA FP4-Local) on our Strata fork, 2 × RTX 3090; T 0.6 / top-p 0.95 / top-k 20; scores after a grader fix; DeepSeek’s agent (dsh) hit the 90-minute cap.

03 · Daily use

One command: the coding agent, with its brain at home

$ claude-qwen
Claude Code · model: Qwen3.8 Flash-Next
server: home, 2 × RTX 3090
plan mode: it proposes, you approve

1M tokens

of context: about eight novels, or a whole codebase

115–190

tokens per second while writing

~5.5 min

to read a full million-token prompt

03 · How far it goes

552 billion parameters at home

DeepSeek V4.1 Flash: the highest-ranked open model that fits the server (index 45).

552B

parameters, plus a 196B lookup table

< 3 min

to read a 256K-token document

~31

tokens per second while writing

3–10 s

per follow-up question

Where it lives:

GPUs: the busiest parts

256 GB RAM: the rest

SSD: the lookup table

For the engineers: maya-ds4, layer split over both cards, expert misses computed from pinned RAM, Engram tables streamed from NVMe, int8 tensor-core indexer; 1M-token prompt in 12.3 min.

03 · Always on

An assistant that never sleeps

Machine 5: a laptop-class chip with no graphics card, always on, running a small local model: 8 billion parameters, about 1 billion used per word.

25 of 25

right tool chosen in our tests

~7 s

to answer a follow-up

4 of 6

real tasks fully right

It also runs the lab

Switches the big AI server on when a job needs it, and off at night.

The honest part

Small models slip on exact numbers. Check them, or hand hard jobs to a bigger model.

03 · A normal PC

An AI workstation on a gaming PC

Machine 3, a friend’s PC: Radeon RX 9070 XT (16 GB) + GeForce RTX 4060 (8 GB), 32 GB RAM.

Private chat site

Personal assistant

Coding agents

One address

1 · Cloud subscription

2 · Free providers

3 · Local model on this PC

~10 s

for a 35B model to read a 40,000-token document

38–47

tokens per second from a 27B model, with guess-and-check

You don’t need a server. My friend’s gaming PC, with one AMD and one NVIDIA card, now runs a private ChatGPT-style chat site, a personal assistant and coding agents. Every app talks to one address that tries the cloud subscription first, then free providers, then the local model, so it always answers. Locally, a 35-billion-parameter model reads a 40,000-token document in about 10 seconds. For the engineers: Qwen3.6-35B-A3B at UD-IQ4_XS on ROCm/HIP split 3:1 across both cards, all experts on GPU, K q8 / V q4 cache: prompt 2,390 → 5,405 tokens/s, decode 60 → 67. Qwen3.8-27B with its own MTP drafter at 131K context: 23 → 38 to 47 tokens/s. Open WebUI, Hermes, OmniRoute, llama-swap, all in Docker. Lesson from this PC: don’t split a model that fits one card (a 4B model dropped from ~138 to ~81 tokens/s when split).

03 · Our own facts

Teaching a small model our own facts

Test: a made-up company of 200 people, tiny models of 2 to 4 billion parameters, no graphics card. Share of questions answered correctly:

Whole org chart pasted into the prompt

34–56%

Only the relevant facts, fetched from a graph database

95–97%

Bars show the range across four small models on a 0 to 100% scale.

At 2,000 people: reading the whole chart took up to 73 minutes. The lookup still answers in seconds, 93 to 99% right.

Next: a small “fact reader” that queries the database itself. Imagine it with our clients, projects and servers.

Small models don’t know your company, and pasting everything into the prompt works badly. We tested it with a made-up company of 200 people and tiny models of 2 to 4 billion parameters, running on the home server without a graphics card. With the whole org chart pasted in, they got 34 to 56% right. When a database looked up just the relevant facts and handed them over, 95 to 97%. At 2,000 people, just reading the chart took up to 73 minutes, while the lookup still answered in seconds. Next we’re training a small “fact reader” so the model can query the database itself. Now imagine that with our clients, projects and servers. For the engineers: Qwen3.5 2B/4B and Gemma 4 E2B/E4B at 4-bit, Neo4j. Harder questions (filters, counts, comparisons) still trip the simple lookup (19 to 28%); letting the model write its own Cypher query reached 93% with Gemma E4B. A change in the database shows up in milliseconds (6.8 ms median). Reader training set: 45,695 checked question-and-query pairs across five domains.

03 · For us

Where it could fit at Websolutions

Coding help on client code that must stay in-house

An internal assistant that knows our clients, projects and servers

Bulk jobs with no per-token bill: docs, reviews, tests, translations

Prototyping AI features without sending client data out

A fallback when cloud tools are down

Some jobs belong at home.

03 · What we can do with it

Training: teach a small model one job

The job: design blocky items and outfits for a game project, from one line of text.

55 min

to train a 0.8B model on one graphics card, from 40,000 examples

8 of 8

designs follow the game's rules after training, 1.7 s each. Untrained 9B with a 3-page rule sheet: 4 of 8

4 sizes

from 0.8B to 9B, same data: the trained 9B was 3.5× slower and no better

Try it yourself

One prompt, every size, before and after training

For the engineers: Qwen3.5 0.8B full fine-tune (4-bit, 529 MB); LoRA 2B / 4B / 9B, same data; JSON under a grammar. Baseline = stock 9B + 4,110-token rules. Medians on Machine 1, 8 ideas, each model alone.

To close this part: the reason I’m most excited about. We can train our own models. We took a tiny model, 0.8 billion parameters, small enough for a cheap computer, and taught it one job: designing blocky items and outfits for a game project we’re working on, from one line of text. Forty thousand examples, 55 minutes on one graphics card. Before training, even a 9-billion-parameter model with a three-page rule sheet followed the rules in only half of our eight test designs. After training, the tiny model gets all eight right, in under two seconds each. We trained four sizes on the same data, from 0.8B up to 9B, and the biggest was three and a half times slower and no better. Scan the code and try it yourself: type an idea and see what each size makes, before and after training. Up to 15 people can use it at once. This is where I think self-hosted AI gets really interesting: a team of small specialists, each very good at one thing. (Page: play.wdays.mywire.org; "/" runs one idea through 11 models, about 20 s alone, about 50 s with 15 people; "/play" is one model at a time, about 2.4 s a design with 15 phones. Keep a recording as backup.)

04 · Case studies

Five machines, from a server to a laptop

Machine 1

AI server

Mine

Threadripper PRO
32 cores
2 × RTX 3090, 48 GB
256 GB of RAM

Runs: DeepSeek V4.1 Flash, Qwen 3.8 Flash Next and 27B, the game-item models

Machine 2

Mini PC

A friend’s

GMKtec EVO-X2
Ryzen AI Max+ 395
64 GB shared memory
120 W

Runs: Qwen 3.6 35B, Qwen 3.8 27B, Gemma 4 26B and more

Machine 3

Gaming PC

A friend’s

Ryzen 7 7800X3D
RX 9070 XT 16 GB
+ RTX 4060 8 GB
32 GB of RAM

Runs: Qwen 3.6 35B, Qwen 3.8 27B, Qwen 3.5 4B

Machine 4

MacBook Pro

Mine

Apple M1 Max
32-core GPU
32 GB shared memory

Runs: Qwen 3.8 27B all day, the game-item models

Machine 5

Home server

Mine

Intel i7 laptop chip
10 cores
No graphics card
23 GB of RAM

Runs: Qwen 3.5 9B, LFM2.5 8B for the assistant, small fact models

For the engineers: “before” = the engine and settings as downloaded, “after” = our tuned setup today; same model file unless noted; one user at a time.

04 · Case studies

Machine 1: the AI server

Threadripper PRO, 32 cores · 2 × RTX 3090 (48 GB) · 256 GB of RAM

Model	What we measured	Before	After	Change
DeepSeek V4.1 Flash · 552B	Writing, tokens/s	11	31	2.8×
DeepSeek V4.1 Flash	Reading a 256K-token document	96 min	2.2 min	43×
DeepSeek V4.1 Flash	Answering a follow-up	15 s	3.6 s	4×
Qwen 3.8 Flash Next · ~180B	Writing, tokens/s	73	118	+61%
Qwen 3.8 Flash Next	Reading 1M tokens	7.5 min	5.4 min	+39%

What made the difference: our own fork of each engine, a better 4-bit file, a small draft model that guesses ahead, and a processor with 60% more memory speed.

For the engineers: before = DeepSeek on llama.cpp, old 12-core CPU; Flash Next on Strata as downloaded. After = our maya-ds4 fork (pinned experts, CPU lane, drafter), 32-core CPU; Strata fork, FP4-Local file, MTP drafter.

04 · Case studies

Machine 2: a mini PC

GMKtec EVO-X2 · Ryzen AI Max+ 395 · 64 GB shared between processor and GPU · 120 W

Model	What we measured	Before	After	Change
All models	Memory the GPU can use	32 GB	54 GB	+69%
Qwen 3.6 35B · 3B active	Writing, tokens/s	59	68.5	+16%
Qwen 3.6 35B	Five coding-agent sessions	12.3 min	10.4 min	−15%
Qwen 3.8 27B	Writing in agent sessions, tokens/s	11.4	23.7	2.1×
Gemma 4 26B · 4B active	Writing in agent sessions, tokens/s	40.6	55.7	+37%

What made the difference: kernel settings to free the memory (every model now holds 262K tokens), one engine for chat and another for agents, and each model’s own draft head.

Lesson: the 27B doubled its speed and is still too slow for agents here; the 35B is this box’s sweet spot.

For the engineers: before = llama.cpp Vulkan, plain, 262K. After = Vulkan or ROCm per model, MTP draft 2 to 3, ubatch 1024; agents = 5 sessions, 23 turns. Still running: Gemma 4 31B, Coder Next, GLM-4.7-Flash.

04 · Case studies

Machine 3: a gaming PC

Ryzen 7 7800X3D · Radeon RX 9070 XT 16 GB + GeForce RTX 4060 8 GB · 32 GB of RAM

Model	What we measured	Before	After	Change
Qwen 3.6 35B · 3B active	Reading, tokens/s	2,390	5,405	2.3×
Qwen 3.6 35B	Writing 32K tokens in, tokens/s	41.8	53.1	+27%
Qwen 3.8 27B	Writing chat / code, tokens/s	23	38 / 47	+62% / 2×
Qwen 3.5 4B	Writing 32K tokens in, tokens/s	70	113	+63%

What made the difference: the right engine for each card, a smaller 4-bit file so the whole model fits on the two GPUs, and the 27B’s own draft head.

Lesson: a model that fits one card got 40% slower split across two; the 27B’s draft head cost half its context.

For the engineers: before = Vulkan, 6 expert layers in RAM, KV q4/q4 (35B); 27B at 262K without drafter. After = HIP + CUDA split 3:1, UD-IQ4_XS, K q8 / V q4; 27B MTP draft 2 at 131K; 4B Vulkan with f16 KV.

04 · Case studies

Machine 4: a MacBook

MacBook Pro · Apple M1 Max · 32-core GPU · 32 GB shared memory

Model	What we measured	Before	After	Change
Qwen 3.8 27B	Writing code, tokens/s	17	28	+66%
Qwen 3.8 27B	First answer, 13K-token prompt	85 s	9 s	9×
Qwen 3.8 27B	Longest conversation that fits	96K	128K	+33%

What made the difference: Apple’s own engine (MLX) with the model’s draft head guessing three words ahead, a prompt cache on the SSD, and 8-bit conversation memory for long chats.

Lesson: a newer draft method that wins on NVIDIA was 25 to 60% slower on this chip. Measure on your own machine.

For the engineers: before = mlx-lm, f16 KV, no drafting. After = MTPLX depth 3 (34 tokens/s on greedy code), SSD session cache, 8-bit KV for 128K, High Power Mode, heat watchdog. It serves the 27B all day during the talk.

04 · Case studies

Machine 5: the home server

Intel i7 laptop chip, 10 cores · no graphics card · 23 GB of RAM · always on

Model	What we measured	Before	After	Change
Qwen 3.5 9B	Reading, tokens/s	23	64	2.8×
Qwen 3.5 9B	Writing, tokens/s	6.6	8.6	+31%
LFM2.5 8B · 1B active	Writing, tokens/s, vs the tuned 9B	8.6	29–37	~4×
LFM2.5 8B	Follow-up, 13K-token prompt	73 s	7 s	10×

What made the difference: a faster engine pinned to the 6 fast cores (adding the slow cores made it slower), and a model that uses only 1B of its 8B parameters per word.

Lesson: without a graphics card, pick a model with few active parameters, not a small total size.

For the engineers: before = Ubuntu’s packaged llama.cpp. After = ik_llama.cpp on the P-cores, -t 12 -tb 6, FA on, --cache-ram 1024; LFM2.5-8B-A1B Q4_K_M, 64K. RAM runs at 4,400 MT/s, not 6,400.

05 · Try it yourself

Same brain, different hands

The model is the brain. The harness is the hands.

Reads and edits your files
Runs commands and tests
Asks the model what to do next, again and again

Harness	From	Talks to Machine 1 as
Qwen Code (easiest)	Qwen team	OpenAI-style API
Claude Code	Anthropic	Anthropic-style API
OpenCode	Open source	OpenAI-style API
Codex CLI	OpenAI	Responses API
Aider	Open source	OpenAI-style API
Cline	VS Code add-on	OpenAI-style API

In our test, four harnesses on the same model all scored 94 to 97, but took 28 to 88 minutes. The harness matters.

One more idea before you try it. The model on its own can only answer text. A harness is the program around it that gives it hands: it reads and edits files, runs commands and tests, and keeps asking the model what to do next. The model is the brain, the harness is the hands. There are many harnesses, and they speak a few different “dialects” to the model. You’ll pick one of six: Qwen Code is the easiest, made by the same team as the model; Claude Code, OpenCode, Codex CLI and Aider run in a terminal; Cline lives inside VS Code. Our server speaks all three dialects, and the instructions page has the setup for each. It matters more than you’d think: in our test, four harnesses on the same model all scored between 94 and 97, but the fastest took 28 minutes and the slowest 88, with 136 versus 389 calls to the model. (Check before the talk: each of the six harnesses in hands-on/README.md connects through the event gateway.)

05 · Try it yourself

Your turn: try it yourself

Developers: a coding agent on your laptop

1

Scan the left code

it opens the instructions

2

Pick a harness

six choices; Qwen Code is easiest

3

Paste address + key

the page fills in the key

4

Give it a small job

watch it write, run and fix

Not a developer? Scan the right code and chat with the same model in your browser.

Developers

wdays.mywire.org/hands-on

key shown at the talk

Everyone

chat.wdays.mywire.org

user shown at the talk

pass shown at the talk

For the engineers: one address + event key (OpenAI, Anthropic, Responses style), Qwen3.8 27B on Machine 1 (vLLM, 2 × RTX 3090); chat = Open WebUI, one guest login, nothing saved; closed after the event.

05 · Try it yourself

Pick a job for your agent

Never coded

A one-page website about your hobby
A tip calculator for the team lunch

Code sometimes

Explain a script, line by line
Add a dark mode to a web page

Developer

Write tests for a sample project
Run the same job in your own harness and in Qwen Code, then compare

House rules: sample code only, no client code or passwords. We all share one server, so answers can take a moment.

06 · Demonstration

Same test, different models, measured the same way

1

Real jobs

A spreadsheet app, cart bug fixes, a SQL report

2

Any model, on demand

Pick a test and the models, press start

3

Hidden grader

The same checks score every round, no human judging

4

Results board

A live race, a leaderboard and every transcript

A fair race: only the model changes.

For the engineers: a logging proxy records every request (tokens, time to first token, speed, tool calls); a power sampler measures energy; agents run headless in a Docker sandbox; tests: gridline (L, 90 min), cart-fix and sql-report (S, 15 min).

06 · Demonstration

What we measure for every run

Measure	What it tells us
Score	Out of 100, from the automatic grader
Time	Per round and in total
Rounds	How many write, test and fix rounds it needed
Calls	How often it asked the model, and how often it used a tool
Tokens	Words in and out, and how much came from the cache
Energy	Watt-hours the run used on the machine
Transcript	Every request and every answer, to see how it worked

For the engineers: from our October runs: Qwen Code 96/100, 59 min, 3 rounds, 213 calls, 23.1M tokens in (90% cached), 263K out · DeepSeek’s agent 97/100, 88 min, 389 calls, 58.2M in (99.6% cached), 433K out.

06 · Demonstration

Live: race, leaderboard, replay

In the demo

Race: two or three models on one test, side by side
Leaderboard: score against time, tokens and energy
Run page: what it missed and the finished app
Replay the transcript

07 · Wrap-up

Six lessons

1

Memory bandwidth predicts speed

2

Software beats hardware

3

Measure everything: most ideas fail

4

4-bit is the sweet spot

5

Mixture-of-experts models suit home hardware

6

Mix cloud and local

07 · Try it

Try it tomorrow

Your machine	What runs well at 4-bit
A laptop with 16 GB	Models of 4 to 9 billion parameters, using LM Studio or Ollama
An Apple laptop with 32 GB	Models up to about 27 billion parameters
One 24 GB graphics card	Qwen3.8 27B at its full context
Two 24 GB cards + 256 GB RAM	DeepSeek V4.1 Flash, 552 billion parameters

Start small: a little model on your own laptop shows you the speed and quality trade-off first-hand.

07 · What’s next

Next in the lab

Faster writing on the biggest model: fewer, fused GPU steps
Training the fact reader on the graph database
A swarm of small specialist models working together

Self-hosted AI

This app was written by an AI in my house

Same kind of model, different address

Cloud AI

Self-hosted AI

Five reasons self-hosted AI is needed

Privacy

Cost

Availability

Control

Training

The honest trade-off

Cloud still wins at

Self-hosted wins at

The hardware keeps getting more expensive

Self-hosted AI is a stack. Every layer can make it fast, slow, or broken.

Memory is the wall

A reader and a bookshelf

Squeezing the hardware

Settings nobody touches, until AI needs them

The patch that changed nothing

Two more traps

Same model, same box, almost 3× faster

Picking the model, and shrinking it

4-bit = JPEG for models

Quality after shrinking

Small gains at the bottom, big gains at the top

Every machine needs its own recipe

Three home models, our coding tests

DeepSeek V4.1 Flash

Qwen 3.8 Flash Next

Qwen 3.8 27B

An AI developer at home

The test

One command: the coding agent, with its brain at home

552 billion parameters at home

An assistant that never sleeps

It also runs the lab

The honest part

An AI workstation on a gaming PC

Teaching a small model our own facts

Where it could fit at Websolutions

Some jobs belong at home.

Training: teach a small model one job

Try it yourself

Five machines, from a server to a laptop

AI server

Mini PC

Gaming PC

MacBook Pro

Home server

Machine 1: the AI server

Machine 2: a mini PC

Machine 3: a gaming PC

Machine 4: a MacBook

Machine 5: the home server

Same brain, different hands

The model is the brain. The harness is the hands.

Your turn: try it yourself

Pick a job for your agent

Never coded

Code sometimes

Developer

Same test, different models, measured the same way

What we measure for every run

Live: race, leaderboard, replay

Six lessons

Try it tomorrow

Next in the lab

Thank you. Questions?