Cameras, a local LLM, and a pile of wrong assumptions

My infrastructure has two halves. The public half is a small HA k3s cluster on Hetzner, all GitOps through ArgoCD, which is where my actual projects live. The other half is home: a machine with an RTX 3090 in it that spends most of its life at six percent GPU load.

That felt wasteful. The 3090 is idle, I have a symmetric gigabit line, and I have TP-Link cameras that dump footage onto SD cards where it goes to die. So the plan was simple. Make the home box the private, heavy, local plane. NVR, camera AI, a voice assistant that never phones home. Nothing that needs to be public, nothing that needs five-nines. That stays on Hetzner.

What started as "let me just run Frigate" turned into a week of finding out which of my assumptions were load-bearing and which were decoration. Most were decoration.

Every "this should just work" in home automation is a small bet against physics or against a vendor. You usually lose both.

What actually got built

The end state, before I get into everything that went sideways:

Frigate on the 3090 box, plain Docker Compose rather than k3s. Frigate wants host networking and GPU passthrough, and fighting Kubernetes for that is not a good time.
Four Tapo cameras pulled over RTSP, detection running on the GPU, 24/7 recording landing on a dedicated 2 TB disk with a 14-day retention window.
Object detection at ~2.4 ms per inference. The GPU does not notice four cameras.
Ollama on the same box doing two jobs: a vision model that writes a sentence about every person the cameras see, and a text model wired into Home Assistant as a conversation agent.
A local voice pipeline. faster-whisper for speech-to-text on the GPU, Piper for text-to-speech, glued to Home Assistant Assist. You talk to it from your phone, it answers, nothing leaves the LAN.
Frigate talks to Home Assistant over MQTT, so every camera, zone, and detection is an entity I can build automations on.

That all works. Getting there is the interesting part.

Wrong assumption #1: "the GPU will just run a good detector"

Frigate 0.17 ships references to nice modern detectors like RF-DETR and YOLO-NAS. I assumed it would download one and go. It does not. Those paths only auto-resolve for paid Frigate+ models. With a plain setup you either bring your own ONNX file or you run the bundled CPU model, which Frigate itself tells you is for testing only.

So I exported my own. YOLO11n, in a throwaway container:

docker run --rm -v "$(pwd)/model_cache:/out" -w /out python:3.11-slim bash -c "
  apt-get update -qq && apt-get install -y -qq --no-install-recommends libgl1 libglib2.0-0 libxcb1 &&
  pip install -q ultralytics onnx onnxslim &&
  yolo export model=yolo11n.pt format=onnx imgsz=320 simplify=True opset=12
"

Point Frigate's onnx detector at that file with model_type: yolo-generic (note the hyphen, the un-hyphenated version fails config validation, which cost me a restart to notice), and it runs on the TensorRT execution provider. YOLO11n at 320px is a tiny model. The input tensor is 320×320 no matter what resolution you feed it, so the detector cost is basically flat whether the camera is 720p or 1080p.

Lesson: the detector was never the hard part.

Wrong assumption #2: "face recognition is a model problem"

This is the one I was most wrong about, and it took real data to accept.

Frigate 0.17 has native face recognition: a face detector plus ArcFace embeddings, all on the GPU. I enrolled myself, my wife, my daughter. It worked occasionally. Most detections came back as unknown with a similarity score around 0.02. That is not "just under the threshold". 0.02 is no relationship to the enrolled face at all.

I dug into the captured crops. They were 25 to 40 pixels of pixelated mush. Here is why, and it has nothing to do with ArcFace.

My cameras are mounted on the eaves, roughly three metres up, wide angle, looking down at the yard. A person walking to the door is five to ten metres away. By this point I had already moved detection to the full main stream, up from the 360p substream. Even so, at 1080p the face in that frame is maybe 70 pixels tall on a good day, viewed from a steep angle so the camera sees forehead and scalp, usually with motion blur because the person is walking.

The one time recognition hit 0.95 was when I deliberately stopped and looked up at the camera during enrollment.

Things that do not fix this:

Labeling the bad captures. Feeding top-down blurry non-frontal crops into the reference set drags the embedding centroid toward garbage and starts producing false positives.
Lowering recognition_threshold. At 0.02 similarity, lowering the bar just means the system confidently mislabels strangers.
"Recording at higher resolution." The Tapo C200/C500 class cameras are 2 MP sensors, so 1080p is the ceiling. There is no higher stream to pull. Asking for more would be upscaling, which adds zero information. Face pixels are a function of sensor resolution × lens FOV × distance × angle, and the stream setting is none of those.

A 70-pixel, top-down, motion-blurred face is a 70-pixel, top-down, motion-blurred face regardless of what you type into the config.

The actual fix is a camera doing a different job. One unit at ~1.7 m, near-horizontal, at a chokepoint everyone walks through facing it, so an approaching face is 100 to 150 px and roughly frontal. That is a doorbell, essentially. The eaves cameras stay as area cameras, doing person/zone/time logic, not identity. Identity becomes per-camera: face recognition at the door, licence-plate recognition on the driveway, and "someone is in the zone after dark while nobody's home" everywhere else.

I had not mounted that camera yet when I stopped, so face-gated automations are on hold. But knowing why it was not working was worth the week on its own.

Wrong assumption #3: "a local 7B model can run the house"

I wired qwen2.5:7b into Home Assistant as a conversation agent with the Assist API exposed, so it could actually control devices. Then I said "turn off the garden lights" six times. Four times it cheerfully said "done" and nothing happened. Twice it said it couldn't find the device.

The model wasn't entirely to blame. Home Assistant's own intent engine gave me the real error:

Sorry, there are multiple devices called garden lights

I had broken template switches. switch.garden_lights, stuck in an unknown state, was shadowing the real SmartThings entity that was also called "Garden lights". Home Assistant refuses to act on an ambiguous name. The LLM, given the same ambiguity, just hallucinated success. On top of that, roughly sixty useless entities were exposed to Assist: dishwasher program toggles, every camera's privacy and patrol switch, "santa". More things for a small model to pick wrong from.

The fixes, in order of impact:

Delete the broken duplicates. Ambiguity gone.
Trim what's exposed to Assist down to real controllables: lights, switches, locks, media players, climate. 78 entities to 34.
prefer_local_intents: true on the Assist pipeline. Now "turn off the garden lights" is matched by Home Assistant's deterministic intent parser, with no LLM in the loop at all. The model only gets involved for actual conversation: "how many lights are on", "is the front door locked".

After that, device control is instant and reliable. The lesson is that a 7B model is a bad switchboard operator and a fine conversationalist. Don't make it do the job the rule engine already does perfectly.

Also, purely operational: Ollama binds to 127.0.0.1 by default, so Home Assistant on another box couldn't reach it. OLLAMA_HOST=0.0.0.0 in a systemd drop-in, plus a ufw rule scoped to the HA box:

sudo ufw allow from 192.168.50.217 to any port 11434 proto tcp comment 'ollama for HA'

I had hit the exact same firewall wall earlier with Frigate's port 5000. The home box runs default-deny inbound, which is correct, and which I forget every single time.

Wrong assumption #4: "the Google speakers can be my voice assistant"

I have Nest speakers in the kitchen and office. The dream was: talk to the room, my local LLM answers through the same speaker, no Google cloud involved.

You cannot do this. "Hey Google" is hardwired to Google Assistant, and Nest speakers do not expose their microphone to Home Assistant or anything else. Sending TTS to them works fine. That half is solved, and it's how the announcements play. But the house cannot hear you through them.

The real path is a dedicated voice satellite: a small mic and speaker running its own local wake word, streaming to Assist, which runs whisper on the GPU, then the LLM, then Piper back to the satellite. Fully local, a couple of seconds round trip. Until that hardware shows up, the phone's Assist button does the same job with no hands-free.

Bonus gotcha from the same evening: media_player.living_room turned out to be an Apple TV, not a speaker. Casting TTS to it threw RuntimeError: not connected to remote and would have interrupted whatever was playing anyway. Half of home automation is discovering what your entities actually are.

What held up

For all the wrong turns, a surprising amount is genuinely good:

GPU object detection. 2.4 ms inference, four cameras, the GPU basically idle. Room for a dozen more.
24/7 recording on its own disk. Frigate self-prunes when the disk fills, so retention is bounded without babysitting.
Semantic search over footage. CLIP embeddings land on every event, so I can type "person with a backpack" and get clips.
The vision descriptions are legitimately useful. qwen2.5vl:7b, running locally, produces things like:
There's a delivery courier in a yellow vest at the door, setting down a large box.
from a single snapshot. That's a phone notification worth reading.
Local voice device control, after the cleanup, is fast and reliable, and every word of it stays on the LAN.

The end goal

The point of all this isn't "cameras". It's a local plane that mirrors the design of the cloud one, just optimised for the opposite constraints.

Hetzner and ArgoCD are for things that must be public and highly available. The home box is for things that must be private, cheap to run heavy, and physically here: the NVR, the footage, the photos, the passwords, the automations that touch the actual house.

Concretely, where this is headed:

A house that understands what it sees. Not "motion detected", but "a courier left a package" or "someone's been standing at the gate for two minutes", with the vision model deciding whether an event is worth interrupting anyone over.
Identity done right, per camera. A real face camera at the door, plate recognition on the driveway, zone-and-time logic everywhere else. No more pretending the eaves cameras can recognise faces.
A voice assistant that never leaves the network. Whisper, an LLM, and Piper all on the 3090, a satellite in the kitchen, Google's speakers demoted to output-only.
Automations that fuse camera events with house state. Presence, the alarm, sunset, who's home. Today either the camera or Home Assistant works blind, and that should not be the case.
The 3090 doing actual work.

None of it depends on a subscription, a cloud, or a vendor deciding to sunset a feature. If the internet drops, the house still watches itself and still answers when you talk to it.

That's the whole idea. The cameras were just where I started finding out how much of it I had wrong.

Till the next one. Stay hydrated.