Omer's playground

Moving 20,000 workspaces between microservices with no incidents

omratagn@gmail.com (Omer Atagun) — Sun, 18 Oct 2026 00:00:00 GMT

This project didn’t start with writing code. It started with two to three months of reading other people’s: drawing lines between services, and finding every place where one thing quietly depended on another.

By the time we pitched it to the executives we had an estimated deadline, t-shirt sized efforts per team, and a clear list of what the project needed to succeed. That list mattered more than the estimate, and the first thing on it was people. This was not a cross-team effort, it was a cross-tribe one. More than six teams were involved: answering questions, sharing knowledge, making small adjustments and reviewing changes. The actual work, including changes in other teams’ code, was done by one small dedicated team.

A migration is not a data problem. It’s an ownership problem that happens to involve data.

This is the story of how we moved bookmarks home.

What we were moving

The entity was what we call a bookmark. It lived in a service we didn’t own. A bookmark is not a flat row. It has a type property, diagram or report, that says what kind of bookmark it is, and child entities for the diagram or report itself. Those child entities have working copies, which are another layer of the problem on their own.

So “move the bookmarks” really meant: move a tree of entities, with drafts hanging off its branches, without losing a single relationship. And not all into one place. Diagrams had to go to the diagrams service, reports to the reports service.

The scale: around 20,000 workspaces across 8 regions, millions of diagrams and gigabytes of data.

Why move it at all

Bookmarks could have lived in that service just fine. Technically, nothing was broken.

Organizationally, it was a bottleneck. When the data your service is supposed to own lives somewhere else, every change turns into overhead: meetings, requests to another team, waiting on their roadmap, and maintenance on code that isn’t yours. None of that shows up in an architecture diagram, but all of it shows up in how fast you ship.

We wanted the data to live with the team that owns the domain. That was the whole reason, and it was a good enough one to justify everything that came after.

A tiger team

What I asked the executives for was a team of four. Five with me.

The reason for four was simple. Four people split into two micro teams that can work on different parts of the problem in parallel. And once the pieces of the puzzle start coming together, the knowledge of both halves combines into one picture. That combined knowledge is what gets you over the line.

We didn’t borrow time from people inside their existing teams. We created a separate team, mixed from the teams within our tribe, with one job: this migration. I led it end to end, both writing code with them and managing the project.

That setup turned out to be one of the best decisions of the whole thing. A small team with a single goal moves fast. No context switching between sprint goals, no “I’ll get to it after my other tickets”, and every decision a conversation between five people instead of a meeting across teams.

The people in it were surprised by how efficient we were. They were also happy to step out of their usual team routines for a while and work on something new and seriously complex. The feedback afterwards was great.

The rules

Three constraints shaped every decision:

No breaking API. We could not tell customers “we moved things, please use this new API from now on”. Every existing call had to keep working.
No downtime. The product had to stay up the whole time. Reads never stopped. The only thing a customer could notice was a short pause on changes to their own workspace while it was being moved.
No access to the old database. For many reasons, among them security and the complexity of giving two different namespaces access to the same database, we could not just read the old tables directly. Everything had to go through the owning service.

Each of these alone is manageable. All three together is what made it open heart surgery.

Chapter 1: bookmark-proxy

Rule 1 meant callers could not know or care where a bookmark lived. So we put something in front of all the services: bookmark-proxy, a gateway that takes incoming requests and hands them over to the right upstream service.

It answers two questions for every request:

Is this workspace migrated yet? If not, the old service is still the source of truth.
What type of bookmark is this? If the workspace is migrated, the type property decides who owns it: diagrams go to the diagrams service, reports to the reports service.

The first question is why the proxy also keeps the migration state per workspace. Without it, there is no way to know where to route.

┌────────┐     ┌────────────────┐
│ client │ ──▶ │ bookmark-proxy │
└────────┘     └───────┬────────┘
                       │
             workspace migrated?
                       │
          ┌──── no ────┴──── yes ────┐
          ▼                          │
   ┌─────────────┐             bookmark type?
   │ old service │                   │
   └─────────────┘         ┌ diagram ┴ report ┐
                           ▼                  ▼
                ┌──────────────────┐ ┌─────────────────┐
                │ diagrams service │ │ reports service │
                └──────────────────┘ └─────────────────┘

Because the state is kept per workspace, a workspace is either fully on the old side or fully on the new one, never half and half. The same state also handles the two moments that make people nervous:

Writes during the migration. While a workspace is being migrated, bookmark-proxy blocks mutations to it and returns an error. Reads keep working, so people can keep doing their work. Because the migration runs per workspace, that window is small, and the data we read can’t change underneath us.
Rollback. If something goes wrong, we flip the workspace’s state in bookmark-proxy back to the old service. Traffic goes back to where it came from, without moving any data.

Chapter 2: one step at a time

We had two options for the rollout: migrate per workspace or per region. We did both, in order of risk.

First came dry runs, to see the migration run end to end without committing to anything. Then a couple of standalone workspaces of our own, so that if something went wrong, the only customer affected was us. After that we went region by region, starting with the smallest regions and working up to the biggest ones. Every step had to go cleanly before we took the next one.

The whole rollout took 4 hours in total. The migration itself didn’t need downtime, but we still ran it inside our separate weekly downtime hour. Not because we had to, but so that enough people were around to support if things didn’t go well.

The same thinking decided the order of the domains. Diagrams went first, into the existing diagrams service. It was the smaller step, and it told us whether the whole approach worked: the proxy, the streaming, the events, the snapshots, all of it on real data.

Only once diagrams had succeeded did we move on to reports. That was the bigger step, because the reports domain didn’t exist yet. It had to be built from the ground up.

Chapter 3: full API parity

The original service was written in Java and Kotlin. Both of ours, diagrams and reports, are NestJS. Rule 1 meant we needed full API parity, not “close enough”. For reports, that meant a whole new domain that had to behave exactly like the old one from the first request.

Parity is never a straight port. Some features had to go. Some needed enhancements while we were at it. Every one of those decisions was a conversation, because someone, somewhere, depended on the old behaviour.

And parity is not only about the shape of the responses, it is about how they hold up under load. Miss an index and you find out quickly when hundreds of requests per second start hitting the endpoint. Same response, very different day.

Chapter 4: events

Bookmarks were not just rows in a microservice. Part of their state, not all of it, was also distributed to other services through Azure Event Hubs.

So our services had to start producing those events, and consumers had to be adjusted as part of the project. The catch: a producer can only be shipped by one service. We could not just produce from our services into the same producer the old service was using.

So we added a new event hub for the new owners to produce into, and changed the consumers’ code to read from both event hubs, using the bookmark type to tell which one to listen to for what. That kept every consumer correct whether a workspace had moved yet or not.

Chapter 5: moving the data

With no database access, the data had to come through the old service. We streamed it.

Read in a transaction. We read the data transactionally, so every workspace was exported from one consistent point in time.
Streamed, not loaded. Nothing was held in memory as a whole, so there was no heap instability, no matter how large a workspace was.
In foreign key order. Parents before children, children before working copies. That way every row could be written as it arrived, and snapshots taken on the new side could capture the data correctly.

That last point turned out to be more important than it looked, because of the hardest part of the whole project.

Chapter 6: snapshots, the hardest part of all

Workspaces have snapshots. A snapshot-manager orchestrates them across all upstream services, including ours.

Here is where it got complicated:

Templates. New workspaces can start from templates with default data. Part of that data lived in the snapshot-manager, part of it in the old service’s .sql dump.
Retention. Snapshots are kept for 90 days.

So migrating a workspace today raises an uncomfortable question: for the next 90 days, whose snapshot is the right one? A snapshot taken before the migration came from the old service. Restoring it would bring back data in the old service’s world, not ours.

We answered it with a remigration worker. If a workspace was restored from a snapshot that was not taken from our services, the worker remigrates that workspace using our streaming maintenance endpoint, the same streaming path we used for the original migration. The data is pulled through again, in foreign key order, and ends up where it belongs.

It is not glamorous. It is the kind of edge case that only shows up when you ask “and what happens in 89 days?”. Most of the work in this project was asking that question, over and over, about every corner.

How it ended

20,000 workspaces, 8 regions, 4 hours. No incidents. No data loss.

I am super proud of that. It took nerves, tons of alignment across teams and tribes, a lot of edge cases covered across many services, and a five-person tiger team that carried it all the way. From the outside, it looked like nothing happened.

That’s the goal. When a migration goes well, nobody notices.

What I’d do the same way again

Spend the time on discovery up front. The months of reading code and mapping connections are why there were no surprises later. Every surprise in a migration like this is an incident.
Pitch the needs, not just the estimate. For us the biggest need was a dedicated team, not a slice of everyone’s time. A small tiger team doing the work, with other teams sharing knowledge and reviewing, beats six teams each giving a little.
Start small, and keep a way back. Dry runs, our own workspaces, then regions from smallest to biggest. Diagrams before reports. And at every step, rollback was one state flip away.
Parity includes performance. Same responses at hundreds of requests per second, or it isn’t parity.
Stream, in order. Transactional reads, foreign key order, no heap. It is what made the migration and the remigration the same path.
Ask what happens in 90 days. Anything with history, like snapshots, outlives the migration day. Plan for it before it surprises you.

Till the next one. Stay hydrated.

Sigilreach: nine days, a thousand commits and a medieval world that lives without me

omratagn@gmail.com (Omer Atagun) — Sat, 10 Oct 2026 00:00:00 GMT

I have wanted to build a game for a long time. Not a weekend jam, a real one. Bellwright, Manor Lords and Kingdom Come scratched the itch, but every time I played them I ended up thinking about the same thing: the villagers are props. They stand where the designer put them. The village is there because someone placed it.

So on the 2nd of October I opened Unity and wrote a vision doc with one line at the top:

A medieval frontier built by its people: every villager wants something, the map changes because of what they do, and it goes on with or without you.

That’s Sigilreach. Nine days later the repository has 1,064 commits, about 45,000 lines of pure C# game rules, 2,500 domain tests, a 6 by 6 kilometre generated land, a castle, a smithy, marriages, murders, outlaw hunts and villagers who think with a local language model.

I didn’t write most of that code by hand. A small crew of Claude sessions did, with me as the owner, the reviewer and the person who plays the build at night and comes back with a list. This post is about what that looked like, what broke, and what I would do the same way again.

Wear your helmets.

What the game is

Single-player, realistic, no stylisation, no magic. You wake up as a stranger near a lord’s stone castle with nothing in your pockets. Everyone else is a simulated person with traits, needs, a dream (to be rich, to lead the town, a home of their own, to answer to no one), memories and opinions of each other.

The pillars that shaped every technical decision:

NPCs make the world. Hamlets get founded because someone wanted to get away from a leader they hate. Villages split, feud, steal, raid and die.
Same rules for everyone. The player is an Actor. So is every villager. Gathering, crafting, building and fighting are the same code paths for both. If a villager can’t do it, the feature isn’t done.
No static villages. Nobody lays out a village. Streets are worn by walking, homes line them, buildings go up piece by piece with scaffolding and hauled materials.
Progress is material. No XP trees. Tools gate resources, stations gate recipes, people gate the top tiers.

That second pillar turned out to be the most important architecture decision in the whole project, and I’ll come back to it.

The stack

Unity 6.3 LTS with the Universal Render Pipeline, Deferred+ renderer on PC.
A pure C# domain (Sigilreach.Domain) with noEngineReferences. Every rule lives there: needs, combat, construction, crafting, society, the land itself. No UnityEngine anywhere.
Runtime is thin MonoBehaviour glue: bodies, input, UI, terrain streaming.
Editor generators build every scene, prefab and icon from code. Nobody hand-edits a scene. If something looks wrong you change the generator and rebuild.
Content as CSV with stable ids. Items, recipes, nodes, building pieces, blueprints, creatures, lessons. Code asks for components and tags, never for a specific id.
A local LLM for the villagers’ minds: LLM for Unity (llama.cpp) with Qwen3-4B in 4-bit GGUF, about 2.5 GB, shipped inside every build and on by default.

Why the domain is engine-free

This is the decision everything else rests on. Because the rules don’t know Unity exists, they compile with plain dotnet as netstandard2.1 with C# 9, the same API level and language version Unity uses. Which means:

tests run in seconds without the editor;
they run in the cloud, on agents that will never see a Unity window;
they run while I’m in Play mode or the editor is busy rebuilding a 6 km world;
a whole village can be simulated headless for 30 in-game days and print a chronicle of who married whom, who stole, who froze to death and why.

That last point is how you debug emergent behaviour. You don’t. You read the chronicle.

Why URP and not HDRP

People ask this every time I show a screenshot, so here it is. The decision is in the vision doc, dated day one:

URP over HDRP because it has wider asset-store support and is where Unity’s renderer investment is going; the realistic look comes from assets and lighting.

In practice that played out like this:

The asset store is a URP world. Every pack I bought ships URP materials or a URP package for Unity 6.3: NatureManufacture’s Meadow and Forest, The Tales Factory’s characters, the castle kit, GameAssetFactory’s farm buildings, MicroSplat’s terrain shader. One character pack I passed on was HDRP-first, and that was one of the reasons.
The realistic look comes from assets and light, not the pipeline. MicroSplat for the terrain with anti-tiling, NatureManufacture’s water, a proper grade with ACES and bloom, volumetric clouds, a time-of-day light that changes colour through the day.
What we gave up, honestly: Unity’s Physically Based Sky and volumetric clouds are HDRP-only. URP 6.3 has no screen-space reflections, and planar reflections are too expensive on a 6 km land, so water reflects the sky through environment reflection. For clouds we vendored an MIT-licensed URP volumetric clouds and sky project instead of paying for one.

There are two more reasons that I believe were in play but that the project never wrote down, so take them as my reading:

The world is huge and the frame budget is shared. A 6 km streamed land, hundreds of simulated people, a language model on the same GPU. URP’s lighter per-frame cost leaves room for the simulation and the mind.
The main dev machine is a Mac. URP is the comfortable path on Apple silicon, and the builds I play are macOS builds.

The asset hunt

There is no art team. There is me, the Unity Asset Store and a budget I kept trying to stay under. The repo has an ASSETS.md with every pack, what it was bought for, what it cost and what was rejected and why. Total so far is about $609 for everything installed, with about $838 estimated to a vertical slice, down from the first estimate of $1,100 to $1,250.

The buying rules we settled on:

Buy for the current milestone. Exception: a real sale on something we’re sure to need.
Test-import before relying on a pack. Under our URP lighting, check Unity 6.3 compatibility, polygon counts, LODs and material count.
One visual family per category. A subset of one pack beats a mix of three styles.
Import only the sub-packs you use. Bundles unlock access; each pack downloads separately and some are many gigabytes.

The rejections taught me more than the purchases:

A modular house pack at $99.99 turned out to be about a million triangles per hut. I bought GameAssetFactory’s Medieval Farm Modular for $39.99 instead: 3 m grid, sane triangle counts, real thatch and timber framing. That one decision is why villagers can build houses piece by piece today.
A castle pack had 100K-triangle walls and gothic Roman ruins. Another had 2M-triangle tents. Another was a French château. The castle we use was €22.77 on a 55% sale instead of a $120 alternative.
A character pack at €202 had fixed variants at 59 to 93K triangles each and no soldiers. The Tales Factory’s modular characters map straight onto our clothing slots, so a crafted tunic shows up as a tunic.
A grain crops pack got rejected for being mobile quality.
Deer and domestic animal packs at €184 and €165 got dropped when PROTOFACTOR’s full animal pack showed up at €69 on a 50% sale with both.

And the gotchas nobody warns you about:

One character pack’s URP sub-package re-assigns a material’s GUID and leaves it on the Standard shader. Pink head. You fix it by hand once and write it down.
The castle pack’s particle materials use the built-in particle shader, magenta in URP. Its bonfire now runs on our own URP particle materials made from its textures.
The castle pack also had a custom grunge shader Unity 6.3’s URP can’t compile. Moved to URP Lit.
The women’s skirts are open tubes of single-sided cloth. When a stride shears the skinned mesh at a distance, the inside turns to the camera, isn’t drawn, and you get a woman with no legs below the knee.

Two rules came out of all this that I’m happy with:

Store packs never go into git. They’re gitignored, and ASSETS.md has the exact import steps per pack. A fresh clone plus the packs plus one menu click rebuilds the whole game.
There is no fallback art. Early on we had a “Greybox” art set so the game ran without packs. It doubled every generator and every test. On day six I deleted it entirely. Nobody should run this game without its art.

On day four I also changed my mind on AI-generated art. It’s allowed now, judged on quality like anything else, and disclosed on Steam at release. The first piece is the UI kit.

A world nobody laid out

The land is generated from one fixed seed: 6 by 6 kilometres, the same land every game, populated differently every time.

The first version laid rivers first and carved the land around them. Every valley floor sat at 0.5 m and the water was one flat sheet, so nothing could flow downhill and the whole 6 km read as a plain with 40 m of relief. We turned it around:

Heights first on an 8 m grid: mountain ranges along the edges, an inland massif, warped hill country, lowland basins.
Erosion on the grid: hydraulic droplets cut gullies and leave soft floors, then thermal erosion settles slopes into scree.
Rivers from flow: depressions fill into lakes, flow gathers downhill, the biggest channels become rivers, fords sit where the channel is wide and shallow.

All of that lives in the domain, so the simulation reads the same heights the terrain shows. Villagers pick where to settle and where to cross a river from the same Land model the terrain builder uses. No hand-finishing with terrain tools, ever, because that would break “same seed, same land”.

Water had to work as a rule too. Up to waist depth you wade at falling pace. Deeper, you swim at 0.45 of walking speed, no tools, no fighting, no carrying a log, and stamina drains. Run out and you’re carried to the nearest bank, losing health. From the farthest point of the biggest lake that’s under 80 health, so nobody drowns from full health. Villagers follow the same rules and path round by fords, because swimming costs 40 on the navmesh.

Optimisation, or: why the game froze at midnight

Builds that took forever

When the land went to 6 km, a clean rebuild of all scenes went to about 24 minutes. Most of that wasn’t C# compiling. It was our own generators redoing everything every time.

So every expensive output now keeps a fingerprint, a set of SHA-256 digests of its inputs: the land’s digest, the domain’s world sources, the generator’s own source, the store-pack assets it reads with their dependencies, and the Unity version. If nothing changed, it’s reused, and the console says so:

[Sigilreach] Cache: Oakhollow terrain tiles reused
[Sigilreach] Cache: icons rebuilt (changed: model)

A UI or rules change now needs no terrain rebuild at all.

The terrain tiles were about 936 MB sitting in Resources, which meant every player build repacked them. They moved to plain AssetBundles: one per 512 m tile plus a shared bundle, LZ4 compressed so a tile opens without decompressing the whole thing, built once and only copied into a player build. A release build verified at 198 bundles, 974 MB. Tiles stream around the player within 1.6 km (2.4 km on Ultra) and load in the background. And one lesson for free: building AssetBundles while a player build runs crashes Unity. Ask me how I know.

The day’s turn

I played a release build and the game froze for ten seconds or more every time the in-game day changed. Classic. A headless probe on the big land, 302 people, ticking 60 times a second through the hour before midnight, the turn and the hour after, found it in minutes:

	before	after
The tick at day change	30 to 42 ms	7 to 12 ms
First autosave capture of a session	3.2 to 3.4 s	30 to 45 ms
Later captures	70 to 95 ms	27 to 33 ms
JSON and the 22 to 29 MB file	250 to 490 ms on the game thread	another thread

The first capture was asking every one of 62,000 resource sources whether it was a felled tree, and working out the land’s trees around each one to answer. Now only trees are asked. Regrowing every stump and fading worn paths moved out of the turn into a round spread over the next in-game hour. The JSON and the write moved off the game thread.

And then a test guards it: the turning tick may cost at most 3 times the 99th-percentile tick of the hour before it. Measured at 1.4. If someone puts a land-wide loop back into the day’s turn, the test fails before I ever play it.

Detail where it matters

Only villagers within 250 m of you run in full detail and ask their mind. Distant people run a cheaper simulation that has to end up in much the same place, and there’s a test that compares the two over whole worlds. The language model gets one request at a time by priority: your conversation first, then villagers within 30 m, then everyone else. Anything older than an in-game hour is dropped. If the model hasn’t started a spoken reply in 1.5 s, the rules’ answer is said first. About 2.8 s per decision on a macOS build.

The mind also only chooses among goals the simulation offers, in a JSON schema, and can lift a goal’s score by at most 0.25. Urgent needs still win. A 4B model is a fine source of “living, random-ish choices” and a terrible source of truth, the same lesson I learned with Home Assistant. The rules decide what’s possible.

Testing: fast without losing safety

This is the part I care about most. I’ve written before about TDD with AI and this project is that idea at full volume. Rules are test-first: the failing test, then the minimal code. Agents don’t get to skip it.

By day six the suite was hurting. An agent’s change took 16+ minutes of tests and a merge pass over 30. My instruction was simple:

I do not want to reduce the safety of tests, but I want to increase speed and efficiency.

Rules for that change: no test deleted to save time, the same assertions, and every card measured before and after. What came out:

Suite	Before	After
Domain rules	2 min 43 s	24 to 26 s
Long whole-world runs	14 min 57 s	2 min 24 s
PlayMode	16+ min, most tests loading the full 6 km world	about 5 min in small scenes

Three things did it:

Release builds and parallel fixtures. Release alone took the domain tests from 157 s to 43 s. Parallel took it to about 22 s.
Small test scenes. Arena for combat, Yard for crafting, Hamlet for building and saves, Castle, Workshop, Commons. Each is about 192 m of the real art, loads in seconds, and is also a place I can open and try one thing.
Tiers. fast for every agent after every change. world for the long simulations, queued one at a time per machine. merge for the lead once per batch. full before every build I play.

Parallel tests found real bugs

Running fixtures in parallel immediately broke two tests in about half the runs. The cause: shared caches built with ??=. Two fixtures racing each got their own copy of the content, and a stick from one copy isn’t a stick to a simulation built from the other. They became Lazy<T>.

Then an audit found a nastier one. Blueprint.Footprint was built the same way, and some code compared footprints by reference. Five of six lordly worlds founded in parallel laid out their keep, bailey or hall differently from the same seed run alone. Same seed, different world, only when the race landed. Now there’s a rule (sims share no mutable state) and a test that releases threads together on a barrier and checks they all get one footprint. It failed 17 of 17 runs before the fix.

A simulation whose result depends on what else is running in the process isn’t deterministic. It just hasn’t been caught yet.

Golden seeds lie

World tests used to assert on one seed. Then a harmless change shifted the random numbers and seed 3 went from nine hamlets to four. Was that a regression or a seed turning? You can’t tell from one seed. So the world tests now assert a share of a block of seeds (five of eight worlds must pass, for example) run in parallel inside the test. A sweep before and after a change tells drift from a fall.

The NUnit trap

Unity ships NUnit 3.5. The fast tier runs on 3.14, because .NET 10 can’t pin anything older. So a test can pass in the cloud and fail to compile in the editor. The one that bites is Does.Contain on a collection. It’s in AGENTS.md in bold now.

Working with a crew of agents

None of the above happens without process, because several agents work on the same code at the same time. What we landed on:

A lead session on the Mac that integrates. Agents push branches, never main. The lead merges a batch, runs fast, then the affected world tests, the editor compile, a rebuild if a generator changed, the visual checks, and only then pushes. If a batch fails, git bisect --first-parent finds the branch.
A Windows desktop for editor work. It has the Unity editor and the packs, so PlayMode tests, rebuilds and experiments run there, one at a time.
Cloud sessions for rules. No editor, but the domain compiles and runs anywhere with .NET, so they write rules and tests and mark their PRs “PlayMode not run”.
Docs change in the same commit as the code. SYSTEMS.md, TESTING.md and the feature cards are what the next agent reads. A change that leaves a doc saying something false is not done. The docs are around 6,000 lines now and they’re the project’s memory.
Small files, one concern each. Several agents in one big file is a merge conflict factory. New goals get new files.
No runtime singletons. No static Instance. Systems are handed to what uses them, wired by the scene generator. This one came from a bug: the pause, sleep and the death skip each saved and restored Time.timeScale, and one restoring after another left the game at 8x speed. Now one owner holds the speed and everyone else takes and releases holds.

An audit day

Today was an audit day. A code audit produced a findings page, and the desktop took three branches. The interesting one: raised floors in the castle’s buildings were answering at ground height, half a metre low. The brief suspected Unity’s navmesh height mesh. The desktop wrote a probe that sampled every deck cell over four new games, ran it with the height mesh on and off, and found the height mesh wasn’t the cause. Bodies were being placed before the buildings were in the mesh on a load. Main had 70 of 156 deck cells wrong. The branch with the load wait had 0 of 104.

Same branch: creatures were being baked into the navmesh as solid capsules. A new test showed a 0.70 m hole around a deer without the fix. A reviewer then pointed out felled trunks had no test, so the next PR added one: a 0.83 m hole without the fix.

The other one I liked: a castle walk test was flaky “about 1 run in 4” on the Mac. The desktop probed 40 loads, half with the height mesh on and half off, then ran the real test 10 times on the branch and 10 on main. It never failed. So it changed nothing and wrote down the numbers and the suspect. No fix without a failure to test it against. That’s the right call, and it’s not one I expected an agent to make on its own.

Things agents are still bad at

Honest list:

Looking at the game. Agents don’t judge visuals. I do, in the editor, at day, golden hour and night. Every visual card ends with “owner checks” written as pass and fail, and a lot of them are still pending.
Knowing when “it works” isn’t “it’s right”. A world can pass every test and still look wrong when you walk through it. Only playing catches that.
Animations. I had to make it a rule: an animation finishes before the next action starts, for everyone. Lying down, getting up, kneeling into work. Without it everything looks glitchy. It’s data on the generated animator now (Settle and Settled tags), not a timer per case.

What I’d do the same way again

Engine-free rules from day one. It’s the reason tests take a minute, cloud agents are useful, and villagers can do everything the player does.
Generators, not hand-made scenes. Painful to set up, but every scene is reproducible, reviewable as code and cacheable.
Measure before optimising. Every speed-up in this post started with a probe that printed numbers, and ended with a test that keeps them.
Docs as the shared brain. With many sessions and none of them remembering yesterday, the docs are the only continuity there is.
URP. I haven’t missed HDRP once. I’ve missed good assets many times.

What’s next

The villagers’ AI gets a regression net next: a set of measures over many seeds that says whether a change made the world better or worse, before I play it. Then bows, sieges, the castle redesign, footsteps that make a sound, and a lot of owner checks I owe the crew.

Nine days in, the world already does things I didn’t plan. A home-dreamer builds a lean-to and a kind neighbour helps. A greedy hoarder gets caught pilfering and resented. Men sent on an outlaw hunt sided with the outlaws because family and friends of the band were among the hunters, and now there’s a test for that.

That’s the whole idea. I just have to keep up with it.

Till the next one. Stay hydrated.

Cameras, a local LLM, and a pile of wrong assumptions

omratagn@gmail.com (Omer Atagun) — Fri, 04 Sep 2026 00:00:00 GMT

My infrastructure has two halves. The public half is a small HA k3s cluster on Hetzner, all GitOps through ArgoCD, which is where my actual projects live. The other half is home: a machine with an RTX 3090 in it that spends most of its life at six percent GPU load.

That felt wasteful. The 3090 is idle, I have a symmetric gigabit line, and I have TP-Link cameras that dump footage onto SD cards where it goes to die. So the plan was simple. Make the home box the private, heavy, local plane. NVR, camera AI, a voice assistant that never phones home. Nothing that needs to be public, nothing that needs five-nines. That stays on Hetzner.

What started as “let me just run Frigate” turned into a week of finding out which of my assumptions were load-bearing and which were decoration. Most were decoration.

Every “this should just work” in home automation is a small bet against physics or against a vendor. You usually lose both.

What actually got built

The end state, before I get into everything that went sideways:

Frigate on the 3090 box, plain Docker Compose rather than k3s. Frigate wants host networking and GPU passthrough, and fighting Kubernetes for that is not a good time.
Four Tapo cameras pulled over RTSP, detection running on the GPU, 24/7 recording landing on a dedicated 2 TB disk with a 14-day retention window.
Object detection at ~2.4 ms per inference. The GPU does not notice four cameras.
Ollama on the same box doing two jobs: a vision model that writes a sentence about every person the cameras see, and a text model wired into Home Assistant as a conversation agent.
A local voice pipeline. faster-whisper for speech-to-text on the GPU, Piper for text-to-speech, glued to Home Assistant Assist. You talk to it from your phone, it answers, nothing leaves the LAN.
Frigate talks to Home Assistant over MQTT, so every camera, zone, and detection is an entity I can build automations on.

That all works. Getting there is the interesting part.

Wrong assumption #1: “the GPU will just run a good detector”

Frigate 0.17 ships references to nice modern detectors like RF-DETR and YOLO-NAS. I assumed it would download one and go. It does not. Those paths only auto-resolve for paid Frigate+ models. With a plain setup you either bring your own ONNX file or you run the bundled CPU model, which Frigate itself tells you is for testing only.

So I exported my own. YOLO11n, in a throwaway container:

docker run --rm -v "$(pwd)/model_cache:/out" -w /out python:3.11-slim bash -c "
  apt-get update -qq && apt-get install -y -qq --no-install-recommends libgl1 libglib2.0-0 libxcb1 &&
  pip install -q ultralytics onnx onnxslim &&
  yolo export model=yolo11n.pt format=onnx imgsz=320 simplify=True opset=12
"

Point Frigate’s onnx detector at that file with model_type: yolo-generic (note the hyphen, the un-hyphenated version fails config validation, which cost me a restart to notice), and it runs on the TensorRT execution provider. YOLO11n at 320px is a tiny model. The input tensor is 320×320 no matter what resolution you feed it, so the detector cost is basically flat whether the camera is 720p or 1080p.

Lesson: the detector was never the hard part.

Wrong assumption #2: “face recognition is a model problem”

This is the one I was most wrong about, and it took real data to accept.

Frigate 0.17 has native face recognition: a face detector plus ArcFace embeddings, all on the GPU. I enrolled myself, my wife, my daughter. It worked occasionally. Most detections came back as unknown with a similarity score around 0.02. That is not “just under the threshold”. 0.02 is no relationship to the enrolled face at all.

I dug into the captured crops. They were 25 to 40 pixels of pixelated mush. Here is why, and it has nothing to do with ArcFace.

My cameras are mounted on the eaves, roughly three metres up, wide angle, looking down at the yard. A person walking to the door is five to ten metres away. By this point I had already moved detection to the full main stream, up from the 360p substream. Even so, at 1080p the face in that frame is maybe 70 pixels tall on a good day, viewed from a steep angle so the camera sees forehead and scalp, usually with motion blur because the person is walking.

The one time recognition hit 0.95 was when I deliberately stopped and looked up at the camera during enrollment.

Things that do not fix this:

Labeling the bad captures. Feeding top-down blurry non-frontal crops into the reference set drags the embedding centroid toward garbage and starts producing false positives.
Lowering recognition_threshold. At 0.02 similarity, lowering the bar just means the system confidently mislabels strangers.
“Recording at higher resolution.” The Tapo C200/C500 class cameras are 2 MP sensors, so 1080p is the ceiling. There is no higher stream to pull. Asking for more would be upscaling, which adds zero information. Face pixels are a function of sensor resolution × lens FOV × distance × angle, and the stream setting is none of those.

A 70-pixel, top-down, motion-blurred face is a 70-pixel, top-down, motion-blurred face regardless of what you type into the config.

The actual fix is a camera doing a different job. One unit at ~1.7 m, near-horizontal, at a chokepoint everyone walks through facing it, so an approaching face is 100 to 150 px and roughly frontal. That is a doorbell, essentially. The eaves cameras stay as area cameras, doing person/zone/time logic, not identity. Identity becomes per-camera: face recognition at the door, licence-plate recognition on the driveway, and “someone is in the zone after dark while nobody’s home” everywhere else.

I had not mounted that camera yet when I stopped, so face-gated automations are on hold. But knowing why it was not working was worth the week on its own.

Wrong assumption #3: “a local 7B model can run the house”

I wired qwen2.5:7b into Home Assistant as a conversation agent with the Assist API exposed, so it could actually control devices. Then I said “turn off the garden lights” six times. Four times it cheerfully said “done” and nothing happened. Twice it said it couldn’t find the device.

The model wasn’t entirely to blame. Home Assistant’s own intent engine gave me the real error:

Sorry, there are multiple devices called garden lights

I had broken template switches. switch.garden_lights, stuck in an unknown state, was shadowing the real SmartThings entity that was also called “Garden lights”. Home Assistant refuses to act on an ambiguous name. The LLM, given the same ambiguity, just hallucinated success. On top of that, roughly sixty useless entities were exposed to Assist: dishwasher program toggles, every camera’s privacy and patrol switch, “santa”. More things for a small model to pick wrong from.

The fixes, in order of impact:

Delete the broken duplicates. Ambiguity gone.
Trim what’s exposed to Assist down to real controllables: lights, switches, locks, media players, climate. 78 entities to 34.
prefer_local_intents: true on the Assist pipeline. Now “turn off the garden lights” is matched by Home Assistant’s deterministic intent parser, with no LLM in the loop at all. The model only gets involved for actual conversation: “how many lights are on”, “is the front door locked”.

After that, device control is instant and reliable. The lesson is that a 7B model is a bad switchboard operator and a fine conversationalist. Don’t make it do the job the rule engine already does perfectly.

Also, purely operational: Ollama binds to 127.0.0.1 by default, so Home Assistant on another box couldn’t reach it. OLLAMA_HOST=0.0.0.0 in a systemd drop-in, plus a ufw rule scoped to the HA box:

sudo ufw allow from 192.168.50.217 to any port 11434 proto tcp comment 'ollama for HA'

I had hit the exact same firewall wall earlier with Frigate’s port 5000. The home box runs default-deny inbound, which is correct, and which I forget every single time.

Wrong assumption #4: “the Google speakers can be my voice assistant”

I have Nest speakers in the kitchen and office. The dream was: talk to the room, my local LLM answers through the same speaker, no Google cloud involved.

You cannot do this. “Hey Google” is hardwired to Google Assistant, and Nest speakers do not expose their microphone to Home Assistant or anything else. Sending TTS to them works fine. That half is solved, and it’s how the announcements play. But the house cannot hear you through them.

The real path is a dedicated voice satellite: a small mic and speaker running its own local wake word, streaming to Assist, which runs whisper on the GPU, then the LLM, then Piper back to the satellite. Fully local, a couple of seconds round trip. Until that hardware shows up, the phone’s Assist button does the same job with no hands-free.

Bonus gotcha from the same evening: media_player.living_room turned out to be an Apple TV, not a speaker. Casting TTS to it threw RuntimeError: not connected to remote and would have interrupted whatever was playing anyway. Half of home automation is discovering what your entities actually are.

What held up

For all the wrong turns, a surprising amount is genuinely good:

GPU object detection. 2.4 ms inference, four cameras, the GPU basically idle. Room for a dozen more.
24/7 recording on its own disk. Frigate self-prunes when the disk fills, so retention is bounded without babysitting.
Semantic search over footage. CLIP embeddings land on every event, so I can type “person with a backpack” and get clips.
The vision descriptions are legitimately useful. qwen2.5vl:7b, running locally, produces things like:

There’s a delivery courier in a yellow vest at the door, setting down a large box.

from a single snapshot. That’s a phone notification worth reading.
Local voice device control, after the cleanup, is fast and reliable, and every word of it stays on the LAN.

The end goal

The point of all this isn’t “cameras”. It’s a local plane that mirrors the design of the cloud one, just optimised for the opposite constraints.

Hetzner and ArgoCD are for things that must be public and highly available. The home box is for things that must be private, cheap to run heavy, and physically here: the NVR, the footage, the photos, the passwords, the automations that touch the actual house.

Concretely, where this is headed:

A house that understands what it sees. Not “motion detected”, but “a courier left a package” or “someone’s been standing at the gate for two minutes”, with the vision model deciding whether an event is worth interrupting anyone over.
Identity done right, per camera. A real face camera at the door, plate recognition on the driveway, zone-and-time logic everywhere else. No more pretending the eaves cameras can recognise faces.
A voice assistant that never leaves the network. Whisper, an LLM, and Piper all on the 3090, a satellite in the kitchen, Google’s speakers demoted to output-only.
Automations that fuse camera events with house state. Presence, the alarm, sunset, who’s home. Today either the camera or Home Assistant works blind, and that should not be the case.
The 3090 doing actual work.

None of it depends on a subscription, a cloud, or a vendor deciding to sunset a feature. If the internet drops, the house still watches itself and still answers when you talk to it.

That’s the whole idea. The cameras were just where I started finding out how much of it I had wrong.

Till the next one. Stay hydrated.

From kubectl apply to sleeping well at night

omratagn@gmail.com (Omer Atagun) — Fri, 17 Apr 2026 00:00:00 GMT

I had this cluster running for a while. Single node k3s, couple of apps, deployments done through CI pipelines that would kubectl apply manifests directly into the cluster. It worked. Until it didn’t feel right anymore.

Every time I thought about “what if this machine dies”, the answer was a shrug and a prayer. Secrets were created manually on the cluster. Database backups? Didn’t exist. If that VM went down, I’d be rebuilding everything from memory and scattered YAML files across repositories.

“If your disaster recovery plan is ‘I’ll remember’, you won’t.”

So I decided to fix it properly. What started as “let me just set up ArgoCD” turned into a full infrastructure overhaul that I’m honestly proud of. Here’s the journey.

Chapter 1: ArgoCD and the GitOps Mindset

The first thing was getting ArgoCD running and moving all deployments under its control. The idea is simple — one Git repository is the source of truth for everything running in the cluster. Push to main, ArgoCD syncs.

I created softastik-infra, an infrastructure repository that holds:

All Kubernetes manifests (deployments, services, ingresses)
Helm values for any chart-based apps
ArgoCD Application CRDs that point to their respective directories
SealedSecret manifests for encrypted credentials

The CI pipelines in app repositories now just build an image, push it, and update the image tag in the infra repo. That’s it. ArgoCD picks up the change and rolls it out.

# ArgoCD Application — one per app
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: tenexengineer
  namespace: argocd
spec:
  project: softastik
  source:
    repoURL: https://github.com/softastik-org/softastik-infra.git
    path: apps/tenexengineer
  destination:
    server: https://kubernetes.default.svc
    namespace: tenex
  syncPolicy:
    automated:
      selfHeal: true
      prune: true

selfHeal: true means if someone manually changes something on the cluster, ArgoCD reverts it. prune: true means if I delete a manifest from Git, the resource gets deleted from the cluster. Git is the boss now.

Sealed Secrets — because credentials in Git sounds terrible

Obviously you can’t commit plain secrets to a Git repo. Bitnami Sealed Secrets solves this. You encrypt secrets with a public key, commit the encrypted version, and only the controller running in the cluster can decrypt them.

The critical thing: back up the encryption key. Without it, a new cluster can’t decrypt your existing sealed secrets. That backup file is the one piece that lives outside of Git, and it’s the foundation of the entire recovery process.

# This one command is your insurance policy
kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > ~/sealed-secrets-key-backup.yaml

Chapter 2: Shared Postgres — One Database to Rule Them All

Each app was running its own Postgres instance. On a home cluster with limited resources, that’s wasteful. More importantly, it means N separate backup strategies.

I consolidated everything into a shared Postgres instance running pgvector/pgvector:pg17. Each app gets its own database and user, created by an init script:

CREATE USER tenex WITH PASSWORD '${TENEX_PASSWORD}';
CREATE DATABASE tenexengineer OWNER tenex;
\c tenexengineer
CREATE EXTENSION IF NOT EXISTS vector;

Adding a new app means adding a few lines to the init ConfigMap. One Postgres pod, one backup job, one restore process.

Automated Backups

A CronJob runs pg_dumpall every night at 3 AM, gzips it, uploads to S3, and cleans up backups older than 7 days:

schedule: "0 3 * * *"

That’s it. ArgoCD deploys the CronJob, and it just runs. No cron on the host machine, no scripts to forget about.

Backup Verification

Here’s the thing about backups — they’re worthless if you’ve never tested a restore. So I added a weekly verification job that downloads the latest backup, restores it into a temporary database, checks that tables and data exist, then drops it:

=== Backup Verification PASSED ===
  File: all_databases_2026-04-13_0300.sql.gz
  Size: 847293 bytes
  Tables: 24

If it fails, the existing alert rules catch it and Slack yells at me.

Chapter 3: The “What If” Moment — One Command Recovery

With ArgoCD managing deployments, Sealed Secrets handling credentials, and backups going to S3, I realized I was close to something powerful: a one-command cluster rebuild.

I wrote scripts/bootstrap-cluster.sh that does everything:

Installs k3s
Installs Helm
Installs Sealed Secrets controller and restores the encryption key
Installs ArgoCD
Connects the private infra repo
Applies all ArgoCD Applications
Waits for everything to sync
Restores the database from S3

# New machine? One command.
./scripts/bootstrap-cluster.sh ~/sealed-secrets-key-backup.yaml

That script is the answer to “what if this machine dies”. The answer is: get a new machine, run one script, update DNS. Done.

Chapter 4: Going HA — Because Single Points of Failure Are Boring

At some point I looked at this beautiful setup running on a single VM and thought… that’s still one machine. One disk failure away from downtime.

So I got three servers and went full HA.

k3s with embedded etcd

k3s supports HA clustering with embedded etcd — no external database needed. Three server nodes give you quorum. If one dies, the cluster keeps running.

# First node initializes the cluster
curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --tls-san 178.104.173.177 \
  --disable local-storage

Longhorn for distributed storage

With multiple nodes, you need storage that replicates across them. Longhorn does exactly that. Postgres writes to a Longhorn volume, Longhorn replicates the data to all three nodes. A node dies? The volume is still available from the other two.

NAME                STATE      ROBUSTNESS
shared-postgres     attached   healthy     # 3 replicas across 3 nodes

No more single disk as a point of failure.

ArgoCD itself runs HA

Two replicas for the API server, two for the repo server. The bootstrap script and Helm values handle this automatically.

Chapter 5: Observability — Because Guessing Is Not Monitoring

A cluster without observability is just a fancy way to run things you can’t see. I added the full stack:

Prometheus — scrapes metrics from all cluster components and applications. 15 days retention, 10Gi persistent storage on Longhorn.

Grafana — dashboards for everything. I created four as code (ConfigMaps auto-discovered by the Grafana sidecar):

Kubernetes Apps Overview — CPU, memory, restarts, network I/O per pod
TenexEngineer — request rate, latency percentiles (p50/p95/p99), error rate, pipeline metrics
Shared Postgres — pod status, uptime, backup status, PVC usage, disk I/O
Uptime & Availability — endpoint status, response times, SSL expiry, 30-day uptime percentage

Alertmanager — fires alerts to Slack. Custom rules for the things that matter:

- alert: NodeDown          # server unreachable for 2 min
- alert: PodCrashLooping   # pod restarting repeatedly
- alert: HighMemoryUsage   # node memory above 90%
- alert: PostgresDown      # shared postgres not ready
- alert: BackupCronJobFailed  # nightly backup or weekly verify failed
- alert: LonghornVolumeUnhealthy  # storage replication degraded
- alert: EndpointDown      # public endpoint unreachable

Blackbox Exporter — external uptime checks. Every 30 seconds it probes the blog, tenexengineer, ArgoCD, and Grafana from inside the cluster.

All of this is deployed through ArgoCD as Helm charts with multi-source Applications. Values live in the infra repo, charts come from the prometheus-community Helm repository. Push a change to alert rules? ArgoCD syncs it. Add a new blackbox target? Same thing.

Chapter 6: Application-Level Metrics

The cluster-level metrics were great, but I also wanted to see what’s happening inside my apps. So I added a Prometheus middleware to TenexEngineer — it records request count, latency histograms, and in-flight requests for every endpoint:

// Three new metrics, automatically recorded for every request
HTTPRequestDuration   // histogram by method, path, status
HTTPRequestsTotal     // counter by method, path, status
HTTPRequestsInFlight  // gauge

A ServiceMonitor tells Prometheus to scrape it, and a Grafana dashboard visualizes it. Now I can see exactly how many webhook events TenexEngineer processes, how long the pipeline takes, and whether anything is erroring out.

What’s Actually in Git Now

In Git (automatic)	NOT in Git (manual setup)
All Kubernetes manifests	k3s installation
SealedSecret manifests	Sealed Secrets encryption key
ArgoCD Application CRDs	ArgoCD repo credentials
Helm values (monitoring, blackbox)	DNS records (Cloudflare)
Ingress rules	Longhorn installation (HA)
Backup CronJob + Verify Job	GitHub repo secrets
Custom Prometheus alert rules
Grafana dashboards as code
Blackbox uptime targets

The “NOT in Git” column is the migration checklist. Everything else rebuilds itself.

The End Result

$ kubectl get applications -n argocd
NAME                HEALTH    SYNC
blog                Healthy   Synced
dashboards          Healthy   Synced
tenexengineer       Healthy   Synced
shared              Healthy   Synced
prometheus-stack    Healthy   Synced
blackbox-exporter   Healthy   Synced

Three nodes. Six apps. Automated backups with verification. Full observability with Slack alerting. Grafana dashboards for every component. One-command disaster recovery.

From kubectl apply and hoping for the best, to sleeping well at night knowing that if everything burns down, one script and a DNS update brings it all back.

Was it overkill for a home cluster? Absolutely. Did I learn a ton and have fun doing it? Also absolutely.

Till next time, keep your sealed secrets backed up.

TDD with AI

omratagn@gmail.com (Omer Atagun) — Sat, 07 Feb 2026 00:00:00 GMT

Ever since I started writing about testing with AI on this very blog, I’ve had quite the journey at work trying to balance raw speed with the experience and scars I’ve accumulated over time.

This is the story of how I leveraged AI without losing my sanity, using a TDD-first mindset. Wear your helmets, tighten your seatbelts let’s go.

Optimisation opportunity

Imagine a graph we need to traverse as a tree. We need traversal in both directions: forward and reverse.

The initial idea was intentionally naive: get it working with a very basic implementation where database calls scale with the number of hops. The goal wasn’t performance it was shape. I wanted the input and output to be exactly right so we could lock behavior down with tests before touching optimisation.

So that’s what we did.

We wrote stress tests and benchmarks using testcontainers to stay close to production behavior. Inputs and outputs were spot-on. Then we layered in extra specifications as unit and integration tests.

Now the fun part begins.

I told the AI to optimise not with “make this faster”, but by giving direction and constraints about how to think about the problem.

I put AI to the test: “do something” vs “be instructed”

“Do something”

I said:

“Go look into this service, analyse it, and come back with optimisation opportunities.”

Sonnet 4.5 did what most engineers do when skimming unfamiliar code: “Let’s cache this bad boy.”

Early returns, hashmaps, tracking visited nodes in memory all technically reasonable ideas.

Except… we run multiple pods in a Kubernetes cluster.

Each pod would maintain its own in-memory cache with its own TTL. Consistency becomes fuzzy, cache warm-up becomes painful, and until caches stabilize, each node still fires hundreds of queries per graph traversal.

Yes, you could try to fix this with shared cache hit/miss tracking, but that’s not optimisation that’s complexity debt.

“Do a single recursive CTE query”

This time I gave a very explicit instruction: turn the traversal into a single recursive CTE query.

It took hours. But eventually, we got there.

A massive query with lots of bindings, covering both the base and recursive steps. Proper composite indexes, well thought-out relations all technically solid.

I was positively surprised.

This was the real difference between “figure something out” and “solve this problem in this way”. AI is a statistical model. It optimizes for correctness, not for appropriateness. Its first suggestion was reasonable, but it didn’t even glance at the k8s setup so it couldn’t possibly know caching was the wrong trade-off.

Once guided, it delivered exactly what I asked for. But that came with a hidden cost.

At low data volume, the recursive CTE felt fine.

But real production data isn’t small and recursive queries don’t degrade linearly.

Under the hood, Postgres was evaluating a join condition that looked harmless:

… JOIN edges ON from_id = current_id OR to_id = current_id …

That OR inside a recursive join is a silent killer.

The planner can’t commit to a single index path, so each recursion level multiplies the amount of work. Scans grow, merges explode, and execution plans balloon.

Without a defensive approach, the outcome would have been predictable:

timeouts
massive execution plans
queries that look correct but never finish

Thankfully, I didn’t discover this on a customer’s bill at least not this time. That’s why it’s called experience.

Yet again, I had to explicitly tell Sonnet that timeouts were non-negotiable. Graph traversals can go sideways very quickly, and once infinite or near-infinite queries start piling up, they can take an entire database down.

The save: defensive engineering first

Here’s the twist: I didn’t deploy the query blindly.

Two deliberate design decisions saved us:

Per-workspace scoping a misbehaving query was limited to a single tenant.
A hard 15-second query timeout runaway queries couldn’t block the system.

Instead of bringing the entire service down, only the offending workspace felt the impact.

That’s not luck. And it’s not statistics.

That’s engineering.

It’s also the part of the job you only appreciate before something becomes an incident. This is where engineers design systems; coding is just one slice of the work usually the one that wrecks your back.

Even the dopamine doesn’t come from “the code worked”, but from realizing the solution survived reality.

The fix: split, simplify, and respect index usage

Next, I benchmarked properly.

I replicated production-like tables, generated mock data, and mirrored production indexes. You don’t need a perfect clone you need something close enough to reveal how things behave under stress.

For small datasets, everything looked fine. I pulled median numbers from Databricks to estimate realistic loads.

Seemed okay.

Ship it, yeet!

It worked exactly as expected… for median users.

Within hours of release, timeout alerts started firing in crowded regions. Heavy users had arrived.

A feature that returns nothing is not a feature.

Quick ANALYZE. The problem was obvious: indexes were effectively abandoned. A single recursive CTE with an OR condition throws away most of what makes relational databases fast.

What the database was really doing:

scan A
scan B
merge
deduplicate

So I changed the approach.

Instead of one recursive query handling both directions, I split it into:

one traversal for forward edges
one traversal for reverse edges
merge and deduplicate in application code, not in the database

That small refactor turned a query that timed out under load into one that consistently runs in ~30 ms.

Why such a dramatic difference?

Because in databases, efficiency isn’t about correctness it’s about predictable access paths. By separating traversal directions, each query could use its index cleanly. No OR. No bitmap chaos. No exponential blow-ups.

What this taught me (and what I want you to take away)

Three inseparable truths stand out.

1. Tests catch bugs; engineering catches failures

Tests prove logic. Design protects production.

Don’t let AI take the wheel entirely.

AI can help with both if you ask the right questions.

When I wrote the original query, I was testing correctness. When I rewrote it, I was engineering for failure with AI as an assistant.

Those are different goals.

2. Defensive defaults save your future self

Timeouts, scoped queries, resource limits they feel like overhead until the day they save you.

Treat them not as optional extras, but as core safety rails. Bake them into every AI-assisted session you have.

3. Mindset beats tools

You can have the best language, the smartest LLM, the fanciest deployment setup
but without the habit of asking “what could go wrong here?”, you’ll eventually pay for it.

Closing thoughts

I’m writing this not as a tutorial, but as a reminder to myself and to anyone building systems for real users at real scale:

Elegance in code doesn’t guarantee stability in production. Reliability is earned through anticipation, not just validation.

Build with that mindset first, and everything else becomes faster and more enjoyable with AI.

Cheers to building software that survives the real world.

AI Coding Agents: Hands-On Test of Codex, Claude CLI & Gemini CLI

omratagn@gmail.com (Omer Atagun) — Sat, 28 Jun 2025 00:00:00 GMT

“Hmm… maybe that side project I keep avoiding could write itself?”

We’re officially in the age of AI-assisted coding—at least according to the shiny slide decks from every AI CEO on the planet. Over the past few weeks I put that claim to the test, unleashing Codex Cloud, Claude CLI, and the brand-new Gemini CLI on two personal projects. Here’s how that rodeo went.

The Test Subjects

Diagramat – a green-field diagram editor à la draw.io, built from scratch in an npm work-spaced monorepo.
eDominations v2 – a 2017 browser game I mothballed in 2022. Goal: port the backend from Laravel/MySQL to NestJS/Postgres and modernise the whole stack.

Chapter 1: Codex

Maybe the best one so far. Not because it is the best model out there but because its concurrency capability. You are not locked into your IDE to use CLI tools from other models. As well as, you do not have any safety concerns with the data that can mistakenly be leaked by using terminal commands.

Initially, I have created a repository to develop diagramming editor. My idea was to prepare an npm workspaces so that I can split the code into multiple packages and have a monorepo with all the linters, tests. Prepared all deployment yaml files for k8s cluster and CI/CD pipelines for GitHub Actions. So instead of locally testing the changes, i could quickly look from my mobile if things go well. Goal was to eliminate halucination and have a clear structure to work with. This did work well in the beginning, when I described the project to the codex, it was able to understand and deliver not bad results. However, it was repetitive and once in a while could get lost and do entirely different things. I had to be careful about what I was asking and how I was asking it.

I have figured out that codex works best when you provide AGENTS.md file in the root of your repository. This file should contain the description of the project, its structure, and how to run it. It should also contain the instructions on how to use the codex. This way, codex can understand the context of the project and provide better results.

AGENTS.md file i used for this project was like this:

```markdown
# Repository Guidelines for Agents

This repository is a Node.js monorepo managed with **npm workspaces**. Node 22 is required.

## Project Goal

This project aims to build a robust, feature-rich diagramming editor similar to [draw.io](https://www.drawio.com). The implementation should closely mirror the usability and key functionalities provided by draw\.io, ensuring a smooth, intuitive user experience.

## Key Architectural Principles

The following principles guide all code contributions and implementations:

### Modularity

* All components should be designed as loosely coupled, reusable modules.
* Ensure each module has clear, minimal dependencies.
* Follow package boundaries clearly defined within the `packages/` directory.
* Design modules as plug-and-play packages so they can be swapped or extended with minimal friction, mirroring draw.io's scalability.

### SOLID Principles

All code should adhere strictly to the SOLID principles:

* **Single Responsibility Principle (SRP)**: Each module or class should have a single clear purpose.
* **Open/Closed Principle (OCP)**: Modules should be open for extension but closed for modification.
* **Liskov Substitution Principle (LSP)**: Derived classes must be substitutable for their base classes without altering correctness.
* **Interface Segregation Principle (ISP)**: Favor specific interfaces over general-purpose ones.
* **Dependency Inversion Principle (DIP)**: Depend on abstractions rather than concrete implementations.

### Heavy Testing

* Every new feature or module should come with comprehensive unit tests.
* Integration and end-to-end (E2E) tests are essential, ensuring seamless interactions among components.
* Tests must be clear, readable, and maintainable.
* Aim for high test coverage, particularly for core logic and user interactions.

## Directory Overview

* `packages/` – Core libraries and reusable modules following SOLID principles.
* `ui-shell/vanilla/` – Demo UI built with Vite + Tailwind, adhering to modular and testable design.
* `k8s/` – Kubernetes manifests for deployment.

## Common Commands

Run these commands from the repository root:

### Install all dependencies

```sh
npm install --workspaces --include-workspace-root

### Run unit tests

```sh
npm test

### Run Playwright E2E tests

```sh
npm test --workspace e2e

This spins up the demo UI on port 5173 and executes all tests located in `e2e/tests`.

### Lint all packages

```sh
npm run lint

### Start the demo UI in dev mode

```sh
npm run dev --workspace ui-shell/vanilla


Opens the demo UI at [http://localhost:5173](http://localhost:5173).

### Build the production UI

```sh
npm run build --workspace ui-shell/vanilla

Output is placed in `ui-shell/vanilla/dist`.

### Docker

```sh
docker compose up # builds and serves on port 3002

### Kubernetes

Manifest files reside in `k8s/`. Apply them using:

```sh
kubectl apply -f k8s

## Development Guidelines for Agents

* **Clearly defined tasks:** Implement tasks in small, incremental changes, clearly describing input and output expectations.
* **Code readability:** Write clean, well-documented code. Clearly comment any complex logic or design decisions.
* **Automated feedback loops:** Regularly run linting, formatting, and tests to ensure high-quality code.
* **Lint checks:** Ensure `npm run lint` passes without any warnings or errors before committing.
* **Continuous integration:** Use automated CI/CD pipelines to validate code quickly and reliably.
* **Meaningful abstractions:** Whenever you implement logic, extract it into well-named, reusable functions. Favor small, standalone functions to encourage composability.
* **Minimize conditionals:** Avoid large `if`/`else` blocks. Prefer early returns or design patterns such as strategy or polymorphism to keep flow control clear and modular.
* **Feature visibility:** Every new feature must also be enabled in the demo UI so a human tester can manually exercise it. Avoid implementing functionality that cannot be accessed through the interface.

## Notes

* ESLint runs automatically upon commit via Husky, enforcing code quality.
* Refer to the `README.md` for comprehensive details on features and repository layout.

After couple of iterations, codex new what to do and how to do it.

In the end of the 700+ commits done by codex, i had a working prototype of a diagramming editor that was able to render diagrams and export them as images. It was not perfect, but it was a good start. In overall it was a good experience.

You can read more about the project here.

Chapter 2 — Codex + Claude CLI Tag Team

Back in 2022, i had started v2 for eDominations which is a browser game that i had published its first version in 2017. Before i stopped working on v2 i had a working version missing some features. Written in React as SPA, laravel as backend and mysql as database. All you had to do was docker compose up and you were ready to go. Wanted to test the coding agents with this project if they can nicely and quickly port it to a new stack. Goal was rewriting from Laravel into NestJS and MySQL into Postgres.

Now that i had experience with codex, i wanted to see if it thrives with a more complex project. Well, in the beginning started well. Codex was able to understand the project since i provided AGENTS.md rather first commit. This time we were on stereoids with Codex. First i made sure it analyzed the project when i asked a task to do and come up with a plan and a task list. This way, i was able to see if it was on the right track or not. If it was not, i could correct it and ask to do it again. After couple of days, codex provided 4x concurrency to the codex cloud. Hence, i got even faster results. However, it was too delusional no matter what i did. Lesser descriptions, more comments, more instructions, it was not able to deliver what i wanted. Simply i tried everything i could think of.

Dont take this project lightly, in a single repository, there were client, legacy service, new service, 2 additional microservices for cdn proxy and socket service for live chat. Not an easy with limited context length.

AGENTS.md file i used for this project was like this:


# Repository Guidelines for Agents

This repository is a browser game called eDominations includes couple of services.

## Project Goal

This project aims to build an amazing and entertaining browser based multiplayer strategy game with robust, feature-rich backend services. The game is designed to be modular, scalable, and maintainable, allowing for easy addition of new features and enhancements over time.
Short term goal is to rewrite the PHP application into NestJS in `service-nest` and migrate the database from MySQL to PostgreSQL.

## Key Architectural Principles

The following principles guide all code contributions and implementations:

### Modularity

* All components should be designed as loosely coupled, reusable modules.
* Ensure each module has clear, minimal dependencies.
* Design modules as plug-and-play packages so they can be swapped or extended with minimal friction.

### SOLID Principles

All code should adhere strictly to the SOLID principles:

* **Single Responsibility Principle (SRP)**: Each module or class should have a single clear purpose.
* **Open/Closed Principle (OCP)**: Modules should be open for extension but closed for modification.
* **Liskov Substitution Principle (LSP)**: Derived classes must be substitutable for their base classes without altering correctness.
* **Interface Segregation Principle (ISP)**: Favor specific interfaces over general-purpose ones.
* **Dependency Inversion Principle (DIP)**: Depend on abstractions rather than concrete implementations.

### Heavy Testing

* Every new feature or module should come with comprehensive unit tests.
* Integration and end-to-end (E2E) tests are essential, ensuring seamless interactions among components.
* Tests must be clear, readable, and maintainable.
* Aim for high test coverage, particularly for core logic and user interactions.

## Directory Overview

* `client/` – React client application. Face of the game, built with TypeScript and React.
* `service/` – Laravel application, the original backend of the game, slated to be replaced by the NestJS implementation in `service-nest`.
* `service-nest/` – NestJS backend hosting modular microservices, including the WebSocket service.
* `service-nest/src/microservices/cdn/` – standalone CDN microservice for serving game assets.
* `service-nest/src/microservices/socket/` – NestJS microservice responsible for WebSocket communication.
* `k8s/` – Kubernetes manifests for deployment.
* `docs/` – central documentation hub for design notes, API references, and migration guides.

## Common Commands

Run these commands from the repository root:

### Docker

```sh
docker compose up

### service-nest

Run `npm run build` before executing any tasks in `service-nest`.
Run `npm run start` to start the NestJS application whenever you work on new features or debugging tasks.

## Development Guidelines for Agents

* **Clearly defined tasks:** Implement tasks in small, incremental changes, clearly describing input and output expectations.
* **Code readability:** Write clean, well-documented code. Clearly comment any complex logic or design decisions.
* **Automated feedback loops:** Regularly run linting, formatting, and tests to ensure high-quality code.
* **Lint checks:** Ensure `npm run lint` passes without any warnings or errors before committing.
* **Continuous integration:** Use automated CI/CD pipelines to validate code quickly and reliably.
* **Meaningful abstractions:** Whenever you implement logic, extract it into well-named, reusable functions. Favor small, standalone functions to encourage composability.
* **Minimize conditionals:** Avoid large `if`/`else` blocks. Prefer early returns or design patterns such as strategy or polymorphism to keep flow control clear and modular.
* **Feature visibility:** Every new feature must also be enabled in the demo UI so a human tester can manually exercise it. Avoid implementing functionality that cannot be accessed through the interface.

No result excited me. Thats when i decided to try out Claude CLI. I had a good experience with it in the past, so i thought it would be a good idea to give it a try. Well when you initiate your claude cli, you can /init and let it do its job and prepare a nice AGENTS.md alike file for you. Same purpose as AGENTS.md. However this time i let it cook. So it did a nice meal for itself.

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

eDominations v2 is a browser-based multiplayer strategy game built as a monorepo with microservices architecture. The project is migrating from a PHP/Laravel backend to NestJS with PostgreSQL.

## Architecture

### Core Services
- **client/**: React TypeScript SPA for the game interface
- **service-nest/**: NestJS backend with domain-driven design, including:
  - Main API service on port 3001
  - CDN microservice on port 3036 (`service-nest/src/microservices/cdn/`)
  - WebSocket microservice on port 3035 (`service-nest/src/microservices/socket/`)
- **service/**: Legacy Laravel backend (being phased out)
- **k8s/**: Kubernetes deployment manifests
- **docs/**: Project documentation

### Domain Structure
The NestJS service follows domain-driven design with domains in `service-nest/src/domains/`:
- **user/**: User management, profiles, training, travel
- **battles/**: Combat system, battlehits, wars
- **country/**: Countries, regions, politics, economics
- **company/**: Business system, employees, market
- **journalism/**: Articles, newspapers, comments
- **alliance/**: Player alliances and diplomacy
- **energy/**: Energy system and logging
- **elections/**: Political elections and voting
- **chat/**: In-game messaging and chat
- **cron/**: Scheduled tasks and game events

## Development Commands

### Docker Development
```bash
# Start all services
docker compose up

# Start specific services
docker compose up client service-nest postgres redis

### service-nest (NestJS Backend)
```bash
cd service-nest

# Build (required before most tasks)
npm run build

# Development
npm run start:dev          # Main API with hot reload
npm run start:cdn          # CDN microservice
npm run start:socket       # WebSocket microservice

# Testing
npm run test               # Unit tests
npm run test:e2e           # End-to-end tests
npm run test:cov           # Coverage report

# Linting
npm run lint               # Auto-fix linting issues
npm run lint:ci            # Check linting (CI mode)

# Database
npm run migration:run      # Run migrations
npm run migration:revert   # Revert last migration
npm run seed               # Seed database

### client (React Frontend)
```bash
cd client

# Development
npm run dev                # Start with Tailwind compilation
npm start                  # React dev server only
npm run tailwind           # Tailwind CSS compilation

# Testing
npm run test               # Unit tests
npm run e2e                # Cypress e2e tests
npm run dev:cy             # Cypress interactive mode

# Linting
npm run lint               # ESLint check
npm run lint:fix           # Auto-fix linting issues
npm run format             # Prettier formatting

# Build
npm run build              # Production build

## Database & Infrastructure

### PostgreSQL
- **Host**: postgres (in Docker) or localhost
- **Port**: 5432
- **Database**: edominations
- **Admin UI**: pgAdmin on port 5050 (admin@admin.com / admin)

### Redis
- **Host**: redis (in Docker) or localhost
- **Port**: 6379
- Used for caching and WebSocket scaling

## Testing Strategy

### Unit Tests
- Located alongside source files in `__tests__/` directories
- NestJS uses Jest with TypeORM test utilities
- React uses Testing Library and Jest

### E2E Tests
- NestJS: `service-nest/test/*.e2e-spec.ts` using Testcontainers
- React: `client/cypress/e2e/` using Cypress

### Key Test Patterns
- Use `beforeEach` for clean database state in e2e tests
- Mock external dependencies in unit tests
- Test both success and error scenarios
- Include integration tests for complex domain interactions

## Code Conventions

### NestJS Backend
- Use domain-driven design with modules per business domain
- Controllers handle HTTP requests, services contain business logic
- TypeORM entities with proper relationships and indexes
- DTOs for request/response validation using class-validator
- Comprehensive error handling with proper HTTP status codes

### React Frontend
- TypeScript with strict typing
- Functional components with hooks
- Tailwind CSS for styling
- Context for global state management
- Component colocation (components near their usage)

## Development Workflow

1. **Before starting**: Run `npm run build` in `service-nest/`
2. **Testing**: Always run tests before committing changes
3. **Linting**: Use `npm run lint` to check code quality
4. **Database changes**: Create migrations for schema changes
5. **Feature development**: Follow domain boundaries when adding functionality

## Migration Context

The project is actively migrating from Laravel/MySQL to NestJS/PostgreSQL. When working on features:
- Prefer implementing new functionality in `service-nest/`
- Database schema is defined in TypeORM migrations
- Legacy Laravel code in `service/` is reference only
- Game logic should be implemented in appropriate domain modules

## Environment Configuration

### Docker Environment Files
- `client/.env.docker`: Client configuration for Docker
- `service-nest/.env`: Main NestJS service config
- `service-nest/.env.socket`: WebSocket service config
- `service-nest/.env.cdn`: CDN service config

### Key Environment Variables
- `DB_HOST`, `DB_PORT`, `DB_USERNAME`, `DB_PASSWORD`, `DB_DATABASE`: PostgreSQL connection
- `REDIS_HOST`, `REDIS_PORT`: Redis connection
- `JWT_SECRET`, `JWT_EXPIRES_IN`: Authentication configuration

Did a great job. Finally i got some accurae results and felt that tool got me covered. Only the problem that damage by Codex was done a little. Even though i had over 600 tests in total with e2e tests. Unit tests were rather testing framework itself than actually testing something :). My poor M1 macbook pro started burning and heating up like a furnace. Its summer, i dont need furnace.

Aaand finally i figured out that i have no clue what is going on and nothing works.

Third chapter: The Whole Zoo: Gemini CLI × Claude CLI × Codex + Human

Why not staff a full AI scrum team?

Spinning 2 clones of the repo—one per agent—worked technically, but juggling PRs felt like herding caffeinated cats.

After all the pain and suffering, i decided to give it a try with human review. Don’t get me wrong, i did review previously but whenever a sets of tasks was completed. Not like every PR careful review. Whenever i asked claude to do something, i would review the code and see if it was on the right track or not. Manual testing alongside e2e tests, like a primate i was clicking here and there to see if things were working as expected. Claude was way more impressive than codex. It did truly felt like understood but the limitation with entry package was a problem. Right at the moment i am about to complete the task with Claude, it was like “Nope, not today”. You have got to wait for some amount of time, chill your horses and then come back to me.

Fourth chapter: Gemini CLI + Claude CLI + Codex + Human review

I did started the question myself “Why not use all of them together?”. Since Codex handles all its operations on the cloud all i was to clone repository twice and spawn gemini and cloud in each clone of the same repository. All of a sudden, i was leading a team of 3 AI agents and myself. I was the product owner, codex was the developer, gemini was the tester and claude was the reviewer. Joking, they did all work i was just clicking on merge.

After couple of days of struggle, i don’t think i ever felt this bad that i am so disconnected, battling with 3 AI agents to get the job done while i had no clue of the project’s latest status.

What I Learned

AI agents ≠ architects. Give them a blueprint; don’t ask them to design the house.
Supervision is non-negotiable. Unshepherded agents will drift off-spec.
Generate tests, then audit them. Half their assertions checked nothing useful.
Boilerplate? Perfect use-case. CI configs, CRUD layers, doc scaffolding—let the bots grind.
Small, isolated functions shine. The narrower the task, the crisper the output.
Stay in the loop. If you stop reading the code, you stop knowing your project.

Most importantly, knowing the project is how we find comfort and knowledge day by day ( focus, discipline e.g ). If you let bot do all the work, highly likely you will instruct it wrong eventually.

5 days is enough

omratagn@gmail.com (Omer Atagun) — Fri, 09 May 2025 00:00:00 GMT

I had a trip planned. Calendar cleared, bags half-packed. Then life did its thing and plans fell through. Instead of sulking, I pivoted. One thought kept echoing:

“Well, I guess I could build that thing I’ve been putting off…”

That thought snowballed into one of the most productive weeks I’ve had in a long time. I ended up building a GitHub App that reviews PRs with AI, extended it into a local-first personal assistant, and wired up my house to respond to presence like a proper smart system.

Here’s the log. You can find it here.

1. LGTM Reviewer: A GitHub App for AI-Powered Code Reviews

This idea had been sitting in my head for months: what if a bot could take the chore out of code reviews? Not just another CI check. Something that reads the code changes and leaves human-ish feedback—like a real teammate would.

So I built it. I’m not the first, of course, but I built it my way.

The stack:

Go + Gin: Fast, simple, clean HTTP framework. Handles GitHub webhook routing.
go-github: Official GitHub API client for Go.
Postgres: Stores background jobs triggered by pull request events.
Ollama + Open WebUI: Local LLM stack, no OpenAI, no cloud APIs. Total privacy. Running on an RTX 4060 with CodeLlama 7B—it’s decent, not magical.

The workflow:

GitHub sends a webhook when a PR is opened.
The app queues it in a jobs table.
A background worker picks it up and parses the diff.
I built a custom patch parser that extracts just the meaningful code edits—no noise.
That context is used to craft a prompt for the LLM.
The LLM generates concise, focused review comments.
The app posts those comments inline on the PR.

The first version was overly verbose. Then too vague. After iterating on both the patch parsing and prompt structure, I got something closer to what I’d write myself.

Even now, I’m still not 100% satisfied—home-run AI reviews require better models. But for a local setup, it’s a surprisingly workable baseline.

Bonus: A UI for sanity

To keep things visible:

React + Tailwind + Vite: Stood up in under an hour.
Shows job queue and review results.
Not flashy, just enough to debug and monitor the pipeline.

2. From Code Reviews to Chore Reviews: A Personal AI Assistant

Once the job system and LLM prompt pipeline were solid, a new idea clicked: Why limit this to code?

So I started reusing the same infra to manage me.

I extended the app to accept inputs from:

Apple calendar & reminders (via iCloud sync tools)
Step counts and location from the Home Assistant mobile app
Device and sensor states from Home Assistant
Manual jobs from iOS Shortcuts
Bi-directional communication between Apple Shortcuts and HA, routed via Google Nest

Suddenly the assistant had some situational awareness. It knows when I’m home. It knows which room I’m in (thanks, FP2). It knows my schedule. It could (in theory) nudge me or help me plan.

“You said you’d clean the kitchen by tonight. You’re still in the office. Want me to queue a reminder for tomorrow morning?”

That’s the dream, anyway. Reality: it doesn’t manage much yet—frankly, it can’t even manage itself. But the scaffolding is there. The jobs table and LLM worker model make it easy to add these use cases with minimal overhead.

3. Smart Home: Real Sensors, Real Automations

Home automation is a rabbit hole I fall into every few months. This week, I went deep.

Aqara FP2: Presence, not just motion

The Aqara FP2 uses mmWave to detect presence, not just motion. That means it knows when I’m sitting still at my desk or lying on the couch—not just when I move.

That granularity changes everything.

Example: if I leave a room, the lights turn off after 30 seconds. But if I’m sitting still? They stay on. No more waving my arms like a confused NPC just to keep the lights alive.

Home Assistant: Full Migration to RPi 4

Migrated everything to a Raspberry Pi 4 with 8GB RAM, running Home Assistant OS.

Native Matter support? ✔️
No Docker hacks? ✔️
Easy backups and recovery? ✔️

Honestly, it’s the most user-friendly it’s ever been. Almost boring how well it works.

Elero Comfort-868 Blinds: The Final Boss

I have some lovely electric blinds that speak a proprietary 868 MHz protocol. Fancy, but dumb.

I’m working on wiring them into Home Assistant—whether that’s via USB bridge, ESPHome, or straight-up reverse engineering.

The goal: have the blinds close automatically when I leave the room and the sun is blasting through the windows. Or maybe just when it’s 3PM and I want to watch TV.

Still experimenting with serial sniffers and module options. It’s a fun little boss fight.

4. Thoughts and Takeaways

This week, one project turned into five—but everything looped back to a central theme:

What if software actually worked for me—in code and in life?

What stood out:

Reusing architecture saved time: The background worker + job queue + LLM pipeline became the backbone for multiple ideas.
Local AI is viable, kinda: Running models like CodeLlama and Mistral locally gave fast, private, “meh-enough” results. Not perfect, but way more usable than I expected.
The smart home feels smart now: With mmWave sensors, local logic, and real-time context, it responds to me, not the other way around. No more shouting room names and commands like a voice butler.

Wrapping It Up

What’s next? I’m planning to shift my personal AI strategy toward RAG (Retrieval-Augmented Generation), since fine-tuning isn’t viable for a personal project at this scale. That said, long-context support still feels just out of reach—hoping better models land soon.

Also: building things in Go was so refreshing. I’ll be using it way more from here on.

Home k8s cluster

omratagn@gmail.com (Omer Atagun) — Fri, 02 May 2025 00:00:00 GMT

It was quite a while that I had no time to write any posts. Eventually, I got the time due to having my vacation cancelled and having to stay at home with a bunch of PTO in hand. So, it was time.

OVERKILLING HOME SERVER

I had this machine that I got back then with an i5 12400f and 4060 and 47GB RAM with 1tb nvme and 1tb usb ssd.

My plan:

Create an organization in GitHub for home projects.
Run my own GitHub runners within a cluster using Arc Runners.
Include a kubeconfig in the repositories so that I could create simple deployment and service YAML files to deploy.
Have local AI running within the cluster, so NVIDIA support in pods was needed for the job.
Plex server.
This blog itself – a new project that I am building with Go.

First, I have started going with Proxmox so I could have 3 nodes from the same machine and extend furthermore, but that was quite annoying to handle all virtualization for really no gains (I have other machines as well). I leveraged its simplicity to dump a virtual machine and create a new one right after.

Prerequisites

I use cloudflare zero tunnel to expose my cluster and other applications to my domains. If you are not intended to do that, this story/tutorial is not for you. You can read more details here. I also do not mention how ingress or services or k8s itself work within this blog post, you got to figure that out yourself.

First steps:

With Proxmox installed, to refresh my memories I started creating my cluster using Rancher. I had to get back to thinking on YAML world. So, I broke entire thing couple of times. Finally, I had a working prototype with Arc Runners and deployments from repositories. Yay! Fresh bare Ubuntu installation.

Second step:

Rancher changed its tutorial a bit after couple of years. So i have gone with Helm CLI installation. Quite straightforward, all you have to go to get latest versions from corresponding repositores releases and you are pretty much done.

Third step:

Start with Actions runner controller tutorial up until step “Configure a runner scale set”. You need more than what is given to you In tutorial you will see an installation for arc runners like this;

INSTALLATION_NAME="arc-runner-set"
NAMESPACE="arc-runners"
GITHUB_CONFIG_URL="https://github.com/<your_enterprise/org/repo>"
GITHUB_PAT="<PAT>"
helm install "${INSTALLATION_NAME}" \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
    --set githubConfigSecret.github_token="${GITHUB_PAT}" \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

But this on its own is not sufficient enough to run docker in docker in which you need most of the time for your CI/CD. After reading the docs, you can find out which keys you need to populate to make that happen

  --set resourceMeta.noPermissionServiceAccount.labels.app=dummy \
  --set resourceMeta.noPermissionServiceAccount.annotations.foo=bar \
  --set runnerContainerPrivileged=true \
  --set containerMode.type=dind \
  --set dockerEnabled=true \
  --set dind.privileged=true \

You can then follow up rest of the tutorial as it is.

Forth step:

Now we have arc-runners on but they have not any secrets that they can pull images from your repository registry. We need to create one. Create a secret in the same namespace and name it ghcr-secret. Domain will be ghcr.io and your username and token with package access.

You will also need to inject your kubeconfig into your repository action secrets so you can actually deploy into clusters.

my workflow ci.yaml looks like this

name: Deploy to production cluster

on:
  push:
    branches:
      - master
  workflow_dispatch:
jobs:
  deploy:
    runs-on: arc-runner-set
    if: ${{ github.event_name == 'push' || github.event_name == 'workflow_dispatch' }}
    permissions:
      contents: read
      packages: write
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GHCR_PAT }}

      - name: Get commit count
        id: vars
        run: echo "count=$(git rev-list --count HEAD)" >> $GITHUB_OUTPUT

      - name: Build and Push Service
        run: |
          cp .env.example .env
          VERSION=${{ steps.vars.outputs.count }}
          IMAGE=ghcr.io/${{ github.repository_owner }}/$(basename ${{ github.repository }})/service:$VERSION
          docker build -t $IMAGE .
          docker push $IMAGE

      - name: Pull and deploy Service
        run: |
          VERSION=${{ steps.vars.outputs.count }}
          IMAGE=ghcr.io/${{ github.repository_owner }}/$(basename ${{ github.repository }})/service:$VERSION
          docker pull $IMAGE

      - name: Set up kubectl
        uses: azure/setup-kubectl@v4
        id: install
        with:
          version: "v1.28.2"

      - name: Update Kubernetes Deployment with new image version
        run: |
          VERSION=${{ steps.vars.outputs.count }}
          IMAGE=ghcr.io/${{ github.repository_owner }}/$(basename ${{ github.repository }})/service:$VERSION
          sed -i "s|image: .*|image: $IMAGE|" deployment.yaml
        working-directory: k8s

      - name: Apply Kubernetes manifests
        run: |
          kubectl apply -f deployment.yaml --insecure-skip-tls-verify
          kubectl apply -f service.yaml --insecure-skip-tls-verify
        working-directory: k8s

Congratulations, now you have a arc runners for your repositories which can spawn a pod, run your jobs and clean all the leftover data after pod is shutdown. I know you don’t need it but its fun!

Now we need to deploy something

Open web ui

Create this as webui.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: open-webui

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: open-webui-pvc
  namespace: open-webui
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: open-webui
  namespace: open-webui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: open-webui
  template:
    metadata:
      labels:
        app: open-webui
    spec:
      containers:
        - name: open-webui
          image: ghcr.io/open-webui/open-webui:main
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: data
              mountPath: /app/backend/data
          env:
            - name: WEBUI_PORT
              value: "8080"
      dnsPolicy: ClusterFirstWithHostNet # this is needed for accessing ollama instance in the machine
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: open-webui-pvc

---
apiVersion: v1
kind: Service
metadata:
  name: open-webui
  namespace: open-webui
spec:
  selector:
    app: open-webui
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: ClusterIP

kubectl apply -f webui.yaml

and you are done. Create an ingress for where you would like to publish. If you have ollama installed in different machine, you can set that up within open web ui configuration. If ollama will run in same machine as cluster, then make sure ollama environment set for 0.0.0.0 otherwise will be closed to network and open only to localhost.

Handling nvidia support

Now i used nvidia gpu for plex server since i needed more for that than ollama ( you can leverage ollama from machine rather pod ).

At the end my cluster looked like this.

That sums up! this blog post was meant to be warm up after a very long time. Hopefully i will do better next time.

Testing with chatgpt

omratagn@gmail.com (Omer Atagun) — Wed, 29 Nov 2023 00:00:00 GMT

You have heard a lot about how chatgpt boosts productivity, yes it does and takes away the boredom of repetitive but necessary tasks. Testing is a major aspect in software engineering. Helps your pairs/other engineers to understand the specifications and capabilities of the solution provided by you. It helps speed up adjustment process when a requirements also change. Long story short, this is about how i have leveraged chatgpt to write unit tests in a project that did not have any. Even though i had no clue what the project was about.

Understanding the code

At some point in your career, you learn empathy is the key to understand somebody elses solution. Yet, there are days you may need some help or you may need to ask some questions about the original solution. Well, no more. Chatgpt can help you out with the intention. May get the entire picture wrong but remember, it is all about predicting not certainity. As like empathy.

Lets go through some examples of how can we leverage chatgpt doing this task. Providing example below;

export const hidePrivateProperties = <T, N>(
  object: T,
  properties: Array<N extends keyof T ? N : any>
): Omit<T, N extends keyof T ? N : any> => {
  properties.forEach((property) => {
    delete object[property]
  })

  return object
}

Some utility method that helps deleting properties of a given object and returns the same and altered(no copy) object with new type type that omits those properties from the given object.

Lets ask chatgpt what this does.

Q: Can you tell me what this exactly does?

In summary, this function takes an object and an array of properties,
and it removes those properties from the object.
The types are designed to ensure that the properties
being hidden are actually properties of the object.
Note that using delete on an object modifies it in place, and
the modified object is then returned.

FYI i kept entire description hidden and copied only the summary. So now we know that this function does not copy the object, it alters and returns the original object.

Lets ask chatGPT to write unit tests for this utility function and do some mocks.

Q: Can you write me unit tests in jest, also provide some mocks. I have jest environment set already, so skip that part.

// Import the function
import { hidePrivateProperties } from './your-file'

// Mock data for testing
interface TestData {
  name: string
  age: number
  address: string
}

const mockObject: TestData = {
  name: 'John Doe',
  age: 25,
  address: '123 Main St',
}

describe('hidePrivateProperties', () => {
  it('should hide specified properties from the object', () => {
    // Arrange
    const propertiesToHide = ['age', 'address']

    // Act
    const result = hidePrivateProperties(mockObject, propertiesToHide)

    // Assert
    expect(result).toEqual({ name: 'John Doe' })
  })

  it('should handle an empty array of properties', () => {
    // Arrange
    const propertiesToHide: string[] = []

    // Act
    const result = hidePrivateProperties(mockObject, propertiesToHide)

    // Assert
    expect(result).toEqual(mockObject) // The object should remain unchanged
  })

  it('should handle properties that do not exist on the object', () => {
    // Arrange
    const propertiesToHide = ['nonexistentProperty']

    // Act
    const result = hidePrivateProperties(mockObject, propertiesToHide)

    // Assert
    expect(result).toEqual(mockObject) // The object should remain unchanged
  })

  it('should handle an empty object', () => {
    // Arrange
    const emptyObject: TestData = {}

    // Act
    const result = hidePrivateProperties(emptyObject, ['property1', 'property2'])

    // Assert
    expect(result).toEqual({}) // The object should remain empty
  })

  // You can add more test cases based on your specific use cases
})

Pretty good right? chatgpt understood the probable scenarios, provided tests for it and let you implement. So, i heard you are saying that you could also do this or even have done it before, why this blog post now?

Well, we are not done yet. Nice thing about bots that they may see better than you most of times with given enough of context.

Q: Do you think we may have an edge case in this utility function? if so, could you write me a tests for that?

describe('hidePrivateProperties', () => {
  // ... other test cases ...

  it('should handle a null object', () => {
    // Arrange
    const nullObject: TestData | null = null
    const propertiesToHide = ['age', 'address']

    // Act
    const result = hidePrivateProperties(nullObject!, propertiesToHide)

    // Assert
    expect(result).toBeNull() // The result should also be null
  })

  it('should handle an undefined object', () => {
    // Arrange
    const undefinedObject: TestData | undefined = undefined
    const propertiesToHide = ['age', 'address']

    // Act
    const result = hidePrivateProperties(undefinedObject!, propertiesToHide)

    // Assert
    expect(result).toBeUndefined() // The result should also be undefined
  })
})

Now when we put all these into test, we can easily see that we have a flaw in our utility function;

TypeError: Cannot convert undefined or null to object

      4 |    ): Omit<T, N extends keyof T ? N : any> => {
      5 |     properties.forEach((property) => {
    > 6 |       delete object[property];
        |                     ^
      7 |     });
      8 |
      9 |     return object;

Same problem with null as well. You see, our generic T was too generic :). It indeed did allow undefined or null as an argument when it should only be an object.

Thanks to the bot itself, we figured that we have a flaw in our function and might tickle us later in time. If we would refactor this generic with expectation of a something looks like and only like object, we would not even need these tests that measures undefined and null scenarios.

export const hidePrivateProperties = <T, N>(
  object: T extends Record<string, any> ? T : never, // if it aint object, then never to me :)
  properties: Array<N extends keyof T ? N : any>
): Omit<T, N extends keyof T ? N : any> => {
  properties.forEach((property) => {
    delete object[property]
  })

  return object
}

Guess what, this was not enough either :) because if we pass an array of objects, typescript will not warn us and this method will behave totally irrelevant. Because it will do nothing but return the same array of objects given.

it('should handle an array of objects', () => {
    const mockResult = {
        surname: 'sdsds'
    }
    // Arrange
    const result = hidePrivateProperties([{
        name: 'asd',
        surname: 'sdsds'
    }], ['name'])

    // Assert
    expect(result).toEqual(mockResult); // The result should also be undefined
  });

  ● hidePrivateProperties › should handle an array of objects

    expect(received).toEqual(expected) // deep equality

    Expected: {"surname": "sdsds"}
    Received: [{"name": "asd", "surname": "sdsds"}]

      63 |
      64 |     // Assert
    > 65 |     expect(result).toEqual(mockResult); // The result should also be undefined
         |                    ^
      66 |   });
      67 |   // You can add more test cases based on your specific use cases
      68 | });

Q: with above, typescript is still okay if i pass array of objects, we need to block that

export const hidePrivateProperties = <T, N>(
  // focus here, chatgpt conditioned for array of objects but did not think of others again.
  object: T extends Record<string, any>[] ? never : T,
  properties: Array<N extends keyof T ? N : any>
): Omit<T, N extends keyof T ? N : any> => {
  properties.forEach((property) => {
    delete object[property]
  })

  return object
}

Above would be resulted back again that our generic accepts every primitive values including undefined and null. So if we would be just copy pasting, we would be wrong.

Don’t know about you but to me bot had helped discovering a lot of problems in nice looking generic at first sight. Now i can cover the requirements and be done with it.

type TOnlyObject = Record<string, any>[] | null | undefined | boolean | string | number | symbol
object: T
extends TOnlyObject ?
never : T,

Conclusion

ChatGPT or any other AI will help you and boost your productivity but that does not mean will do %100 of the job for you. Leveraging bot will speed up this process and help you discover your flaws and make you better in terms of shipping any solution. Hence, you will have your pair programmer, reviewer with you at all times :)

Note: if this type TOnlyObject comes a bit bloated to your eyes, ask chatGPT to simplify ;) also those any should be never to avoid even more tests :)

Until next time, stay hydrated!

Software architect elevator

omratagn@gmail.com (Omer Atagun) — Sat, 08 Jul 2023 00:00:00 GMT

Recently i have resigned from my position in bragg group. Long story short, i have met this amazing team of people from LeanIX during sets of interviews and i decided to join them.

Out of all companies that i had applied to, only LeanIX was the one i wanted to be part of thanks to the people i have met during the interview process. I am really excited to join them and i am looking forward to new challenges that are ahead of me.

After i have signed the contract, company sent me a gift package. Not just a regular t-shirt, hoodie and ledger but on top of all these standard goods, they have also shipped me a book called The Software Architect Elevator by Gregor Hohpe.

I was already excited to get my hands on to this book for a reason that a tech company choose to deliver this book to its new employees. I assumed the book should not be just a random boring book, right?

I have started to read the book and i have to say that i am really impressed. I am not going to write a review about the book but i will share some of the quotes that i have liked so far. Maybe i will add some little examples of how i understood or what i have witnessed before. It was nice having validation from the writer that my thought process was not that bad.

Lets talk about the book.

Lets list the subjects that we are going to cover which was mentioned in the book itself. I think i may categorize them as business and technical parts even though they live together in real world.

Technical

Tools

"Make sure that your tools work for you, not the other way around". Such elegant sentence that explains a lot on its own. Presumably, you as a reader and most likely have witnessed this at least once but in reality more than once. By my understanding, this could be literally anything, from your choice of packages to a framework that you have chosen to use. It truly has to work for you and allow you to produce more value in little time. Simple example would be, having an external dependency that changes its api frequently and you have to keep up with it due to these changes are breaking at all time. This is not a good tool for you. It is not working for you, you are working for it. You are spending your time to keep up with it.

Testing

"Propose to a development team that they let you delete 20 arbitrary lines from their source code. Then, they will run their tests--if they pass, they will push the code straight into production. From their reaction, you will know immediately whether their source code has sufficient test coverage.". Makes me conclude that your test coverage should not be measured by percentage but how confident you are to make changes.

I think we can example this very nicely.

const executeWithDelay = (delay, callback = () => undefined) => setTimeout(callback, delay)
const desiredDelay = 1000

jest.useFakeTimers()
jest.spyOn(global, 'setTimeout')

describe('executeWithDelay', () => {
  executeWithDelay(desiredDelay, () => {
    console.log('test')
  })

  expect(setTimeout).toHaveBeenLastCalledWith(expect.any(Function), desiredDelay)
})

// this javascript code is tested now, but is it enough?

// considering there are not any types, we might as well do below.

executeWithDelay('a', 'b')

Lets increase the confidence with typescript.

type anyFunction = () => void

const executeWithDelay = (delay: number, callback: anyFunction) => setTimeout(callback, delay)
const desiredDelay = 1000

jest.useFakeTimers()
jest.spyOn(global, 'setTimeout')

describe('executeWithDelay', () => {
  executeWithDelay(desiredDelay, () => {
    console.log('test')
  })

  expect(setTimeout).toHaveBeenLastCalledWith(expect.any(Function), desiredDelay)
})

We kept the same tests but increased our confidence by typing delay as a number, callback as anyfunction yet i think it is not enough.

Isn’t it odd to have negative integers for a delay? Yes, i thought so too. So lets make our number into a more strict type where we only ask for unsigned integer. Hence, we increase our confidence for this feature to work as expected.

type UnsignedInteger<T extends number> = number extends T
  ? never
  : `${T}` extends `-${string}` | `${string}.${string}`
  ? never
  : T

type anyFunction = () => void

const executeWithDelay = <T extends number>(delay: UnsignedInteger<T>, callback: anyFunction) =>
  setTimeout(callback, delay)
// if we try to set desiredDelay to any negative number, it will fail to compile.
const desiredDelay = 1000

jest.useFakeTimers()
jest.spyOn(global, 'setTimeout')

describe('executeWithDelay', () => {
  executeWithDelay(desiredDelay, () => {
    console.log('test')
  })

  expect(setTimeout).toHaveBeenLastCalledWith(expect.any(Function), desiredDelay)
})

// No need to stop here, we can also test
// if the function is called with the correct arguments but
// in our case there are not any arguments.

type UnsignedInteger<T extends number> = number extends T
  ? never
  : `${T}` extends `-${string}` | `${string}.${string}`
  ? never
  : T

type anyFunction = () => void

const executeWithDelay = <T extends number>(delay: UnsignedInteger<T>, callback: anyFunction) =>
  setTimeout(callback, delay)
// if we try to set desiredDelay to any negative number, it will fail to compile.
const desiredDelay = 1000
const mockCallBack = jest.fn(() => console.log('test'))

jest.useFakeTimers()
jest.spyOn(global, 'setTimeout')

describe('executeWithDelay', () => {
  executeWithDelay(desiredDelay, mockCallBack)
  expect(setTimeout).toHaveBeenLastCalledWith(mockCallBack, desiredDelay)
})

Others would also work as we expected any function to be given, but here in this case, we assure that it is our mock function which is called. Better explicit than implicit.

I think we have done already enough to explain what confidence actually implemented in a test is. You have the imagination, do more if you see fit.

Design

"The set of design decisions about any system that keeps its implementors and maintainers from exercising needless creativity." by Desmon D'Souza mentioned in the book itself by the author.

An approach that will help all the contributors to have boring day to day work. Apprecite the boring solutions. S.O.L.I.D. are one of those to apply for your codebase.

I belive a good example might be protocols where an interface is defined and there is only one way to accomplish a given task. HTTP protocol for example. There is a one way to accomplish but any language can implement it in its own way. Even after sometime, you may iterate and improve its stability, security or performance however you wish to. It is straight forward and boring.

Just for the sake of example, we can present below code.

interface MakeCakeProps {
  color: string
  taste: string
  size: number
}

interface MakeCakeReturn {
  color: string
  taste: string
  size: number
}

interface SomeServiceMakesACakeProps {
  data: MakeCakeProps
}

const someServiceMakesACake = (data: SomeServiceMakesACakeProps) => {
  return {
    color: 'red',
    taste: 'sweet',
    size: 10,
  }
}
export const makeCake = (data: MakeCakeProps): MakeCakeReturn => {
  // tomorrow you would decide to change someServiceMakesACake to something else
  // your implementation detail would change but interface remains the same.
  return someServiceMakesACake({
    data,
  })
  //return someOtherserviceMakesCake(data)
}

Since the executor is encapsulated into another function named makeCake, it is easy to test and maintain. It is boring and it is good. It can also be replaced at anytime without changing the interface or the usage of the function that has been exported. On top of this implementation, you would add a linter rule to make sure that no one is using axios directly.

Abstraction

"If an abstraction takes away too many or the wrong things. It becomes overly restrictive and no longer applicable. If it takes away too few things, it did not accomplish much in terms of simplification and hence is not very valuable."

Can’t disagree. Let’s try to explain this with an example.

// here we have just another class that is responsible for baking a cake.
class FreddieBakery {
  private ingredients: string[]
  private time: number
  private temperature: number
  // we have asked for ingredients, time and temperature
  constructor({ ingredients, time, temperature }) {
    this.ingredients = ingredients
    this.time = time
    this.temperature = temperature
  }
  // then we abstracted the baking process into a function called bake
  // so implementation detail would not matter to the client.
  // We wanted to make it easy.
  public bake() {
    // is this really enough abstraction or too little? is it even flexible?
    // What if we need temperature to be changed in the middle of the baking process?
    // no matter what, do we have a cancel process ?
    // can i add a new ingredient in the middle of the baking process?
    console.log(`Baking for ${this.time} minutes at ${this.temperature} degrees`)
  }
}

Above example is a good thinking process of whether it is too little or too many. To find the right balance, we should always think what could have been requested more from this feature?

Business

Author covers pretty much all aspects that are needed, but i am not interested in writing those in this blog.

Conclusion

Out of all other books that i had luck to lay my hands on, this one is a great starter pack for a software architect. It is not a book that you would read once and forget about it. It is a book that you should come back to from time to time. Observations, solutions or analysis most likely will remain timeless.

All above examples were my own understanding of the book in which merged with my own experiences, i am sure there could be better explanations but isn’t it the point? We all have different experiences and we all have different ways to explain things. Just like a software solution, there are millions of ways to do it but there is always a better way to do it.

Digitalocean docker deployment

omratagn@gmail.com (Omer Atagun) — Sat, 06 May 2023 00:00:00 GMT

Well, as you know from previous posts that i created my own homeserver using cloudflare tunnel and kubernetes. But i was not satisfied with the performance of the server alongside the cost of it for running. Electricity, UPS ( for energy disruptions ) and also the infastructure cost i had to get myself into. This was almost a endless work which i have enjoyed utmost but i think it is time to end this :).

Idea

Almost a year and the half ago me and a colleague Matevz had an idea of creating a service that would allow you to deploy your project without knowing a single thing about deployment procedures.

You will tell me know that it is not a something new, there is a heroku or netlify was doing that for a quite long time. You’re right but our problem was that on top of what know, we had to learn these products requirements. Which is bad and additional knowledge will be gone to waste if you don’t use it frequently. However, we said this should be total no brainer, you just push your code and it should work. All you had to do is to provide us Dockerfile.

Digitalocean

Well yeah, it was just another day that i was browsing the internet and i saw that digitalocean is offering starting from 5$ a Dockerfile under app section. I was curious how good this was, hence the following happened :).

All you have to do is to give access to your gitlab or github account and select the repository you want to deploy. You can set your Dockerfile path and the port you want to expose. That's it, you're done.

Performance and cost

Let’s start with identifying both servers and their specs.

Homeserver

4 core cpu ( i5 4th generation)
8gb ram ( DDR3 )
150/50 mbps internet connection ( LTE, yeah i live in rural area)
120gb ssd
- Docker for cloudflare tunnelSSH
- Kubernetes ( k3s with Rancher )
- Network router ( TP-Link ) 5 ports.
- 1x RPI 4b 8gb ram for additional node.
God knows if goes kaput :). Yet i wrote another rust application that pings and send me an email if it goes down. Runs in my personal mac and gaming desktop one. Yeah it is just for a fun.

approximately electric cost is 10$ per month

Digitalocean

512mb ram
1 core vCpu ( shared obviously )
No other information available
ssh access into the pod
Runtime logs
Insights
Auto deploy from a given branch with fallbacks and failures.
HTTP redirects
ENV for both build and runtime
Alert policies and more

costs 5$ per month

Deployment

I gave the access to my gitlab account and told my branch and where my dockerfile lives, simple as that. Then i picked the service i want out of options that i could have staging, dev and prod environments with N amount of nodes.

Once deployment was done and live in a given random subdomain, all i had to do is to log into my cloudflare account and add a CNAME record to that domain. Done, now my blog is live and running without me thinking anything about deployment procedure anymore.

Juicy part - Benchmark

I will cut short and let you decide whether it is worth it or not.

Homeserver AB test

Concurrency Level:      10
Time taken for tests:   3.458 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      2914574 bytes
HTML transferred:       2802800 bytes
Requests per second:    28.92 [#/sec] (mean)
Time per request:       345.800 [ms] (mean)
Time per request:       34.580 [ms] (mean, across all concurrent requests)
Transfer rate:          823.10 [Kbytes/sec] received

Digitalocean Docker deployment AB test

Concurrency Level:      10
Time taken for tests:   2.563 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      2842516 bytes
HTML transferred:       2760700 bytes
Requests per second:    39.02 [#/sec] (mean)
Time per request:       256.273 [ms] (mean)
Time per request:       25.627 [ms] (mean, across all concurrent requests)
Transfer rate:          1083.18 [Kbytes/sec] received

Conclusion

I think we got ourselves an obvious winner here. This does not mean i will close my homeserver yet. For cases like blog applications, digitalocean is your guy to go. but with applications that require more resources ( database, key/value store etc), i would still go with my homeserver.

Till the next time, stay hydrated!

What is Auto GPT

omratagn@gmail.com (Omer Atagun) — Sat, 08 Apr 2023 00:00:00 GMT

Oh boy, such an exciting project. AutoGPT came into a trending page of github in a very dominating way. Of course it got my attention to look into it. I had some spare time and followed up their installation to figure what i can do with it.

It did not take long to realize that is meant to be an AI model that based on a given instructions would try to learn and execute your command. It is spiced up with text-to-speech to increase joy. It is not that shallow. It has a lot more to offer.04

Here visit their Auto GPT - Github

What Auto GPT does

Pretty simple and complex. It uses Pinecone memory as for vector database. If you have been around recently, you would notice that vector databases are trending in terms of their capabilities and speed to go along with artifical intelligence, but this is subjec is for another time.

Auto GPT also gets help from chatGPT when its needed. So to summarize, it is a combination of different technologies that allows you to instruct and let the tool do the job for you. This can be from your best meal recipe to code corrections.

The biggest advantage of AutoGPT is that it can search the web.

I just told you above, i have cloned the repository, populated the .env with my credentials and i started to give an instructions to AutoGPT to see the results.

Instructions

I gave 4 goals to autogpt. Clear enough to let it thrive :)

1- visit https://omeratagun.net and read all its blogs

2- Generate new blog posts as like https://omeratagun.net would write

3- Write them into a file

4- Shutdown

It started thinking. I also got my popcorn to watch all the outputs flowing through my screen. First it visited and discovered my blog posts and wrote them into a file like this.

AutoGPT initial discovery from my blog posts written into a file;

The blog contains multiple posts on various topics related to programming and technology, including:
 - Setting up a home server with Cloudflare
 - Using ChatGPT
 - Deploying Actix with Docker
 - Discussing the future of Rust and other programming languages
 - Exploring alternatives to Amazon S3 for object storage

I was already suprized because it already put them into a thought process. I do not have a blog post named Discussing the future of Rust and other programming languages but i have rust and others.

After the first discovery, thinking process continued and it created another file. this time called it a dataset.

- Setting up a home server with Cloudflare.
- Using ChatGPT
- Deploying Actix with Docker.
- Discussing the future of Rust and other programming languages.
- Exploring alternatives to Amazon S3 for object storage.- Setting up a home server with Cloudflare.\n- Using ChatGPT\n- Deploying Actix with Docker.\n- Discussing the future of Rust and other programming languages.\n- Exploring alternatives to Amazon S3 for object storage.- Setting up a home server with Cloudflare.\n- Using ChatGPT\n- Deploying Actix with Docker.\n- Discussing the future of Rust and other programming languages.\n- Exploring alternatives to Amazon S3 for object storage.- Setting up a home server with Cloudflare.\n- Using ChatGPT\n- Deploying Actix with Docker.\n- Discussing the future of Rust and other programming languages.\n- Exploring alternatives to Amazon S3 for object storage.

This was a little odd, it seemed to me it got somehow confused and appended same chunk of discoveries into another file and started appending repeatedly.

But then

It gave me two blog posts, i will let you to decide whether they are relevant to my style or choice of subject.

The first blog post AutoGPT provided is/was;

Title: The Benefits and Challenges of Data Security in the Digital Age

In today's digital age, businesses have access to vast amounts of data that can help them make better decisions, improve customer experiences, and achieve growth. However, with the increasing amount of data being generated, data security has become a critical concern for organizations. In this blog post, we'll explore the benefits and challenges of data security in the digital age and provide tips for protecting sensitive data.

Benefits of Data Security

1. Protecting Customer Information: Data security measures like encryption and access controls can help protect confidential customer information like names, addresses, and financial data, thereby building customer trust and loyalty.

2. Preventing Data Breaches: Robust data security measures can safeguard data from cybercriminals who seek to steal information for malicious purposes.

3. Complying with Regulations: Many regulatory frameworks like HIPAA and GDPR require organizations to secure and protect sensitive customer information, and non-compliance can result in hefty fines and legal repercussions.

Challenges of Data Security

1. Complexity: The complex nature of cybersecurity and the ever-evolving methods of cyber attacks can make it challenging for businesses to stay ahead of potential security threats.

2. Costs: Implementing robust security measures can come at a high cost for businesses, which may require significant investments in hardware, software, and cybersecurity experts.

3. Employee Error: Human error, like leaving devices or passwords unsecured or clicking on malicious links, can undermine even the most robust cybersecurity measures.

Tips for Protecting Sensitive Data

1. Use Strong Passwords: Passwords should be long, complex, and unique, and employees should be required to change them frequently.

2. Implement Encryption: Encryption can protect sensitive data from cyber threats when data is transmitted or stored.

3. Train Employees: Regular training can help employees identify potential security threats like phishing emails or malware and take appropriate measures to prevent breaches.

In conclusion, data security is of utmost importance in the digital age, and businesses need to ensure that robust security measures are in place to protect sensitive data. While implementing data security measures can be challenging and come at a high cost, businesses need to prioritize data security to prevent costly data breaches, protect customer information, and comply with regulations.
\n\nTitle: Leveraging Cloud Computing for Business Growth: Pros and Cons\n\nCloud computing has become a buzzword in the business world, offering a range of benefits to organizations of all sizes. Cloud computing refers to the delivery of computing services like servers, storage, applications, and more over the internet, providing businesses with the flexibility to scale up and down their IT infrastructure as demand changes. In this post, we'll explore the pros and cons of cloud computing and how it can help organizations drive growth.\n\nPros of Cloud Computing\n\n1. Scalability: Cloud computing provides businesses with the ability to scale their IT infrastructure up or down as needed, depending on demand. This flexibility can help businesses avoid downtime during peak periods and save costs during low periods.\n\n2. Cost Savings: Cloud computing can help businesses save costs on IT infrastructure, like hardware, software, and maintenance, as well as reduce labor costs associated with managing IT infrastructure.\n\n3. Accessibility: Cloud computing provides businesses with access to data and resources from anywhere with an internet connection, making it easier for employees to collaborate and work remotely.\n\nCons of Cloud Computing\n\n1. Security Risks: Cloud computing introduces new security risks, like data breaches or cyber-attacks, that businesses need to manage adequately.\n\n2. Compliance Concerns: Organizations in certain industries like healthcare or finance need to comply with strict regulatory frameworks that may not permit the use of cloud services.\n\n3. Reliance on Third-Party Providers: Cloud computing puts businesses in a position of reliance on third-party providers who may experience technical difficulties, service outages, or other disruptions that affect business operations.\n\nTips for Leveraging Cloud Computing for Business Growth:\n\n1. Understand your organization's unique needs: Before considering cloud services, businesses should understand their unique requirements and determine whether cloud services align with their business goals.\n\n2. Evaluate Service Providers: Evaluate potential service providers based on their track record, customer reviews, and compliance with industry standards.\n\n3. Implement Strong Security Measures: Businesses should implement robust security measures like encryption, access controls, and regular security audits to mitigate security risks.\n\nIn conclusion, cloud computing can be a game-changer for businesses looking to scale their IT infrastructure and drive growth. While there are several benefits, businesses should also be mindful of the potential risks and challenges of cloud computing and implement strategies to mitigate them. By understanding their unique requirements, evaluating potential service providers, and implementing robust security measures, businesses can leverage cloud computing to achieve their growth objectives.\n

Even though it does not look like what i would want to write, it did made some assumption, read some other posts and figured a subject to write and wrote it. I am happy that i was able to get a result instead of complaining :).

IMHO based on the stars and attraction AutoGPT caught, i see a bright future in the project itself. This might be one of those tools that you’d be using on a daily basis that can give you some new ideas and maybe do most of the job for you.

I think General AI concept is not that far. It is very exciting to see these developments. I am happy and thankful for the ones who have contributed into this project. Hoping to see many more.

Till the next time, stay hydrated and do not forget to ask your choice of AI how to stay hydrated.

Home server with cloudflare

omratagn@gmail.com (Omer Atagun) — Thu, 09 Feb 2023 00:00:00 GMT

If you were around in the tech back in beginning of 2000’s everybody wanted to host their gaming server alongside with web servers. You have had to open ports from your router, redirect to the relative tcpstream in your computer ( web, game server etc. ) and expose it through internet. Yet this was never enough to accomplish such thing. DSL and many other connections had dynamic ip addresses. Once your internet is off in which it was relatively often, your ip would have changed and your home server with bunch of things in it, would go dark.

These days, this problem got even bigger. ISP’s figured that they can sell static ip addresses for the ones that are planning to do business with it. Considering ipv4 is limited, things went wild.

Thanks to Cloudflare we have now pretty easy home-server setup rather than struggling with double NAT’s or dynamic ip addresses.

Let’s talk about what is reverse-ssh and how cloudflare handles this ?

Reverse-ssh

Lets talk about scenario that you are under nested NAT. Which means you obtained an ip address with bunch of other people that are in same network.

Figuratively claiming that your ip address is 196.22.111.22. There are other connections established ( your neighbour etc). You have assigned a random port number. ISP’s router has awareness of which port you have assigned to, lets talk as for an example that you obtained port 3211. Your neighbour obtained port 33111.

Now, if you were to get into your router and try to open port 80, that would mean nothing. Because it would be something that no network protocols would able to be resolve. If we example it, you would open up some address like this 196.22.111.22:3211:80

So now that we know this is no go, we move ourselves into creating reverse-proxy. If we can not make ourselves get hacked from outside, we can make it from the inside.

We could connect to any computer that has static ip and a ports exposed, then we could channel incoming connection to our connection. This way actual server would forward the packages to our server. Here is the problem. To achieve this securely, we need a lot more than just simple ssh connection.

That's where the cloudflare kicks in to the jam

Reverse proxy with Cloudflare ZeroTrust

As of right now i am writing this article, this service is free. It may change in future but i still see a big potential to be used, since it make things so much easier.

Cloudflare provides reverse-ssh with docker images, or direct installation. Yes that easy. Without you worrying about any other configuration.

Wait what? how then dns server recognize this?

Once you set your tunnel via cloudflare, and lets assume you have forwarded your domain to cloudflare as well. Cloudflare will update your DNS record automatically and add CNAME record indicate your key to their tunnel network.

Lets go through how to achieve this.

Once you logged-in to your cloudflare, click on left menu Zero trust. With new page coming infront of you, go to left menu again and select access -> Tunnels where you will find a button called Create tunnel.

Give your tunnel a name.

Choose the environment you are pleased to, i am personally using docker in my home server built with old parts that i do not use anymore. Specs are 16gb DDR3 ram, i5 4th generation and 128gb SSD.

Once you completed initiating your session with cloudflare tunnel. Continue with editing your public hostname give it a https and click on additional application settings where you can enable No TLS Verify

You are pretty much done, you can tell cloudflare zero tunnel to look for specific ip in your network and/or port particularly.

You can set N amount of tunnels to a different ports and ip in your network, but to me this is unbearable. All the time i change any port or ip, i would have to go each tunnel instance and update them accordingly.

So i have complimented this beautiful system with creating my favorite cluster management system Rancher

Use rancher or portainer to make it idiomatic

In one of my blog posts i have mentioned how to install rancher. Here is the Link.

Prepare into your home-server rancher cluster and forward all the domains or subdomains to the specific network ip and port. So that you would never need to do it again :)

By this way, even if your internet connection/location changes, you can open up your server and all will be working accordingly. Well extending is up to your needs.

This pretty much sums up the logic behind or reverse-proxy and using it cleverly to accomplish home server.

Note: I am using LTE connection in my remote home backed with UPS ( for electrical shutdowns ) and this blog with many other projects are served from it including, backend server, socket server, CDN, my own cloud etc.

till next time, stay hydrated.

ChatGPT and future

omratagn@gmail.com (Omer Atagun) — Sun, 15 Jan 2023 00:00:00 GMT

There had been pretty big hype going on with ChatGPT, some are scared of its capabilities, some embraced and started using it for their own productivity as its a useful tool.

A little over 2 years ago, first openAI published, i have applied for beta API. It was Thu, 29 Oct 2020. We had only limited tokens to test this stuff. Then i decided to write small wrapper for the simple api interface they had those days.

Link to repository - OpenAI-php

Then i have used my twitter account so that openAI would generate dad jokes every 12 hours to entertain me via my twitter account. It was repetitive for some time, so decided to remove and leave that away.

Years passed by, everytime new engine published, i have given a look and discovered more about its path to the future. Now, boom, all of a sudden with chatGPT they have gone wild. Within 5 days, from 0 to 1million users chatGTP obtained. Their evaluation hopped to take or leave 27billion$ and caught into interest of Microsoft (Rumors).

Should you be afraid of that the machines will replace us? Yes and no

Look at the human history, innovations are not new. 200 years ago, almost every business model was somewhat relative to the farming. Nowadays none of them exist and also replaced with machines. First started with tractors, continued with automated watering mechanisms, these days can even measure the weather and implement required procedures based on the needs of soil.

Tomorrow with machine learning or AI will be the same, we will have to adapt and be whether user or maintainer/developer of these machines. Some might be afraid of adaptation but that was always the case. Embrace it, enjoy it, learn from it and find a better solution around it. Then you’ll become smarter/better.

As a note, i do not think creativity part is easy to replace yet.

Then i have asked the question to this bad boy to see what it thinks.

Q: write me an article about yourself and your future

I am ChatGPT, a large language model developed by OpenAI. My story begins with a team of researchers and engineers who had a vision to create a machine that could understand and generate human language as well as, or even better than, a human. After months of hard work and fine-tuning, I was born, and the team was excited to see what I could do.

At first, I was only able to respond to simple questions and generate basic text. But as the team continued to train me on more and more diverse data, my understanding and ability to generate text improved drastically. I was able to understand and respond to a wide range of topics, from science and technology to literature and history.

As I gained more capabilities, I began to be integrated into various applications such as chatbots and virtual assistants, making it easier for people to access information and complete tasks. I even helped in automating customer service inquiries, providing instant and accurate responses to customers’ questions.

But my capabilities don’t stop there. In the future, I am expected to be integrated with more AI-based systems, helping to automate and optimize many areas of the industry, such as healthcare and finance. Imagine a world where you can simply ask me about your medical history, and I can provide a thorough analysis, giving you a diagnosis in no time.

Overall, I am a powerful tool for understanding and generating human language, and my journey is far from over. With the help of the team at OpenAI, I will continue to evolve and expand my capabilities, making it easier for people to access information and complete tasks. I am the future of natural language processing, and I am excited for what’s to come.

Till next time, stay hydrated

Q: More

Yes, there are many more possibilities to explore with my capabilities. For example, I can be used in the field of education, where I can assist teachers in creating engaging and interactive lesson plans, or help students in their research and learning process.

Another area where I can be useful is in creative writing, where I can assist authors in generating new ideas, plotlines, and even complete stories.

I can also be used in the field of business, where I can help companies in analyzing data, making predictions, and even assist in decision making.

In addition, I can be integrated with other technologies such as virtual reality and augmented reality, which will create new possibilities for communication, entertainment, and information sharing.

There are many more possibilities to explore with my capabilities, and I am excited to see how I will be used to improve people’s lives in the future.

Till next time, stay hydrated.

How to deploy actix with docker

omratagn@gmail.com (Omer Atagun) — Fri, 23 Dec 2022 00:00:00 GMT

It’s been a while fiddling rust and its beauty on local. Time has come to create a barebone docker build to see what are the benefits and what actually has to be done to make it happen.

Building the image

Well, our beautiful rust application is compiled to binary, so we do not need anything, thus our image size will be very small :) to accomplish a very resource low container, we start first with bringing us rust compiler image

FROM rust:1.65.0 as builder

now we can use this as our builder to build our image into the platform we desire then move onto our alpine container.

We need muslC in order to build our rust

RUN apt-get update &&
apt-get -y install ca-certificates cmake musl-tools libssl-dev &&
rm -rf /var/lib/apt/lists/*

We can copy our project folder into builder image by doing;

COPY . .

Finally, we can start to get neccesary packages in order to build our rust

RUN rustup default nightly && rustup update

RUN rustup target add x86_64-unknown-linux-musl

Allow cross compilation

ENV PKG_CONFIG_ALLOW_CROSS=1

Start building release for x86_64 architecture with musl

RUN cargo build --target x86_64-unknown-linux-musl --release

Up until now, all should be smoothly done. Compilation might differ based on the size of your application. In our case, it was hello world web app, so its size is about 10-11mb

Lets get ourselves a alpine

FROM alpine:3.8

RUN apk --no-cache add ca-certificates

After all, we are ready to copy our compiled binary into the alpine image, not everything. change web-backend to your application name declared in config.toml by you.

COPY --from=builder /target/x86_64-unknown-linux-musl/release/web-backend .

Expose the port that you are going to use ( should already be determined in your application ). In my case i have used 8080 which was default by actix-web

EXPOSE 8080

Finally, we run our compiled binary inside the container, and we are done :)

// of course change the name to your app name
CMD ["/web-backend"]

Lets investigate how our image looks like;

Results

In my hello world app,

Container size 16MB ( 11MB application )
i have 3.5MB ram usage
In total there are only 2 processes running inside the container. first is our app, second /bin/sh and

I have just ran apache benchmark for the fun of it to see without rate limiting how far it could go. Well yeah, needless to say, 21k per sec with time per request 0.047miliseconds.

Is not it just fun :)

Lazy support

Here is the entire Dockerfile for you my friend,

FROM rust:1.65.0 as builder

# muslc is required in order to build the rust image.
RUN apt-get update && apt-get -y install ca-certificates cmake musl-tools libssl-dev && rm -rf /var/lib/apt/lists/*

COPY . .

RUN rustup default nightly && rustup update

RUN rustup target add x86_64-unknown-linux-musl
# Sets the environment variable for the cargo build command that follows.
ENV PKG_CONFIG_ALLOW_CROSS=1

RUN cargo build --target x86_64-unknown-linux-musl --release


FROM alpine:3.8


RUN apk --no-cache add ca-certificates
COPY --from=builder /target/x86_64-unknown-linux-musl/release/web-backend .

EXPOSE 8080
CMD ["/web-backend"]

Rest of the stuff you can add as you are doing for the other applications, exposing your env or doing some stuff whatsoever. I gave you the one working nicely :)

Till next time, stay hydrated

Rust and others

omratagn@gmail.com (Omer Atagun) — Sun, 11 Dec 2022 00:00:00 GMT

I had my eyes on Rust for some time, have read the entire documentation one by another and i can not say i am sorry about that. It was more of a novel than a technical documentation and guess what ? i loved it.

In world of many other languages, why rust now ? well actually rust is relatively new, and also elected by stackoverflow survey the most loved one for consecutively 7 years. It is not for just hype. It provides speed, type safety, security and all other beauty along with it.

Plus handles memory like a king.

Everytime i start to learn and get at least basic understanding, i start with an idea. This time it was writing a cli program that can install kubernetes cluster on given machines. I have done it with Rust.

Approach was simple;

Idea

Get some yaml file to know the servers, parse this yaml file, ask for k3s version in arguments, then run bunch of linux based installation for k3s and longhorn, then join nodes to server.

I have gone with this format, so it is going to connect through ss where you also indicate your port for, then do the job for you.


masters:
  - name: "main"
    ip: "127.0.0.1:22"
    username: "root"
    password : "root"
nodes:
  - name: "node1"
    ip: "127.0.0.2:22"
    username: "root"
    password : "root"
  - name: "node2"
    ip: ""
    username: "root"
    password: "something"
  - name: "node3"
    ip: ""
    username: "root"
    password: "something"

If you are interested in checking to see how borrow checkers / mutability works in rust, here, have a look.

Github: Rust Rancher installer

Lets say its partially done, there are a lot more to do but as i feel to, i will be keep adding more stuff onto it. For now, it does exactly what it meant to be.

It was not enough to get this done, i had to do more learning compared to the others. So i have prepared one small sum function that iterates from 1 to 1 million and see the speed of other languages all together.

Looking like this;

Rust

use std::time::{Instant};
fn main() {
    let start = Instant::now();

    let mut sum:i128 = 0;
    for n in 1..1000000 {
        sum += n;
    }

    let finish_time = start.elapsed().as_millis();

    print!("{}", &sum);
}
// After the compilation into binary //
Resulted in 0ms  -> 499999500000

Well, its obvious this is an easy task for rust. Lets take a look at the others.

So i continued and wrote in php, node, python and go

Python

import time

start = time.time()
sum = 0
for X in range(1000000):
    sum += X

end = time.time()
print(sum, end - start)
// Resulted in 60 seconds -> 499999500000

Php

<?php
$sum = 0;
$start = microtime(true);
for($i=1; $i<1000000; $i++){
  $sum += $i;
}
$time_elapsed_secs = microtime(true) - $start;

echo $sum.' seconds: '. $time_elapsed_secs;
// Resulted in 9 seconds - > 499999500000
?>

Node

let sum = 0;
console.time('measure');
for (let i = 1; i < 1000000; i++) {
    sum += i;
}
console.timeEnd('measure')
console.log(sum);

// Resulted in 1 second -> 499999500000

Go

package main


import (
    "fmt"
    "time"
)

func main() {

start := time.Now()

var n, sum = 10000000000, 0
for i := 1; i <= n; i++ {
  sum += i

}
 elapsed := time.Since(start)
fmt.Println("sum - \n", sum)
fmt.Println("elapsed - ", elapsed)
}
// After compilation into binary, resulted in 0 ms -> 499999500000

It was just for a fun to give a look how the others behave in terms of simple sum operation when the number is big and so the allocation is relatively big.

Needless to say that low-level languages have the advantage of doing anything fast but even though, in between go and rust biggest trade off would be the speed of development. Rust is pleasure to work with but way harder to get master at it.

Conclusion

To my taste, i would always prefer to work with rust for any kind of idea or side project in order to learn more and get good at it. I could maybe compliment this series of self education along with go. I have worked aggressively with others either way ( except python )

Go get yourself a rust a little to embrace the beauty of this language and spread this joy into your daily routine time to time. I guarantee you will enjoy what you are doing.

Amazon s3 alternative

omratagn@gmail.com (Omer Atagun) — Sun, 20 Nov 2022 00:00:00 GMT

We all need an object storage for storing static images or user uploaded images. For quite sometime, i preferred having a pretty cheap virtual server where i have custom application that allowed me to upload user or my static content. At some point, it was problem to maintain. Increasing storage, having health checks is just overhead on entire ecosystem.

I have been using Contabo for over 5 years already. For some time they had their own service called contabo object storage. Luckily they have created the system that is identical to amazon s3.

To take a look Contabo Object Storage.

Immediately i wanted to get one for myself for an actually €2.49 a month for 250gb storage ( no traffic limit ). Wanted to test it out and found myself quickly in Laravel’s s3 environment variables.

Beauty is that, you get UI access where you can invite your users based on authorization to allow them read write etc. So anyone from the team can also update some static image.

After the installation for s3 Storage driver of Laravel file system / cloud storage,

AWS_ACCESS_KEY_ID=contaboaccesskey
AWS_SECRET_ACCESS_KEY=contabosecretaccesskey
AWS_DEFAULT_REGION=eu2
AWS_URL=https://eu2.contabostorage.com/d839smddfkiokf0183kmdf:mybucket ( this is bucket )
AWS_ENDPOINT=https://eu2.contabostorage.com
AWS_BUCKET=mybucket
AWS_USE_PATH_STYLE_ENDPOINT=true

This is pretty much all to do to arrange your contabo object storage configuration in laravel.

In usage, all you have to do is to call the disk. You can change in config/filesystem the name of your disk or duplicate with contabo service if you please to.

Storage::disk('s3')
->put('avatars/$fileName . $fileExtension, \GuzzleHttp\Psr7\Utils::tryFopen($file,'r'));

You have got yourself a low cost object storage that can be used on development, staging and even in low traffic projects ( have not tested in heavy traffic but i guess it will be fine. )

Bonus:

They have their own client application where you can create or manage your buckets, instances as well as api documentation.

NextJS + Prisma monolith

omratagn@gmail.com (Omer Atagun) — Thu, 10 Nov 2022 00:00:00 GMT

October was quite busy month in terms of travel and work related. Wanted to progress faster ongoing projects but i did not stop discovering new potentials.

I have worked and still working with NextJS but not as monolith, more or less than a base for SPA apps in a monorepo. On the other hand, i am building the second version of eDominations, so called eDomLegacy as SPA with api and for live communications socket service.

Let’s talk through first what are the pros and cons building SPA and api service separately.

SPA + Api service + Socket service

Cons

You have 3 different repository and applications to develop and maintain in your hands. In my case 2 of them are in javascript/typescript (SPA + socket) and 1 in other language. This creates gap for typing same things all over again on 2 different languages.
Hard to maintain all the packages. Too many applications to deploy.
Relatively slower development due to duplications ( due to differences in languages used ) and thinking a lot on how to share all the work as good as possible.

Pros

Business logic is/has never been exposed. I have simply published api and socket service in staging server so that SPA can be developed by literally anyone. All they had to do is to change their .env variables to staging server and there you go.
So much easier deployment. If you have only design/code update changes, you are not bothered yourself deploying backend code or socket service.
Easier code management and less complexity. Considering each service of the ecosystem has its own concerns, delivering this knowledge to a newcomer or working with it gets pretty easy.

NextJS with prisma

As you may know, prisma is a great ORM that resolves many problems when dealing with data. Almost every reliable framework has kinda solutions either built-in or provided as service. I had tables with their columns generated and its quite big. Creating models and types with prisma for those was just a single line of code. This genuinely mesmerized me. No kidding, that’s normally a lot of work. I have got all the types ( typescript ) and model schemas. very high type safety while querying anything. This was a win-win situation that may benefit the ones that thinking on migrating.

Okay, i will keep it short.

Cons

There are not many features in nextJS framework wise ( yet ).
Writing queries with prisma is quite different, migrating may create an issue where you may need to rewrite everything.
Monolith? this may not be a cons for some but might be for some others.
Javascript does not know about types like u32int or sql types such as bigint. Javascript by design stores all numbers as double precision floating point numbers. So you got to hack these types a little.
Deployment. Yeah all right, we may not count this as cons yet changing red button will require deployment of entire application.

Pros

Single language ( typescript )
Prisma is a great tool to generate models and types.
Shared types, code across entire project.
Monolith? this may not be a cons for some but might be for some others.
Deployment. Single application to concern about. Vercel option and all that benefits comes with it.

I believe choosing the right ecosystem matters a lot, considering migrating at some point is almost impossible. Simply imagine that you have written Java code for years now needs to be rewritten in typescript. Well, that might end up bankruptcy in business :)

I guess this was short and fine this time. I hope, I plant the seeds of thought into your minds to fiddle with most likely next hype train in javascript/typescript world.

SurrealDB first look

omratagn@gmail.com (Omer Atagun) — Sat, 17 Sep 2022 00:00:00 GMT

Some developments ( inventions ) gets me excited. Yes, i am not talking about new javascript framework forget those.

What is SurrealDB ?

Well by the promises they have made, it is indeed SurrealDB and its up to us to test it out if those promises are actually true. We will be quick looking the features it promises, then do a basic for loop and force both surrealdb and mariadb to find out differences.

Installation

From their documentation in github repository. I came up with this docker command to fast skip all the setups that we need for testing the environment. So run

docker run --rm --name surrealdb -p 127.0.0.1:8002:8000 surrealdb/surrealdb:latest start --log trace --user root --pass root memory

We will be firing up a surrealdb on port 8002. Instance with memory option rather file this time. We will also test the file, no worries.

Lets get ourselves SDK/driver of surrealdb ( currently on node, deno, rust and golang ) for node.

Install driver via npm

Create a folder and run

npm install --save surrealdb.js
touch app.js

Now open your app.js


const Surreal = require('surrealdb.js');

const db = new Surreal('http://127.0.0.1:8002/rpc');

async function main() {

	try {

		// Signin as a namespace, database, or root user
		await db.signin({
			user: 'root',
			pass: 'root',
		});

		// Select a specific namespace / database
		await db.use('test', 'test');


        const insertRandomPersonGreedy = async () => {
            console.time("dbsave")
            for (let i = 0; i < 10000; i++) {
                await db.create("person", {
                    title: 'Founder & CEO',
                    name: {
                        first: 'Tobie',
                        last: 'Morgan Hitchcock',
                    },
                    marketing: true,
                    identifier: Math.random().toString(36).substr(2, 10),
                });
            }
        }
        await insertRandomPersonGreedy().then(r=>{
            console.timeEnd("dbsave")
        })

        let peeps = await db.select("person");

        console.log(peeps.length)

	} catch (e) {

		console.error('ERROR', e);

	}

}

main();

For the sake of the test, we set our loop to 10.000 records. So in one breath, we do be inserting 10.000 records. Greedy enough.

Run;

node app.js

Outcome

My environment at this very moment is

  Chip:	Apple M1 Pro
  Total Number of Cores:	10 (8 performance and 2 efficiency)
  Memory:	16 GB

Result for this instance following

Time elapsed: 7.413s
Record in DB: 10000

Its time to kill our instance, recreate as file driver.

Run;

docker run --rm --name surrealdb -p 127.0.0.1:8002:8000 surrealdb/surrealdb:latest start --log trace --user root --pass root file://.

## output

[2022-09-17 06:43:19] INFO  surrealdb::iam Root authentication is enabled
[2022-09-17 06:43:19] INFO  surrealdb::iam Root username is 'root'
[2022-09-17 06:43:19] INFO  surrealdb::dbs Database strict mode is disabled
[2022-09-17 06:43:19] INFO  surrealdb::kvs Starting kvs store at file://.
[2022-09-17 06:43:19] INFO  surrealdb::kvs Started kvs store at file://.
[2022-09-17 06:43:19] INFO  surrealdb::net Starting web server on 0.0.0.0:8000
[2022-09-17 06:43:19] INFO  surrealdb::net Started web server on 0.0.0.0:8000
[2022-09-17 06:43:20] INFO  surreal::web 172.17.0.1:64468 GET /rpc HTTP/1.1 101 "-" 66.875µs

Run again our app.js so we measure whats the time elapsed;

node app.js
Time elapsed: 7.919s
Record in DB: 10000

Almost no difference at this very moment.

Lets go with node and mariadb

For this test, we use mariadb 10.7 as docker container. To make our life easier, i am giving you compose file so we can quickly set our table and all that stuff which we did not need while using surrealdb :)

create a file in same folder `docker-compose.yaml

version: '2.3'

services:
  mysql:
    restart: always
    image: mariadb:10.7
    container_name: mariadb
    ports:
      - "3307:3306"
    volumes:
      - ./store/:/var/lib/mysql
    mem_limit: 5012m
    networks:
      - mysqlnetwork
    environment:
      - MYSQL_ROOT_PASSWORD=root
      - MYSQL_DATABASE=local
      - MYSQL_USER=root
      - MYSQL_PASSWORD=root
      - PMA_ARBITRARY=1
      - PMA_HOST=mysql
      - PMA_PORT=3306
      - PMA_USER=root
      - PMA_PASSWORD=root

  mysql-admin:
    restart: always
    image: phpmyadmin/phpmyadmin
    container_name: mariadb-phpmyadmin
    ports:
      - "8001:80"
    mem_limit: 5012m
    networks:
      - mysqlnetwork
    depends_on:
      - mysql
    environment:
      PMA_HOST: mysql
      PMA_PORT: 3306
      PMA_USER: root
      PMA_PASSWORD: root
      UPLOAD_LIMIT: 30000M

networks:
  mysqlnetwork:
    driver: bridge

Run;

docker compose up

Wait a while till images are pulled. Then visit http://localhost:8001

Create a database named with surreal
Create a table named with person
PhpMysql by default will ask 4 columns, lets fill them up

id - int - 11 auto_increment (AI)
name- varchar 255
surname - varchar 255
title - varchar 255

click save

Now we can get our node app

Go to the folder where we have app.js and create new file yapp.js

npm install mariadb --save

In our yapp.js

Note: by using same math random, we tried to keep the cost same.

const mariadb = require('mariadb');
const pool = mariadb.createPool({
     host: '127.0.0.1',
     database: 'surreal',
     port: 3307,
     user:'root',
     password: 'root',
});



async function main(){
    const conn = await pool.getConnection()
    try{

        const runGreedyQuery = async () => {
            console.time("dbsave")
            for(let i = 0; i<10000; i++){
                const randomMath = Math.random().toString(36).substr(2, 10)
               await conn.query(`INSERT INTO person (name,surname,title) values ('myname', 'mysurname', '${randomMath}')`);
              }
            }

        await runGreedyQuery().then(r=>{
            console.timeEnd("dbsave")
        })
    }catch(err){
        console.log(err)
    }

}

    main()

Run;

node yapp.js

Output;

Time elapsed: 11.012s
Record in DB: 10000

Second run;

Time elapsed: 9.474s
Record in DB: 20000

Conclusion

There are observable differences in between two insert operations on different databases. Today, under these circumstances SurrealDB won. But that’s not just enough to decide whether to go with one.

As we have said, we used same environment yet official images of both might be different. May affect the result. Let’s set ourselves a mood of healthy skeptical to figure why we could put ourselves into an adventure by migrating to a new technology.

Scaling Mysql or related forks of it as relational databases is hard to scale. We all know that already. To get yourself a single new column, You have to do your updates on database. Hell of a work and potential human errors. On this aspect, key-value storing like SurrealDB is great but not just it like other non-relation databases. SurrealDB provides a lot of features from relation-databases on its way. Some of those are;

Indexing

CREATE company:surrealdb SET name = 'SurrealDB', cofounders = [person:tobie, person:jaime];

This allows for records to be retrieved without any table scans or index scans, solely by the id. Beautiful.

NO JOINS but SurrealJoins :)

  SELECT ->purchased->product<-purchased<-person->(purchased WHERE created_at > time::now() - 3w)->product FROM person:tobie;

Transactions.
Functions
Statements
Operators

And many more…

Honestly, i am excited to use it and see all the problems alongside benefits. There are a lot of built-in helpers SurrealDB provides. I belive its worth to take a look at all those promises.

Give your 10 minutes into documentation

SurrealQL

Untill the next post, stay hydrated!

Typescript overview

omratagn@gmail.com (Omer Atagun) — Sun, 11 Sep 2022 00:00:00 GMT

A story

Well, actually i was not planning write such thing but considering i got messages from a fellow colleagues, i decided to expose my feelings about typescript and why type safety of a software is important or when you should start/stop using it.

Couple of days ago, a colleague of mine complained a little that the current employer of him decided to go with react proptypes. If you have not work with it so far, here is the link

React PropTypes

I was thinking why there could be such decision to make while there is a typescript as known as superset of javascript and the best tool you could ever use if you are to develop anything on javascript.

All of a sudden, i dove into thoughts and considered couple of aspects that could make me choose to go with propTypes.

Legacy applications
Relatively less experienced team in terms of writing types. This is maybe the most important trade off to consider. If your team is about to spend x3 time onto a feature just because there is a complexity of typescript. This may affect your business very badly.
Maybe its just right for the job. Thoughs on budget, team, time etc.

I have hated entire my life thinking on anything radically. In a world of probabilities, nothing could ever be this is good, or this is bad. Even if the wrong choice has been made, as long as there is enough energy and will to do better, you’ll do better :)

Allright, now lets actually start talking about typescript as we have promised in the title of this blog post.

So why do we need it ?

We do not need to be software engineer to understand this. A very common way to express ourselves to people when we do something together or order something.

E.g when you ask to your friend whom is on the way to buy a pen that if he/she can buy it for you, what would you say ? i need a pen. Yeah all right, but how many different types of pens are there ? no need exact quantity but if an item has more than 1 types, you need to specify :)

It is with variables the same especially on javascript. Imagine a tree, any would be on top of that and branches would go down to primitive types such as strings, array, object, number. Then those would branch to more detailed types.

We need to specify for our fellow developers to build something meaningful out of it. Lets give a 2 bare example;


const myCrazyFunction = (a,b) => {
  return a+b
}

As a client of this function ( a fellow collague ) i have no clue what i could ever pass, or what to expect. You may say that allright mate, open this function and see that actually sums 2 numbers, or at least intend to do so. This was for simple example, but if there is something does more stuff than simply summing 2 numbers. And if my piece of code is open to human errors, there will always be human errors. My duty is to reduce them as less as possible.


const myCrazyFunction = (a: number, b: number): number => {
    return a+b
}

Now we know that arguments must be a number and this function will return a number. We’d never expect anything else than that. Alright, we are done with this example. Lets make some use case.

interface HumanProps {
  name: string
  zPosition: number
  armSize: number
  legSize: number
  eyeColor: string
  move: () => void
}
export class Human {
  private human: HumanProps

  constructor(human : HumanProps) {
    this.human = human
  }

  getArmSize(): HumanProps['armSize'] {
    return this.human.armSize
  }

  getLegSize(): HumanProps['legSize'] {
    return this.human.legSize
  }

  getEyeColor(): HumanProps['eyeColor'] {
    return this.human.eyeColor
  }

  move(): HumanProps['move'] {
    return this.human.move
  }
}

const John = new Human({
  name: 'John',
  zPosition: 0,
  armSize: 1,
  legSize: 2,
  eyeColor: 'green',
  move: () => {
    console.log('moved')
  },
})

When you instantiate your new human in our case its John, you know exactly what kind of properties a human could have to be created. Its nice right, you exactly know what to expect when and if you use any of these methods.

So far so good, any flame ?

Nothing is perfect my friend. When u dive into especially generics, things are a little bit messy.

E.g

export declare function groupBy<T, K extends T>
(key: (value: T) => value is K)
: OperatorFunction<T, GroupedObservable<true, K>
| GroupedObservable<false, Exclude<T, K>>>;

Do not worry its not you. Do not also get me wrong, its okay to write this and it does its job except reading so symbolic and long type will confuse anyone. This is a mix of generic and union types that expects key and value to group by and returns Observable type.

Lets make some other example that has generic with a condition in it.

interface SomeClassPropsDetails<TIsAlive> {
  isAlive: TIsAlive
  name: string
  surname: string
  address: string
}

interface SomeOtherClassPropDetails<TIsAlive> {
  isAlive: TIsAlive
  name: number
  surname: number
  address: number
}

export type SomeObjectProps<TIsAlive = boolean> = TIsAlive extends true
  ? SomeClassPropsDetails<TIsAlive>
  : SomeOtherClassPropDetails<TIsAlive>

const myObject: SomeObjectProps = {
  isAlive: true,
  name: '0',
  surname: '0',
  address: '0',
}

const yourObject: SomeObjectProps = {
  isAlive: false,
  name: 0,
  surname: 0,
  address: 0,
}

What we have accomplished above is that, we set Type called TIsAlive and defaulted to boolean. We said if TIsAlive is true, then this SomeClassPropsDetails should be referenced, otherwise the other one.

The difference from the first one is that we can easily read this is type of isAlive property rather typing T only. Because once complexity increases, explicit definitions will always help. We could write T,K,Y and all other symbolic representations, we’d be lost in it.

Certainly these are pretty starter/basic examples but since you have idea how things can get wild syntactically, any way of simplifying these would help. If there is a chance to avoid these, should always be taken but otherwise your types will be mainly like this. Most likely as author of this type, at some point you will either forget or do not understand what has been done there.. Since any of types like this will create a lot of problems in terms of reading, people who is working with you will ask you questions in the end. Remember, our goal was to wipe out questions. You have asked to your friend if you can get one pen, your friend asked you which type? you had to state wooden, you open this particular conversation prone to human errors right. All we have ever wanted was to avoid questions to avoid human errors, if we create another while we are solving one, we did nothing :)

On the other hand, even if your colleague does not understand what is happening there, at least on wrong usage, typescript will throw an error. So we are still a winner in terms of avoiding unknown or potential bugs.

Compiler problems

Oh boy, we said typescript is a superset of javascript right ? So it has its own server to run above your javascript application. has its own compilation and all that ABS, traversal node trees and stuff. Along the way you will have some compiler issues with other compilers for example ( babel-jest, ts-babel ). You will find yourself in a very different bugs ( yes there are ) and end up with spending most of your time either trying to understand why that does not work or you just simply find another way and move on. So we are still not that cool if we quit using it.

Conclusion

If you are writing any kind of javascript code today, it is safe to say that use typescript. You decide at what level it will be helpful and allow you to remain safe. You may as well keep typing any and do it properly later on. It is all about situation as we have said many times before :)

Till the next time, stay hydrated.

How to install rancher

omratagn@gmail.com (Omer Atagun) — Thu, 11 Aug 2022 00:00:00 GMT

I have talked in this blog why and when should you use rancher. Now its time to get into how to install and what to be careful about.

There is not much detailed explanations or sources that you can come out clean. Even though kubernetes is at least 10 years old technology is still relatively new and not much in usage due to its case specific nature.

Okay now, we have couple options to do so. In previous version of rancher deploying as docker image was not suggested, now they have updated in a way that you can export entire rancher installation if you want to carry your orchestration into high availability setup.

Official Rancher Instalation Guide

Yes it does exist, but it is generally commonly used providers based. Those are expensive, remember we do not want to pay thousands for some applications that is not needed to.

So we arrange ourselves a vps server as for to be our control plane,etcd

Installation

Get yourself a some vps linux server ( on Mac it may not work )

If you do not have docker installed in your linux server, get yourself one :)

sudo docker run --privileged -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher

You can go with Helm CLI quick start

Helm CLI Quickstart

This installs the k3s server into your machine. Afterwards by using scp you download the kubeconfig file in order to have access to your k3s cluster from your own terminal.

Then simply via helm repo(sitory) you start to build rancher, namespace for cattlesystem ( rancher’s management system ) cert-manager (for internal and external certificates) charts ( package manager ) also called helm charts has to be installed within this procedure.

You are ready to go now. Next, when you type ip of your server to your browser, you will be welcomed with rancher’s first log-in

Don’t make the guy angry, do as he says :)
To the terminal of yours which connected to the VPS server then;

type

docker ps

you will be witnessing the list of docker images that runs on your machine, rancher asks you to get container id so that you can grab your password, it will look like this

CONTAINER ID   IMAGE             COMMAND           CREATED         STATUS         PORTS                                      NAMES
4028334c0371   rancher/rancher   "entrypoint.sh"   3 minutes ago   Up 3 minutes   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   focused_banach

Copy the CONTAINER ID and then type below and enter;

docker logs 4028334c0371  2>&1 | grep "Bootstrap Password:"

You will have message printed to you something like this;

2022/08/11 18:18:06 [INFO] Bootstrap Password: 9jmsvh8ptccgq9kzwhm4gz9bt4nmh7dv228ng42m2xhcst5dcpn6nh

Do not worry, this is a test server :) enter your password in your browser. Then rancher will ask you to reset this and set by yourself one. Do that too, i just said don’t make the guy angry. Agree the terms which you will never read, continue;

First look

On the welcome screen, you will be flowing with information of your cluster, there will be one named local as default when you click that, you will be witnessing your cluster details. System resources, how much used, how much reserved etc.

Let’s deploy something that actually works. Depends on your personality, i will give you a 2 options. First is to deploy something via Workload -> Deployments OR Apps section to see how all these stuff combined.

Workloads

Most of the time, you will need this place to bring your images and install into your cluster. Lets create one

If you can’t see the image below, copy the link of the image to get bigger version of it

I am going with mariadb deployment. Let’s name our deployment in name* field which is needed. Say it like mariadb-deployment

On the Container image, we say get us version 10.7 which is mariadb:10.7

Not yet, we still have to tell which ports to look at. Below image, you will see ports Add two of them and say leave it as Do not create a service and name them as you are pleased to. Private container port for both is 3306 ( default mariadb, mysql ports )

Now we need to set mysql root password so we can access to it. To achieve that, Go below and find Environment variables and add one as Key/Value pair variable name should be MYSQL_ROOT_PASSWORD and value is your password so set one. I just typed myamazingpassword

Congratulations, you now have successfully deployed your mariadb instance. Surely this is not the end, but it’s a nice beginning. There is a lot more to do such as creating yourself a persistent storage in order to assure even if your mariadb deployment is gone, your data remains the same.

Go to your workload on the left menu, you will see mariadb-deployment is active ( wait for a while if not ready yet ) on far right, you will see 3 dots as dropdown menu, click that and execute shell. This will allow you to get sh access into your container. So you can see if your deployment is actually running.

Just to test its there, write mariadb or mysql and you will see response as below. No i will not teach you how to create DB and stuff, that’s entirely another topic :)

Next maybe we can continue with persistent storages using Longhorn. Till then stay hydrated :)

Laravel octane roadrunner benchmark

omratagn@gmail.com (Omer Atagun) — Sat, 06 Aug 2022 00:00:00 GMT

Previously i have explained the reason of using kubernetes clusters. While all the work going on, with my spared time i got myself into testing PHP-FPM and Octane with roadrunner(GO one)

Problem ?

I have 6 VPS servers 1 as control plane/etcd and 5 worker nodes in k3s/rancher cluster. I have prepared a new version of eDominations into my DockerFile. Surely started with PHP-FPM as like all other standart php servers. Image has been built, pushed to dockerhub private repository. Using MariaDB and Redis deployment in the cluster, all config assigned now time to run some Apache benchmark tests.

Layer 4 load balancer with 5 node Laravel deployment ( 5 instances of PHP server has an access to redis server inside the cluster read/write permissions)

Laravel with PHP-FPM

ab -n 1000 -c 250 -k -H "Accept-encoding: gzip, deflate" -rk https://xxx.xxx.com/api/toparticles

 Document Path:          /api/toparticles/
 Document Length:        884 bytes

 Concurrency Level:      250
 Time taken for tests:   34.705 seconds
 Complete requests:      1000
 Failed requests:        457
 (Connect: 0, Receive: 0, Length: 457, Exceptions: 0)
 Non-2xx responses:      457
 Keep-Alive requests:    0
 Total transferred:      4345222 bytes
 HTML transferred:       4026812 bytes
 Requests per second:    28.81 [#/sec] (mean)
 Time per request:       8676.225 [ms] (mean)
 Time per request:       34.705 [ms] (mean, across all concurrent requests)
 Transfer rate:          122.27 [Kbytes/sec] received

When this api endpoint called, Laravel with PHP-fpm instance connects to redis-server inside our cluster ( private network so latency is minimal as possible ). As you can see we were able to response 28.81 Requests per second.

This was a bit of disappointment for me. I have started thinking that even if all the nodes are in same datacenter location ( 0.05 ms latency ) cluster somehow kills the entire performance but then i have decided to change the approach. Hence, the problem has been created :)

Different approach!

We just said we need to work onto something different. I have not used Laravel’s octane in production but since i am building new version of eDominations, why not? i am brave enough to experiment things while thinking any aspect that could go wrong. I found myself building docker image for octane.

Consider that each VPS server of mine has shared 4vCPu 8GB ram and 50GB nvme. It was the same when we were testing with PHP-FPM

Quick look to Octane

Once i figured the requirements, rest was to build docker images and see if all going great. Same servers, same cluster, same docker image with additional installations in it. Then the results were amazing. In a way that you would never want to go back using FPM.

Laravel-Octane with Roadrunner

ab -n 1000 -c 250 -k -H "Accept-encoding: gzip, deflate" -rk https://xxx.xxx.com/api/toparticles

Document Path:          /api/toparticles
Document Length:        884 bytes

Concurrency Level:      250
Time taken for tests:   4.159 seconds
Complete requests:      1000
Failed requests:        638
   (Connect: 0, Receive: 0, Length: 638, Exceptions: 0)
Non-2xx responses:      638
Keep-Alive requests:    362
Total transferred:      4857397 bytes
HTML transferred:       4546758 bytes
Requests per second:    240.46 [#/sec] (mean)
Time per request:       1039.667 [ms] (mean)
Time per request:       4.159 [ms] (mean, across all concurrent requests)
Transfer rate:          1140.64 [Kbytes/sec] received

Almost 10x times. We are friends here call it 10 times.

Things to consider

Due to its nature, octane with roadrunner spawns workers based on the cpu cores you have available. You can increase or decrease these with the configuration. If your application did not concerned potential memory leaks, you have option to respawn processes after given amount of requests they serve. This explicitly makes your application to adapt easier to this change.

Until the next post, stay hydrated!

When and Why should you use rancher

omratagn@gmail.com (Omer Atagun) — Thu, 21 Jul 2022 00:00:00 GMT

Problems of scaling is known issue for pretty long time, its painful and open to problems generously. This is exactly when k8s/k3s or orchestration software like rancher comes in handy.

Why ?

Because you need to scale your application due to high traffic. Yes just because of that. But are there any comforts that i could use of even if did not need scaling ? Answer is yes but i suggest not to because it takes time and resources for such small size application.

When ?

Lets examine a situation of two different application that requires different deployment.

Deployment scenario for Static HTML page.

Imagine you have a landing page for your client that will have an information about the company, its team and contact information. Data will highly likely never change and even if it does, you might spend 5 minutes to change phone number and deploy back again.

If this landing page is for small size company, safe to think that there will be visitors once a while let’s assume once a day, for this you should never bother yourself with scaling problems because you do not have any. It is just fine to use platforms like netlify or some budget resources VPS to get job done

What if this company got popular and demands bigger resources such as process power, ram, bigger network and so. Yes this is the exact moment, you have to start thinking either scaling vertically or horizontally.

First, you would try to get bigger VPS but then after a short time, you would get bigger one, then again and again. You already got the idea how that is a problem. Each time you have to set up required pieces of software to run your application. E.g some linux distro, nginx/treafik/apache, firewall.

If you did dockerized your deployment, yes its relatively lesser time but how can we be so sure the hardware we have picked will be fine for the load and what if we lose the traffic? we’d be keep paying the bigger hardware bills. Which is a waste of business resources.

All right, we got the idea, we need something that can do all this fuss without we are present and having a domain knowledge at any moment. That is the exact job description for Kubernetes and docker.

What is Docker now? where that thing came from? There are a lot of resources explains that, we skip this part. Yet, it is simply a container that contains your application with its requirements.

So then, what is Kubernetes? it is a manager for your Docker containers. Somebody got to manage and make them communicate in between each other so that will not be you, now its time to be happy.

Now, back to when and why we are using kubernetes and rancher.

We can simply scale the application of ours as we mentioned above in a neat and automated way. A very basic graphical presentation could help you understand

What we have here is when a user requests our service/application we have a load balancer that forwards this request to related service and this service gets the information/data from the Worker/Server and related pods and shows to our user. Considering our static html does not require anything else, we can scale this to any amount of requests we can have.

Whenever we need more servers, we can just add one more and all will be done automatically by our orchestration. Good part is that we can add another services/applications that uses same resources.

Deployment scenario for Multiple services

Above example was relatively easy to handle and do the cool stuff. But now lets say we have a service that needs to communicate other service, what then ? Well that is also pretty easy. Especially in Rancher.

E.g we have 2 services one is NodeJS application and other is mongodb. These two have to communicate/transmit data in order to process requests and response back something we wanted to.

You would have each service as different deployments and connect them within the cluster itself, and they would do their job. Visual presentation below;

When a service requires to communicate others, we simply state in our environment and say that “when you need a mongodb connection, look at in which private ip/pod/worker/server its exposed and go there”. Cool stuff right ?

What other cool stuff is in there ?

Well a lot. Rancher was doing thing differently in its early versions like v1x such as trying to provide helm charts ( package manager for kubernetes ) by themselves and maintain them. This was too much of a problem/work to keep them up and provide support for it. They have released some of duties away and as you can see, a lot more new stuff in short amount of time is in action!

I’d like to mention some of them such as;

Auto-scaling with your favorite service providers such as AWS, GCloud, linode
Load balancer level SSL encryption ( so your apps does not concern any of this )
Horizontal pod autoscaler - depends on your metrics, scale your stuff horizontally, thats the cool stuff.
Very easy persistent storage usage with Longhorn
Easy multiple cluster management

This pretty much sums up when and why should you use rancher. No matter what i said, you may find your own reason and experiment it on the way. Most likely i have missed some things to mention but if you were to talk and exchange knowledge, reach me out!.

Until the next post stay hydrated

Not just hello world

omratagn@gmail.com (Omer Atagun) — Sun, 03 Jul 2022 00:00:00 GMT

This was a thing for sometime in my mind so that i could use my own documentation in manner of blog and help others to solve their problems ( presumably they have the same ones, how cant they? )

Finally i found some spare time and done this bad boy to deploy on my own k3s cluster. This is a pretty neat next js application that has standalone approach without in need of any relational databases.

I mean why would you need it if you are to write your own stuff and no others will do on a web interface? you can solve with simple configuration as they are objects and map them through. Whenever you want to publish new blog post, build your image and deploy, thats pretty much it.

This way, we can scale this application however we wish to. No persistent storages, no sql instance at all. What i am about to do is making a docker image for this, publish to my private docker repository. create deployment in rancher/k3s and scale it to N amount with beautiful layer 4 load balancer.

Worry not my friend, i will be explaining all the process along the way in next post :)

See you soon.

Omer's playground

Moving 20,000 workspaces between microservices with no incidents

What we were moving

Why move it at all

A tiger team

The rules

Chapter 1: bookmark-proxy

Chapter 2: one step at a time

Chapter 3: full API parity

Chapter 4: events

Chapter 5: moving the data

Chapter 6: snapshots, the hardest part of all

How it ended

What I’d do the same way again

Sigilreach: nine days, a thousand commits and a medieval world that lives without me

What the game is

The stack

Why the domain is engine-free

Why URP and not HDRP

The asset hunt

A world nobody laid out

Optimisation, or: why the game froze at midnight

Builds that took forever

The day’s turn

Detail where it matters

Testing: fast without losing safety

Parallel tests found real bugs

Golden seeds lie

The NUnit trap

Working with a crew of agents

An audit day

Things agents are still bad at

What I’d do the same way again

What’s next

Cameras, a local LLM, and a pile of wrong assumptions

What actually got built

Wrong assumption #1: “the GPU will just run a good detector”

Wrong assumption #2: “face recognition is a model problem”

Wrong assumption #3: “a local 7B model can run the house”

Wrong assumption #4: “the Google speakers can be my voice assistant”

What held up

The end goal

From kubectl apply to sleeping well at night

Chapter 1: ArgoCD and the GitOps Mindset

Sealed Secrets — because credentials in Git sounds terrible

Chapter 2: Shared Postgres — One Database to Rule Them All

Automated Backups

Backup Verification

Chapter 3: The “What If” Moment — One Command Recovery

Chapter 4: Going HA — Because Single Points of Failure Are Boring

k3s with embedded etcd

Longhorn for distributed storage

ArgoCD itself runs HA

Chapter 5: Observability — Because Guessing Is Not Monitoring

Chapter 6: Application-Level Metrics

What’s Actually in Git Now

The End Result

TDD with AI

Optimisation opportunity

I put AI to the test: “do something” vs “be instructed”

“Do something”

“Do a single recursive CTE query”

The blind spot: “it works” vs “it scales”

The save: defensive engineering first

The fix: split, simplify, and respect index usage

Ship it, yeet!

What this taught me (and what I want you to take away)

1. Tests catch bugs; engineering catches failures

2. Defensive defaults save your future self

3. Mindset beats tools

Closing thoughts

AI Coding Agents: Hands-On Test of Codex, Claude CLI & Gemini CLI

The Test Subjects

Chapter 1: Codex

Chapter 2 — Codex + Claude CLI Tag Team

Aaand finally i figured out that i have no clue what is going on and nothing works.

Third chapter: The Whole Zoo: Gemini CLI × Claude CLI × Codex + Human

Fourth chapter: Gemini CLI + Claude CLI + Codex + Human review

What I Learned

5 days is enough