Moving 20,000 workspaces between microservices with no incidents

This project didn’t start with writing code. It started with two to three months of reading other people’s: drawing lines between services, and finding every place where one thing quietly depended on another.

By the time we pitched it to the executives we had an estimated deadline, t-shirt sized efforts per team, and a clear list of what the project needed to succeed. That list mattered more than the estimate, and the first thing on it was people. This was not a cross-team effort, it was a cross-tribe one. More than six teams were involved: answering questions, sharing knowledge, making small adjustments and reviewing changes. The actual work, including changes in other teams’ code, was done by one small dedicated team.

A migration is not a data problem. It’s an ownership problem that happens to involve data.

This is the story of how we moved bookmarks home.

What we were moving

The entity was what we call a bookmark. It lived in a service we didn’t own. A bookmark is not a flat row. It has a type property, diagram or report, that says what kind of bookmark it is, and child entities for the diagram or report itself. Those child entities have working copies, which are another layer of the problem on their own.

So “move the bookmarks” really meant: move a tree of entities, with drafts hanging off its branches, without losing a single relationship. And not all into one place. Diagrams had to go to the diagrams service, reports to the reports service.

The scale: around 20,000 workspaces across 8 regions, millions of diagrams and gigabytes of data.

Why move it at all

Bookmarks could have lived in that service just fine. Technically, nothing was broken.

Organizationally, it was a bottleneck. When the data your service is supposed to own lives somewhere else, every change turns into overhead: meetings, requests to another team, waiting on their roadmap, and maintenance on code that isn’t yours. None of that shows up in an architecture diagram, but all of it shows up in how fast you ship.

We wanted the data to live with the team that owns the domain. That was the whole reason, and it was a good enough one to justify everything that came after.

A tiger team

What I asked the executives for was a team of four. Five with me.

The reason for four was simple. Four people split into two micro teams that can work on different parts of the problem in parallel. And once the pieces of the puzzle start coming together, the knowledge of both halves combines into one picture. That combined knowledge is what gets you over the line.

We didn’t borrow time from people inside their existing teams. We created a separate team, mixed from the teams within our tribe, with one job: this migration. I led it end to end, both writing code with them and managing the project.

That setup turned out to be one of the best decisions of the whole thing. A small team with a single goal moves fast. No context switching between sprint goals, no “I’ll get to it after my other tickets”, and every decision a conversation between five people instead of a meeting across teams.

The people in it were surprised by how efficient we were. They were also happy to step out of their usual team routines for a while and work on something new and seriously complex. The feedback afterwards was great.

The rules

Three constraints shaped every decision:

1. No breaking API. We could not tell customers “we moved things, please use this new API from now on”. Every existing call had to keep working.
2. No downtime. The product had to stay up the whole time. Reads never stopped. The only thing a customer could notice was a short pause on changes to their own workspace while it was being moved.
3. No access to the old database. For many reasons, among them security and the complexity of giving two different namespaces access to the same database, we could not just read the old tables directly. Everything had to go through the owning service.

Each of these alone is manageable. All three together is what made it open heart surgery.

Chapter 1: bookmark-proxy

Rule 1 meant callers could not know or care where a bookmark lived. So we put something in front of all the services: bookmark-proxy, a gateway that takes incoming requests and hands them over to the right upstream service.

It answers two questions for every request:

• Is this workspace migrated yet? If not, the old service is still the source of truth.
• What type of bookmark is this? If the workspace is migrated, the type property decides who owns it: diagrams go to the diagrams service, reports to the reports service.

The first question is why the proxy also keeps the migration state per workspace. Without it, there is no way to know where to route.

┌────────┐     ┌────────────────┐
│ client │ ──▶ │ bookmark-proxy │
└────────┘     └───────┬────────┘
                       │
             workspace migrated?
                       │
          ┌──── no ────┴──── yes ────┐
          ▼                          │
   ┌─────────────┐             bookmark type?
   │ old service │                   │
   └─────────────┘         ┌ diagram ┴ report ┐
                           ▼                  ▼
                ┌──────────────────┐ ┌─────────────────┐
                │ diagrams service │ │ reports service │
                └──────────────────┘ └─────────────────┘

Because the state is kept per workspace, a workspace is either fully on the old side or fully on the new one, never half and half. The same state also handles the two moments that make people nervous:

• Writes during the migration. While a workspace is being migrated, bookmark-proxy blocks mutations to it and returns an error. Reads keep working, so people can keep doing their work. Because the migration runs per workspace, that window is small, and the data we read can’t change underneath us.
• Rollback. If something goes wrong, we flip the workspace’s state in bookmark-proxy back to the old service. Traffic goes back to where it came from, without moving any data.

Chapter 2: one step at a time

We had two options for the rollout: migrate per workspace or per region. We did both, in order of risk.

First came dry runs, to see the migration run end to end without committing to anything. Then a couple of standalone workspaces of our own, so that if something went wrong, the only customer affected was us. After that we went region by region, starting with the smallest regions and working up to the biggest ones. Every step had to go cleanly before we took the next one.

The whole rollout took 4 hours in total. The migration itself didn’t need downtime, but we still ran it inside our separate weekly downtime hour. Not because we had to, but so that enough people were around to support if things didn’t go well.

The same thinking decided the order of the domains. Diagrams went first, into the existing diagrams service. It was the smaller step, and it told us whether the whole approach worked: the proxy, the streaming, the events, the snapshots, all of it on real data.

Only once diagrams had succeeded did we move on to reports. That was the bigger step, because the reports domain didn’t exist yet. It had to be built from the ground up.

Chapter 3: full API parity

The original service was written in Java and Kotlin. Both of ours, diagrams and reports, are NestJS. Rule 1 meant we needed full API parity, not “close enough”. For reports, that meant a whole new domain that had to behave exactly like the old one from the first request.

Parity is never a straight port. Some features had to go. Some needed enhancements while we were at it. Every one of those decisions was a conversation, because someone, somewhere, depended on the old behaviour.

And parity is not only about the shape of the responses, it is about how they hold up under load. Miss an index and you find out quickly when hundreds of requests per second start hitting the endpoint. Same response, very different day.

Chapter 4: events

Bookmarks were not just rows in a microservice. Part of their state, not all of it, was also distributed to other services through Azure Event Hubs.

So our services had to start producing those events, and consumers had to be adjusted as part of the project. The catch: a producer can only be shipped by one service. We could not just produce from our services into the same producer the old service was using.

So we added a new event hub for the new owners to produce into, and changed the consumers’ code to read from both event hubs, using the bookmark type to tell which one to listen to for what. That kept every consumer correct whether a workspace had moved yet or not.

Chapter 5: moving the data

With no database access, the data had to come through the old service. We streamed it.

• Read in a transaction. We read the data transactionally, so every workspace was exported from one consistent point in time.
• Streamed, not loaded. Nothing was held in memory as a whole, so there was no heap instability, no matter how large a workspace was.
• In foreign key order. Parents before children, children before working copies. That way every row could be written as it arrived, and snapshots taken on the new side could capture the data correctly.

That last point turned out to be more important than it looked, because of the hardest part of the whole project.

Chapter 6: snapshots, the hardest part of all

Workspaces have snapshots. A snapshot-manager orchestrates them across all upstream services, including ours.

Here is where it got complicated:

• Templates. New workspaces can start from templates with default data. Part of that data lived in the snapshot-manager, part of it in the old service’s .sql dump.
• Retention. Snapshots are kept for 90 days.

So migrating a workspace today raises an uncomfortable question: for the next 90 days, whose snapshot is the right one? A snapshot taken before the migration came from the old service. Restoring it would bring back data in the old service’s world, not ours.

We answered it with a remigration worker. If a workspace was restored from a snapshot that was not taken from our services, the worker remigrates that workspace using our streaming maintenance endpoint, the same streaming path we used for the original migration. The data is pulled through again, in foreign key order, and ends up where it belongs.

It is not glamorous. It is the kind of edge case that only shows up when you ask “and what happens in 89 days?”. Most of the work in this project was asking that question, over and over, about every corner.

How it ended

20,000 workspaces, 8 regions, 4 hours. No incidents. No data loss.

I am super proud of that. It took nerves, tons of alignment across teams and tribes, a lot of edge cases covered across many services, and a five-person tiger team that carried it all the way. From the outside, it looked like nothing happened.

That’s the goal. When a migration goes well, nobody notices.

What I’d do the same way again

1. Spend the time on discovery up front. The months of reading code and mapping connections are why there were no surprises later. Every surprise in a migration like this is an incident.
2. Pitch the needs, not just the estimate. For us the biggest need was a dedicated team, not a slice of everyone’s time. A small tiger team doing the work, with other teams sharing knowledge and reviewing, beats six teams each giving a little.
3. Start small, and keep a way back. Dry runs, our own workspaces, then regions from smallest to biggest. Diagrams before reports. And at every step, rollback was one state flip away.
4. Parity includes performance. Same responses at hundreds of requests per second, or it isn’t parity.
5. Stream, in order. Transactional reads, foreign key order, no heap. It is what made the migration and the remigration the same path.
6. Ask what happens in 90 days. Anything with history, like snapshots, outlives the migration day. Plan for it before it surprises you.

Till the next one. Stay hydrated.