← Back to Blog

When Moving a Database Tenant Becomes an Ownership Problem

Designing safe tenant migration across Kubernetes data-plane clusters when failure can leave ownership ambiguous.

Sun Aug 09 2026

When Moving a Database Tenant Becomes an Ownership Problem

Moving a database tenant sounds like a data-movement problem. In production, the harder problem is moving authority safely.

Migrating a serverless database tenant from one Kubernetes data-plane cluster to another sounds straightforward.

Stop writes. Move the tenant. Register it on the destination. Start traffic again.

The happy path is not the difficult part.

The difficult part is what happens when something fails halfway through the transition.

If the source believes it still owns the tenant while the destination also believes it owns it, we risk divergent histories.

If neither side believes it owns the tenant, the system can become unavailable and recovery may require reconstructing state from multiple subsystems.

That led to a fundamental invariant for the migration:

At any point in time, a tenant must be safely activatable in exactly one place — either the source or the destination, never both and never neither.

Everything else in the migration design followed from that principle.

1. The System

The system operated a serverless database platform where tenants were logical constructs rather than resources permanently tied to a particular compute cluster.

Customers did not need to know which Kubernetes data-plane cluster hosted their tenant.

That abstraction allowed us to move tenants between clusters when the platform needed to:

Conceptually, the migration looked like:

                 Control Plane
                       |
              Tenant ownership
                       |
          +------------+------------+
          |                         |
          v                         v
   Data Plane A              Data Plane B
   +-----------+             +-----------+
   | Tenant A  |   ------>   | Tenant A  |
   +-----------+             +-----------+

The important detail is that this wasn’t simply copying bytes from one location to another.

The migration was transferring ownership and authority from one data plane to another.

That distinction became critical when failures occurred.

2. The Real Problem Wasn’t Throughput

This migration was not primarily a performance problem.

We were not trying to optimize how quickly data could be copied between clusters.

The difficult questions were:

These are correctness and recoverability problems.

The migration had to operate safely despite distributed-systems realities:

There was no single global transaction that could roll everything back.

3. The Fundamental Invariant

The most important design decision was to make ownership explicit.

We defined a single active ownership invariant:

At any point in time, a tenant must be safely activatable in exactly one place.

There are two dangerous states.

Split ownership

             Tenant
             /    \
            /      \
           v        v
      Source       Destination
      ACTIVE         ACTIVE

Both sides believe they can serve the tenant.

That creates the possibility of divergent histories.

Lost ownership

             Tenant
             /    \
            v      v
         Source   Destination
          OFF        OFF

Neither side can safely activate the tenant.

Availability is lost and recovery becomes an investigation.

The safe states are therefore much more constrained:

ACTIVE_SOURCE
      OR
ACTIVE_DESTINATION

The migration process existed to move between those states without accidentally creating an unsafe intermediate state.

4. Why the Happy Path Wasn’t Enough

The initial workflow was designed around the expected sequence of operations.

A simplified version looked like:

Prepare
  |
  v
Enter Maintenance
  |
  v
Stop Writes
  |
  v
Unregister Source
  |
  v
Register Destination
  |
  v
Validate
  |
  v
Activate Destination

On the happy path, this worked.

The problem appeared when an operation in the middle failed.

For example:

Source
  |
  | unregister
  v
?????
  |
  | timeout
  v
Workflow stops

The workflow knew that an operation had timed out.

It did not necessarily know the exact truth of the underlying distributed system.

That distinction matters.

A workflow timeout does not necessarily mean that the operation did not happen.

The request may have reached the target.

The target may have partially completed the operation.

A controller may have observed the change.

Another subsystem may still have stale information.

The workflow itself cannot safely infer the final state from the timeout alone.

5. The Failure That Exposed the Design Gap

One migration encountered a flaky tenant unregistration from the source.

The workflow eventually timed out during the transition.

The original orchestration model did not have durable checkpointing for every irreversible ownership transition.

That created an uncomfortable state:

The immediate instinct in an automation-heavy system is often:

Retry the failed step.

But retrying an ownership-changing operation without knowing the current state can make the situation worse.

The system needed to answer a more fundamental question first:

Who currently owns this tenant?

We had designed the workflow around migration steps.

We had not made ownership enforcement a first-class system primitive.

That was the real design gap.

6. Turning the Workflow Into a State Machine

The migration became easier to reason about when we modeled it explicitly as a finite state machine.

The important states were:

ACTIVE_SOURCE
      |
      v
MIGRATION_PREP
      |
      v
MAINTENANCE_MODE
      |
      v
UNREGISTERING_SOURCE
      |
      v
OWNERSHIP_TRANSITION
      |
      v
REGISTERING_DESTINATION
      |
      v
ACTIVE_DESTINATION

And there was an important escape state:

                    +----------------------+
                    | RECOVERY_REQUIRED    |
                    +----------------------+
                         ^            ^
                         |            |
                    failure        ambiguity
                         |            |
                         +------------+

The value of the state machine was not merely organizational.

It made irreversible boundaries explicit.

It allowed us to define:

7. Not All Failure Windows Are Equal

One of the most useful ways to reason about the migration was to divide failures into windows.

Failure Window A — Before the irreversible boundary

The source is still authoritative.

SOURCE = AUTHORITATIVE
DESTINATION = NOT ACTIVE

If preparation fails here, the migration can usually be aborted safely.

Traffic can return to the source.

This is a relatively safe failure window.

Failure Window B — During the ownership transition

This is the dangerous part.

SOURCE = ?
DESTINATION = ?

The system may no longer have enough information to determine ownership automatically.

This is where automation must stop.

Trying to “continue and see what happens” is precisely the wrong behavior.

The correct response is:

STOP
  |
  v
FREEZE
  |
  v
INSPECT
  |
  v
DECIDE
  |
  v
ENFORCE
  |
  v
VALIDATE

Failure Window C — After destination activation

Once the destination is authoritative:

SOURCE = INACTIVE
DESTINATION = AUTHORITATIVE

The system can validate the destination and resume traffic.

The critical point is that the ownership boundary has already been crossed successfully.

8. The Most Important Recovery Operation: Freeze

When ownership becomes ambiguous, the first objective is not recovery.

It is containment.

The first question should be:

How do we prevent the system from making the situation worse?

Ideally, the platform should support a per-tenant freeze.

             Controllers
                  |
        +---------+---------+
        |                   |
        v                   v
    Tenant A             Tenant B
      FROZEN              ACTIVE

Tenant A is prevented from further automated mutations while Tenant B continues operating normally.

This is much safer than shutting down an entire controller.

The principle is:

Contain the affected tenant, not the entire platform.

9. When Humans Must Decide

Once ownership becomes ambiguous, there may not be enough reliable information for an automated system to choose the correct side.

A human operator may need to evaluate:

The goal isn’t to make humans part of every migration.

The goal is to make the system recognize when automation no longer has enough information to make a safe decision.

That distinction is important.

Manual intervention is not necessarily a failure of automation.

In an ambiguous ownership state, deliberately stopping automation can be the safety mechanism.

10. Ownership Enforcement

After deciding which side should be authoritative, the system needs a mechanism to enforce that decision.

The recovery process becomes:

Phase 1 — Freeze

Stop automation for the affected tenant.

Prevent controllers from continuing to mutate state.

Tenant
  |
  v
FROZEN

The objective is simple:

Stop making it worse.

Phase 2 — Decide

Determine whether the source or destination should become authoritative.

The decision can depend on:

Phase 3 — Enforce

Apply a hard ownership lock.

Disable the losing side.

Force the system into a known state.

Then validate before traffic is restored.

Conceptually:

Ambiguous
    |
    v
  Freeze
    |
    v
  Decide
    |
    +------------------+
    |                  |
    v                  v
Source wins       Destination wins
    |                  |
    v                  v
Disable dest      Disable source
    |                  |
    +--------+---------+
             |
             v
          Validate
             |
             v
        Resume traffic

Without enforcement mechanisms, humans are forced to guess and repeatedly inspect the system.

That is not a reliable recovery strategy.

11. Stopping Automation Safely

There are several levels of containment.

Per-tenant freeze

The preferred option.

Only the affected tenant is blocked.

Other migrations and tenants continue operating.

Traffic isolation

If ownership is uncertain, routing for the affected tenant can be blocked to limit customer impact and prevent additional writes.

Break-glass controller shutdown

As a last resort, a controller can temporarily be scaled to zero.

Controller replicas
        |
        v
       0
        |
        v
No automated mutations

This is effective but risky.

It also affects unrelated tenants.

For that reason, it is better viewed as a temporary emergency mechanism than the normal recovery model.

12. Metadata Backups Matter

A migration isn’t only about the database contents.

Metadata defines important parts of the system’s understanding of a tenant:

If that metadata is lost or becomes inconsistent, recovery can turn into inference.

Inference is dangerous during an incident.

The goal of backups is therefore not simply:

“Restore the database.”

It is also:

Preserve enough authoritative metadata to reconstruct the intended state safely.

That distinction becomes particularly important when the system is already ambiguous.

13. Why Stateless Orchestration Eventually Breaks Down

The original migration workflow was intentionally stateless.

There were good reasons for that.

Migrations were relatively rare and operator-driven.

A stateless workflow provided:

But ownership transfer changes the equation.

A long-running migration with irreversible side effects needs durable knowledge of where it stopped.

Without durable checkpoints:

Step 1 ✓
Step 2 ✓
Step 3 ✓
Step 4 ???
Step 5 ?

The system cannot safely distinguish between:

That is why durable state becomes valuable at irreversible boundaries.

The lesson wasn’t:

Stateless workflows are bad.

It was:

When an operation can permanently change ownership, the system needs durable knowledge of the transition.

14. Designing Recovery Before Automation

One of the biggest lessons from the migration was that recovery should not be an afterthought.

A safer development sequence is:

1. Define invariants
       ↓
2. Model failures
       ↓
3. Build observability
       ↓
4. Design recovery
       ↓
5. Implement happy path
       ↓
6. Add automation
       ↓
7. Roll out gradually

This is intentionally different from:

Build happy path
       ↓
Add retries
       ↓
Discover failures
       ↓
Invent recovery

The second approach is tempting because it produces visible progress quickly.

The first approach reduces the probability of discovering an unrecoverable state in production.

15. Planning and Estimation

This type of migration also changed how I think about engineering estimates.

Early in the project, there were many unknowns:

Giving a precise delivery estimate before answering those questions would have created false confidence.

A better approach was to estimate around risk reduction.

First understand the invariants.

Then improve observability.

Then prove the happy path.

Then design operator-assisted recovery.

Then gradually increase automation and rollout scope.

Confidence increases as unknowns are retired.

16. Getting Stakeholder Buy-In

A migration like this affects more than the engineering team.

Stakeholders naturally care about:

The strongest way to communicate the design was not to promise that migration would never fail.

Instead, the design needed to make the failure behavior explicit.

For example:

If the migration fails before the ownership boundary, we abort safely.

If it fails during the ownership transition, we freeze the tenant and require recovery.

If the destination becomes authoritative, we validate it before restoring traffic.

That makes the blast radius understandable.

It also makes the tradeoff visible:

We intentionally optimize for safety over migration speed.

17. What I Would Change in the Design

The experience highlighted several capabilities that are valuable for future migrations.

Explicit ownership state

Ownership should be represented as a first-class concept rather than inferred from multiple subsystems.

Durable checkpoints

Irreversible transitions need durable progress information.

Per-tenant freeze

Operators should be able to stop automation for one tenant without stopping the entire platform.

Recovery tooling

The platform should provide explicit tools for:

Strong observability

During an incident, operators should be able to answer:

Who owns this tenant?

What was the last completed transition?

Which side is serving traffic?

Which controllers are acting on it?

What changed recently?

Can I safely resume?

Those questions should not require reconstructing the system from logs.

18. The Broader Distributed Systems Lesson

This experience reinforced a principle that applies far beyond database migrations.

Whenever a system moves ownership of something important, ask:

What happens if the transition stops halfway?

This applies to:

The happy path is usually easy to describe.

The difficult engineering work lives at the boundaries.

Especially the irreversible ones.

19. Lessons Learned

Ownership must be explicit

If ownership is inferred from several independent systems, recovery becomes difficult.

Make it a first-class concept.

Irreversible operations need boundaries

Know exactly where rollback stops being safe.

Recovery must be designed

A retry is not a recovery strategy when the current state is unknown.

Automation needs an escape hatch

A good automation system knows when it does not have enough information to continue safely.

Containment comes before recovery

When the system is ambiguous, stop further mutations before attempting to repair it.

Safety can be more valuable than progress

A migration that takes longer but has a well-defined recovery path is preferable to a fast migration that can leave ownership ambiguous.

Conclusion

The most important lesson from this migration wasn’t about Kubernetes.

It wasn’t about workflows.

It wasn’t even about databases.

It was about ownership under failure.

Moving a tenant from one cluster to another means transferring authority from one part of a distributed system to another.

That transfer has an irreversible boundary.

Before that boundary, the source can still be authoritative.

After it, the destination can become authoritative.

In the middle, the system must be extremely careful.

If that transition becomes ambiguous, the right response isn’t to blindly retry.

It is to:

STOP
  ↓
CONTAIN
  ↓
DETERMINE AUTHORITY
  ↓
ENFORCE OWNERSHIP
  ↓
VALIDATE
  ↓
RESUME

The most important engineering principle I took away is simple:

Recovery should be designed before the system needs it.

Because in production, the difficult question isn’t whether the happy path works.

It’s whether you know what to do when it doesn’t.