When the Most Scalable Design Wasn’t the Right One

TL;DR

We wanted to modernize TLS certificate management for a multi-tenant Kubernetes platform using cert-manager. The proposal looked technically elegant, but it introduced a larger question: where should certificate signing authority live? In the end, we chose a simpler operational improvement that preserved the existing trust model instead of redesigning our PKI architecture.

The Background

Every customer on our platform could configure a custom domain.

Traffic terminated at an NGINX ingress layer, where every customer certificate was stored as a Kubernetes Secret.

The architecture was intentionally simple.

Customer Domain
      │
      ▼
+-------------------+
| NGINX Ingress     |
+-------------------+
      │
      ▼
TLS Secret
      │
      ▼
Kubernetes API
      │
      ▼
etcd

This model worked well for a long time.

Growth Changed the Problem

As more customers onboarded, the number of TLS certificates increased.

Every certificate required:

• A Kubernetes Secret
• Storage in etcd
• Watches from Kubernetes components
• Mounts into ingress
• Rotation during renewals

Nothing had failed.

However, the operational overhead of managing thousands of Secrets was steadily increasing.

The obvious question became:

Can we automate the certificate lifecycle?

The Attractive Solution

The natural answer seemed to be cert-manager.

Using cert-manager would allow certificates to be:

• Automatically issued
• Renewed before expiry
• Rotated without manual intervention
• Managed consistently across the platform

On paper, it solved almost every operational problem.

Customer Request
       │
       ▼
 cert-manager
       │
       ▼
Certificate Signing
       │
       ▼
TLS Secret
       │
       ▼
NGINX

From an operational perspective, this was a clear improvement.

Or so it seemed.

The Conversation Changed

As the design review progressed, the discussion gradually shifted away from Kubernetes.

Instead, it became a conversation about trust.

cert-manager does not create certificates by itself.

It needs access to a certificate signing authority.

That raised an architectural question:

Where should certificate signing happen?

Our existing PKI model intentionally kept certificate signing capabilities isolated from the Kubernetes data plane.

Moving certificate issuance into the cluster would expand the trust boundary.

At that point the discussion stopped being about automation.

It became a security architecture discussion.

Understanding Trust Boundaries

The Kubernetes data plane runs workloads that process customer traffic.

Compared to dedicated security infrastructure, it generally has:

• More exposed services
• A larger attack surface
• More operational change
• Different ownership

Introducing certificate-signing capabilities into that environment changes the security model.

Even if only an intermediate CA is used instead of a Root CA, the platform now has to answer questions like:

• Who owns the intermediate CA?
• How is it rotated?
• How is issuance audited?
• What is the blast radius if the cluster is compromised?
• How is trust revoked?

Those questions are often harder than writing the controller itself.

Why We Didn’t Build It

The proposal was technically sound.

However, solving one operational problem introduced another architectural concern.

The security team’s feedback wasn’t “cert-manager is bad.”

Instead, it was:

Introducing certificate signing capabilities into the data plane changes the platform’s trust model.

That distinction mattered.

The disagreement wasn’t about Kubernetes.

It was about where security responsibilities should live.

The Alternative

Instead of redesigning our PKI architecture, we stepped back and reconsidered the original problem.

The issue wasn’t that certificate issuance was impossible.

The issue was the operational cost of managing large numbers of Secrets.

We adopted a much smaller change:

• Shard TLS Secrets
• Reduce operational pressure
• Preserve the existing trust boundary
• Avoid introducing signing authority into the cluster

This solved the immediate operational concern without expanding the security model.

Sometimes incremental improvements are exactly what a system needs.

Lessons Learned

This project changed the way I think about architecture.

1. Scalability isn’t the only design goal

Performance and automation matter, but so do ownership, auditing, governance, and operational risk.

2. Trust boundaries are architectural boundaries

Moving a security capability into a different environment can be a much larger decision than changing code.

3. Security reviews improve designs

Initially I viewed the proposal as an engineering optimization.

The review helped reveal assumptions about trust that weren’t obvious during implementation.

4. Simpler solutions are often better

The most technically sophisticated design isn’t always the right engineering decision.

Sometimes solving today’s problem without expanding the system’s complexity is the better choice.

Final Thoughts

Looking back, I don’t remember this project because we considered cert-manager.

I remember it because it reinforced an important engineering principle.

Good architecture is rarely about finding the most advanced technology.

It is about balancing scalability, operational simplicity, and security while respecting the trust boundaries of the system.

Sometimes the best design is the one you intentionally decide not to build.