Abstract
Every certificate automation tool makes one architecture decision before it does anything else. It decides where the private key gets generated. A reader who used to be “the certificate guy” at his organization emailed me this week to ask about exactly that.
Why not create the key locally and only ship the CSR back to go through the signing process? Why have the private key anywhere it doesn’t need to be?
It’s the right instinct, and it’s how most automation tools work. Certbot generates the key on the server, builds a certificate signing request, and the private key never leaves the machine.
That’s the textbook answer, and for one server it’s the right one. The key never crosses the network, never sits in a backup it shouldn’t, and can only be stolen from the server itself. The textbook assumes one server, one name, and one certificate. It falls apart the first time a name has to live on a second machine.
Perfect forward secrecy already made private keys less dangerous to lose. With a forward-secret key exchange (all of TLS 1.3, and ECDHE in TLS 1.2), a stolen key can’t decrypt recorded traffic. It can only impersonate a server, and only for an attacker who can pull off a man-in-the-middle attack.
The same SSL certificate on multiple servers
Most organizations have at least one wildcard certificate, whatever anyone thinks of wildcards as a practice. *.example.com is one certificate on many hosts. If keys can only be generated where they’re used, one of those hosts has to generate it, and that host is now the master node. The master needs a way to share the certificate and key with every other host. Someone has to decide what happens when the master is offline during a renewal, whether another node takes over or the renewal just fails, and how anyone finds out either way.
You don’t need a wildcard to hit this. Any high-availability setup puts the same name on more than one server. my.trackjs.com runs on three servers behind a load balancer, and all three present a certificate for the same name.
If one of the three generates the key, you’re back to the master node. If each generates its own, you have three different certificates for one name. Monitoring gets noisier, verifying a renewal means checking every server separately, and every rebuilt server is another request against the CA’s rate limits.
Once a certificate is shared, the private key has to move between machines. The security property that justified generating it locally is gone, and it left through scp in a cron job or a shared folder.
A thread on the Let’s Encrypt community forum asks how to put one wildcard certificate on dev, QA, and production servers. Issuing the certificate was the easy part. Getting it everywhere it needs to go is the part nobody solved.
The rate limit problem
Let’s Encrypt’s rate limits allow 5 certificates for the exact same set of names every 7 days, refilling at one every 34 hours. Renewals coordinated through ARI are exempt from every limit. Any other request for the same names counts, whether the client calls it a renewal or not.
That’s what catches a fleet where every server gets its own certificate. Six servers renewing the same names within a day, without ARI, is one too many. So is an autoscaling group that launches six nodes in an afternoon, a container that requests a certificate on every rebuild, or an engineer re-running a renewal script while debugging it. Once the limit trips, the next certificate for those names waits about 34 hours, and Let’s Encrypt doesn’t grant overrides on that limit.
Pick where the private key lives, on purpose
Generate the certificate in one place, and treat distribution as the engineering problem it is. Keys move either way. The choice is whether they move through a system someone designed, encrypted and logged, or through whatever got bolted on when the second server showed up.
That’s the tradeoff we made with CertKit. CertKit generates keys and certificates centrally, holds them encrypted, and the agent pulls them to every server that needs them, whether that’s 1 node or 10. There’s no master node to elect, no scp jobs, and one issuance per renewal regardless of fleet size.
If the key can’t leave your network, the Keystore (available on Enterprise plans) generates it on your own hardware and only the CSR reaches CertKit. That’s the reader’s design, applied to the network instead of to each server.
CertKit handles certificate distribution, so where the private key lives is a decision, not an accident.