Near the end of 2018, I was working on a project in which we had to generate ssh keys dynamically in code to be used in connecting to a number of remote servers. At some point, this stopped working and it was hard to figure out why. The public and private keys were being created, stored, and used in the right places, however, we constantly saw Permission denied (publickey) showing up in our logs. It was perplexing what had worked fine for months seemingly randomly stopped working. But in the world of software engineering, nothing is ever random.
Like any good software engineering team, we tried to break down the issue to pinpoint the problem spot.
- Can we reproduce it locally? √
- Does it happen on each team member's machine? √
- Can we manually generate an ssh key, add the public key to the remote server, and connect to it with the private key via normal
sshmethods? √ - Is our manual usage the same as in the running app? Hmmm....
As we took a closer look and started breaking down our assumptions, we noticed a very important detail. The keys that had worked in the past all had began with:
-----BEGIN RSA PRIVATE KEY-----
...but our new keys all began with:
-----BEGIN OPENSSH PRIVATE KEY----- (!!!)
SSH Key Format
We came to find out that the default format for ssh-keygen (OpenSSH) had recently changed to something proprietary and was no longer compatible with a library we were using to connect to remote servers. The "key" for us to get an ssh key that worked with our solution, was to use the, now legacy, PEM (Privacy Enhanced Mail) format. Here is the command to do it:
ssh-keygen -m PEM
This change is described in the OpenSSH release notes for v7.8 on 8-24-2018:
ssh-keygen(1): write OpenSSH format private keys by default instead of using OpenSSL's PEM format. The OpenSSH format, supported in OpenSSH releases since 2014 and described in the PROTOCOL.key file in the source distribution, offers substantially better protection against offline password guessing and supports key comments in private keys. If necessary, it is possible to write old PEM-style keys by adding "-m PEM" to ssh-keygen's arguments when generating or updating a key.
Food for thought: this is where integration and end-to-end testing provides a lot of value. Upgrading versions of operating systems, dependencies, etc. might not show immediate side effects with day-to-day work or unit tests. Testing the whole system, in detail, would have yielded this problem earlier and helped us pinpoint "what changed" without as much digging.