Deploy really hot code with habanero 🌶️
How can you dramatically decrease time-to-deploy? Wouldn’t it be cool if your CI/CD pipeline took very little time? Wouldn’t it be cool if it took zero time? Is that even possible?
One way to minize deploy time is to follow the standard procedure and use solid engineering to parallelize your tests, get beefy CI machines, add lots of caching etc. Another way is to throw all the best practices out of the window and YOLO hot-code deploy whenever your local code changes. Guess which path this post is about.
Luckily, Elixir has very good hot-code-deploy capabilities. Unluckily, hot code deploys are pretty complicated if you want to do them right with no errors in production. Luckily, I don’t have production constraints on my side projects, so I can do it the easy way anyway.
There are several ways of going about it, but at the heart of it all is the fact that you can access an existing Elixir deployment via a REPL and update any existing code without restarting anything. The only question is how and when to trigger this process and I just release a tiny Elixir package habanero, which has some answers to these questions.
It exposes an HTTP endpoint in your app, which accepts your updated code and brutally replaces the existing, running code with this new version. I say “brutally”, because if you’re updating e.g. a GenServer and change how state is handled, you will probably get a crash. Hence YOLO, don’t do this to apps, where you care about reliability.
The code update is triggered by a watcher you’re running locally: any time you save a new version of your code, habanero will send that updated code in a request to your server and the new code will be deployed.
How to
To install it against your better judgement, follow the getting started guide.
Trade-offs
Besides the obvious, there are some trade-offs here that I think are interesting. First, security: it feels pretty careless to expose an HTTP enpoint that’s basically built for Remote Code Execution. Wouldn’t it be better to go over e.g. SSH? And the answer is that it might be better from the “I feel dirty, security-wise” perspective, but it would also complicate things. HTTP is easy if you’re already running Phoenix!
Another is that deploying on every save has the potential to just kill the app on your server? What’s stopping you from deploying broken code? And the answer there is yes, this is a terrible idea, you should definitely run some tests at least before you deploy. Exercise for the reader.
Finally, the way things are now, many kinds of updates just won’t work. E.g. if you install a new Elixir package or a new OS-level package, or update your config, you will need to restart the app to get the new code. Also, as soon as you restart your app all your hot changes melt away and you’re left with whatever you cold-deployed last. Isn’t that great?
Conclusion
Stick with blue-green deploys, unless you like living on the edge.