I don’t believe in ghosts. But something spooky happened to me only a few minutes ago, and sitting in the melancholic darkness of 3AM, I couldn’t think of a more fitting name than “The Ghost in the CI.”
It started with a change I made in the HTML template of a Flask app. This app spawns some other apps on-demand with Nomad (which has similar capabilities as Kubernetes). The app accesses Nomad through a third-party wrapper library for the API. This library has a method to get the status of a deployment. The app calls this method as part of a health check for the deployed app during the tests. This method shouldn’t ever return None as per the source code. But it did. My code didn’t handle that case and it crashed.
This was interesting to me because I had run tests locally before pushing it. My initial guess was that I had forgotten to commit something. But the tests were passing in CI previously, and the only change that was made was a typo fix in a page’s HTML.
Very unexpectedly, the tests passed when I ran them again in CI, with the same code. No explanation comes to mind other than the the bit flip. I had only heard of it in this video and never in real life, like a mythical creature which exists purely for imagination and nerd talk. It’s hard to believe.
The issue is documented here: https://github.com/pipalacademy/capstone/issues/74 The source code is referenced in the comments and CI logs can be accessed from the mentioned commit.