So today I discovered that there’s a cron job that holds non-reproducible state that died, and now our system is fucked.
The cron job doesn’t live inside any source control. This morning it entered a terminal state, and because it overwrites its state there’s no way to revert it.
I’m currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.
What’s extra frustrating is the previous guy did create a git repo of these types of hacks, but this one doesn’t live in it for no discernible reason.
Job security
He does charge a consulting fee to “fix” these issues
Almost all of them are dumb shit like this, where something is built in super hacky and dumbass ways.
It’s his kill switch and he forgot to check in.
Super hacky and dumb? Sign me up 😂
Me running all my services in tmux
Ah yes, good old dependency.
This is almost exactly what happened to me on Monday, resulting in a fifteen hour day.
My particular jenga piece was an Access query that none of my predecessors had deigned to document or even tell me about… but was critical to run monthly or you had obsolete data embedded deep within multi-million dollar reports.
Thank god I don’t work on salary anymore, or I’d be really upset.
I stopped reading at “Access” and just wept a silent tear for you.
Cron job that evals some base64 encoded string which is actually downloading a script from a personal GitHub repo of an IT guy who left…
And just started cleaning up their GitHub account…
I have also mixed up
crontab -lwithcrontab -r. 😔Let this be a lesson to start versioning your crontabs.
So do you work for Spotify or Zoom?
Probably DeepSeek.
