Wow, what a fucking mess this sunday was.
My boss wrote me an email that one route of a RESTful API we wrote for a customer was not working anymore and puking back a status 500 with some error mentioning invalid UTF-8 characters.
Not one single person has had touched nor changed the code on production in some 6 months, so what the fuck could it be?

Phpunit did not give any errors (running only locally), the code had no syntax errors and the DB dump did not contain any invalid bytes (tested with a hex editor).

WHAT THE FUCK?!

OK so I started to comment out lines (all tested directly on production of course) until the error vanished.

Guess what was the culprit?
.
.
.
.
.
.
In the code (PHP) we used strftime(...) to get nice time strings. Of course we set the correct locale on the server, thus having months and days formatted in German.
So, in Geman there is this one mysterious month called "März" which contains an umlaut character.
Calling strftime generated the date with März in it, but the server locale was de_CH.iso-8859-1 and not fucking de_CH.utf8, so the "ä" was returned as 0xE4 instead of 0xC3A4 (valid UTF-8), which json_encode(...) did not want to swallow but instead threw an exception.

"Stop throwing errors in your library! They cause my app to crash"

Who amongst us hasn't hear of the popular maxim: "Fail late even after you know something is wrong, and then make sure the failure cascades over the whole app in mysterious ways"

I'm trying out Picolisp. Cool, I think, an OpenGL library. I'll try the example program.

(Clicks mouse on program window)
(A wild SIGILL appears!)

Two hours later, still trying to figure out why it's doing that, with Google and DuckDuckGo returning no helpful results whatsoever. This is very annoying.