Mystery
Today at work, I came across what’s gotta be the strangest bug (software wise, not real life!) that I’ve ever seen.
It started innocently enough - the ICAP server I wrote dies under heavy request modification stress. I realized that one of the required headers called Encapsulated was not being sent properly. Since Encapsulated tells the server the location of boundaries of different data sent over in a single request, it’s pretty important. 
So, I start sniffing packets… Packet captures on client side showed that right data is being sent over… Packet captures on the server side (on the networking layer) showed that right data is being sent over… but once it got to my program, the Encapsulated header would mysteriously change to another syntactically valid but nevertheless wrong value. And I wasn’t using any libraries to connect either! It was all low level socket programming (you know, select() and stuff) so nothing should’ve gotten in anywhere…
Anyhoo, I figured out the pattern that caused the server to crash thereby sort of jumping over the sections that caused the crash (at a slight Performance hit… it’s a QA tool, so compliance is more important than performance anyway)… but I still don’t understand why it worked the way it did. Not a slightest clue. Bah!