Newlines in bad places

So it turns out that if you violate standards, bad things happen. Who woulda thought?

For the longest time, I’ve had this little problem with my web server, in that some clients would refuse to load it. The major ones worked – Firefox, Chromium, cURL, Android apps – all of them good. The first time I encountered this problem was when I was working on hardening the web server’s security (TLS, CSP, that kind thing). I would test the site with Mozilla Observatory, only for it to fail with a vague error message that the site couldn’t be loaded.

Since everything else worked, I didn’t put too much thought to it back then. Maybe there was some weird issue that maybe the Observatory had Scaleway—my current VPS host—banned for some reason? I left it at that for some time.

But then, the issue kept turning up in other places. I’ve been setting up OpenGraph for the website, testing it with various tools. Some would work well, but others would fail. Again, with no helpful messages, everything along the lines of “something went wrong” or “is the site down?”. No, it isn’t! I’m looking at it in my browser, dammit! Mastodon would seemingly ignore my OpenGraph tags, while other services happily showed pretty links. I kept debugging, to no avail. SslLabs worked fine. Nginx’s debug log turned up nothing useful – clients would do the proper TLS handshake, receive the HTTP headers, then close the connection. I couldn’t, for the life of me, figure it out. So I gave up for several more months. My actual use cases weren’t hindered, so I just couldn’t be bothered.

Fast-forward to today. Out of curiosity, I tried to open my website in a nightly build of Servo. It, like other random clients, would refuse to load up the site:

Servo could not load the requested page: client error (SendRequest)

As with all the other offenders, not a very helpful message. I tried searching for the message. Turns out the format is common for various Rust projects using the reqwest crate, among them Deno, a JavaScript runtime. I found an issue on Deno’s GitHub about a similar, but ultimately unrelated problem – something about a proxy, not interesting to me. But I got curious. I took the reporter’s script, removed the proxy option and swapped the URL to my website:

using client = Deno.createHttpClient({});
await fetch("https://spiffyk.cz", { client });

I ran it with Deno and… bingo!

error: Uncaught (in promise) TypeError: error sending request from *snip* for https://spiffyk.cz/ (*snip*):
       client error (SendRequest): invalid HTTP header parsed

I started digging through my Nginx configs, then I found it:

add_header "Content-Security-Policy"
    "default-src 'self';
    object-src 'none';" always;

In an attempt (a bad one at that, now that I’m looking at it) to make the config more readable, I’ve split the CSP directives onto multiple lines. Since neither Nginx nor my web browsers ever complained (not even in devtools), I never gave it much more thought and assumed Nginx would replace the newline with a space. It wouldn’t. It just pasted that newline right into the HTTP response. Many clients, adhering to the infamous “Postel’s law¹,” just swallowed that right up. Others—rightfully, in my opinion—refused. Turns out violating RFCs just doesn’t pay off.

Well, easy fix. I merged the directives onto a single line, and one systemctl reload nginx later, everything works! So I can finally be proud of my A+ score in Observatory, and show you our cat every time I share a link to my blog:

1. “Be liberal in what you accept, and conservative in what you send.” A relic of the past. I don’t particularly like it, but we have to live with it when working the likes of HTTP.

   ↩