←back to thread

CRLF is obsolete and should be abolished

(fossil-scm.org)
422 points km | 8 comments | 13 Oct 24 19:16 UTC | HN request time: 0s | source | bottom
Show context
michaelmior ◴[13 Oct 24 20:03 UTC] No.41831072[source]▶
> various protocols (HTTP, SMTP, CSV) still "require" CRLF at the end of each line

What would be the benefit to updating legacy protocols to just use NL? You save a handful of bits at the expense of a lot of potential bugs. HTTP/1(.1) is mostly replaced by HTTP/2 and later by now anyway.

Sure, it makes sense not to require CRLF with any new protocols, but it doesn't seem worth updating legacy things.

> Even if an established protocol (HTTP, SMTP, CSV, FTP) technically requires CRLF as a line ending, do not comply.

I'm hoping this is satire. Why intentionally introduce potential bugs for the sake of making a point?

replies(13): >>41831206 #>>41831210 #>>41831225 #>>41831256 #>>41831322 #>>41831364 #>>41831391 #>>41831706 #>>41832337 #>>41832719 #>>41832751 #>>41834474 #>>41835444 #
FiloSottile ◴[13 Oct 24 20:39 UTC] No.41831391[source]▶
Exactly. Please DO NOT mess with protocols, especially legacy critical protocols based on in-band signaling.

HTTP/1.1 was regrettably but irreversibly designed with security-critical parser alignment requirements. If two implementations disagree on whether `A:B\nC:D` contains a value for C, you can build a request smuggling gadget, leading to significant attacks. We live in a post-Postel world, only ever generate and accept CRLF in protocols that specify it, however legacy and nonsensical it might be.

(I am a massive, massive SQLite fan, but this is giving me pause about using other software by the same author, at least when networks are involved.)

replies(7): >>41831450 #>>41831498 #>>41831871 #>>41832546 #>>41832632 #>>41832661 #>>41839309 #
tptacek ◴[13 Oct 24 20:47 UTC] No.41831450[source]▶
This would be more persuasive if HTTP servers didn't already widely accept bare 0ah line termination. What's the first major public web site you can find that doesn't?
replies(5): >>41831506 #>>41831717 #>>41832137 #>>41832555 #>>41832731 #
LegionMammal978 ◴[13 Oct 24 23:08 UTC] No.41832555[source]▶
Going down a list of top websites, these URLs respond with HTTP 200 (possibly after redirections) when sent an ordinary HTTP/1.1 GET request with 0D0A line endings, but respond with HTTP 400 when sent the exact same request with 0A line endings:

  https://br.pinterest.com/ https://www.pinterest.co.uk/
  https://apps.apple.com/ https://support.apple.com/ https://podcasts.apple.com/ https://music.apple.com/ https://geo.itunes.apple.com/
  https://ncbi.nlm.nih.gov/ https://www.salesforce.com/ https://www.purdue.edu/ https://www.playstation.com/
  https://llvm.org/ https://www.iana.org/ https://www.gnu.org/ https://epa.gov/ https://justice.gov/
  https://www.brendangregg.com/ http://heise.de/ https://www.post.ch/ http://hhs.gov/ https://oreilly.com/
  https://www.thinkgeek.com/ https://www.constantcontact.com/ https://sciencemag.org/ https://nps.gov/
  https://www.cs.mun.ca/ https://www.wipo.int/ https://www.unicode.org/ https://economictimes.indiatimes.com/
  https://science.org/ https://icann.org/ https://caniuse.com/ https://w3techs.com/ https://chrisharrison.net/
  https://www.universal-music.co.jp/ https://digiland.libero.it/ https://webaim.org/ https://webmd.com/
This URL responds with HTTP 505 on an 0A request:

  https://ed.ted.com/
These URLs don't respond on an 0A request:

  https://quora.com/
  https://www.nist.gov/
Most of these seem pretty major to me. There are other sites that are public but responded with an HTTP 403, probably because they didn't like the VPN or HTTP client I used for this test. (Also, www.apple.com is tolerant of 0A line endings, even though its other subdomains aren't, which is weird.)
replies(1): >>41832634 #
1. tptacek ◴[13 Oct 24 23:22 UTC] No.41832634[source]▶
You sure about this? www.pinterest.com, for instance, does not appear to care whether I 0d0a or just 0a.
replies(1): >>41832865 #
2. LegionMammal978 ◴[14 Oct 24 00:03 UTC] No.41832865[source]▶
My apologies, I was using a client which kept the connection alive between the 0D0A and 0A requests, which has an effect on www.pinterest.com. Rerunning the test with separate connections for 0D0A and 0A requests, www.pinterest.com and phys.org are no longer affected (I've removed the two from the list), but all other URLs are still affected.
replies(1): >>41832909 #
3. tptacek ◴[14 Oct 24 00:12 UTC] No.41832909[source]▶
I picked one at random --- hhs.gov --- and it too appears to work?

For what it's worth: I'm testing by piping the bytes for a bare-newline HTTP request directly into netcat.

replies(1): >>41833406 #
4. LegionMammal978 ◴[14 Oct 24 01:39 UTC] No.41833406{3}[source]▶
Make sure you're contacting hhs.gov and not www.hhs.gov, the www. subdomain reacts differently.

  $ printf 'GET / HTTP/1.1\r\nHost: hhs.gov\r\n\r\n' | nc hhs.gov 80
  HTTP/1.1 302 Found
  Date: Mon, 14 Oct 2024 01:38:29 GMT
  Server: Apache
  Location: http://www.hhs.gov/web/508//
  Content-Length: 212
  Content-Type: text/html; charset=iso-8859-1
  
  <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>302 Found</title>
  </head><body>
  <h1>Found</h1>
  <p>The document has moved <a href="http://www.hhs.gov/web/508//">here</a>.</p>
  </body></html>
  ^C
  $ printf 'GET / HTTP/1.1\nHost: hhs.gov\n\n' | nc hhs.gov 80
  HTTP/1.1 400 Bad Request
  Date: Mon, 14 Oct 2024 01:38:40 GMT
  Server: Apache
  Content-Length: 226
  Connection: close
  Content-Type: text/html; charset=iso-8859-1
  
  <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>400 Bad Request</title>
  </head><body>
  <h1>Bad Request</h1>
  <p>Your browser sent a request that this server could not understand.<br />
  </p>
  </body></html>
replies(1): >>41833624 #
5. tptacek ◴[14 Oct 24 02:18 UTC] No.41833624{4}[source]▶
Ahh, that was it, thanks.
replies(1): >>41837585 #
6. shadowgovt ◴[14 Oct 24 13:56 UTC] No.41837585{5}[source]▶
And this whole exercise is an example of why this is a non-starter proposal (at least the "change existing implementations" part).

How much do we expect the domain owners to invest in changing an implementation that already works? Hint: it's a number smaller than epsilon.

Google might, but their volume is so high they care about the cost of individual bytes on the wire.

replies(1): >>41838741 #
7. tptacek ◴[14 Oct 24 15:57 UTC] No.41838741{6}[source]▶
This exercise was about demonstrating that our security can't rely on making sure there's a carriage return in HTTP line termination, because there is no such norm. See the root of the thread, where I asked the question.
replies(1): >>41839176 #
8. shadowgovt ◴[14 Oct 24 16:38 UTC] No.41839176{7}[source]▶
Oh, I agree it's about that too, but my point is you've already volunteered more time and resources investigating the situation than most companies would be willing to spend.