Most active commenters
  • p_l(7)
  • WorldMaker(3)
  • zzo38computer(3)

←back to thread

309 points StalwartLabs | 28 comments | | HN request time: 0.942s | source | bottom
Show context
9dev ◴[] No.45673491[source]
While JMAP seems to scratch every itch of a sucker for proper web API design, I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP. Is there really any new-ish binary protocol these days? Stuff like file sharing or groupware, mail, calendars, and so on—these things could be a lot more efficient and don’t really need the overhead of JSON as the message interchange format, IMHO. Then again, a lot of solid thinking went into these things, so there probably are a lot of good reasons that I’m not aware of.

Still, it’s an interesting space, I think.

replies(8): >>45673994 #>>45674417 #>>45675659 #>>45675911 #>>45676495 #>>45676769 #>>45677107 #>>45678657 #
1. WorldMaker ◴[] No.45673994[source]
> binary protocol

Email was never a binary protocol. Notoriously so, it's why MIME types and MIME encodings get so complicated.

Most of the "old internet" protocols (email, FTP, even HTTP itself) were bootstrapped on top of built-mostly-for-plaintext Telnet. HTTP as the new telnet has a bunch of improvements when it comes to binary data, request/response-based data flows, and some other considerations. HTTP/3 is even inherently a binary protocol, it's lack of "telnet-compatibility" one of the concerns about switching the majority of the web to it.

vCard/vCal/iCard/iCal were also deeply "plaintext formats". JSON is an improvement because it is more structured, even more efficient, than those predecessors. JSON may not look efficient, but it compresses extremely well and can be quite efficient in gzip and Brotli streams.

I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet", even if it doesn't sound like "binary protocol efficiency" at first glance. Especially as HTTP/3 pushes HTTP more efficient and more "binary", and arguably "more fundamental/basic" with HTTP/3 even taking over more roles in the TCP/UDP layer of the internet stack. (Telnet would never try to replace TCP.) HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of. Sure, it would be neat to see more variety and experiments outside of the HTTP stack, too, but HTTP is too useful at this point not to build a bunch of things on top of it instead of as their own from-scratch protocol.

replies(6): >>45674072 #>>45674139 #>>45675430 #>>45675548 #>>45675668 #>>45676137 #
2. p_l ◴[] No.45674072[source]
A lot of the textual nature of older IETF protocols, including the CR LF line endigns, can be probably traced to how easy it was to bang out a bad implementation full of subtle problems that could be debugged by sitting an undergrad student at a teletype instead of spending time on having some binary serializer (that telecom companies definitely had money for)
replies(2): >>45674085 #>>45674657 #
3. JoshTriplett ◴[] No.45674085[source]
Yeah, a fair bit of email protocol reeks of "is this tolerant of `telnet mailserver 25` and whatever garbage that might produce".
replies(1): >>45674113 #
4. p_l ◴[] No.45674113{3}[source]
A lot of old RFCs explicitly mention running on top of TELNET.

Additionally, as much people like to harp about "telcos focusing on connection-oriented protocols while we ran loops around them with packets", the reality is that NCP and later TCP pretty much focused on emulating serial lines around, and one of the earliest ways to access ARPAnet outside of machines directly on it was through calling into a TIP which set up bidirectional stream from your modem to a port on some host.

replies(1): >>45677797 #
5. p_l ◴[] No.45674139[source]
Another point is that the use of HTTP for everything, outside of the issue of middle boxes breaking protocols for everyone, is that it's essentially capitulation to the wisdom of OSI multi-layered protocols - we replicate their feature sets by reusing bits and pieces of HTTP spec all the time.
6. cyberax ◴[] No.45674657[source]
It's also a reflection of the state-of-the-art at the time. Binary protocols were an unmitigated disaster, with standards bodies thinking that applying to the ISO for your organizational ID is a perfectly fine step that anyone does anyway.
replies(1): >>45676125 #
7. frumplestlatz ◴[] No.45675430[source]
HTTP is also a large, complex stack for any server/client that isn’t already a web server or a browser.
replies(1): >>45675698 #
8. 9dev ◴[] No.45675548[source]
It’s not that I cannot appreciate the improvements in the space, I’m just wondering if there might be a big part of the design space for widely used protocols that ends up unexplored because the default for almost anything now is HTTP. It has basically become OSI layer 8 at this point.
replies(2): >>45676247 #>>45678227 #
9. zzo38computer ◴[] No.45675668[source]
> Notoriously so, it's why MIME types and MIME encodings get so complicated.

I made up ULFI because I thought MIME has some problems.

> JSON may not look efficient

Efficiency is not the only issue; there is also the consideration of e.g. what data types you want to use. JSON does not have a proper integer type, does not have a proper binary data type (you must encode it as hex or base64 instead), and is limited about what character sets can be used.

(Also, like other text formats, escaping will be needed.)

> I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet"

I think it can be, depending on the specific use; sometimes it isn't, and will make it worse. (HTTP does have the advantage of having URLs and virtual hosting, although I think it adds too much complexity more than should be needed.) However, I still think that DER is generally better than JSON.

> HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of.

I think it depends on the specific application. However, even then, I think there are better ways than using HTTP with the complexity that it involves, most of which should not be necessary (even though a few parts are helpful, such as virtual hosting).

replies(1): >>45675977 #
10. zzo38computer ◴[] No.45675698[source]
Yes, this is the other issue with using HTTP.
11. dotancohen ◴[] No.45675977[source]

  > JSON does not have a proper integer type
What are the drawbacks to using the JavaScript Number (really a double float I think) datatype as an integer in an object representation language such as JSON? I've never seen a use case where e.g. 42 (int) could be confused with 42.0 (float). If your application needs specifically an int or a float, then the ingesting application knows that.

If the answer is monetary values, then those should never be floats, and should not be represented in JSON as such. E.g. a dollar and a half should be represented as 150 cents. This follows even for sub-cent precision.

replies(3): >>45676282 #>>45677914 #>>45678201 #
12. p_l ◴[] No.45676125{3}[source]
That's why there was an entire section for unassigned numbers.

Binary protocols just meant you actually needed to implement serialiser/deserialiser and similar tooling instead of writing dumbest possible riff on strtok() and hoping your software won't be used anymore once DoD internet becomes mature

replies(1): >>45679069 #
13. 7bit ◴[] No.45676137[source]
> Most of the "old internet" protocols (email, FTP, even HTTP itself) were bootstrapped on top of built-mostly-for-plaintext Telnet.

That's just not true. Telnet and SMTP are built on top of TCP. They live on the same layer. They were originally both protocols that transmitted data with printable ascii, hence why they look similar. There are many other protocols like Telnet and SMTP that worked like that, auch as nntp, irc, and yes, even http.

replies(1): >>45676588 #
14. p_l ◴[] No.45676247[source]
MIME/Content-Negotiation is essentially OSI Presentation Layer

HTTP sorta acts as stump of ROSE with bit of ACSE. In addition it provides a bit of basic layer for passing some extra attributes that might be considered in-band or out (or side?) band to the actual exchange.

15. zzo38computer ◴[] No.45676282{3}[source]
I don't mean 32-bit integers, but rather, 64-bit (and sometimes bigger) integers. JSON already works OK for 32-bit integers (although it is not ideal).
16. 20after4 ◴[] No.45676588[source]
I think that the point is this: You could literally use a telnet client to talk to all of those servers and it was considered a good thing, maybe even an essential feature. So protocol design was somehow influenced by the need to be fully plain text ASCII streams (and maybe that other options weren't adequately explored due to this restriction)
replies(1): >>45677108 #
17. dhosek ◴[] No.45677108{3}[source]
You can also telnet into port 80 and communicate with an HTTP server.
replies(1): >>45678236 #
18. tbrownaw ◴[] No.45677797{4}[source]
> Additionally, as much people like to harp about "telcos focusing on connection-oriented protocols while we ran loops around them with packets", the reality is that NCP and later TCP pretty much focused on emulating serial lines around, and

The idea with packets is that you don't need to reserve N bit/s of each link along the route to whatever system you're talking to; instead you just repeatedly say "here's a chunk of data, send it to X". It's not really relevant that the typical thing to do with these packets is to build a reliable stream on top of them, what matters is that everything except the endpoints can be a lot dumber.

replies(2): >>45679227 #>>45679304 #
19. RedShift1 ◴[] No.45677914{3}[source]
Much easier would just be native decimal support in JS and we can represent a decimal using d as the decimal point, for example 15d0 (15.0) or 384d25 (384.25).

Using cents instead of dollars sounds fine until you have to do math like VAT, you really need decimal math for that.

20. cstrahan ◴[] No.45678201{3}[source]
JSON numbers are not JavaScript Numbers.

While the grammar is specified (that’s what JSON is, after all), the runtime representation is unspecified. A conformant JSON parser can parse “1” as 1.0. They can be backed by doubles, or singles, or arbitrary precision.

21. WorldMaker ◴[] No.45678227[source]
HTTP/3 replaces parts of OSI layer 4 (TCP/UDP). There's a perspective that HTTP has been trying to lower its layer for decades. Especially the way that TLS originated as HTTPS but now is almost a "universal" vertical chunk of Layers 4, 5, and 6. In that perspective HTTP is probably the new Layer 6.
replies(1): >>45679109 #
22. WorldMaker ◴[] No.45678236{4}[source]
Unless it is HTTP/3.
replies(1): >>45679321 #
23. cyberax ◴[] No.45679069{4}[source]
> That's why there was an entire section for unassigned numbers.

That's also why the majority of OIDs in SNMP are rooted in the 1.3.6 hierarchy, which belongs to the DoD.

replies(1): >>45679312 #
24. jeroenhd ◴[] No.45679109{3}[source]
HTTP/3 uses QUIC as a layer 4 equivalent (which is actually just UDP with extra implications in practice). Not many services leverage standards-compliant QUIC other than HTTP/3, but you don't need to do HTTP to get the same protocol working.

If anything, HTTP/3 running on top of QUIC forced shitty middlebox vendors to de-ossify by permitting any QUIC-based protocol, as they cannot practically distinguish a new HTTP/3 connection from a QUIC connection.

25. miki123211 ◴[] No.45679227{5}[source]
There's something in-between traditional (IP style) packets and traditional (TDM / ISDN style) circuit switching, where you use packets with no bandwidth guarantees, but address them by connection number, not endpoint address.

This still requires you to set up a connection beforehand, but doesn't require you to reserve resources you might not be using.

26. p_l ◴[] No.45679304{5}[source]
And while certain telcos had packet network focused on that (Bell and Datakit, for example), and generally OSI cared about including QoS indicators, CLNS and its specific on-the-wire implementation CLNP is so similar to IP that IPv9 explicitly called it out in the proposal to replace IPv4 with it (because of ISO 160bit addressing)
27. p_l ◴[] No.45679312{5}[source]
SNMP OIDs are in 1.3.6 because TCP/IP stack was once more well known as "DoD Internet"
28. gsich ◴[] No.45679321{5}[source]
or 2.