How IMAP works under the hood

Interesting no attempt has been made to make it at least be less heavy on networked bytes. Especially since it is old and was meant to be used on a connection with no compression or encryption.

HasChildren could have been Parent, HasNoChildren could have been Leaf or Child. And so many more things.

The protocol has ossified and been entrenched. In general more efficient usage of IMAP relies on extensions to the protocol.

A modern replacement (JMAP) hasn't been adopted by major providers.

If you really cared about data transfer size you'd use something like Protobuf.

Networked bytes generally don't matter, networked packets do. Would your proposed change move the needle on that?

It could have, but for stuff you only do once per session that seems excessive. Better to have names that need no explanation, especially for stuff that I think is completely optional.

Despite people always complaining about their perceived inefficiencies, standard protocols like IMAP or XMPP always seem to work on a crappy connection, when most of the modern web doesn't.

Probably wrong context, but the more code I wrote, more I like the these `Has`+Noun style naming than just Noun. Reading `HasChildren` will give you a clearer expectation of what the function would do and return, while `Parent` gives far weaker indication.

Maybe they thought the same when they were designing the protocol.

Also, in the context of email, given the size of each mail (including headers and body), these bytes "waste" maybe insignificant.

IMAP COMPRESS?

> Interesting no attempt has been made to make it at least be less heavy on networked bytes.

Kind of a surprising observation given the first spec (IMAP2) was released in 1998, when dial-up was still a thing:

* https://datatracker.ietf.org/doc/html/rfc1064

IMAP4 was in 1994:

* https://datatracker.ietf.org/doc/html/rfc1730

ITU V.32 gave us 9.6 kbit/s in 1998, and V.34 was 28.8 kbit/s in 1994:

* https://en.wikipedia.org/wiki/List_of_ITU-T_V-series_recomme...

When every new Gmail client ships with an entire web browser embedded to load their hundred megabytes of JavaScript, I think we've long jumped the shark on caring about brevity in the length of information in the protocol itself.

It might have mattered back then but now it would be less than a rounding error.

JMAP is fully supported (since 2019) for Fastmail

https://www.fastmail.com/dev/

Well and you could of course server JMAP over something like Protobuf, JSON is used for ease of use but it's not like thats required feature.

The bigger benefit of JMAP is it's well designed for dealing with deltas and changes and syncing data requiring less roundtripping than IMAP.

Back in the day when I worked at a WISP with 2.4Ghz mountain top antennas the one thing that would work when we were having massive icing issues was email. It was very impressive how it would get through even with massive packet loss and transfer rates measuring in bytes per second.

Jmap was developed by Fastmail, meanwhile all primary FLOSS mail servers do not support it and just new projects like Stalwart are starting to bring it up.

s/1998/1988/

IMAP had its day in the sun, but the advent of big webmail providers (especially gmail) has killed off the advancement of email clients. Now all major development is focused on trying to recreate Gmail to varying degrees of success. It all ends up internal to one or another corporation so they are just all endlessly reinventing the wheel with IMAP just being relegated to a afterthought front end to some sort of search-based backend.

Actually having a email client software running on your machine is extremely nitch and is mostly in the realm of self-hosters and legacy holdouts that won't let their clients go.

A most advanced modern approach is to just use POP3 to download your emails to a local Maildir and have them indexed there non-destructively. And then sync between your various machines that you want your email accessible using some sort of file sync or P2P solution.

I use notmuch for this. It automatically indexes and tags emails and thus enables much more advanced email management solutions then what can be offered over something like IMAP.

The main advantage of this is that 'folders' are managed virtually. There is no shuffling or copying or editing of emails done normally. I only have to worry about backing up my emails and notmuch config as all the rest can be regenerated relatively quickly.

This is more or less replicating what Gmail and other webmail providers do server side.

Where as the traditional approach shuffling and moving and deleting of emails on some imap server is fairly dangerous and expensive operation. Mistakes can lead to data loss and are often very difficult to reverse.

Can confirm that (implemented JMAP). Deltas and asking for the entire mailbox instead of folder by folder is really good. And as a side effect: 1. Less client-side logic for sync. 2. There are many JSON parsers compared to IMAP, making it way easier to use. For example, in C++, you only need a JSON headers-only library, whereas IMAP is meh—only Linux, or clunky usage. Btw, I made one from scratch (IMAP Parser). 3. Goodies like offloading HTTP to the mobile network stack, which supports TLS 1.3 and offline (background) sync, compared to manually extracting certificates from an IMAP connection to validate against the device keychain. It's just 10x easier.

There is an extension that adds compression. [0] I think most widely-used servers implement it (at least Gmail and Dovecot do)

[0] https://www.rfc-editor.org/rfc/rfc4978.html

IMAP 4rev2 merged in a most of the common extensions into the base protocol. Sadly they left out the THREAD, QRESYNC and OBJECTID, though they reference them.

Of course, major providers like Gmail don't support that as well.

It's designed well, from the ground up with a lot of lessons learned from some pretty serious email junkies. Anyone hemming and hawing about JSON being selected as the transport encoding or whatever is just raising silly points. You could do JMAP over protobuf too probably if you cared, but the whole point (IMO) is to get the hell away from unstructured protocols.

Yep, there's always someone ranting about JSON inefficiencies, which I don't observe at all :)

Is this a joke? Both XMPP and IMAP are terrible over an unreliable connection.

oh i mean i rant about it too but its like, a byproduct of the real goal, and since most clients are running some type of webapp it makes sense. you could definitely use something like messagepack or protobuf if you had an extension that negotiates it.

Cyrus supports it. https://www.cyrusimap.org/imap/developer/jmap.html