I decided to put together my own class to do the job, starting with an easy, low-overhead way to move bytes in and out of arbitrary buffers. Along the way, it picked up useful bits and pieces, such as buffer structures and allocators that made the byte shuffling faster, often being able to do it with zero allocations and zero copies. Safety features came along to make sure that malicious packet data or mistakes in the code wouldn't result in segfaults or vulnerabilities.
It's become useful enough to me that I've packaged it up in its own standalone library on the chance that it might be useful to others. It has zero dependencies other than the standard library and has been designed for quick integration into any project within minutes, or seconds with a copy paste of the amalgamated header. It can be used in production code but it's also ideal for for those that want to quickly hack away at binary data with minimal fuss.
I'd like to read an even more thorough overview of how it works and all the gotchas before I'd consider using this 'in production' but the API looks very easy to use and very elegant.
EDIT: just hit the section on portability, seems like you would always have to use that API, yeah? I feel like when you are writing network code you simply have to make it portable from the get-go. I guess I'm always thinking about having it run on client machines.
You'd likely want to always use that API (or layer something on top of it) unless you're in control of both ends and know they were built with the same toolchain & settings. One area where I've skipped over it is by writing a basic code gen tool (albeit unfinished as most personal projects) that generates the serialisation functions at compile-time from a very basic DSL that describes the network structures (of a game protocol I don't control). If it detects that the current toolchain is going to generate a binary-compatible struct layout and there aren't any variable length fields in there (no strings, basically), it'll generate a memcpy (via using get/put on the stream) rather than per-field (de)serialisation. If it can guarantee alignment of the buffer, which is a tougher requirement to meet, it'll give you a view directly into the network buffer so you effectively have zero-overhead deserialisation. Very much a work in progress but there's scope for making things quite efficient with just a few basic building blocks.