The Single Byte That Kills Your Exploit: Understanding Endianness

What first confused me about endianness is that it is about byte order, not bit order. The latter would have seemed more logical, or is this just me?

This is a weird take. I've never put together this kind of exploit, but still I know enough to not buy this. Do people ever really craft exploits that are perfectly valid except for using the wrong endianness?

> If you’ve ever crafted a perfect shellcode and ROP chain only to have your exploit immediately crash with a SIGSEGV(a signal sent by the operating system to a program when it attempts to access a protected or invalid memory location) or EIP(a 32-bit CPU register in the x86 architecture that holds the memory address of the next machine instruction to be executed) pointing to garbage, you’ve likely met the silent killer of beginners: Endianness.

Aren't there a million other ways to get addresses wrong?

> Using x86/x86_64 gadgets and packers on a MIPS/PowerPC target (different endianness and instruction set) will not work.

"and instruction set" is carrying a lot of weight here.

This isn't like a coin flip thing: even considering architectures with configurable endianness, in 2025 it's overwhelmingly likely both host and target are little-endian. And on old, big-endian platforms, that's just one of many things you have to get right.

Learning this initially was confusing for me too, aren't we arranging bits?

it does seem like the audience the article is explicitly targeted for is an edge case. people who understand enough to be writing an exploit but are somehow unaware of their target architecture works.

but i guess the real target audience is probably people that are just starting out on CTFs and just trying to string stuff together without a proper understanding of the fundamentals. everyone has to start somewhere and i guess if people are just using packers and tools to generate exploit code then its quite easy to use the wrong flags and not know what is going on.

Little endian does appear strange at first, but if you consider the motivation it makes a lot of sense.

Little endian's most valuable property is that an integer stored at an address has common layout no matter the width of the integer. If I store an i32 at 0x100, and then load an i16 at 0x100, that's the same as casting (with wrapping) an i32 to an i16 because the "ones digit" (more accurately the "ones byte") is stored at the same place for both integers.

Since bits aren't addressable, they don't really have an order in memory. The only way to access bits is by loading them into a register, and registers don't meaningfully have an endianness.

You can't address individual bits. There is no way of telling if the LSBit is "left" or "right" of the MSBit. So endianness can't be about that.

For bytes, you can distinguish them, as you can look at the individual bytes produced from a larger-than-byte store.

> Disclaimer: This article was written with AI assistance, for a bit of brainstorming and proofreading.

I suspect the target audience is "whoever will subscribe on Substack" more than someone who has ever written or contemplated writing shellcode. I'm seeing more and more articles like this that focus the prose on some weird subset-of-a-niche aspect of a subject, then end with a set of bullet points for fixing the problem as if this is something one regularly encounters.

Your CPU (probably) has left and right variants for shift and rotate operations, which is certainly an avenue for confusion. There's a "logical" bit order that these operations follow, which starts with the MSBit and ends with the LSBit, even when the physical connections are all parallel and don't really define a physical bit order.

People learn things in different orders. For many people, low-level CTF challenges are their introduction to computer architecture (a good way to learn if you ask me!) If so, endianness is a novel concept to them.

While I personally learned about endianness before writing my first exploit, I've definitely made endianness-related mistakes before.