Encoder pipeline order: why your second pass breaks the first

Encoder stacks are seductive. You fix nulls, then fix newlines, then fix uppercase, and suddenly your decoder stub is half the payload. I have seen more failed labs from encoder order than from wrong syscall numbers.

Encoders change more than bytes

Each pass typically adds:

• a decoder stub that mutates memory at runtime
• register pressure on x86/x64
• alignment requirements you did not have in the raw shellcode

If your exploit lands with a misaligned stack, a decoder that worked in isolation still faults.

Order matters because constraints compound

Think of encoders as filters:

1. Start from raw shellcode that matches OS/arch/payload
2. Apply the hardest bad-char filter that reflects the delivery channel
3. Apply size-shaping passes only if you still fit the buffer

Reversing that order often means pass two undoes pass one's guarantees. Example pattern I see: xor encoder removes nulls, then a formatting export reintroduces ASCII parsing issues when someone wraps output in quotes incorrectly. The encoder did its job. The human undid it.

Size budget is a first-class input

Before stacking encoders, write the max size next to the max bad-char list. If the decoder plus encoded body exceeds the overflow length, no order is correct.

I measure:

• raw length
• length after each pass
• offset where execution must begin (entry point is often the decoder, not the original shellcode)

Export format is part of the pipeline

C array, Python bytes, raw hex, and escaped byte strings each have footguns. A perfectly encoded blob pasted into a broken C snippet introduces null terminators via string literals. That is not the encoder's fault.

Pick export format based on the injector, not based on what looks nice in blog screenshots.

Debugging stacked passes

When a stacked payload fails:

1. Execute only the decoder plus first N bytes in a controlled debugger
2. Confirm the decoder loop terminates (watch for infinite xor loops on wrong lengths)
3. Compare memory after decode against known-good raw shellcode hash

Hashes beat vibes.

sha256sum payload.bin

When to stop stacking

If you need four encoders, ask whether the engagement should use staging: a tiny loader reads the rest from a file descriptor you control. Stacking is not a virtue. It is debt.

Tooling note for shellcodes users

The builder UI exposes encoder pipelines visually. Use that to snapshot what you used for a given runbook entry. Future you will not remember whether shikata_ga_nai was before or after the custom filter unless you wrote it down.