THE GARDEN

Self-Modifying Shellcode

Updated 13 Sep 2026

• •

  What it is

  • •

    Shellcode that patches instruction bytes in memory before executing them. This can avoid an input-time opcode filter when the patched location remains writable and executable.

• •

  How it works

  • •

    Reserve a placeholder (e.g. two NOPs) at a known, RIP-relative address.

  • •

    Write an intermediate value into that address that is itself filter-safe.

  • •

    Apply arithmetic that turns the intermediate value into the forbidden bytes, in place, right before execution reaches them.

  • •

    Before patch (address labeled `open_syscall`):
      [ 90 90 ]              ← NOP NOP placeholder, filter-safe
    
    Patch instructions:
      mov word ptr [rbx], 0x0610
      sub word ptr [rbx], 0x0101      ; 0x0610 - 0x0101 = 0x050F
    
    After patch (little-endian bytes at same address):
      [ 0F 05 ]              ← this is the `syscall` opcode
    

  • •

    Execution then falls through into the now-patched bytes and runs them.

• •

  When to use it

  • •

    Filter scans the delivered bytes for specific opcode sequences (not just data bytes) — e.g. "no syscall/sysenter/int allowed."

  • •

    The location being patched must be writable, and it must be executable when control reaches it. See NX-DEP (No-eXecute).

  • •

    If only part of the buffer stays writable, put the bytes being patched beyond that boundary. The patching instructions themselves only need to be executable.

• •

  Examples

  • •

    Write 0x0610 to a two-byte placeholder, then subtract 0x0101. The result is 0x050f, stored as the 0f 05 syscall bytes.

• •

  pwntools

  • •

    patch = asm('mov word ptr [rbx], 0x0610; sub word ptr [rbx], 0x0101')
    # rbx must point to the writable placeholder.
    

• •

  Debugging

  • •

    x/2xb $rbx  # Inspect the placeholder before/after the patch.
    stepi
    

• •

  Variants

  • •

    Can be extended to patch entire multi-byte instructions, not just a single opcode, and combined with a NOP-sled when the landing address isn't precisely predictable.

• •

  Related

  • •

    Shellcode

  • •

    Bad Character Filtering Bypass

  • •

    NX-DEP (No-eXecute)

• •

  Cards

  • •

    How can shellcode execute an instruction whose opcode a scanner blocks?

    • •

      Write the instruction's bytes into writable+executable memory at runtime (e.g. via mov + sub) so the forbidden bytes only exist after execution begins, then fall through into them.

  • •

    If only part of the shellcode buffer stays writable, where must a self-modifying patch live?

    • •

      The placeholder must remain writable and later executable. The patching instructions may live elsewhere in executable memory.

◌ Explore connections in Graph view