Reported by @oakkaya
If a decode hook (object_hook / object_pairs_hook / list_hook / ext_hook) calls .feed()
on the same Unpacker while it is still unpacking, append_buffer() may reallocate
(PyMem_Free) the internal buffer that the in-progress unpack_execute() is still reading from.
When the hook returns, the parser keeps reading the remaining bytes from the freed buffer →
use-after-free. On a stock build this is a hard crash (SIGSEGV).
The application supplies the (re-entrant) hook; the attacker controls the bytes, which decide when
the hook fires and how large the re-entrant feed grows the buffer.
Reproduction
import struct
from msgpack import Unpacker
up = None
def ext_hook(code, data):
# re-entrant feed on the SAME unpacker, large enough to force a buffer realloc
up.feed(b"\xc0" * 8_000_000)
return 0
up = Unpacker(ext_hook=ext_hook, max_buffer_size=64 * 1024 * 1024)
# array(200): [ ext (fires the re-entrant hook), then 199 more elements ]
up.feed(b"\xdc" + struct.pack(">H", 200) + b"\xd4\x05A" + b"\x2a" * 199)
for _ in up: # SIGSEGV
pass
Under ASan:
ERROR: AddressSanitizer: heap-use-after-free READ of size 1
#0 unpack_execute msgpack/unpack_template.h:162
freed by: PyMem_Free <- Unpacker.feed -> append_buffer
Root cause
Unpacker._unpack runs the parser over the internal buffer:
ret = execute(&self.ctx, self.buf, self.buf_tail, &self.buf_head)
unpack_execute keeps local p / pe pointers into self.buf. A decode hook is invoked from
inside execute (at map/array end or for ext). If the hook calls up.feed(...),
append_buffer (_unpacker.pyx) reallocates the buffer:
new_buf = <char*>PyMem_Malloc(new_size)
...
memcpy(new_buf, buf + head, tail - head)
PyMem_Free(buf) # <-- frees the buffer the outer execute() is reading
After the hook returns, unpack_execute continues reading from the now-freed p/pe.
(The same applies to a file_like.read() that re-enters feed()/unpack(); the unpacker is not
re-entrant but does not guard against it.)
Suggested fix (verified)
Add a re-entrancy guard so a buffer-mutating call during an active parse fails cleanly instead of
corrupting memory. Set a flag around the execute(...) call and reject feed() while it is set:
# field: cdef bint _in_exec (init False in __init__)
def feed(self, next_bytes):
...
if self._in_exec:
raise RuntimeError("Unpacker.feed() called re-entrantly during unpacking")
...
# in _unpack, around the execute call:
self._in_exec = True
try:
ret = execute(&self.ctx, self.buf, self.buf_tail, &self.buf_head)
finally:
self._in_exec = False
Verified: the PoC now raises RuntimeError instead of crashing (clean under ASan), and normal
streaming (feed() between objects), iteration, and the file_like path are unaffected
(read_from_file calls append_buffer between execute() calls, where the flag is not set, so
there is no false positive). A broader guard could also reject re-entrant unpack()/skip().
Reported by @oakkaya
If a decode hook (
object_hook/object_pairs_hook/list_hook/ext_hook) calls.feed()on the same
Unpackerwhile it is still unpacking,append_buffer()may reallocate(
PyMem_Free) the internal buffer that the in-progressunpack_execute()is still reading from.When the hook returns, the parser keeps reading the remaining bytes from the freed buffer →
use-after-free. On a stock build this is a hard crash (SIGSEGV).
The application supplies the (re-entrant) hook; the attacker controls the bytes, which decide when
the hook fires and how large the re-entrant feed grows the buffer.
Reproduction
Under ASan:
Root cause
Unpacker._unpackruns the parser over the internal buffer:unpack_executekeeps localp/pepointers intoself.buf. A decode hook is invoked frominside
execute(at map/array end or for ext). If the hook callsup.feed(...),append_buffer(_unpacker.pyx) reallocates the buffer:After the hook returns,
unpack_executecontinues reading from the now-freedp/pe.(The same applies to a
file_like.read()that re-entersfeed()/unpack(); the unpacker is notre-entrant but does not guard against it.)
Suggested fix (verified)
Add a re-entrancy guard so a buffer-mutating call during an active parse fails cleanly instead of
corrupting memory. Set a flag around the
execute(...)call and rejectfeed()while it is set:Verified: the PoC now raises
RuntimeErrorinstead of crashing (clean under ASan), and normalstreaming (
feed()between objects), iteration, and thefile_likepath are unaffected(
read_from_filecallsappend_bufferbetweenexecute()calls, where the flag is not set, sothere is no false positive). A broader guard could also reject re-entrant
unpack()/skip().