Tags: shellcoding assembly 

Rating: 3.4

TLDR;
1) Find out that the program is computing a recursive function inefficiently (the function has overlapping subproblems, fibonacci-like)
2) Get an overview of the recursive function (the parameters changing between the calls)
3) Perform memoization from a shellcode in assembly, by patching the function prologue and epilogue, and caching the computed results (without paying attention to the details of how the function works)

Original writeup (https://red0xff.github.io/writeups/volgactf_fhash/).