Long story short: we used frida to patch the code and let it compute the solutions. You can find all the details by following the provided link (https://zenhack.it/writeups/XmasCTF2018/catana/)