TL;DR: We have a binary executable for custom virtual machine, and IDA processor module for this VM. By reverse engineering processor module we can recover information about VM instruction set. After that, we reverse engineer the binary and conclude that we need to run it somehow to get a flag. Since used algorithms are very inefficient, we reimplement binary logic in some programming language while performing optimizations and run the code to get a flag.