tl;dr: See the `encryptFlag` function in Ghidra, try possible candidates for decryptions of the individual characters.

Original writeup (https://dunsp4rce.github.io/rgbCTF-2020/pwn-rev/2020/07/15/Advanced-Reversing-Mechanics-2.html).