Self-modifying executable with 3-layer substitution cipher hiding the key. GDB past the self-modifying executable and create a lookup table for the flag.
Original writeup (https://advancedpersistentjest.com/2018/04/09/writeups-hastad-triptych-uiuctf/).