[YauzaCTF - 2019] Crypto challenge: enJoy!

Task

Solution

The challenge gives us a .rtf (Rich Text Format) file. I don't know much about this file type. First open this file with Word Editor, it looks like this:

This site suggests convert to HTML file and view with browser. So I did, and I got a Base64 encoded text (Note: the encoded text has been shortened for readable):

Decode the above text with Base64 decoder, I got the second Base64 encoded text (Note: the encoded text has been shortened for readable):

Decode again, now I got a string of hexadecimal (Note: the text has been shortened for readable):

Using Hex to Text converter, I got a substitution ciphertext:

Using Substitution solver to decrypt it:

So, Tupper's self-referential formula right? This formular visually represents itself when graphed at a specific location in the (x, y) plane. From here, we got the formula:

Maybe the very long number which represented in binary string we have received from the plaintext is the given number N for the formula. Then I found this site which will draw the graph according to given N using Tupper's self-referential formula. Using the given binary string, here is the flag:

Original writeup (https://gitlab.com/145giakhang/playing-ctf/tree/30682f8fd2ec1408a8ba6cf23cda1755da3561d1/CTF-events/2019/YauzaCTF/Crypto/Enjoy).