Tags: misc re
Rating:
The villains are communicating with their own proprietary file format. Figure out what it is.
$ nc proprietary.ctfcompetition.com 1337
The attachement is a file named flag.ctf - looking at it in a hex editor I saw four magic bytes GCTF at the
start, then little endian 32 bit values corresponding to 600 and 400 - so I assumed it was some kind of
picture format.
At first I completely missed that there was a network address in the challenge description, so I spent a lot of
time doing analysis of the content of the file. It seemed like some kind of stream format - each "element" was a
byte in the range 0x00-0x0f, where all elements except 0x0f was followed by a 3-byte value. Interpreting
these 3-byte values in isolation as RGB pixels gave some garbled pastel colored gradients, which suggested that
the picture format assumption was correct.
Due to lack of progress I dropped the challenge until near the end of the contest, when someone made me aware of
the network address. After a bit of error message guided fuzzing, I found out that the service first required
three lines each terminated by LF, first the format specifier P6, then the width and height of the picture,
like 320 240, and finally some unknown value that always had to be 255. Then it read in width*height raw RGB
pixel values, and in return it sent the picture encoded in the proprietary format.
With a reference encoder, I finally made some progress. I quickly realized that it was some kind of 2D based lossy encoding, but couldn't figure out the specifics before time ran out.
Some days after the contest I realized what was missing from the puzzle and managed to write a decoder. First,
find the largest power of 2 square that can fit the entire picture - in the case of flag.ctf this is a
1024x1024 pixel square. Then, split the picture into quadrants, and read the first "command" byte. Here, each
bit corresponds to a512x512 pixel quadrant:
1 2
4 8
where a set bit means that you should recursively parse that quadrant, and a cleared bit means that you should
paint the entire quadrant with the color following after the command byte. (Since 0x0f means recursively parse
all quadrants, no color follows it.)
As an example, this is the format of the sample file, excluding the 12 byte header:
03 000000 [upper left quadrant data] [upper right quadrant data]
This means that the definition of the upper left 512x512 block comes first, then the definition of the upper
right block. The bottom blocks are painted entirely black, as that's the color following the command. As the
original picture was only 400 pixels high, this makes sense.
With this information, I was able to write a picture decoder, and finally exctract
the flag CTF{P1c4Ss0_woU1d_B3_pr0UD}
