Rating:

Request /robots.txt with the User-Agent header set to GoogleBot.

Original writeup (https://www.nevivur.net/writeups/2019/02/tamuctf-19/#robots-rule).