The technique used in the end is Filippo Valsorda's CVE-2016-1494, by way of Bleichenbacher's 2006 attack (low public exponent allows signature forging).

Original writeup (http://karabut.com/google-ctf-2017-quals-rsa-ctf-challenge-writeup.html).