The picture from the hint shows an interpreter, very similar to sh. Trying send sh by first message and got it.
Without a hint, try the overflow, it doesn't work, try the format strings, it doesn't work, so most likely there is some kind of handler on the server, we try various commands in the shell calling style.