[link to original writeup](https://github.com/babaiserror/ctf/tree/main/%5B210813-16%5D%20ReallyAwesomeCTF%202021#Packed-PwnReversing-350-pts)

General gist of it:

The exe file "unpacks" data within it by subtracting 5 and xoring 0x80 to every bit in some section of 0x3000 bytes. With these bytes, the .exe file creates another .exe file in the temp directory and executes it, where the actual checking of license key happens.

The executed exe file has a basic debugger check by calling `IsDebuggerPresent` and also its own debugger check, by calling

```
try {
asm{
pushfd
or [esp], 0x100
popfd
nop
}
} except (EXCEPTION_EXECUTE_HANDLER) {
exceptionFlag = true
}
```

which prevents you from proceeding if you're in a debugger going through the instructions step by step. (Of course that code isn't given, and I didn't know before I saw the source code either)

Without the debugger, the code takes the output from `IsDebuggerPresent` and the input name, and generates a license key, which is compared to the input license key.

Solving this involves using a debugger (I used x64dbg), and setting breakpoints right after `IsDebuggerPresent` to set `eax` to 0, and after the `vsprintf` call to see the generated license key. After changing the value of `eax`, just run through it and don't step through individual instructions.

More specifics in the link above.