Rating:

This was a very tricky chall. It all starts from just one line which gives the files their names.

masked_file_name = "".join([chr((((c - ord("0") + i) % 10) + ord("0")) * int(chr(c) not in string.ascii_lowercase) + (((c - ord("a") + i) % 26) + ord("a")) * int(chr(c) in string.ascii_lowercase)) for c, i in zip([ord(a) for a in password], range(0xffff))])

which can be also written as:

for c, i in zip([ord(a) for a in password], range(0xffff)):
if(chr(c) not in string.ascii_lowercase):
s+=chr(((c - ord("0") + i) % 10) + ord("0"))
else:
s+=chr(((c - ord("a") + i) % 26) + ord("a"))

From here we can notice that for ascii-lowercase it is applying caesar shift to each character. Non-ascii charcters are almost not recoverable.
Running the script given below will give us many decoded passwords.

import glob
def decode(enc):
rev=""
for c, i in zip([ord(a) for a in enc], range(0xffff)):
rev+=chr((c-97-i+26)%26+97)
return rev

lis=glob.glob("*.enc")
for i in lis:
try:
decode(i)
except:
pass

Now, I had all the passwords. I noticed that they are first 400 strings from the wordlist rockyou.txt.
After recovering all the passwords, we have to xor them with their contents and somehow get the key. One important thing to notice in the key is that the key is '8'*<8th character of the flag> which implies that for each password array-S is same for each of its character.
It took me almost 5-6 hours to write a perfect script coz there are many missing files in the extracted folder.

import string
import glob
lis=glob.glob("*.enc")
lis.sort()
def final(l):
for ke in range(256):
S = [i for i in range(256)]
j = 0
out = bytearray()

for i in range(256):
j = (j + S[i] + ke) % 256
S[i] , S[j] = S[j] , S[i]

i = j = 0
temp=0
for t in range(len(l)):
i = ( i + 1 ) % 256
j = ( j + S[i] ) % 256
S[i] , S[j] = S[j] , S[i]
if(S[(S[i] + S[j]) % 256] ==l[t]):
temp=1
else:
temp=0
break
if(temp==1):
print chr(ke),

def rc4(text, s):
l=[]
for i in range(min(len(text),len(s))):
l.append(ord(text[i]) ^ ord(s[i]))
final(l)

def main():
for i in range(420):
for j in range(len(lis)):
if(lis[j].split('_')[0]==str(i)):
#print lis[j].split('_')[0]
try:
rc4(f[i], g)
except:
pass

if __name__ == "__main__":
main()

which gives us the output as:
ysnnltu_idr_aoug_iy_fptd_ics_htaopidr_aoug_iy_fptd_ics_htaopps}
ysnnp{idtsltu_idr_aoug_iy_fptd_ics_htaopps}
ysnnp{idtsltu_idr_aoug_iy_fptd_ics_htaopps}
ysnnp{idtsltu_idr_aoug_iy_fptd_ics_htaopps}
ysnnp{idtsltu_idr_aoug_iy_fptd_ics_htaopps}
ysnnp{idtsltu_idr_aoug_iy_fptd_ics_htaopps}
ysnnp{idtsltu_idr_aoug_iy_fptd_ics_htaopps}
ysnnp{idtsltu_idr_aoug

But it's still not the flag, because the key has been made using every 8th character of the flag. At this point we are not sure of the flag's length but do have a rough guess of length to be around 43. So, I wrote a bruteforce script to determine the length and the flag.

s="ysnnp{idtsltu_idr_aoug_iy_fptd_ics_htaopps}"*100
for l in range(10,100):
f=list("0"*l)
k=0
for i in range(len(s)):
f[k%l]=s[i]
k+=8
print l, ''.join(f)

which gives us the flag: flag{crypto_is_stupid_and_python_is_stupid} at len=43