Rating:
Flagsifier
----------
This challenge consisted in finding the correct input for a Deep Neural Network
that classifies images (size: `1064x28`) and has 40 output neurons.
There were some example images, composed of 38 letters from what looked like the EMNIST dataset.
All of them activated the fourth neuron, therefore being classified as the fourth class.
Some quick tests by moving around random letters and removing some others (plus, the structure
of the network) hinted us that there was a softmax and the classes were represented as one-hot
encoding. Therefore, the network classifies images into 40 classes. Time to discover what they are!
So, at first we transcribed the sample images and used the combination
of tile + corresponding text as dataset.
```python
dataname=["RUNNEISOSTRICHESOWNINGMUSSEDPURIMSCIVI",
"MOLDERINGIINTELSDEDUCINGCOYNESSDEFIECT",
"AMADOFIXESINSTIIIINGGREEDIIVDISIOCATIN",
"HAMIETSENSITIZINGNARRATIVERECAPTURINGU",
"ELECTROENCEPHALOGRAMSPALATECONDOIESPEN",
"SCHWINNUFAMANAGEABLECORKSSEMICIRCLESSH",
"BENEDICTTURGIDITYPSYCHESPHANTASMAGORIA",
"TRUINGAIKALOIDSQUEILRETROFITBIEARIESTW",
"KINGFISHERCOMMONERSUERIFIESHORNETAUSTI",
"LIQUORHEMSTITCHESRESPITEACORNSGOALREDI"]
```
(Finding typos in this transcription is left as an exercise to the reader
:smile: )
After that we divided all the sample images in 38 28x28-tiles (one tile per
letter).
We have done that using this script:
```python
dataset=[]
datalet=[]
datax={}
for i in range(0,8):
img = Image.open('./sample_%d.png'%i)
for j in range(0,38):
x=img.crop((j*28,0,(j+1)*28,28))
dataset.append(x)
datalet.append(dataname[i][j])
let=dataname[i][j]
if let not in datax:
datax[let] = []
datax[let].append(len(dataset)-1)
```
* `dataset` contains the images.
* `datalet[i]` contains the corresponding text of `dataset[i]`.
* `datax` contains the mapping between letters and array of samples. Basically
it answers to questions like: "which dataset entries correspond to a particular
letter?"
Then, we experimented as follows: for each letter, starting from a black image,
put the letter in position 0...38, and classify these images. We saved all the
predictions, and then averaged them to see the most likely class for each letter.
We discovered that neurons 14...40 clasified letters: neuron 14 activated for A,
neuron 15 for B, up to neuron 40 for Z.
We then need to discover what the neurons 1...14 classify, as some of them probably
classify the flag.
To do that, we need to try to find inputs that maximize the activation of these, one at
a time. Another thing that we can leverage is that the flag likely starts by `OOO`.
So, what would one usually do here, with a "real" network? Decide which neuron (e.g.,
the first) to try to activate, then create random 38-letters inputs, and then use the
log-likelihood of that neuron for that input as the fitness function for his favourite
optimization algorithm (e.g., this problem looked perfect for genetic algorithms).
But before throwing cannons to the problem, let's try something simpler (and if it fails,
move to more advanced but computationally intensive stuff).
The suspect here is that the network is trained on a small dataset, and is strongly
overfitting the flag on some of the "low" neurons. This could maybe mean that
the function we need to optimize is not crazily non-linear and with tons of local optima
that require complex optimization algorithms to escape from. Therefore we tried with
a simple greedy strategy: for each of the 38 positions, pick the letter that maximizes the
output of the target neuron. And it worked!
Trying `OOO` as a test string showed activation of neurons 2 and 3 - let's focus on them.
### Results
Neuron 2 has been our first guess, which gave us these (highly noisy) strings,
with the greedy strategy:
```
OOOOTHISISBYKCOZMEKKAGETONASTEYOUATIMW
OOOOTHISISBYKCOZMKYKAGZTONBSTWVOUATIWM
OOOOTNISISBDKCOZMKSGBGETONMSTXVOUWTIRR
OOOOTOISISOYECOIUEYSOGETONOSTNVOUOTIWW
```
We tried (failing) to submit some flags like
* `OOOOTHISISBYKCOZMESSAGETOHASTEYOLATINE`
* `OOOOTHISISBYKCOZMESSAGETOHASTEYOLATINW`
* ...
After realizing that neuron 2 was just a test-neuron, we changed output neuron
from 2nd to 3rd and we got sentences like:
```
OGOSOMEAUTHTNTICINTEIXIGXNCCISVEGUIWEG
OOOSOMRAUTHGNTICINTGIIIGGNGGISMRGUIWEG
OOOSOMXAUTHENTICINTEKXIGXNCRISRRRRIRER
OOOSOYEOLTUTNTICINTEIIIGCNCEIIETOLIRTI
RROSOMEAUTHTNTICINTEIXIGXNCCISVEGUIWEG
```
We obtained `OOOSOMEAUTHENTICINTELLIGENCEIS........` by averaging (and manually
correcting) them and after few tries of guessing ( :bowtie: ) the last word we
obtained the correct flag: `OOOSOMEAUTHENTICINTELLIGENCEISREQUIRED`.
### Python script
You can find here the full python script we have used (keras + tensorflow-gpu):
```python
#!/usr/bin/env python
import numpy as np
from keras.models import load_model
from keras.preprocessing import image
from keras.datasets import mnist
from keras.applications.resnet50 import preprocess_input, decode_predictions
from PIL import Image, ImageDraw, ImageFont
import string, random, sys
dataset=[]
datalet=[]
datax={}
dataname=["RUNNEISOSTRICHESOWNINGMUSSEDPURIMSCIVI",
"MOLDERINGIINTELSDEDUCINGCOYNESSDEFIECT",
"AMADOFIXESINSTIIIINGGREEDIIVDISIOCATIN",
"HAMIETSENSITIZINGNARRATIVERECAPTURINGU",
"ELECTROENCEPHALOGRAMSPALATECONDOIESPEN",
"SCHWINNUFAMANAGEABLECORKSSEMICIRCLESSH",
"BENEDICTTURGIDITYPSYCHESPHANTASMAGORIA",
"TRUINGAIKALOIDSQUEILRETROFITBIEARIESTW",
"KINGFISHERCOMMONERSUERIFIESHORNETAUSTI",
"LIQUORHEMSTITCHESRESPITEACORNSGOALREDI"]
for i in range(0,8):
img = Image.open('./sample_%d.png'%i)
for j in range(0,38):
x=img.crop((j*28,0,(j+1)*28,28))
dataset.append(x)
datalet.append(dataname[i][j])
let=dataname[i][j]
if let not in datax:
datax[let] = []
datax[let].append(len(dataset)-1)
def genImg(n):
img = Image.new('1', (1064,28), color='black')
#for i in range(max(0,len(n)-1),len(n)): # only this letter and everything else black
for i in range(0,len(n)):
img.paste(dataset[n[i]], (i*28,0))
return img
model = load_model('model.h5')
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
def eeval2(a, op):
img = genImg(a)
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
classes = model.predict(x)
score = float(classes[0][op])
return score
for oo in range(2,40): # start from 2, this one seems correct
# out=[datax[x][0] for x in 'OOOSOMEAUTHENTICINTELLIGENCEIS'] #do not start from zero
out=[]
for i in range(len(out),38):
maxv=([], -99999)
for j in datax:
for k in datax[j]:
out.append(k)
score = eeval2(out, oo)
if score > maxv[1]:
maxv = (0, score, j)
out.pop()
sys.stdout.write("[%d] %38s : %.10lf \r" % (oo, ''.join([datalet[x] for x in out]), maxv[1]))
sys.stdout.flush()
out.append(datax[maxv[2]][0])
print("")
print("--Neuron %d: %s" % (oo, ''.join([datalet[x] for x in out])))
```
