Step 1: Calculating embeddings from the image base64 strings

As, image strings are stored in csv format so we decided to go with pandas and loaded the csv file into pandas Dataframe. After this we extracted the column of image base64 representation. Now, after getting Image base64 strings we passed the strings to generate_image function which used these strings to generate image array. We then finished out final step to calculate image embeddings which then again stored in csv file with one additional column of image urls.

Step 2: Creating Index tree using embeddings

After getting the embedding of the images, we created index tree using AnnoyIndex from annoy. This index tree will we used to get similar images at the time of APIs calling.

Timeline

1. Investigate the best tools and methods to efficiently compute image similarity at scale
2. Implement a first prototype for large-scale image similarity
3. Iterate on the prototype and implement a public-facing tool

Experiment 1: Model selection for Embedding generation

We mainly want to test EfficientNet against the ResNet50, that is why we only took variants of these models.
Observation: While, doing experiment we observe one strange thing variants of EfficientNet taking much time to load on memory otherwise they are faster in terms of model inference.

Experiment 2: Finding the scalability of AnnoyIndex tree

Experiment 3: Selecting the n_components of PCA for embeddings

To selected the best PCA n_components, we experimented with different models by selecting different n_components. To compare different models with different PCA components we used to matrix
1. class_accuracy and
2. overlap_acc.

Columns of Table

1. name: Name of the model used in the experiment.
2. n_components: It defines the number of principle components we used for PCA.
3. variance_ratio: It is the ratio of the total information to the information represented by principle components of PCA on images.
4. class_accuracy: It the the accuracy of the predicted class of top 10 predicted image of belongs to same class as the source image. It is done for all images in testing dateset.
5. overlap_acc: It is quite interesting, start with assumption that images predicted by tree(annoy) of original images(let's say ideal_list) without any PCA is ideal. Now, images predicted after applying PCA with different n_components, if image present in ideal_list, we will call it Hit otherwise Miss, then sum of all hits divided by total predicted images(we took 100) gave us this accuracy for that n_component of PCA.
6. tree_size: We also keep track of size of tree(annoy) created after applying PCA.

EfficientNetB3V2 PCA Experiment

name	n_components	variance ratio	class_accuracy	overlap_acc	tree_size
EfficientNetB3V2	-1	100.00%	48.42%	100.00%	726.94MB
EfficientNetB3V2	1024	94.47%	48.25%	81.85%	525.67MB
EfficientNetB3V2	512	83.62%	47.96%	80.22%	334.65MB
EfficientNetB3V2	256	73.22%	47.50%	77.42%	240.83MB
EfficientNetB3V2	128	60.28%	46.32%	69.26%	194.31MB
EfficientNetB3V2	64	46.31%	40.97%	52.70%	166.54MB

EfficientNetB4 PCA Experiment

name	n_components	variance ratio	class_accuracy	overlap_acc	tree_size
EfficientNetB4	-1	100.00%	42.33%	100.00%	825.95MB
EfficientNetB4	1024	93.45%	42.32%	85.10%	526.46MB
EfficientNetB4	512	85.37%	42.25%	84.09%	336.73MB
EfficientNetB4	256	74.33%	42.08%	76.84%	241.28MB
EfficientNetB4	128	58.49%	40.48%	67.35%	194.53MB
EfficientNetB4	64	43.50%	34.98%	50.88%	168.76MB

EfficientNetB7 PCA Experiment

name	n_components	variance ratio	class_accuracy	overlap_acc	tree_size
EfficientNetB7	-1	100.00%	41.60%	100.00%	1132.17MB
EfficientNetB7	1024	92.78%	41.69%	85.67%	532.97MB
EfficientNetB7	512	87.51%	41.90%	85.05%	343.51MB
EfficientNetB7	256	76.62%	40.76%	75.54%	250.43MB
EfficientNetB7	128	60.11%	38.07%	63.64%	201.94MB
EfficientNetB7	64	44.96%	36.08%	53.85%	174.78MB

ResNet50V1 PCA Experiment

name	n_components	variance ratio	class_accuracy	overlap_acc	tree_size
ResNet50V1	-1	100.00%	40.51%	100.00%	917.29MB
ResNet50V1	1024	93.57%	40.31%	83.25%	523.87MB
ResNet50V1	512	84.63%	40.10%	80.77%	331.79MB
ResNet50V1	256	73.91%	39.24%	74.97%	238.33MB
ResNet50V1	128	61.83%	38.88%	69.67%	190.20MB
ResNet50V1	64	49.22%	35.58%	55.98%	162.61MB

ResNet50V2 PCA Experiment

name	n_components	variance ratio	class_accuracy	overlap_acc	tree_size
ResNet50V2	-1	100.00%	40.14%	100.00%	916.58MB
ResNet50V2	1024	95.73%	40.25%	82.71%	526.60MB
ResNet50V2	512	86.23%	39.92%	80.06%	334.51MB
ResNet50V2	256	74.25%	39.01%	73.46%	239.71MB
ResNet50V2	128	60.57%	38.21%	66.84%	192.16MB
ResNet50V2	64	46.84%	35.47%	55.34%	164.16MB

Experiment 4: End-to-End(in reference to Time) testing of models

In this experiment what we want to test, is mainly timing of the model. Like how much time it will take to generate result. In this experiment time is calculated from passing to url to getting the recommendation of similar images.

Experiment 5: accuracy of selected model with different classes

After selecting the model we decided to do one class accuracy check with model.

After seeing these low accuracy of classes, we were confident that there is something wrong class labels, to confirm this we perform one more experiment Experiment 6

Experiment 6: Testing model with critical classes(having low accuracy)

These are some sample images of this experiment where we can easily observe that although images are similar to each other content-wise but they are present under different categories.

Above (source) image belongs to Food category, notice last three images contains food but they belongs to different category which was the cause of low score in preview experiment.

This is second sample of recommendation (source)image belongs to House category.

1. PCA generation for mapping embeddings to [256,] vector

When we were starting with this step there were mainly two choices available -
1. pyspark.ml.feature.PCA
2. sklearn.decomposition.IncrementalPCA
and both were have there problems -
The problem with pyspark.ml.feature.PCA was that it required spark context everytime even at the time of inference which was not possible in our case. And, the problem with sklearn.decomposition.IncrementalPCA was that it can't be distributed with spark.
We choose sklearn.decomposition.IncrementalPCA because we could fill it with batches of data, which will reduce the spark efficiency but it would not be required spark context at the time of inference which was required in our case.

# Created the instance of PCA with 256 components
pca = IncrementalPCA(n_components=256, batch_size=50000)

# this function will fill the batches of data to pca
def fill_pca(df):
    global pca
    df_list = df.collect()
    features = []
    for row in tqdm(df_list):
        features.append(list(row['features']))
    arr = np.array(features)
    pca = pca.partial_fit(arr)

These are the part of code which is used to fill pca with required data batch by batch

with open('pca256.pkl', 'wb') as f:
    pickle.dump(pca, f)

At the end we just dumped the pca instance for using it later for PCA generation of embeddings.

2. Index to URL map file generation

OK, we had the small problem is that annoy will return index of the matched embeddings of image, but the limitation with annoy was that only integers can be used as index but we needed URL of the matched image.
SO, to solve this problem we came up with a map which will accept index of the image and return URL of the image. To generate this map we used following code -

tree = AnnoyIndex(256, 'angular')

# will work as map file for index->URL
d = dict()

ite = 0
def fill_tree(df, ite):
    df_list = df.collect()
    # following will iterate batch row by row
    for row in tqdm(df_list):
        data = np.array(list(row['features'])).reshape(1, -1)

        # this will map image_url from index  
        d[ite] = row['image_url']

        tree.add_item(ite, pca.transform(data).reshape(-1))
        ite += 1
    return ite

At last we dumped the map file to use at the time of inferencing the API.

with open('./idx2url.pkl', 'wb') as f:
    pickle.dump(d, f)

3. Annoy tree generation for recommendation

After generating pca and idx2url map, it was time to complete the final step of generating annoy tree for making recommedation of similar image. Nothing much left to tell about annoy as we discussed every thing about this in this document, we will direct jump to code we use to fill embedding to tree:

# this is a instance of annoy tree
tree = AnnoyIndex(256, 'angular')
d = dict()
ite = 0

# this function will fill embeddings to tree row by row
def fill_tree(df, ite):
    df_list = df.collect()
    for row in tqdm(df_list):
        data = np.array(list(row['features'])).reshape(1, -1)
        d[ite] = row['image_url']
        
        # notice we are adding embeddings to tree after applying pca on it 
        tree.add_item(ite, pca.transform(data).reshape(-1))

        ite += 1
    return ite

In order to understand why 100 is passed in build method, you can refer to following paragraph which was taken from the annoy github page.
a.build(n_trees, n_jobs=-1) builds a forest of n_trees trees. More trees gives higher precision when querying. After calling build, no more items can be added. n_jobs specifies the number of threads used to build the trees. n_jobs=-1 uses all available CPU cores.

We saved tree to use it in API

tree.build(100)
tree.save('./tree.ann')

4. Testing everything till now, before moving further

# We took url of image
url = 'https://www.malaysia-today.net/wp-content/uploads/2019/11/Taj-Mahal-Agra-India.jpg'
# downloading image data form URL
img_data = urllib.request.urlopen(url).read()
# image data to image conversion
image = generate_image(img_data)

# first we passed image to model(efficientNetB3)
features = model.predict(np.expand_dims(image, axis=0))
# here we are applying pca to embeddings
feature_pca = loaded_pca.transform(features)

# using tree to generate index of similar images which then mapped to image_url
res = [d[i] for i in tree.get_nns_by_vector(feature_pca.reshape(-1), n=10)]