Search photos by description, and why it plateaus

How a sentence gets compared to a photo with one matrix multiply, how to read the scores, and why text search plateaus.

You have an index. Now you can search it with a sentence. This guide covers how that comparison works, how to read what comes back, and the specific limitation that motivates everything in the next guide.

The search itself#

# search.py
import os
os.environ.setdefault("HF_HUB_OFFLINE", "1")

import sys, json
import numpy as np
import torch
import open_clip

model, _, _ = open_clip.create_model_and_transforms(
    "ViT-B-32", pretrained="laion2b_s34b_b79k")
tok = open_clip.get_tokenizer("ViT-B-32")
model.eval()

vecs  = np.load("vecs.npy")
paths = json.loads(open("paths.json").read())

query = " ".join(sys.argv[1:])
with torch.no_grad():
    q = model.encode_text(tok([query]))
    q = q / q.norm(dim=-1, keepdim=True)
q = q.numpy()

scores = (vecs @ q.T).ravel()
for i in scores.argsort()[-25:][::-1]:
    print(f"{scores[i]:.3f}  {paths[i]}")

python search.py a photo of a welding table

That is the whole search. One matrix multiply and a sort.

Why this works#

Nothing is searched in the sense you probably imagine. There is no keyword index, no tags, no scan of your folder. When you indexed, every photo became 512 numbers and that happened once, before you had typed anything.

Your sentence goes through the same model's text encoder and comes out as 512 numbers in the same format. The comparison is then arithmetic: vecs @ q.T multiplies your query against all rows at once. Because everything is unit length, each result is the cosine of the angle between two vectors, bounded between -1 and 1.

The reason that arithmetic carries meaning is the training objective. CLIP saw about two billion image and caption pairs with one job: move an image and its real caption toward the same coordinates, and push mismatched pairs apart. Do that at sufficient scale and "close together in this space" and "means the same thing" become the same statement.

So your sentence lands wherever pictures of welding tables already live, and the dot product is high for those and low for everything else.

How to read the scores#

The numbers are compressed and not intuitive. As a rule of thumb, 0.30 and above is usually a real match, 0.20 to 0.25 is weak, and below 0.20 is noise. These are experience-based bands, not measurements, and they shift with the query.

Trust the ranking over the absolute number. A query where the best result scores 0.24 might still have found exactly what you wanted, because some phrasings just sit further from everything.

Phrasing changes the results more than you expect#

"a photo of a welding table" beats a bare "welding table" fairly consistently. That is not superstition. The captions CLIP trained on were web alt text and descriptions, which are full phrases, so queries shaped like captions land closer to where images actually sit.

Be concrete. "a rusty metal gate" retrieves better than "old stuff outside." Describe what is visibly in the frame, not the context or the reason you took the photo. CLIP has no idea it was your birthday.

Search by example instead#

Sometimes you already have a picture of the thing you want. Swap the encoder:

from PIL import Image
_, _, preproc = open_clip.create_model_and_transforms(
    "ViT-B-32", pretrained="laion2b_s34b_b79k")

with torch.no_grad():
    img = preproc(Image.open("example.jpg").convert("RGB")).unsqueeze(0)
    q = model.encode_image(img)
    q = q / q.norm(dim=-1, keepdim=True)
q = q.numpy()

The scoring line underneath is unchanged. This often works better than text, because you skip the step of translating a visual idea into words.

Where it plateaus#

Run a few queries against your own photos and you will hit this. The top ten results are decent. Somewhere around result fifteen the quality falls off a cliff, and things you know are in the folder never surface at all.

The reason is structural, not a phrasing problem.

Text search measures distance from one fixed point. Your sentence produces exactly one vector, and results are ranked by proximity to it. But that point sits wherever the internet's captions put it, which is usually offset from where your specific photos cluster.

"A welding table" on the internet means catalog product shots and tidy workshop stock photography. A welding table in your folder means your bench, your lighting, your phone, half obscured by a project. Those are related regions of the space but they are not the same point.

So you get two failure modes at once. Photos near the internet's idea of the concept but not yours rank too high. Your photos that sit further from that generic anchor never make the cut, no matter how you rephrase.

You cannot fix this by writing a better sentence. You are choosing among points in a space where the right answer is not a point at all. It is a region, and the region has a shape that no single query vector describes.

What actually fixes it#

Stop measuring distance from a fixed point and start fitting a boundary around the region you care about. That needs examples rather than a description, and examples means you clicking yes and no on a few dozen photos.

The surprising part is how few you need and how fast it trains, because the expensive work is already done. The embeddings never change again. Everything that follows is a small classifier fitted on top of numbers you already have.

That is the next guide.

Where this stops being the right tool#

Text search is genuinely enough for a one-off question. If you need to find that one picture of a bicycle and you are never going to ask again, do not build a classifier. Type the sentence, skim twenty results, move on.

The correction loop earns its cost when you will ask the same question repeatedly, or when you need results complete rather than merely good, or when you are sorting rather than finding.