findintermediate9 min readAug 29, 2026

Why CLIP Finds a Welding Table But Not Your Welding Table

Text search ranks by distance to one fixed point that sits in the wrong place, so fit a boundary on your own labels instead.

Before you start

Python3.10 or newer
torch2.2+ (CPU build is fine)
open_clip_torch2.24+
scikit-learn1.4+
numpy2.x
An existing CLIP index (vecs.npy and paths.json)

Type "welding table" into a CLIP search over your own photos and you get welding tables. Some of them are yours. Plenty are not, and several of yours are missing. By the end of this you will understand exactly why that happens, and you will have a classifier fitted on your own 30 labels that fixes it, reusing the embeddings you already computed.

This assumes you already have vecs.npy and paths.json from indexing a folder. If your index files are named something else, pass --vecs and --paths. Nothing here re-runs CLIP over your images.

The shape of the problem#

CLIP was trained on roughly two billion image and caption pairs with one objective: pull an image and its true caption together, push mismatched pairs apart. Captions on the internet say "a welder working on a steel frame". They almost never say "Josh's welding table, the one with the vise bolted to the left end".

So the model learned the category and had no reason to learn the instance. Your table and a stranger's table sit close together because the caption that describes them is the same caption. Meanwhile your table shot at night with the shop lights on and your table shot in daylight can sit noticeably apart, because lighting and background do move the vector even though the object did not change.

That is the whole failure in one sentence. The thing you care about, identity, is not what the training objective rewarded.

Step 1: Look at the scores, not just the ranking#

Save this as text_rank.py.

import argparse
import json

import numpy as np
import torch
import open_clip


def load_index(vecs="vecs.npy", paths="paths.json"):
    emb = np.load(vecs)
    with open(paths, encoding="utf-8") as f:
        return emb, json.load(f)


def encode_text(query):
    model, _, _ = open_clip.create_model_and_transforms(
        "ViT-B-32", pretrained="laion2b_s34b_b79k"
    )
    model.eval()
    tokenizer = open_clip.get_tokenizer("ViT-B-32")
    with torch.no_grad():
        feat = model.encode_text(tokenizer([query]))
        feat = feat / feat.norm(dim=-1, keepdim=True)
    return feat.cpu().numpy().astype("float32")[0]


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("query")
    ap.add_argument("--vecs", default="vecs.npy")
    ap.add_argument("--paths", default="paths.json")
    ap.add_argument("--k", type=int, default=20)
    args = ap.parse_args()

    emb, paths = load_index(args.vecs, args.paths)
    q = encode_text(args.query)
    scores = emb @ q
    order = np.argsort(-scores)

    print(f"n={len(scores)}  max={scores.max():.3f}  "
          f"mean={scores.mean():.3f}  spread={np.ptp(scores):.3f}")
    for rank, i in enumerate(order[: args.k], 1):
        print(f"{rank:2d}  {scores[i]:.3f}  {paths[i]}")


if __name__ == "__main__":
    main()

Run it and read the header line before the results.

python text_rank.py "a welding table in a workshop"

Why this works#

The spread is the interesting number. Across a folder of shop photos, text query scores usually live in a band a few hundredths of a point wide. A strong match sits around 0.30estimatestrong match, rule of thumb, a weak one around 0.20 to 0.25estimateweak match, rule of thumb, and noise is below that. These come from working with this model rather than from a benchmark, and they shift with the phrasing of your query. Treat the ranking as the signal and the absolute score as a rough sanity check.

A narrow spread tells you the honest situation: the text anchor is one fixed point in a 512-dimensional space, and your photos are scattered around it at almost the same distance. You are sorting by tiny differences. That is why the ordering feels arbitrary near the top, and why rewording the query shuffles the results so dramatically.

The anchor also sits wherever internet captions put it, which is usually offset from where your specific photos actually cluster. You are ranking by distance to a point that is close to correct and not correct. There is no phrasing that fixes an offset anchor, which is why prompt tweaking plateaus fast.

Step 2: Label about thirty images#

Save this as label_cli.py. It opens candidates one at a time in your default image viewer and records your answer.

import argparse
import json
import os
import subprocess
import sys
from pathlib import Path

import numpy as np

from text_rank import load_index, encode_text


def open_image(path):
    if sys.platform.startswith("win"):
        os.startfile(path)
    elif sys.platform == "darwin":
        subprocess.run(["open", path])
    else:
        subprocess.run(["xdg-open", path])


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("query")
    ap.add_argument("--vecs", default="vecs.npy")
    ap.add_argument("--paths", default="paths.json")
    ap.add_argument("--concept", required=True)
    ap.add_argument("--n", type=int, default=30)
    ap.add_argument("--labels", default="labels.json")
    args = ap.parse_args()

    emb, paths = load_index(args.vecs, args.paths)
    scores = emb @ encode_text(args.query)
    order = np.argsort(-scores)[: args.n]

    lf = Path(args.labels)
    store = json.loads(lf.read_text(encoding="utf-8")) if lf.exists() else {}
    entry = store.setdefault(args.concept, {"good": [], "bad": []})
    seen = set(entry["good"]) | set(entry["bad"])

    for i in order:
        p = paths[i]
        if p in seen:
            continue
        open_image(p)
        ans = input(f"[{scores[i]:.3f}] {p}\n  y / n / q > ").strip().lower()
        if ans == "q":
            break
        if ans == "y":
            entry["good"].append(p)
        elif ans == "n":
            entry["bad"].append(p)

    lf.write_text(json.dumps(store, indent=2), encoding="utf-8")
    print(f"{args.concept}: {len(entry['good'])} good, {len(entry['bad'])} bad")


if __name__ == "__main__":
    main()

python label_cli.py "a welding table in a workshop" --concept my_table --n 30

Step 3: Fit a boundary and re-rank#

Save this as fit_rank.py.

import argparse
import json

import numpy as np
from sklearn.linear_model import LogisticRegression

from text_rank import load_index


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--vecs", default="vecs.npy")
    ap.add_argument("--paths", default="paths.json")
    ap.add_argument("--concept", required=True)
    ap.add_argument("--labels", default="labels.json")
    ap.add_argument("--k", type=int, default=20)
    args = ap.parse_args()

    emb, paths = load_index(args.vecs, args.paths)
    with open(args.labels, encoding="utf-8") as f:
        entry = json.load(f)[args.concept]
    pos, neg = set(entry["good"]), set(entry["bad"])

    idx = {p: i for i, p in enumerate(paths)}
    rows, y = [], []
    for p in pos:
        if p in idx:
            rows.append(idx[p])
            y.append(1)
    for p in neg:
        if p in idx:
            rows.append(idx[p])
            y.append(0)

    if len(set(y)) < 2:
        print("need at least one good and one bad label")
        return

    X = emb[rows]
    clf = LogisticRegression(max_iter=1000, class_weight="balanced")
    clf.fit(X, y)

    probs = clf.predict_proba(emb)[:, 1]
    order = np.argsort(-probs)
    print(f"fitted on {len(y)} labels")
    for rank, i in enumerate(order[: args.k], 1):
        print(f"{rank:2d}  {probs[i]:.3f}  {paths[i]}")


if __name__ == "__main__":
    main()

python fit_rank.py --concept my_table

Why this works#

The fit takes about a second on a few thousand rows, because logistic regression on 512 features with 30 examples is a trivially small optimization. This is the payoff of freezing the embeddings. CLIP never runs again. Ten concepts over one folder cost one index, not ten.

The difference in output is not subtle. Text search asks "how close is this photo to the phrase". The classifier asks "which side of the line separating the photos you said yes to from the ones you said no to". The line is fitted to where your photos actually sit, not where captions of similar photos sit. The offset anchor problem disappears because there is no anchor any more.

class_weight="balanced" matters as soon as your labels are lopsided, which they will be. If you marked 6 good and 24 bad, the unweighted fit can score well by calling everything bad. Balanced reweights each class by its inverse frequency so the six positives carry equal total weight.

What breaks and how to tell#

Only one class labeled. If you answered "y" to everything, scikit-learn stops you:

ValueError: This solver needs samples of at least 2 classes in the data, but the data contains only one class: 1

The guard above catches it first and prints a plain message. Keep the guard. Without negatives, there is no boundary to fit, only a cloud of points. Go back and label some near misses "n" specifically.

NumPy 2.0 removed the ndarray method form of ptp. If you write scores.ptp() instead of np.ptp(scores):

AttributeError: 'numpy.ndarray' object has no attribute 'ptp'

The function form survives, the method form was removed in NumPy 2.0. This bites a lot of code written before 2024, not just this project. The text_rank.py above uses np.ptp(scores) for that reason.

Labels pointing at paths that are no longer in the index. The loop above silently skips any labeled path missing from paths.json, then you get a confusing "fitted on 4 labels". That happens after you move or rename a folder, because the index stores absolute paths and nothing else. Re-index, or prune. Keep labels.json when you do, since your labels are the one file here you cannot regenerate.

The ranking looks great and is wrong. If you fit and evaluate on the same 30 photos, of course the boundary separates them. The honest check is to look at results 20 through 40, which you never labeled. If those degrade sharply, you have a boundary memorizing 30 points rather than a concept.

Where this stops being the right tool#

The classifier holds for photos that look like the ones you labeled. Change the camera, the lighting, or the shop and the new embeddings can land far enough away that the boundary sits wrong. The fix is another 20 to 30 labels from the new conditions, not starting over.

It cannot separate two instances that genuinely look alike. If your welding table and your neighbor's are the same model with the same vise, no number of labels will help, because CLIP encoded no feature that distinguishes them. Look for a feature that is actually visible, like a specific sticker or the layout of the wall behind it, and accept that you are now searching for the sticker.

It does not do people. Identity of faces is a different model family trained on a different objective, and no amount of labeling on CLIP embeddings gets you there.

And if you have fewer than about ten positives, stay with text ranking and keep labeling. A boundary fitted on four points is a line through noise that will look confident and be wrong, which is worse than a mediocre ranking you already distrust.

More on finding things in your own files