teachintermediate11 min readAug 29, 2026part 3 of 6

Teach it what you want with a correction loop

Fit a classifier on frozen embeddings in about a second, correct the cases it is least sure about, and know when to stop.

Before you start

a working vecs.npy and paths.json
scikit-learn installed

Text search ranks by distance from one point. This guide replaces that with a boundary fitted to your own examples. It trains in about a second, needs roughly thirty labels to start being useful, and gets better every time you correct it.

The part that surprises people#

You are not training CLIP. CLIP never changes. Every photo's 512 numbers were computed once during indexing and they are frozen forever.

What you train is a logistic regression on top of those numbers. That is a model with 513 parameters, fitted on a few dozen rows, and it takes about a second on a CPU.

import json
import numpy as np
from sklearn.linear_model import LogisticRegression

vecs  = np.load("vecs.npy")
paths = json.loads(open("paths.json").read())
pos   = {p: i for i, p in enumerate(paths)}

good = ["D:/pics/a.jpg", "D:/pics/b.jpg"]   # you said yes
bad  = ["D:/pics/c.jpg", "D:/pics/d.jpg"]   # you said no

X = vecs[[pos[p] for p in good + bad]]
y = [1] * len(good) + [0] * len(bad)

clf = LogisticRegression(max_iter=2000, class_weight="balanced").fit(X, y)
probs = clf.predict_proba(vecs)[:, 1]

for i in probs.argsort()[-25:][::-1]:
    print(f"{probs[i]:.3f}  {paths[i]}")

Which photos to correct#

This is where most of the value is, and it is counterintuitive.

Do not label random photos. Do not label the top-ranked results. Label the ones the model is least sure about, the ones sitting near probability 0.5.

labeled = set(range(len(X)))                 # indices you have already done
free = np.array([i for i in range(len(paths)) if i not in labeled])

p = probs[free]
n = int(24 * 0.7)
uncertain = free[np.abs(p - 0.5).argsort()[:n]]      # closest to the fence
confident = free[p.argsort()[::-1][:24 - n]]         # its strongest guesses
batch = list(dict.fromkeys(list(uncertain) + list(confident)))

False positives teach more than true positives#

If it keeps dragging workbenches into your welding table results, feeding it twenty workbenches marked "no" sharpens the boundary faster than twenty more welding tables marked "yes".

The reason is the same as above. It already knows welding tables are welding tables. What it does not know is where the edge of that category is, and the edge is defined by the nearest things that are not in it.

Knowing when to stop#

Track whether the model already predicted what you ended up saying.

history = []                     # 1 if it agreed with you, 0 if not

for photo_index, model_prob, you_said_yes in corrections:
    history.append(1 if (model_prob >= 0.5) == you_said_yes else 0)

window = history[-60:]
if len(window) == 60 and sum(window) / 60 >= 0.95:
    print("converged")

Why this works#

Accuracy on your labeled set is misleading, because the model was fitted on that set. Agreement measures something more useful: whether your corrections are still adding information. Once it predicts your answer 95 times out of 100, you are clicking a button at a machine that already knows.

Be honest about what this claim covers. It means the model predicts you correctly on photos like the ones you have already labeled. It says nothing about photos unlike those. That is a narrow claim and it is the only one the metric supports.

In practice, expect somewhere in the range of 150 to 400 corrections for a visually distinct concept. That range is a rule of thumb, not a measurement. Something subtle, like separating a good product photo from a mediocre one, can sit at 88% after four hundred and never converge, which usually means the distinction is not really visual.

Picking the cutoff#

Do not guess a threshold. Sweep it.

for t in (0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95):
    print(f"{t:.2f}  {int((probs >= t).sum())} matches")

Look for where the count stops climbing steeply. A jump from 4 matches at 0.7 to 33 at 0.4 tells you the real boundary is somewhere in between, and that seeing the shape beats picking 0.6 because it sounds reasonable.

Once it is trained#

Index new photos, score them, done. No clicking.

probs = clf.predict_proba(new_vecs)[:, 1]
matches = [p for p, s in zip(new_paths, probs) if s >= 0.6]

The boundary is a rule about a region of the space, not a list of approved files. Anything landing inside that region scores high whether you labeled it or not, which is why it generalizes to photos taken at a different angle, in different light, with a different camera.

When it needs a top-up#

The failure mode is a shift in what the photos look like, not how many there are. New camera, outdoor instead of indoor, someone else's photos mixed in. Those land in a different part of the space and your boundary sits wrong for them.

Twenty or thirty labels on the new batch fixes it, and you are extending the same label set rather than starting over. The signal to watch for is the sweep: if your match count at a fixed threshold suddenly jumps out of proportion to how many photos you added, something drifted.

Multiple concepts, one index#

Every concept reads the same vecs.npy. Adding a tenth concept costs nothing in disk or embedding time, just the clicks. Ten concepts over 20,000 photos is one 40 MB file, not ten.

They are fully independent. Labeling a photo yes for one concept has no effect on any other, and the same photo can be yes in three and no in two, which is correct since a rusty welding table in a shop genuinely is all of those.

Where this stops being the right tool#

If your two classes are not linearly separable in CLIP space, a logistic regression will not find them. Symptom: agreement stalls in the 80s no matter how much you label. Try a small MLP or an RBF-kernel SVM on the same embeddings before concluding the task is impossible.

If you need to distinguish specific individuals, this whole approach fails and no amount of correction fixes it. That is a different model and a different guide.

Next: wrapping all of this in a local dashboard so you stop typing commands.