Teach It What You Want
findintermediate10 min readpart 5 of 6

Find people in your photos without labeling anything

Why CLIP cannot tell people apart, what to use instead, and how clustering discovers everyone in your library with almost no input.

Before you start

  • a working photo index
  • insightface, onnxruntime and opencv-python installed

Everything so far used CLIP. For faces it fails completely, and understanding why explains what to use instead.

Why CLIP cannot do this#

CLIP learned from web captions, and captions almost never name individuals. It saw millions of images described as "a man in a hard hat" and essentially none described as "Josh". So it learned the category and not the identity.

The practical result is that two different people photographed in similar poses and lighting land closer together in CLIP space than one person photographed in two different settings. That is exactly backwards for this task, and no amount of correction fixes it, because the information was never in the representation.

Use a model trained on identity#

pip install insightface onnxruntime opencv-python

InsightFace bundles a detector and ArcFace, an embedding model trained directly on identity with many images per person. First use pulls buffalo_l, roughly 330 MB, into ~/.insightface. Note this is a separate download from the CLIP weights and it ignores HF_HUB_OFFLINE, so do it while connected.

from insightface.app import FaceAnalysis

app = FaceAnalysis(name="buffalo_l", providers=["CPUExecutionProvider"])
app.prepare(ctx_id=-1, det_size=(640, 640))

ctx_id=-1 means CPU. Use 0 for the first GPU.

Index faces, not images#

import cv2, json
import numpy as np
from pathlib import Path

DET_MIN, MIN_FACE_PX = 0.60, 40
rows, vecs = [], []

for p in Path("D:/pics").rglob("*.jpg"):
    bgr = cv2.imread(str(p))
    if bgr is None:
        continue
    for f in app.get(bgr):
        if f.det_score < DET_MIN:
            continue
        x1, y1, x2, y2 = f.bbox
        if min(x2 - x1, y2 - y1) < MIN_FACE_PX:
            continue
        vecs.append(f.normed_embedding.astype(np.float32))
        rows.append({"path": str(p), "box": [float(v) for v in f.bbox],
                     "score": float(f.det_score), "cluster": -1, "person": None})

np.save("face_vecs.npy", np.vstack(vecs))
Path("faces.json").write_text(json.dumps(rows))

Discover people automatically#

Here is the part that makes this different from the CLIP workflow. You do not label faces. You cluster them.

from sklearn.cluster import DBSCAN

vecs = np.load("face_vecs.npy")
labels = DBSCAN(eps=0.55, min_samples=3, metric="cosine").fit_predict(vecs)

for row, label in zip(rows, labels):
    row["cluster"] = int(label)

Tuning eps, the one dial that matters#

eps is the maximum cosine distance for two faces to be considered neighbours. It is the only setting you will realistically adjust.

Too tight and one person splits into four groups. Too loose and siblings merge. Start at 0.55 and move in steps of 0.05 based on what you see.

Why this is the hard part#

DBSCAN uses one global eps for everybody, which assumes every person in your library has the same spread. That is false. Someone with 200 varied photos occupies more space than someone with 6 similar ones, so a single threshold tends to over-split the first person and over-merge the second at the same time.

This is a limitation of the clustering, not of the embeddings. The embeddings are good. The one-size-fits-all threshold is the weak link.

Name each group once#

Store the name on the face record, not on the cluster id.

def name_group(cluster_id, name):
    for row in rows:
        if row["cluster"] == cluster_id:
            row["person"] = name

Search by reference photo#

def search_face(image_bytes, threshold=0.35):
    arr = np.frombuffer(image_bytes, np.uint8)
    bgr = cv2.imdecode(arr, cv2.IMREAD_COLOR)
    found = [f for f in app.get(bgr) if f.det_score >= DET_MIN]
    if not found:
        return {"error": "no face detected in that photo"}

    found.sort(key=lambda f: -(f.bbox[2] - f.bbox[0]))   # largest face
    q = found[0].normed_embedding.astype(np.float32)

    sims = vecs @ q
    hits = {}
    for i in np.argsort(sims)[::-1]:
        if sims[i] < threshold:
            break
        p = rows[i]["path"]
        hits.setdefault(p, float(sims[i]))
    return hits

How accurate is it, honestly#

ArcFace scores around 99.8% on LFW, a standard verification benchmark. Do not put much weight on that number for your folder. LFW is mostly frontal, well-lit celebrity photography, which is not what a personal photo library looks like.

Where it genuinely struggles: profile and three-quarter angles, faces under about 80 pixels, hard backlighting, sunglasses, and a beard grown or shaved between photos. Children are the worst case, since the same face at 8 and at 13 may not match at all. Siblings and parent-child pairs are the classic false merge.

The embeddings are rarely your problem. The clustering threshold usually is.

Where this stops being the right tool#

Face embeddings are biometric data, and several jurisdictions regulate them specifically. Running this on your own photo library on your own machine is one thing. Shipping it, or running it on images of people who have not agreed to it, is a different question worth answering before you build on it.

If you need reliable identification rather than convenient sorting, this is not enough on its own. Convenient sorting is what it is genuinely good at.

Next: every error we actually hit building this, with the literal messages.

More on finding things in your own files