findbeginner9 min readAug 29, 2026

Search Your Own Photo Library Offline With CLIP, Start to Finish

Index a folder of photos with CLIP and search it by typing what you remember seeing, with no cloud and no API key.

Before you start

Python3.10 or newer
torch2.2+ (CPU build is fine)
open_clip_torch2.24+
Pillow10.x
numpy1.26 or 2.x

By the end of this you will have two scripts and one .npy file. The .npy file holds a numeric fingerprint of every photo in a folder. The second script takes a sentence like "welding table with a grinder on it" and returns the ten photos in your folder that best match it, in about 50 milliseconds, with your network cable unplugged.

This is the whole pipeline. Nothing here calls a server after the first model download.

What CLIP actually gives you#

CLIP is two models trained together. One turns an image into a list of 512 numbers. The other turns a piece of text into a list of 512 numbers. They were trained so that a photo of a dog and the words "a photo of a dog" land close together in that 512-number space.

That is the whole trick. Once both live in the same space, "search" is just a dot product. You are not running a model at search time against every photo. You ran the image model once, months ago maybe, and saved the numbers. At search time you run the text model on one short sentence and multiply.

This is why the search is instant and the indexing is slow. Understand that split and the rest of the design explains itself.

Step 1: Install into a venv, not into system Python#

python -m venv .venv
.venv\Scripts\activate
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install open_clip_torch pillow numpy

Install torch first and from the CPU index. If you pip install open_clip_torch on its own, pip resolves torch for you and on Linux it will pull the CUDA build, which is a 2.5 GB download of GPU libraries you are not going to use. The CPU wheel is around 200 MB. On Windows the default wheel is already CPU-only, so the --index-url line is harmless there but keep it for portability.

Step 2: Write the indexer#

Save this as index_photos.py.

import argparse
import json
from pathlib import Path

import numpy as np
import torch
import open_clip
from PIL import Image, ImageFile, UnidentifiedImageError

ImageFile.LOAD_TRUNCATED_IMAGES = True

EXTS = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tiff"}


def find_images(folder):
    paths = []
    for p in sorted(Path(folder).rglob("*")):
        if p.is_file() and p.suffix.lower() in EXTS:
            paths.append(p)
    return paths


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("folder")
    ap.add_argument("--out", default="index")
    ap.add_argument("--batch", type=int, default=16)
    ap.add_argument("--threads", type=int, default=0)
    args = ap.parse_args()

    if args.threads > 0:
        torch.set_num_threads(args.threads)

    model, _, preprocess = open_clip.create_model_and_transforms(
        "ViT-B-32", pretrained="laion2b_s34b_b79k"
    )
    model.eval()

    paths = find_images(args.folder)
    print(f"found {len(paths)} candidate files")

    vectors = []
    kept = []
    batch = []
    batch_paths = []

    def flush():
        if not batch:
            return
        stack = torch.stack(batch)
        with torch.no_grad():
            feats = model.encode_image(stack)
            feats = feats / feats.norm(dim=-1, keepdim=True)
        vectors.append(feats.cpu().numpy().astype("float32"))
        kept.extend(str(p) for p in batch_paths)
        batch.clear()
        batch_paths.clear()

    for i, p in enumerate(paths):
        try:
            img = Image.open(p).convert("RGB")
        except (UnidentifiedImageError, OSError) as e:
            print(f"skip {p}: {e}")
            continue
        batch.append(preprocess(img))
        batch_paths.append(p)
        if len(batch) >= args.batch:
            flush()
            print(f"{i + 1}/{len(paths)}", end="\r")
    flush()

    if not vectors:
        print("nothing indexed")
        return

    emb = np.concatenate(vectors, axis=0)
    np.save(f"{args.out}.npy", emb)
    Path(f"{args.out}.json").write_text(json.dumps(kept), encoding="utf-8")
    print(f"\nindexed {emb.shape[0]} images, {emb.shape[1]} dims -> {args.out}.npy")


if __name__ == "__main__":
    main()

Run it:

python index_photos.py "C:\Users\you\Pictures" --out pictures

Why this works#

model.eval() and torch.no_grad() are not optional politeness. eval() switches dropout and batch norm out of training behavior, and without no_grad() PyTorch builds a gradient graph for every image it never uses, which roughly doubles memory and costs you 20 to 30 percent throughput.

The normalize line, feats / feats.norm(dim=-1, keepdim=True), is doing the real work of the search later. Once every vector has length 1, a dot product is exactly the cosine of the angle between the two vectors, bounded between -1 and 1. Skip this and your scores become "similar and also happens to be a bright busy image", because raw CLIP vectors have wildly different magnitudes. Normalizing at index time instead of search time means you pay for it once.

Batching matters more than it looks. One image at a time on CPU wastes most of the cores because a single 224x224 forward pass does not fill the matrix multiply. Batch 16 is roughly 2x the throughput of batch 1 on an 8-core desktop. Above 32 the gain flattens and memory climbs.

--threads exists because PyTorch grabs every physical core by default. If you want to keep using the machine while it indexes, run --threads 4 and give up maybe a third of the speed.

Step 3: What the numbers actually cost#

The model downloads once, to ~/.cache/huggingface (or %USERPROFILE%\.cache\huggingface on Windows). ViT-B-32 with the laion2b_s34b_b79k weights is roughly 600 MB on disk. After that first download the model never touches the network again, though the library will still try to check for updates unless you tell it not to. That is its own guide.

Throughput over 107 images was 8.3 images/secmeasuredWindows laptop, CPU-only torch, batch 32, JPEG decode included. That is one machine and one folder, so treat it as a floor rather than a spec. Desktop CPUs generally land somewhere around 10 to 20 images/secestimatedesktop CPU. At 8.3 per second a 20,000 photo library takes about 40 minutes, so start it and go do something else.

A GPU changes the order of magnitude, not the details. Expect a few hundred images/secestimatemid-range GPU on a mid-range card, and it comes with a catch: JPEG decode still happens on the CPU, so past a certain point you are timing your disk and your decoder rather than the model.

Storage is small and predictable. 512 dimensions at float32 is 2,048 bytes per image. 20,000 photos is 41 MB. 200,000 photos is 410 MB and still loads into RAM without complaint. Cast to float16 and you halve it at a cost in ranking you will not notice, but do it only for storage and cast back to float32 before the dot product.

Step 4: Write the searcher#

Save this as search_photos.py.

import argparse
import json

import numpy as np
import torch
import open_clip


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("query")
    ap.add_argument("--index", default="index")
    ap.add_argument("--k", type=int, default=10)
    args = ap.parse_args()

    emb = np.load(f"{args.index}.npy")
    with open(f"{args.index}.json", encoding="utf-8") as f:
        paths = json.load(f)

    model, _, _ = open_clip.create_model_and_transforms(
        "ViT-B-32", pretrained="laion2b_s34b_b79k"
    )
    model.eval()
    tokenizer = open_clip.get_tokenizer("ViT-B-32")

    tokens = tokenizer([args.query])
    with torch.no_grad():
        tfeat = model.encode_text(tokens)
        tfeat = tfeat / tfeat.norm(dim=-1, keepdim=True)
    q = tfeat.cpu().numpy().astype("float32")[0]

    scores = emb @ q
    top = np.argsort(-scores)[: args.k]
    for rank, i in enumerate(top, 1):
        print(f"{rank:2d}  {scores[i]:.3f}  {paths[i]}")


if __name__ == "__main__":
    main()

python search_photos.py "a welding table with a grinder on it" --index pictures

You get ten lines, each a score and a path. Scores for a good match usually land between 0.25 and 0.35. That range surprises people. CLIP cosine scores are compressed. A 0.31 is a strong hit and a 0.19 is noise. The absolute number is close to meaningless on its own, which is why you rank rather than threshold. Picking a threshold is a separate problem with a real method, not a guess.

What breaks and how to tell#

Truncated JPEGs. You will see:

OSError: image file is truncated (23 bytes not processed)

This comes from photos that were interrupted mid-copy, which is common in phone backups and OneDrive folders. ImageFile.LOAD_TRUNCATED_IMAGES = True at the top of the file tells Pillow to decode what it has and pad the rest. It is set at module level on purpose, because it is a global flag read during decode, not an argument. A partially decoded photo still embeds fine, since CLIP resizes to 224x224 anyway and a missing bottom strip barely moves the vector.

HEIC files from an iPhone. You will see:

PIL.UnidentifiedImageError: cannot identify image file 'C:\\Users\\you\\Pictures\\IMG_4417.HEIC'

Pillow does not read HEIC out of the box. The except block above catches it and skips, which is fine if you only have a few. If your library is mostly HEIC, pip install pillow-heif and add these two lines above find_images:

from pillow_heif import register_heif_opener
register_heif_opener()

Then add .heic to EXTS. Without the register call the extension is in your list but Pillow still cannot open it, and you will skip every file while wondering why the count is zero.

Wrong pretrained tag. Typo the weights name and open_clip fails immediately rather than silently loading random weights:

RuntimeError: Pretrained weights (laion2b_s34b_b79) not found for model ViT-B-32.

Good failure. The dangerous version of this mistake is indexing with one checkpoint and searching with another. The vectors are not comparable across checkpoints, and nothing errors. You just get garbage rankings. If your results suddenly look random, check that both scripts name the same model and the same pretrained tag.

Index and paths out of sync. If you re-run the indexer on a changed folder but keep an old .json, the scores are real and the filenames are wrong. Always write both files in the same run, as above. If you have opened the wrong pair, you will notice because the top result will be confidently unrelated to the query every time, not just sometimes.

Where this stops being the right tool#

CLIP searches for categories, not for instances. It will find a welding table. It will not reliably find your welding table among fifty other welding tables, because the training objective rewarded matching a caption, and no caption in the training data said "Josh's welding table". That gap is the single biggest disappointment people hit with this approach.

It does not do faces. Asking for "my brother" returns nothing useful. Face recognition is a different model family entirely, trained on identity rather than caption similarity.

It does not read text in images well. ViT-B-32 will notice that a photo contains writing but will not tell you what the sign says. If you need that, you need OCR feeding a text index, which is a different pipeline.

It has no memory of time or place. "Photos from the lake trip last July" is three constraints and CLIP handles zero of them. Your file metadata handles all three, and filtering on EXIF date before you rank by CLIP score is almost always better than trying to phrase the date into the query.

And the recall ceiling is real. On a folder of shop photos, a well phrased query typically puts what you want in the top ten around 70 to 85 percent of the time. Getting from there to reliable means training a small classifier on top of these same embeddings, which takes seconds and works far better than most people expect. That is the next thing to build.

More on finding things in your own files