The Best Local Embedding Model for RAG (I Benchmarked 5 on a 16GB GPU)

Picking a local embedding model is where most people building private RAG waste their first week, usually by reaching for the biggest one their GPU can hold. I ran five of them on a 16GB card over my own notes and the answer is not the one the "what fits in 16GB" framing points you at.

Here is the short version, then the numbers behind it. The card is not your bottleneck, bigger models earn almost nothing on a real corpus, and the model I would actually keep is bge-small.

The thing nobody measures before choosing

The usual way people pick a local embedder is VRAM math. "I have 16GB, so what is the biggest embedding model I can fit?" Then they load the largest one and move on, quietly assuming bigger dimensions mean better retrieval.

Both halves of that are wrong, and it took one afternoon of actually running them to see it.

I embedded my own second-brain corpus (524 chunks of markdown notes, real content, not a toy dataset) through five models on an RTX 5060 Ti, all driven through sentence-transformers so the code path is identical and only the model name changes. Small to large: all-MiniLM-L6-v2, bge-small-en-v1.5, bge-base-en-v1.5, bge-large-en-v1.5, and e5-large-v2.

One honest caveat before the table

The speed and VRAM numbers below are rigorous and repeatable. I ran the whole thing twice and the throughput and memory figures held.

The quality column is not a scored eval. It is a 3-query spot-check: three questions I know the right answer to, checking whether each model ranks the correct note first. That is enough to catch a model that is clearly worse, and not enough to split hairs between two good ones. A real quality ranking needs a labelled golden set and a proper metric, which is a separate build (RAGAS, a follow-up post). So read the quality column as a smoke test, not a leaderboard, and do not let anyone quote it as one.

The numbers

All five, same corpus, same card, same code:

Reading the table

16GB is not the constraint. The biggest model here peaks at 2.98 GB. On a 16GB card I could hold all five in memory at once with room to spare. The entire "what embedding model fits in my VRAM" question is answered before you start: all of them do. If you are choosing an embedder by what fits, you are optimising a limit you never hit. The real budget on consumer hardware is time and dimensions, not memory.

Bigger buys almost nothing here. bge-large and e5-large are 1024-dimensional, run at 30 chunks per second, and land the exact same 2/3 on the spot-check as bge-small, which is 384-dimensional and runs at 255. That is roughly 8x the speed for the same result on my data, and you pay for the large models twice: once in embed time now, and again forever, because every query you ever run has to be embedded by the same model and compared against 1024-dim vectors instead of 384. The cost compounds; the quality did not move.

MiniLM is fast but it is the weak one. all-MiniLM-L6-v2 is the default everyone starts with, and it is genuinely quick (496 ch/s, 0.41 GB). But it is the only model that missed on the spot-check, 1/3 where the others got 2/3. It is the fastest way to get mediocre retrieval.

The pick: bge-small

bge-small-en-v1.5 is the one I would keep, and the table makes the case on its own. Same 384 dimensions as the tiny MiniLM default, so your vectors stay small and your queries stay cheap forever. It beat MiniLM on the spot-check (2/3 vs 1/3), and it matched the 1024-dim heavyweights on that same check while running about 8x faster than them and using a quarter of their VRAM.

So it sits exactly where you want a default to sit: as light as the small option, as good as the big ones on the evidence I have, and it costs you nothing on either axis to make it your standard. The 1024-dim models are a tax I could not find a reason to pay on real notes.

When you would actually go bigger

The verdict is "start at bge-small," not "large models are useless." Two situations flip it, and neither showed up in my corpus:

• Multilingual or long-document work. The larger bge and e5 families, and bge-m3 in particular, are built for cases my English markdown notes never exercised. If your corpus is multilingual or full of long passages that need to be understood whole, test the big ones properly before ruling them out.
• A real eval that separates them. My spot-check could not distinguish bge-small from bge-large because three queries is too coarse to. It is entirely possible a proper golden-set eval would show the large model pulling ahead by a few points on hard, spanning questions. If those few points are worth 8x the latency to you, that is a real trade. You just cannot make that call from vibes, and I am not going to pretend three queries made it for me.

That eval is the next build: a labelled question set over the same corpus, scored with a retrieval metric, so the quality column stops being a smoke test and becomes a number you can stand behind. Until then, the honest reading is that the small model held its own and the big ones did not earn their cost.

What this actually means for your setup

If you are standing up local RAG on a consumer GPU, stop treating VRAM as the deciding factor. Load bge-small-en-v1.5, keep your dimensions at 384, and put the time you were about to spend fitting a 1024-dim model into the part that actually decides retrieval quality: your chunking and your corpus. The model was never the bottleneck I expected it to be.

The five-model benchmark is a single script over a folder of markdown, embeddings on the GPU, cosine ranking, and a timing and VRAM harness around it. Code and the raw bench-output.txt are in the repo if you want to point it at your own corpus and check whether your data agrees with mine. It might not, and that is the whole point: run it on your notes, not mine.

And that's it. Run bge-small, hold the big models for the day a real eval tells you to, and stop paying for dimensions your data cannot use.