Vector Databases: What They Are and When You Need One

Vector databases have become a near-default component in AI application stacks, especially for retrieval-augmented generation (RAG) systems. But the proliferation of dedicated vector stores — each promising scale, speed, and relevance — has created a common confusion among builders: do you actually need a separate database for vectors, or can you work with what you already have?

The honest answer depends on your data volume, query patterns, latency requirements, and how much operational complexity you're willing to take on. This guide walks through what vector databases do, when a dedicated one earns its place, and when simpler alternatives are the better engineering choice.

What a Vector Database Actually Does

A vector database stores and retrieves high-dimensional vectors — typically the output of an embedding model that converts text, images, or other data into arrays of floating-point numbers. Each vector represents the semantic meaning of its source content. The core job of a vector database is similarity search: given a query vector, find the stored vectors that are closest to it using a distance metric such as cosine similarity, dot product, or Euclidean distance.

What makes this non-trivial is scale. Exact nearest-neighbor search over millions of vectors by comparing each one individually becomes slow. Dedicated vector databases use approximate nearest neighbor (ANN) algorithms and indexing structures like HNSW (Hierarchical Navigable Small World graphs) to return results in milliseconds rather than seconds. They also typically combine vector search with metadata filtering — letting you restrict results to a specific user, category, or time range, which is essential in multi-tenant applications.

This is a different problem from keyword search. A traditional search engine matches terms; a vector search matches meaning. "How do I reset my password" and "recover account access" share no significant words but should produce similar embeddings. That semantic gap is why vector search became central to modern AI applications.

When a Dedicated Vector Store Earns Its Place

Dedicated vector databases — options include Qdrant, Weaviate, Milvus, and managed services like Pinecone — are built specifically for the workload described above. They become worth the added infrastructure when you're operating at scale: typically hundreds of thousands to millions or more vectors, with concurrent query load and strict latency requirements.

They also make sense when you need features that go beyond raw similarity search. Multi-tenant isolation, hybrid search (combining keyword and vector retrieval), built-in quantization to reduce memory footprint, and sharding across nodes are all areas where purpose-built systems have an edge. If your application's core function is search or retrieval — a support knowledge base, a product recommendation system, a large document corpus — a dedicated vector store is a reasonable architectural choice.

Operational maturity matters too. Managed vector databases handle backups, scaling, and uptime. For a small team without dedicated infrastructure engineers, that trade — paying a premium for a managed service instead of running your own — can be worth it. If you're scoping out roles or contracts in this space, the Neural Goldmine jobs feed regularly lists positions focused on RAG infrastructure and retrieval system design.

When You Don't Need One

Here's the part that often gets skipped: many applications don't need a dedicated vector database at all. If you're working with thousands or tens of thousands of vectors, brute-force similarity computation in memory is fast enough and trivially simple to implement. You can store vectors in a list, compute cosine similarity against a query vector, and sort. For prototyping and many production workloads, that's sufficient.

More importantly, you may already have a database that supports vectors. Postgres, through the pgvector extension, lets you store and index embeddings alongside your relational data using the same database you're already running. SQLite has vector extensions. Redis has vector support. MongoDB added vector search capabilities. If your application already uses one of these, adding a separate vector database introduces a new failure surface, a new data sync problem, and a new operational burden — often without meaningful benefit.

A common pattern that works well for teams building their first RAG application: start with pgvector on your existing Postgres instance. When and if query latency or index size becomes a real problem — measurable, not anticipated — migrate to a dedicated solution. Premature adoption of a dedicated vector database is a frequent source of unnecessary complexity in early-stage AI projects.

Key Tradeoffs to Keep in Mind

The choice between a dedicated vector database and an existing-store extension involves several axes. On scale and performance, dedicated systems are optimized for the vector search workload and generally outperform general-purpose databases at high vector counts; at small to moderate scale, the difference is rarely noticeable. On operational complexity, each additional data store is another thing to deploy, monitor, back up, and keep consistent — a single database with vector support is simpler to operate than two separate systems that must stay in sync.

Query flexibility also matters. Relational databases excel at complex filtering, joins, and transactional consistency. If your retrieval needs involve rich structured queries alongside vector search, keeping everything in one database can be easier than coordinating across two. On cost, managed dedicated vector databases charge based on storage and compute, often at a premium; self-hosted options are free but require your time. For smaller workloads, pgvector on a Postgres instance you're already paying for is effectively zero additional cost. Finally, dedicated vector stores often have richer APIs for hybrid search, reranking integrations, and observability. Whether those matter depends on what you're building.

A Practical Starting Point

If you're unsure, start simple. Use the database you already have with a vector extension, build your retrieval pipeline, and measure. Only when you hit a concrete limit — latency, scale, or a feature gap — does the added complexity of a dedicated vector database justify itself. Engineering decisions should follow from observed constraints, not anticipated ones.

The vector database ecosystem is mature and the options are solid, but "mature and solid" is not the same as "always necessary." The best architecture is the one that solves your actual problem with the least moving parts.