The Open-Source AI Ecosystem, Mapped

The open-source AI landscape can feel like an endless scroll of GitHub repositories with overlapping purposes. Every week brings a new model, a new framework, or a new database claiming to solve the same problem slightly differently. But if you strip away the branding, the ecosystem falls into a handful of clear categories that work together in a predictable pattern. Understanding that pattern makes it far easier to pick tools for your own projects — and to explain your choices clearly in interviews or team discussions.

This guide maps the main layers of the open-source AI stack, what each one does, and where the boundaries between them sit. It's not exhaustive, and most real projects don't need every layer. But knowing the shape of the ecosystem helps you reason about tradeoffs instead of defaulting to whatever tool has the most stars this month.

Models and Weights

At the foundation are the models themselves — or more precisely, the trained weights released by teams like Meta (Llama), Mistral, and the Qwen project. These are the parameter files you load into memory to generate text, produce embeddings, or handle other tasks. Hugging Face has become the central registry for open-weight models, hosting model cards, licenses, and download counts that help you compare options before committing.

The key decisions at this layer are size, license, and intended use. A 7-billion-parameter model can run on a single consumer GPU; a 70-billion-parameter model needs significantly more memory but may produce better results on harder tasks. Not every model labelled "open" is truly open — many restrict commercial use, require acceptance terms, or limit redistribution. Reading the license before building around a model saves real problems later.

Serving and Inference

A model file on disk doesn't do anything until you run it through an inference engine. This is the layer that turns weights into an API your application can call. Ollama makes it straightforward to run models locally with a single command, which is ideal for development and prototyping. vLLM, by contrast, is built for high-throughput serving and is commonly used in production deployments where latency and concurrency matter.

Most serving tools expose an OpenAI-compatible HTTP interface, which means your application code can often treat open and closed models interchangeably. This matters more than it sounds: it lets you prototype with a local model and switch to a hosted API later without rewriting your pipeline. Some teams also use managed inference endpoints, which handle the operational burden of running models in exchange for per-usage pricing.

Frameworks and Orchestration

Above the serving layer sit the frameworks that help you compose applications — chaining calls together, managing conversation history, connecting tools, and routing between components. LangChain and LlamaIndex are the two most widely adopted, and their feature sets overlap considerably. Both let you build pipelines from prompts, retrievers, and model calls. LlamaIndex tends to focus more on retrieval-heavy use cases, while LangChain covers a broader range of agent and chain patterns.

Neither framework is mandatory, and plenty of production teams skip them entirely once application logic stabilizes. Frameworks are most useful during prototyping, when you're still figuring out what your pipeline should do and want to iterate quickly. Once the flow is settled, calling model APIs directly with your own glue code is often simpler and easier to debug. The risk of frameworks is that they can obscure what's actually happening between your data, your prompts, and your model.

Vector Stores and Retrieval

If your application needs to search over documents, code, or a knowledge base, you need a vector store — a database optimised for embedding similarity search. Options range from lightweight embedded libraries like Chroma to distributed production systems like Qdrant, Weaviate, and Milvus. There are also extensions that add vector search to databases you may already run, such as pgvector for PostgreSQL.

The choice depends on scale and what infrastructure you already have. If you're already running Postgres, pgvector lets you add semantic search without introducing a new system. If you need purpose-built performance at higher scale, a dedicated vector database is the more common path. The embeddings themselves — numeric representations of text that get stored and searched — come from either open embedding models or API-based services, and that choice is largely independent of which store you use.

Evaluation Tooling

The category that gets the least attention in demos but matters most in production is evaluation. Once your pipeline is assembled, you need to measure whether outputs are correct, relevant, and safe — not just in a few examples you tested by hand, but across the range of inputs real users will send. Ragas focuses on RAG-specific metrics like retrieval relevance and answer faithfulness. DeepEval and promptfoo offer broader testing frameworks for prompts and model behaviour, often using LLM-as-judge patterns to score outputs at scale.

For more deterministic checks, simple assertion-based testing against a fixed set of golden examples still works well and is often the first thing to set up. Evaluation is where many teams discover the gap between a demo that looks impressive and a system that holds up with actual users. It's also the layer that most reliably tells you whether a framework swap or model change actually improved your application — or just made it different.

How the Layers Fit Together

The typical open-source stack flows in one direction: a model provides capability, a serving layer exposes it, a framework orchestrates it, a vector store grounds it in your data, and eval tooling tells you whether any of it works as intended. You don't need every layer for every project — a local model and Ollama may be enough for a prototype — but knowing where each piece sits helps you reason about tradeoffs and isolate problems when something breaks.

If you're looking for specific tools to try alongside real projects, the Neural Goldmine tools section tracks open-source projects, and the jobs feed regularly lists roles where this stack is the daily working environment. Experimenting with each layer directly — even just getting a local model running and querying a vector store — is the fastest way to make the abstraction concrete.