How Acceldata Fixed the Scattered Engineering Context Problem: A Deep Dive

Why Vector Search Isn't Enough While Building a Living Knowledge System for Engineers

Acceldata maintained deep technical information across its multiple tech platforms; however, that context was fragmented across disparate systems. Crucial knowledge remained trapped in source code, documentation, tickets, wikis, and meeting minutes. As the organization expanded, the primary operational challenge shifted from creating information to simply knowing where to locate it.

Whether it involves an engineer searching for a code reference, a support representative tracing a fix across product releases, an RFP response buried in an old email, or a cross-functional team waiting on an engineer to provide a timely answer, the information typically exists somewhere within the organization. The challenge has always been finding it. Even when an answer was eventually found, there was no way to know whether it was accurate, current, or complete, and getting to it rarely happened quickly. Matrix was built to solve exactly that problem: delivering answers that are fast, correct, and grounded in evidence, every time.

This blog walks you through the architecture, retrieval patterns, and engineering principles behind Matrix, giving you a practical blueprint to build a similar living knowledge system in your own organization.

Architectural Decision

The first architectural decision was to avoid creating separate knowledge systems for every team or use case.

Instead, Matrix was designed around a shared knowledge foundation that could be accessed differently depending on the task. The architecture therefore separated four concerns: the intelligence store that makes knowledge searchable, the connected sources that continuously provide knowledge, the personas that influence reasoning and tool selection, and the surfaces through which users interact with the system. This separation matters because knowledge and presentation have very different lifecycles.

A source code repository may change many times in one day. A persona can change instantly depending on whether the user is investigating an engineering issue or preparing customer facing material. A Slack interface and a web interface may look completely different while asking questions against the same underlying information. If those concerns are tightly coupled, every new interface or use case eventually becomes another knowledge silo.

Matrix instead keeps the underlying knowledge independent from the way that knowledge is consumed. Conceptually, the first layer of the architecture looks like this:

This gives us the overall structure, but it immediately introduces a more difficult engineering question.

How should all of these different sources actually be searched?

Why Not Everything Behind Vector Search

The obvious answer was retrieval augmented generation.

Matrix does use RAG. An embedding model and vector store sit at the core of the knowledge retrieval architecture. Source content can be divided into chunks, converted into vector representations, and retrieved according to semantic similarity. This works extremely well for prose.

Imagine that somebody asks:

Can we catch bad data before it lands downstream?

The relevant documentation may describe “data quality validation rules.” Those phrases contain very few matching words, but they express closely related concepts.

Traditional keyword retrieval can struggle with this type of question. Vector retrieval can find the relationship because it operates on semantic similarity rather than exact vocabulary. This was one of the reasons vectors became an important part of Matrix. However, the moment we applied the same idea to every source, a limitation appeared.

Consider this question:

Where is resolveUserById implemented?

Semantic similarity is no longer the primary requirement. We want the actual identifier. More importantly, we want the implementation that exists in the repository now.

If source code is periodically embedded into a vector database, the vector representation describes the repository at the time of the most recent synchronization. The actual repository may already have changed.

The same problem appears with work items. A work item can change state between two synchronization cycles. If somebody asks for its current status, retrieving an embedded copy is unnecessary when the system of record can provide the exact answer. This led to one of the central engineering principles behind Matrix:

Different kinds of knowledge require different retrieval strategies.

The Matrix architecture therefore does not treat RAG as another name for vector search. Instead, vector search is one retrieval mechanism inside a broader retrieval system.

Retrieval Modes

Once we separated retrieval according to the nature of the source, three major patterns emerged.

Prose knowledge is generally retrieved semantically. Source code is searched against the live repository using lexical retrieval. Work items and release information can be queried through structured interfaces when exact current state or aggregation is required.

We drew a simple line: use vectors when meaning matters, and use live lexical or structured retrieval when freshness and precision matter.

The architecture therefore becomes:

This architecture allows the user to ask one natural language question without knowing which retrieval mechanism is appropriate. That raises the next problem of figuring out what has to decide which path to use.

Making Retrieval Agentic

A conventional RAG pipeline usually defines retrieval before the question arrives. The application embeds the question, retrieves a configured number of chunks, adds those chunks to the context, and asks the model to answer.

Matrix changes this relationship.

The model receives retrieval tools and chooses which tools are required for the current question. We call this agentic retrieval: the model selects tools, inspects the results, and keeps retrieving when the first pass does not give enough evidence.

Consider a question such as:

Where is resolveUserById implemented, and did work item 2780 fix it?

There are at least two information requirements inside this single sentence.

Matrix can search the live repository for the implementation while separately retrieving information about the work item.

For the code request, the system can expand the natural language request into likely identifiers and perform lexical search against the current repository.

For the work item, semantic retrieval can provide surrounding context while a structured lookup obtains the current record from the live system.

These operations can happen independently and their results can later be combined.

That is the flow we implemented. Lexical code results, semantic work-item context, and live structured data are gathered independently, then injected together before Matrix synthesizes the answer.

The resulting retrieval loop looks like this:

There is an important engineering detail here. The loop cannot be unlimited.

We bound the agent loop at sixteen iterations. That is enough room to investigate a hard question without letting latency and cost run unbounded. This gives the model enough freedom to investigate a complex question while keeping execution bounded.

That balance is important in any agentic architecture. Giving a model tools without execution boundaries creates unpredictable latency and cost. Making the retrieval sequence completely static removes much of the value of agentic reasoning.

A bounded loop sits between those two extremes.

What Happens to Prose During Ingestion

Agentic retrieval changes the query path, but prose still needs an ingestion pipeline before semantic retrieval becomes possible.

Matrix follows the standard RAG sequence for this part of the architecture.

Content is divided into overlapping chunks. Each chunk is converted into an embedding. Those embeddings are stored in the vector store. When a question arrives, the question is converted using the same embedding model and compared with the stored vectors.

The important relationship is that ingestion and querying share the same embedding model and vector space.

This is our vector retrieval path. We use it for conceptual questions such as 'How does ADOC handle reconciliation?' Retrieved chunks become evidence in the model context, and the answer is generated from those passages rather than from model memory.

This is textbook RAG, and there was no reason to replace it where it worked well.

The more important architectural decision was knowing where not to use it.

Problem Solving with Structured Knowledge

Some questions cannot be answered reliably through similarity retrieval regardless of embedding quality.

Consider:

Which release had the most fixes?

The word “most” changes the computational problem.

The system cannot retrieve a few semantically similar release notes and infer the answer. It needs to inspect the relevant population and calculate the result.

Similarly:

What changed between version A and version B?

This requires comparison.

And:

Was this issue fixed in version B?

This requires an authoritative release record.

Matrix therefore represents release information as structured knowledge that can support comparisons, migration information, compatibility checks, and exact release facts. We specifically designed this to avoid inferring shipment from the state of a work item.

This gives us a useful design rule for building similar systems.

If the question contains concepts such as how, why, or explain, semantic retrieval is often appropriate.

If it contains concepts such as exactly which, current, latest state, how many, between these versions, or most, direct or structured retrieval should at least be considered.

This is not a routing rule that must be implemented literally. It is a way of thinking about retrieval architecture.

Connecting the Knowledge Sources

Once multiple retrieval modes existed, the next problem was source management.

Matrix connects several categories of knowledge. Product truth includes documentation, capability information, and source code. Work systems include work tracking and internal wikis. Field knowledge includes previous RFP responses, emails, and call transcripts. Release and support knowledge includes release history, migration information, compatibility facts, and API changes.

These sources have different update characteristics. That means synchronization cannot simply be treated as one nightly batch job. A document may be indexed and embedded. A repository may be queried live.

A structured system may expose both historical content for semantic retrieval and a live interface for exact queries. The resulting architecture is therefore better understood as a knowledge fabric than as a single database.

This distinction is useful when replicating the architecture.

Do not begin by asking, “How do we embed all our enterprise knowledge?”. Begin by asking, “What guarantees does each source need when somebody retrieves from it?” That question usually produces a better system.

Keeping Product Knowledge Intact Across Contexts

Once Matrix could search many sources, another problem appeared. More knowledge does not automatically produce better answers. Sometimes the system must deliberately know less.

Acceldata has multiple product lines. A question about one product should not silently retrieve implementation details belonging to another product simply because the passages happen to be semantically similar.

Matrix therefore treats product as a top level scoping entity. Sources are associated with the appropriate product, and retrieval is constrained according to the product context of the conversation. The isolation is enforced on the server rather than being left entirely to prompting.

This creates an important separation:

This is a subtle but useful architectural principle.

Reasoning scope and knowledge scope should not be the same control.

The persona determines how the system approaches a question. The product scope determines which knowledge the system is allowed to use. Keeping those controls independent means the same engineering persona can operate across different products without mixing their knowledge.

Understanding Personas as Reasoning Configurations

This separation also explains how Matrix implements personas.

A persona does not create another copy of the knowledge base.

Instead, it changes the reasoning priorities, available tools, and presentation appropriate for a particular task while continuing to use the shared knowledge foundation. We treat workspace and persona as different controls. A workspace decides where someone is working. A persona decides how the question is answered.

For example, an engineering persona may need implementation details, limitations, root cause information, and the source files inspected during retrieval. Another persona may use the same underlying facts but should not expose internal implementation details. Architecturally, this is preferable to maintaining independent assistants for every role.

Independent assistants tend to create duplicated indexes, duplicated synchronization pipelines, inconsistent facts, and increasingly difficult maintenance.

Matrix instead follows this pattern:

The knowledge remains shared. The reasoning changes. This is what allows the system to add new ways of using knowledge without creating a new knowledge architecture every time.

Grounding as an Architectural Property

Once a system can retrieve from documentation, code, work items, and release data, generating an answer is comparatively straightforward. Trusting that answer is harder.

Matrix therefore treats grounding as part of the architecture rather than as a final instruction added to a prompt. The system verifies claims against sources, cites the evidence used, and is designed to avoid inventing an answer when the available evidence does not support one. We also built structural code tracing and an evaluation harness so we can measure whether answers stay grounded in the evidence we actually retrieved.

For code related questions, this becomes particularly concrete.

The hybrid retrieval path can return a verification footer containing the files that were actually inspected while producing the answer.

The difference is significant.

“Generated using information about the codebase” is difficult to verify.

“Generated after reading these source files” is inspectable.

For engineering systems, inspectability is often more valuable than confidence.

The System Also Needs to Observe Itself

Once retrieval becomes agentic, the cost of an answer is no longer constant. One question may require a single vector search. Another may require semantic retrieval, repository searches, structured queries, and several iterations through the agent loop.

Matrix therefore records the cost associated with generating answers. We track cost at the answer level, so the operational expense of each response stays visible.

This is an important part of productionizing agentic systems.

Quality, latency, and cost are connected.

Increasing the number of retrieval iterations may improve evidence coverage but increase latency. Adding more retrieved context may improve recall while increasing token consumption. Running several tools concurrently may reduce wall clock time while increasing total compute. These tradeoffs should be measurable. An agent that can observe the enterprise but cannot be observed itself becomes difficult to operate.

The Complete Matrix Architecture

Putting these decisions together gives us the full system.

• This architecture is recognizably RAG, but RAG is only one part of the system.
• The vector store solves semantic retrieval.
• The live repository solves code freshness.
• Structured systems solve exact operational questions.
• Product scoping controls which knowledge is available.
• Personas control how the available knowledge is interpreted.
• The agent loop decides which retrieval operations are required.
• Grounding connects the generated answer back to the evidence.
• Cost and evaluation instrumentation tell us whether the system remains practical to operate.
• Each component exists because a simpler version of the architecture eventually encounters a specific limitation.

What We Would Recommend Building First

If another engineering team wanted to build a similar system, we would not begin with agents.

• We would begin by mapping the knowledge.

Identify the sources that contain descriptive knowledge. Identify the sources that contain rapidly changing operational state. Identify which questions require semantic understanding, which require exact identifiers, and which require calculations across complete data sets. Only then should retrieval mechanisms be selected. You must:

• Build semantic retrieval for the sources that benefit from semantic retrieval.
• Keep live access to sources where freshness matters.
• Expose structured query capabilities where exact computation matters.
• Once those primitives are reliable independently, expose them as tools to an agent.
• Then place boundaries around the agent.
• Finally, build grounding, evaluation, tracing, scoping, and cost measurement before treating the system as production ready.

That sequence matters because agentic orchestration cannot compensate for weak retrieval primitives. A model can choose between tools intelligently only when those tools themselves return trustworthy information.

What Building Matrix Taught Us

The most useful lesson from Matrix was not about embeddings or language models. It was about retrieval.

Enterprise knowledge is heterogeneous. Documentation, source code, work items, release records, external meeting minutes, and internal knowledge do not have the same semantics, freshness requirements, or query patterns. Trying to normalize all of them into one retrieval mechanism makes the architecture simpler at first and less reliable later.

Matrix takes the opposite approach.

It maintains a common reasoning layer while allowing the retrieval layer to remain heterogeneous. Vectors retrieve by meaning. Lexical search retrieves current code. Structured queries retrieve exact operational facts. Product scope limits what can be retrieved. Personas determine how the evidence should be reasoned over. The agent decides which combination is required for the question in front of it.

That architecture is what turns Matrix from a search interface over enterprise documents into something more useful: a system that can determine where an answer should come from before attempting to generate the answer itself.

And for us, that became the more interesting engineering problem to solve.

To see how these architectural components come together in practice, watch the following demonstration of Matrix in operation.

For more engineering deep dives into the architectures, design choices, and practical lessons behind production scale data systems, follow engineering.acceldata.io, where engineers globally share what they learn while building and solving complex systems in practice.