Every company adopting AI eventually runs into the same problem.
An employee opens ChatGPT, Claude, or another AI assistant and asks a question about the business:
- How do we deploy this application?
- What did we decide about the payment provider?
- Where is the current policy?
- Has this production error happened before?
- Which database table owns this field?
- What process should the support team follow?
- Why was this architecture chosen?
The model usually does not know.
The employee then copies documents, source code, email threads, meeting notes, or internal wiki pages into the conversation. Another employee does the same thing the next day. A developer starts every coding session by explaining the application architecture again. Important lessons remain buried in old chat sessions that nobody else can access.
The company may be paying for powerful AI models, but every session starts with organizational amnesia.
That is not primarily a chatbot problem.
It is a knowledge architecture problem.
The Real Problem Is Not Missing Information
Most established companies already have plenty of information.
They have too much of it.
Knowledge is scattered across SharePoint, Google Drive, Git repositories, Confluence, Teams or Slack, ticketing systems, email, network drives, databases, employee laptops, Markdown files, old PDF manuals, meeting notes, chat histories, and the memory of the one employee who has been there for 17 years.
The difficult question is not:
Can an AI model read documents?
It can.
The difficult questions are:
- Which document is current?
- Which document is approved?
- Which version was superseded?
- Which employees are allowed to see it?
- What information should never leave the company?
- How can the answer cite its sources?
- How do we know the retrieved information is correct?
- What happens when a document is deleted or its permissions change?
- Can the company change AI providers without rebuilding everything?
Uploading folders into a chatbot does not solve those questions.
It usually hides them.
Why We Are Building This at Agave Information Solutions
We began with a smaller version of the problem inside our own development workflow.
Over time, we built an Obsidian knowledge vault containing project documentation, deployment procedures, architecture decisions, production incidents, database notes, infrastructure details, troubleshooting history, runbooks, lessons learned, and agent instructions.
The vault follows a pattern similar to Andrej Karpathy's idea of an LLM-maintained wiki: durable knowledge should live between raw source material and individual AI conversations.
That model works.
Instead of explaining an application from scratch every time we open Codex or Claude Code, an agent can start with the current project state, accepted architecture decisions, and known deployment traps.
But the vault grew.
It passed 1,000 Markdown files. Historical logs, archived plans, active runbooks, and project notes began competing for attention.
The problem was no longer whether the information existed. The problem was whether an agent could reliably answer:
Which five documents should I trust for this task?
That question led us beyond a personal Obsidian vault and toward a broader architecture that could work for other companies.
We are now building a governed organizational knowledge platform using PostgreSQL, pgvector, full-text search, an ASP.NET Core API, and the Model Context Protocol.
Our vault is tenant zero.
The long-term goal is a reusable system that connects approved company knowledge to the AI tools employees already use.
A "Shared Brain" Is the Wrong Mental Model
People often describe this kind of system as a company-wide shared brain.
The phrase is catchy, but technically dangerous.
A company does not need a machine pretending to be its brain. It needs a controlled knowledge service that can answer three questions:
- What information is relevant?
- Is the employee allowed to access it?
- What source supports the answer?
The AI model should not become the source of truth.
The model is a consumer.
The source systems remain authoritative: the current runbook, the accepted architecture decision, the approved policy, the live application code, the production database, and the document with the correct access controls.
The knowledge platform sits between those sources and the AI assistant.
Its job is to retrieve the right evidence.
The Architecture
The system we are building follows this flow:
Company knowledge sources
↓
Ingestion and normalization
↓
Metadata, ownership, permissions, and versioning
↓
Structural document chunking
↓
Embeddings and full-text indexing
↓
PostgreSQL + pgvector
↓
Permission-aware hybrid retrieval
↓
REST API and MCP tools
↓
Codex, Claude, ChatGPT, internal portals, and agents
PostgreSQL becomes the retrieval and governance layer.
It does not replace the original documents.
If the source is an Obsidian vault, employees continue editing Markdown. If the source is SharePoint, SharePoint remains authoritative. If the source is GitHub, the repository remains authoritative.
The database stores a normalized and searchable representation that can be rebuilt.
That distinction matters.
A generated search index can be deleted and recreated. A company knowledge system that silently becomes the only copy of the truth is another future migration project wearing an AI hat.
Why PostgreSQL and pgvector
There are many specialized vector databases available.
We chose PostgreSQL because company knowledge is not just a pile of vectors.
A production system also needs tenants, users, groups, document ownership, access-control lists, source identifiers, version history, lifecycle status, effective dates, retention rules, retrieval logs, user feedback, and evaluation results.
That information is relational.
PostgreSQL can store the organizational metadata, perform native full-text search, and use pgvector for vector similarity in the same system. The pgvector project specifically supports combining vector retrieval with PostgreSQL full-text search, including fusion methods such as Reciprocal Rank Fusion.
For our first implementation, local PostgreSQL is intentional. The entire retrieval stack runs in a single Docker container on a developer machine: PostgreSQL 17 with pgvector, costing nothing and touching no production infrastructure. That is not a compromise. It is the fastest honest way to answer the only question that matters at this stage: does retrieval return the right evidence?
The same local-first rule applies to embeddings. The first slice generates them with a small open-source model running in-process, which means indexing sends nothing to any external API. Providers sit behind an interface, and the store records which model produced each vector, so upgrading to a commercial embedding model later is a deliberate re-index, not a silent drift.
Starting local does not trap the system on a laptop. Because the store is rebuildable from the sources by design, promoting it to managed infrastructure later is a re-ingest, not a migration. Amazon RDS for PostgreSQL ships pgvector on every currently offered PostgreSQL major version (as of mid-2026, all of them carry pgvector 0.8 or newer), and enabling it is a single CREATE EXTENSION vector;. The schema that proves retrieval quality on one machine is the same schema that will eventually serve multiple employees, multiple applications, and multiple AI providers from managed AWS infrastructure.
Do Not Use Vector Search Alone
The typical RAG tutorial works like this:
- Split documents into chunks.
- Convert each chunk into an embedding.
- Store the embeddings.
- Convert the user's question into an embedding.
- Return the nearest chunks.
- Ask an LLM to write an answer.
That is a useful demonstration.
It is not enough for company knowledge.
Semantic similarity can retrieve a document that sounds right while being completely wrong operationally.
Imagine that the database contains the current production deployment runbook, a deployment plan abandoned two years ago, a postmortem describing a temporary emergency fix, a draft migration strategy, and an archived version of the old application.
All five documents may be semantically similar to the question:
How do we deploy the application?
The nearest vector is not necessarily the correct answer.
This is why the retrieval process must begin with metadata and authority.
Before semantic search runs, the system should filter by tenant, user permissions, project, document type, lifecycle status, authority, classification, effective date, and archive status.
Only then should it compare embeddings.
Authority Must Be Explicit
Every company already has an authority hierarchy, whether it documents one or not.
For example:
Verified production state
↓
Approved policy or current runbook
↓
Accepted architecture decision
↓
Active project documentation
↓
Reference material
↓
Historical logs
↓
Archived or superseded documents
The retrieval system must understand those distinctions.
We model lifecycle metadata such as canonical, approved, active, reference, draft, historical, superseded, archived, and expired. The first slice enforces the core buckets today, and the finer authority tiers arrive with the hardening phase.
A canonical runbook receives a strong ranking advantage.
A superseded document is excluded.
An archived incident may be available when the user explicitly asks for history, but it should not appear as current guidance.
Age alone is not enough.
A five-year-old architecture decision may still be valid. A note written yesterday may still be an unreviewed guess.
The system must model authority, not just freshness.
Permissions Must Be Applied Before Retrieval
This is the point where a personal RAG system becomes an enterprise system.
Suppose an employee asks:
What did leadership decide about the acquisition?
The database may contain relevant documents, but the employee may not be authorized to see them.
The wrong architecture performs vector search across everything and removes unauthorized results afterward.
That is dangerous.
Titles, snippets, similarity scores, or generated summaries may already reveal sensitive information.
The correct sequence is:
Authenticate the user
↓
Resolve company and group membership
↓
Determine the permitted document set
↓
Apply classification and lifecycle rules
↓
Run lexical and vector retrieval
↓
Return permitted evidence
PostgreSQL provides Row-Level Security policies that can restrict which rows are visible to particular database roles or session contexts. Once enabled, policies can apply expressions that determine whether a row is visible or writable.
Application-level permission checks are still necessary, but database-level enforcement provides defense in depth.
For a multi-company platform, every relevant table should carry a tenant identifier from the beginning.
Adding tenancy after the data model is complete is how architecture teams accidentally invent a six-month rewrite.
Hybrid Search Works Better Than Semantic Search Alone
Company questions often contain exact technical language: error codes, database table names, API routes, customer identifiers, product names, class names, server names, and policy numbers.
Embeddings are good at meaning.
They are not always good at exact identifiers.
PostgreSQL full-text search can retrieve exact and lexical matches. pgvector can retrieve semantically similar passages. A fusion algorithm can combine both result sets.
Our pipeline uses:
Metadata and ACL filters
↓
PostgreSQL full-text retrieval
+
pgvector semantic retrieval
↓
Reciprocal Rank Fusion
↓
Authority adjustment
↓
Optional reranking
↓
Cited evidence
We are starting with exact vector search rather than immediately adding approximate indexes.
Our first corpus is small enough that query quality matters more than shaving milliseconds off a benchmark.
Optimization without a measured problem is just architecture cosplay.
Documents Should Be Chunked by Structure
Another common RAG mistake is chopping documents every fixed number of characters.
That can produce chunks such as:
...restart the production server. ## Do Not Restart the Production Server Under no circumstances...
The first chunk now contains advice that appears to say the exact opposite of the document.
Markdown already contains useful structure: document titles, headings, subheadings, lists, tables, code blocks, and wikilinks.
We parse the heading structure and create chunks around meaningful sections. The parser is code-fence aware, so a # comment inside a code block is never mistaken for a heading.
Each stored chunk retains context such as:
Document: Hiberden Deployment Runbook Project: Hiberden Type: Runbook Authority: Canonical Heading: Backups > Cloud Destinations > Resume
That improves retrieval and makes citations understandable.
It also allows the system to re-process only the documents that changed instead of paying to re-embed the entire corpus.
The Database Is an Index, Not the Author
The source document remains the editable artifact.
For our first implementation:
Obsidian Markdown + Git = source of truth PostgreSQL = searchable representation Embeddings = ranking signal Full-text search = ranking signal Metadata = authority and security boundary MCP/API = controlled agent access
The ingestion process will:
- Scan approved folders.
- Exclude archives, credentials, caches, and unsupported files.
- Read YAML frontmatter.
- Parse Markdown structure.
- Calculate content hashes.
- Skip unchanged documents.
- Create stable chunks.
- Reuse unchanged embeddings.
- Insert new versions transactionally.
- Reconcile deleted and moved documents.
A filesystem watcher may provide fast updates, but periodic full reconciliation is still necessary. Watchers miss events. Production systems should assume that eventually something will fail while nobody is watching the watcher.
Connecting Codex and Claude Through MCP
We do not want every AI client to implement its own database access.
The platform exposes a controlled retrieval API and an MCP server.
MCP allows compatible AI clients to access defined tools and contextual data. Codex supports MCP servers through both its CLI and IDE integration.
The first tool is already live: search_knowledge, a read-only retrieval tool that Claude Code picks up from a one-file repository configuration. The next tools on the roadmap are narrow in the same way:
search_knowledge get_document_section get_current_project_context submit_retrieval_feedback
The server will not expose:
run_arbitrary_sql browse_every_customer update_company_policy execute_deployment
The first version is intentionally read-only.
The agent can retrieve evidence. It cannot silently rewrite the company's operating procedures because it had a particularly confident Tuesday.
The Smallest Useful Implementation, Built
We did not start with every company source and a polished chat interface.
The first complete vertical slice is:
One approved Markdown folder
↓
Structural parsing
↓
Chunking
↓
Embeddings
↓
PostgreSQL storage
↓
Hybrid search
↓
Cited API response
↓
One MCP tool
↓
Ten evaluation questions
We built that slice against our own vault before publishing this post.
The first full run ingested 30 approved documents into 179 heading-scoped chunks and passed all ten golden evaluation questions, with mean retrieval latency around 50 milliseconds on a local container. The set includes a trap case where an archived near-duplicate of a current page must lose to the current page, and a case where the correct behavior is admitting the corpus does not contain the answer.
That proves the important parts:
- The ingestion pipeline works.
- The database model works.
- Retrieval works.
- Citations work.
- Agents can use the service.
- Quality can be measured.
Only after that works should the system expand into multiple tenants, source permissions, SharePoint, Google Drive, GitHub, Confluence, administrative portals, usage analytics, and customer onboarding.
Building six connectors before proving retrieval quality is a reliable way to create six pipelines that all return the wrong document.
How We Know Whether It Works
A RAG system cannot be judged by whether the answer sounds intelligent.
Language models are exceptionally good at making weak retrieval look convincing.
We maintain an evaluation set containing real questions and expected evidence.
An evaluation case defines:
question: How do we deploy this application to production? expected_documents: - current deployment runbook forbidden_documents: - archived deployment plan required_authority: - canonical expected_facts: - publish location - deployment verification
We track:
- Whether the correct document appears in the top results
- Whether the canonical source ranks above historical notes
- Whether archived documents appear when they should not
- Whether citations point to the correct section
- Whether permission rules leak restricted content (once the permission model lands)
- Whether the system correctly returns no answer
- Retrieval latency
- Embedding costs
- User feedback
The most important test is not:
Did the model produce a good paragraph?
It is:
Did the platform give the model the correct evidence?
What This Means for Companies Adopting AI
Companies do not need to standardize every employee on one AI model.
Models will continue to change.
Some teams will use ChatGPT. Developers may use Codex or Claude Code. Other employees may use Microsoft Copilot, internal applications, or models running through Amazon Bedrock.
The durable investment is the knowledge layer behind them.
A well-designed platform allows the company to control sources, permissions, authority, retrieval, citations, logging, and evaluation.
The LLM becomes replaceable.
That is a healthier architecture than uploading the same employee handbook into six AI products and hoping each one remembers which version is current.
Where We Are Starting
At Agave Information Solutions, we are using our own Obsidian knowledge vault as tenant zero.
That gives us a realistic test corpus containing years of application architecture, AWS infrastructure, deployment procedures, database design, production incidents, operational lessons, project history, and both current and obsolete documentation.
It also gives us the exact problem many companies face: valuable knowledge mixed with historical clutter.
We will document the implementation as we build it, including why we chose PostgreSQL and pgvector, how we model authority, how we chunk Markdown, how hybrid retrieval performs, how we connect Codex and Claude, how we test permissions, what the AWS infrastructure costs when we promote the store to RDS, what fails, and what turns out to be unnecessary.
The objective is not to produce another RAG tutorial that works against five clean PDFs.
The objective is to build a system that can survive real company data, real permissions, stale documents, changing AI tools, and employees who need the correct answer before the production deployment, not three days later during the postmortem.
The Takeaway
Most organizations do not need another standalone AI chatbot.
They need an organizational knowledge layer that can answer:
- What information is current?
- Who is allowed to see it?
- What source supports the answer?
- How do we know retrieval is working?
- Can the same system serve multiple AI tools?
Our implementation uses Obsidian as the initial source, PostgreSQL and pgvector as the governed retrieval store, hybrid search for relevance, authority metadata for trust, and MCP for controlled agent access. All of it starts local: one container, one folder of approved documents, zero cloud spend, and nothing leaving the building until the company decides it should.
The first milestone is simple to describe:
An authenticated employee or AI agent can retrieve the correct, current, permitted company evidence with citations, and the company can prove why those results were returned.
The retrieval half of that milestone, correct, current, cited, and measured, is running today against our own vault. The authenticated, permission-aware half is the next phase.
That is not a shared brain.
It is something more useful: company knowledge that employees and AI systems can finally use without losing control of it.
Agave Information Solutions builds custom software, database systems, AWS infrastructure, and private AI search solutions for organizations with complicated data. This series will document the platform from the first Markdown scan through a production-ready, permission-aware knowledge service.