RAG System for SMEs: How to Build Your Own AI Knowledge Base

Knowledge is not the problem in most SMEs. The problem is where it is: in PDFs nobody can find anymore, in SharePoint folders that have not been cleaned up in years, and in the heads of employees who are about to retire. A RAG system changes this fundamentally—without an IT department, without a developer team, and without month-long projects.

RAG stands for Retrieval Augmented Generation. A system like this gives your knowledge base a search function that actually delivers answers. No hit lists, no ten documents to click through. Instead, a precise answer with source citation.

What RAG Actually Solves in SMEs (and What It Does Not)

At its core, RAG works simply: instead of teaching the AI model everything in advance, you retrieve the relevant documents for each query and pass them to the model as context. The model then responds based on those passages, not based on general internet knowledge.

Practical example: A plumbing shop with 18 employees had a classic problem. New employees constantly asked for maintenance instructions, measurement forms, and supplier terms. The answers were somewhere, but “somewhere” is not a helpful address. After setting up a RAG system, an internal chat tool answers these questions in seconds—with source citations. The team asks, the answer comes, work done.

What RAG does not solve: Unstructured chaos. If your documents are outdated in content, contradictory, or only exist as unreadable scanned PDFs, no system in the world will help. Good document preparation comes before good retrieval—that is the most important insight.

The Architecture: 2 Workflows, One Database

The target picture is manageable. You do not need a complex IT infrastructure. Two workflows in n8n and one database in Supabase are sufficient to get started.

Workflow 1—Ingestion: Read documents, prepare them, convert to vectors, and store in Supabase. This workflow runs either once or each time a change occurs in the source system (folder, SharePoint, DMS).

Workflow 2—Q&A: The user asks a question, the workflow searches for matching text passages in Supabase, passes them to the language model as context, and the answer comes back—including sources.

A timber construction entrepreneur with a team of 22 people implemented exactly this structure. He connected the ingestion workflow to his Google Drive directory containing construction plans and supplier lists. The system has been running for months without significant maintenance.

Connecting Data Sources: Where Your Knowledge Lives Today

For getting started, one folder is enough. A local folder or a Google Drive with the most important 20 to 50 documents is a legitimate starting point.

For scaling, SharePoint, a DMS, or structured folder hierarchies are added. n8n has native connectors for all these systems. The important point: you need a way to detect changes. Either on a schedule (“check daily for new files”) or event-driven (“as soon as a file changes, update the index”). For SME entry, the scheduled approach is entirely sufficient.

Document Preparation: The Underestimated Lever

This is where most people lose the game. Not at the model, not at retrieval—at document preparation.

Extraction

PDFs with a text layer are easy. Scanned PDFs need OCR. DOCX files are unproblematic. Emails often have a lot of noise (signatures, forwarding chains) that should be filtered out. Tables in PDFs are tricky—they are often extracted as unstructured string chaos.

Cleaning

Automatic header detection is not natively available in n8n but can be solved with regex. More important are duplicates: if the same document exists in two versions, the system needs to know which one applies. Stable document IDs help with this.

Metadata: Required Fields from the Start

Recommended minimum standard: tenant, department, docType, validFrom, validUntil, sourceUrl. These fields allow targeted filtering of search queries—for example: “Show me only documents from the Assembly department that are still valid.” This is the difference between a toy and a production system.

Chunking: Dividing Text Sensibly

A document is not stored as a whole. It is split into sections—called chunks—that are individually indexed.

Recommended Starting Values

For getting started: chunk size 512 to 800 tokens, overlap 10 to 20 percent. Adjacent chunks overlap slightly so that semantic connections are not cut off at chunk boundaries.

• Chunks that are too small (under 100 tokens) create retrieval noise—many short fragments without sufficient context.
• Chunks that are too large (over 1,500 tokens) dilute relevance because the embedding represents a mixture of multiple topics.

Structure-Aware Chunking

The best chunk boundary is a heading or section break. SOPs and checklists usually have clear steps that make good natural boundaries. Tables should be treated as standalone chunks, with a description line before them.

Embeddings: Cost, Quality, and GDPR

An embedding is a numerical representation of a text passage. This is how the system searches not by keywords but by meaning.

Which Model?

OpenAI offers two relevant models: text-embedding-3-small (default: 1,536 dimensions) and text-embedding-3-large (default: 3,072 dimensions). The dimensions parameter allows reducing the vector size—this saves storage and speeds up queries but costs some quality.

For SME entry, text-embedding-3-small with 1,536 dimensions is a good compromise. If retrieval quality is insufficient, you can switch to text-embedding-3-large.

Regarding GDPR: OpenAI provides a Data Processing Addendum (DPA) that is needed as part of the data processing documentation. It is not sufficient on its own but is an important artifact.

Supabase as a Vector Database

Supabase with pgvector is the most pragmatic choice for SMEs without their own database team: managed service, good n8n integration, and solid documentation.

The Table Model

The chunk table needs at minimum: id, content (text content), embedding (vector), metadata (JSONB for required fields), document_id (foreign key to the source document), created_at.

Index Choice: HNSW

pgvector offers two index types. HNSW delivers better query performance than IVFFlat—shorter response times and a better recall/speed ratio. The downside: more storage and longer build times during index construction. For SME data volumes (under 100,000 chunks), this is not a problem.

The Retrieval Pipeline: From Question to Answer

Pre-Filter and Top-K

First, you filter by metadata—for example: only documents from the Assembly department, only valid versions. Then a vector search runs on this filtered subset, returning the top-K most similar chunks (typical: 5 to 20).

Re-Ranking: The Biggest Quality Booster

After retrieval comes re-ranking. Cohere Rerank v4.0 is multilingual, processes up to 32,000 tokens of context, and works well for German documents and mixed formats. The principle: first quickly retrieve top-K via vector search, then re-sort the most relevant hits with the reranker before the language model responds. In practice, this is often the step that turns a mediocre system into a good one.

Response Template with Sources

The language model should respond exclusively based on the provided chunks—not based on general knowledge. You achieve this through a clear system prompt:

“Answer the question exclusively based on the following documents. Always cite the source. If no relevant information is available, respond: ‘I have no verified information on this in the knowledge base.’”

The second sentence is not optional. It prevents hallucinations and makes the system trustworthy.

Permissions and Governance: RLS in Supabase

Row Level Security (RLS) in Supabase is the most practical way to enforce access rights at the chunk level. Policies act like implicit WHERE clauses—a user only sees what their policy allows. Despite RLS, explicit filters should still be set in queries. Supabase recommends wrapping auth.uid() as (select auth.uid()) for significantly better query plans.

Audit logging—who asked what and when—is initially optional for SMEs but should be planned early for compliance-sensitive industries.

Evaluation: Does the System Work?

A golden set of 10 to 30 questions with known correct answers is the benchmark. Ask the system, compare the answers, and classify errors into three categories:

• Retrieval error: Wrong document retrieved
• Ranking error: Correct document retrieved but not prioritized
• Hallucination: Answer without basis in the document

For SME systems, two hours and a spreadsheet are enough. But without this step, you do not know whether your system is good or just looks good.

Common Mistakes and Quick Fixes

• Chunks too small with too much overlap: Retrieval noise, poor answers. Fix: increase chunks to 400 to 800 tokens.
• Metadata not structured: Retroactively re-indexing everything is tedious. Fix: establish a metadata standard before the first ingestion run.
• System prompt too open: The model hallucinates. Fix: explicitly constrain to source-based responses.
• RLS forgotten or misconfigured: Data privacy problem. Fix: test policies before go-live, explicit filters in all queries.

GDPR and EU AI Act: What SMEs Need to Know

For operating a RAG system with third-party models (OpenAI, Cohere), you need data processing agreements (DPA). OpenAI and Cohere provide corresponding agreements.

Regarding the EU AI Act: in force since August 2024, fully applicable from August 2026. Relevant for SMEs: AI literacy obligations have been in effect since February 2025, GPAI obligations since August 2025. An internal knowledge system for employees is typically not classified as a high-risk system—but documentation and transparency toward the system’s users are still advisable.

Frequently Asked Questions

What is a RAG system and why do I need one as an SME?

RAG (Retrieval Augmented Generation) is a method where an AI model provides answers based on your own documents—not based on general internet knowledge. Employees can ask in natural language about instructions, processes, and supplier information and immediately receive an answer with source citation.

Do I need programming skills?

No. With n8n as the workflow tool and Supabase as the database, you can build a functional RAG system without code. n8n offers ready-made workflow templates that can be adapted to your own sources and requirements.

How long does setup take?

For a working prototype with a document folder as the source, one to two days are sufficient with prepared documents. A production-ready system with access management, metadata standards, and an evaluation process can be realized in two to four weeks.

What does operation cost?

For a typical SME setup with OpenAI embeddings, Supabase Free/Pro, and n8n Cloud, ongoing costs are often in the low double-digit euro range per month. The effort for setup and initial document preparation is the larger cost factor.

References

• OpenAI Embedding Models - Information on text-embedding-3-small and text-embedding-3-large
• OpenAI Data Processing Addendum (DPA) - Data processing agreement for GDPR compliance
• pgvector GitHub - Vector database extension for PostgreSQL
• Supabase RLS Documentation - Row Level Security in Supabase
• Cohere Rerank - Multilingual re-ranking model
• EU AI Act - EU regulatory framework for artificial intelligence
• n8n Company Knowledge Base Agent (RAG) - Workflow template for RAG in n8n