Apache Tika: From Document Extraction to AI-Powered Search
Who this is for: Software engineers and data engineers who want to build production-grade document ingestion pipelines — from raw files all the way to LLM-powered semantic search.
Architecture Overview
The end-to-end pipeline this article builds:
┌─────────────────────────────────────────────────────────────────────────┐
│ DOCUMENTS (Input Layer) │
│ PDF │ DOCX │ XLSX │ Images │ HTML │ Audio/Video │
└──────────────────────────────┬──────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ APACHE TIKA (Extraction Layer) │
│ │
│ ┌─────────────────┐ ┌──────────────────┐ ┌───────────────────┐ │
│ │ MIME Detector │──▶│ Parser Router │──▶│ Content Handler │ │
│ │ (magic bytes, │ │ (AutoDetect, │ │ (SAX events → │ │
│ │ extensions) │ │ 1000+ parsers) │ │ plain text) │ │
│ └─────────────────┘ └──────────────────┘ └───────────────────┘ │
│ │ │
│ ┌──────────────┴───────────┐ │
│ │ Metadata Object │ │
│ │ (Dublin Core, XMP, │ │
│ │ EXIF, Office props) │ │
│ └──────────────────────────┘ │
└──────────────────────────────┬──────────────────────────────────────────┘
│
┌───────────────┴───────────────┐
│ │
▼ ▼
┌──────────────────────────┐ ┌─────────────────────────────────────┐
│ RELATIONAL DATABASE │ │ VECTOR DATABASE │
│ (PostgreSQL) │ │ (ChromaDB / Pinecone) │
│ │ │ │
│ • file path │ │ chunk_1 → [0.12, -0.45, ...] │
│ • author │ │ chunk_2 → [0.33, 0.81, ...] │
│ • created_date │ │ chunk_3 → [-0.07, 0.24, ...] │
│ • content_type │ │ (text-embedding-ada-002) │
│ • language │ │ │
│ • raw_text │ └────────────────┬────────────────────┘
└──────────────────────────┘ │
│ semantic search
▼
┌─────────────────────────────────────┐
│ LLM QUERY LAYER │
│ │
│ User question │
│ │ │
│ ▼ │
│ Embed question │
│ │ │
│ ▼ │
│ Retrieve top-k chunks │
│ │ │
│ ▼ │
│ Build prompt + context │
│ │ │
│ ▼ │
│ GPT-4 / local LLM (Ollama) │
│ │ │
│ ▼ │
│ Grounded answer │
└─────────────────────────────────────┘
1. What is Apache Tika?
Apache Tika is a content detection and extraction toolkit maintained by the Apache Software Foundation. It solves one of the most tedious problems in data engineering: how do you extract plain text and metadata from an arbitrary file when you don't know what format it is?
Before Tika, engineers writing document processing systems had to wire together a different library for each format — PDFBox for PDFs, Apache POI for Office files, ImageIO for images, JAudio Tagger for MP3s, and so on. Each library had a different API, different error handling, and different metadata conventions. Maintaining that collection at scale was a serious engineering burden.
Tika collapses all of that behind a single, consistent API. You hand it a file (or a stream), and it hands back plain text and a metadata map. It does not matter whether the file is a Word document, a ZIP archive, a JPEG with GPS data, or an MP4 video — the calling code is identical.
Core components
| Component | Responsibility |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| MIME Detector | Identifies the file type using magic bytes, file extension, and XML root element hints. Returns a MIME type string such as application/pdf. |
| Parser Router | Maps the detected MIME type to the correct parser implementation. Uses Java's ServiceLoader SPI so parsers can be plugged in without changing core code. |
| ContentHandler | A SAX-style event receiver that collects the extracted text. The most common implementation, BodyContentHandler, builds a plain text string. |
| Metadata | A mutable key-value map populated during parsing. Keys follow Dublin Core, XMP, and format-specific vocabularies. |
| ParseContext | A thread-local configuration bag for passing options into parsers (passwords, locale, rendering hints). |
Tika is written in Java and lives at tika.apache.org. It is released under the Apache 2.0 licence. The project has been active since 2007 and is a direct dependency of Apache Solr, Apache Nutch, and many enterprise search products.
2. Key Advantages of Apache Tika
Format breadth
Tika ships with parsers for more than 1,000 file formats out of the box:
- Documents: PDF (via PDFBox), DOCX/XLSX/PPTX (via Apache POI), ODF, RTF, EPUB, Pages
- Images: JPEG, PNG, TIFF, WebP — including full EXIF metadata extraction
- Audio/Video: MP3 (ID3 tags), MP4, Ogg, FLAC, WAV
- Archives: ZIP, TAR, GZip, 7-Zip — with recursive extraction of nested content
- Web/Data: HTML, XML, JSON, CSV, RSS/Atom feeds
- Source code: Java, Python, C++, and dozens of other languages
Single unified API
No matter the format, extraction always looks the same:
Tika tika = new Tika();
String text = tika.parseToString(new File("report.pdf"));
This uniformity makes it trivial to add new file types to an existing pipeline.
Language detection
Tika bundles Apache OpenNLP and the Optimaize language detector. After parsing, you can call LanguageIdentifier to get an ISO 639-1 language code with a confidence score — useful for routing multilingual content to the right downstream model.
OCR via Tesseract
For scanned PDFs and images containing text, Tika integrates with Tesseract OCR. When Tesseract is installed and Tika detects an image-only PDF, it automatically rasterises each page and runs OCR. No code change required.
⚠ Gotcha: OCR is slow. On a standard laptop, expect 3–8 seconds per page. Use the Tika server with a dedicated OCR worker, or pre-filter documents to only OCR when normal text extraction returns empty content.
REST server mode
The tika-server JAR exposes all extraction functionality over HTTP. Any language — Python, Node.js, Go, Ruby — can call Tika by sending a file as a POST body and receiving text or JSON back. This also provides process isolation: a malformed file that crashes the JVM does not take down your main application.
How it compares
| | Apache Tika | Unstructured.io | AWS Textract | | --------------- | --------------------- | --------------------- | ----------------------- | | Hosting | Self-hosted | Self-hosted / cloud | Managed cloud | | Formats | 1,000+ | ~30 common | PDF, images | | OCR | Tesseract | Tesseract + layout AI | AWS proprietary | | Cost | Free | Free / paid tiers | Pay-per-page | | LLM integration | Manual | Built-in chunking | Manual | | Best for | Broad format coverage | Modern RAG pipelines | Regulated AWS workloads |
3. Real-World Organizational Use Cases
Enterprise search
A large organisation typically has documents spread across SharePoint, file shares, email archives, and legacy ECM systems in dozens of formats. Tika is the extraction layer in front of Apache Solr or Elasticsearch. It normalises everything to plain text and a structured metadata envelope, which the search engine then indexes. Without Tika, the ingest pipeline would require separate connectors for every format.
Legal e-discovery
Law firms and corporate legal teams use Tika to process "custodian data" — everything on a person's laptop or email account. E-discovery platforms need to extract text from every file regardless of format (often millions of files), detect language, and extract creation/modification dates for privilege review timelines. Tika's recursive archive extraction is particularly important here: a single ZIP may contain nested emails with embedded attachments.
ML training data preparation
Building a large text corpus for pre-training or fine-tuning requires ingesting raw files from the web and internal repositories. Tika converts heterogeneous file dumps into clean UTF-8 text streams that can be fed directly to tokenisers. It also provides content-type metadata that lets data engineers filter out binary files or non-text formats before they enter the training loop.
RAG / LLM pipelines
Retrieval-Augmented Generation systems need a reliable way to ingest enterprise documents and make them searchable by meaning. Tika is the first stage: it extracts text from PDFs, Office files, and other formats, which is then chunked and embedded into a vector store. This article's pipeline (sections 7–9) is a working implementation of this pattern.
Content migration
When moving from one CMS or ECM to another, teams need to extract content from the old system's proprietary formats, transform it, and load it into the new system. Tika handles the extraction step, including stripping formatting and preserving structural metadata like headings and table structure in its XHTML output mode.
Digital forensics
Forensic investigators use Tika to extract text and metadata from evidence drives at scale. The metadata Tika extracts — GPS coordinates from camera photos, author fields from Office documents, creation timestamps — is often legally significant. The tika-app JAR can be scripted from the command line to produce machine-readable output for forensic toolchains.
4. Installation and Setup
Maven
<dependencies>
<!-- Core Tika API -->
<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-core</artifactId>
<version>2.9.1</version>
</dependency>
<!-- All standard parsers (PDF, Office, images, audio, archives, etc.) -->
<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-parsers-standard-package</artifactId>
<version>2.9.1</version>
</dependency>
</dependencies>
Why
tika-parsers-standard-package?tika-corecontains only the interfaces and the detection/routing engine. Without a parsers JAR, Tika can detect MIME types but cannot extract any content.tika-parsers-standard-packagebundles all Apache-maintained parsers and their transitive dependencies (PDFBox, POI, etc.) into a single convenient dependency. For a leaner build, you can instead include only the individual parser modules you need.
Gradle
dependencies {
implementation 'org.apache.tika:tika-core:2.9.1'
implementation 'org.apache.tika:tika-parsers-standard-package:2.9.1'
}
CLI (tika-app JAR)
Download the fat JAR from the Apache Tika downloads page:
wget https://dlcdn.apache.org/tika/2.9.1/tika-app-2.9.1.jar
# Extract text from a PDF
java -jar tika-app-2.9.1.jar --text report.pdf
# Extract metadata as JSON
java -jar tika-app-2.9.1.jar --metadata --json report.pdf
# Detect MIME type only
java -jar tika-app-2.9.1.jar --detect report.pdf
# Detect language
java -jar tika-app-2.9.1.jar --language report.pdf
Java "Hello World"
import org.apache.tika.Tika;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.parser.AutoDetectParser;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.sax.BodyContentHandler;
import java.io.FileInputStream;
import java.io.InputStream;
public class TikaHelloWorld {
public static void main(String[] args) throws Exception {
// High-level facade — simplest possible usage
Tika tika = new Tika();
String text = tika.parseToString(new java.io.File("report.pdf"));
System.out.println("=== Extracted Text (first 500 chars) ===");
System.out.println(text.substring(0, Math.min(500, text.length())));
// Lower-level API — gives you metadata too
AutoDetectParser parser = new AutoDetectParser();
BodyContentHandler handler = new BodyContentHandler(-1); // -1 = no size limit
Metadata metadata = new Metadata();
ParseContext context = new ParseContext();
try (InputStream stream = new FileInputStream("report.pdf")) {
parser.parse(stream, handler, metadata, context);
}
System.out.println("\n=== Metadata ===");
for (String name : metadata.names()) {
System.out.println(name + ": " + metadata.get(name));
}
}
}
5. Starting the Tika REST Server
Running Tika as an HTTP server is the recommended deployment pattern for any multi-language environment, and it provides critical process isolation: a file that triggers a JVM crash does not kill your application server.
Via JAR
wget https://dlcdn.apache.org/tika/2.9.1/tika-server-standard-2.9.1.jar
# Start on default port 9998
java -jar tika-server-standard-2.9.1.jar
# Custom host and port
java -jar tika-server-standard-2.9.1.jar --host=0.0.0.0 --port=9998
# Enable forked child process (recommended for production — isolates crash-prone parsers)
java -jar tika-server-standard-2.9.1.jar --fork
Via Docker
# Pull and run the official image
docker pull apache/tika:2.9.1
docker run -d \
--name tika \
-p 9998:9998 \
apache/tika:2.9.1
# With OCR (Tesseract included in the full image)
docker pull apache/tika:2.9.1-full
docker run -d \
--name tika-ocr \
-p 9998:9998 \
apache/tika:2.9.1-full
Key REST endpoints
| Endpoint | Method | Returns |
| ------------------------------------ | ------ | ------------------------------- |
| PUT /tika | PUT | Plain text body of the document |
| PUT /meta | PUT | Metadata as JSON |
| PUT /detect/stream | PUT | MIME type string |
| PUT /language/stream | PUT | Detected language code |
| PUT /tika with Accept: text/html | PUT | Structured XHTML output |
curl examples
# Extract plain text
curl -T report.pdf http://localhost:9998/tika
# Extract metadata as JSON
curl -T report.pdf \
-H "Accept: application/json" \
http://localhost:9998/meta
# Detect MIME type
curl -T report.pdf http://localhost:9998/detect/stream
# Detect language
curl -T report.pdf http://localhost:9998/language/stream
# Get XHTML output (preserves headings, tables, lists)
curl -T report.pdf \
-H "Accept: text/html" \
http://localhost:9998/tika
🔒 Security note: Do not expose the Tika server directly to the internet. It accepts arbitrary files and has no authentication. Place it behind an internal network boundary or a reverse proxy with access controls.
6. Extracting Data from a PDF — Python
Using tika-python
pip install tika
from tika import parser
# Parse a single PDF
parsed = parser.from_file("report.pdf")
# parsed is a dict with two keys: 'content' and 'metadata'
text = parsed["content"] # str — the full extracted text
metadata = parsed["metadata"] # dict — all metadata fields
print("=== First 500 characters ===")
print(text[:500] if text else "(no text extracted — likely a scanned PDF)")
print("\n=== Key Metadata Fields ===")
interesting_keys = [
"dc:title", "dc:creator", "dcterms:created",
"dcterms:modified", "pdf:PDFVersion",
"Content-Type", "language"
]
for key in interesting_keys:
if key in metadata:
print(f"{key}: {metadata[key]}")
ℹ️ Note: tika-python automatically downloads and starts a Tika server JVM on the first call. Set
TIKA_SERVER_ENDPOINTto point at an existing server instead:import os os.environ["TIKA_SERVER_ENDPOINT"] = "http://localhost:9998"
Using the REST API directly (requests)
This is the recommended approach for production — you control the server lifecycle.
import requests
import json
TIKA_URL = "http://localhost:9998"
def extract_text(file_path: str) -> str:
with open(file_path, "rb") as f:
response = requests.put(
f"{TIKA_URL}/tika",
data=f,
headers={"Accept": "text/plain"}
)
response.raise_for_status()
return response.text
def extract_metadata(file_path: str) -> dict:
with open(file_path, "rb") as f:
response = requests.put(
f"{TIKA_URL}/meta",
data=f,
headers={"Accept": "application/json"}
)
response.raise_for_status()
return response.json()
if __name__ == "__main__":
text = extract_text("report.pdf")
meta = extract_metadata("report.pdf")
print(text[:500])
print(json.dumps(meta, indent=2))
Common gotchas
Scanned PDFs return empty text. If content is None or an empty string, the PDF contains only images. Enable OCR by using the apache/tika:2.9.1-full Docker image (which includes Tesseract) and passing the X-Tika-OCRLanguage header:
response = requests.put(
f"{TIKA_URL}/tika",
data=open("scanned.pdf", "rb"),
headers={
"Accept": "text/plain",
"X-Tika-OCRLanguage": "eng",
"X-Tika-PDFextractInlineImages": "true"
}
)
Encoding issues. Always decode responses as UTF-8. The requests library does this automatically when you access .text, but if you read raw bytes, call .decode("utf-8", errors="replace").
Large files. Tika's default in-memory limit is 10 MB of extracted text. Increase it by passing writeLimit=-1 in ParseContext (Java) or by setting the Content-Length header appropriately from the client.
7. Loading Extracted Data into PostgreSQL
This script walks a folder of PDFs, extracts text and metadata from each, and loads everything into a PostgreSQL table.
Schema
CREATE TABLE IF NOT EXISTS documents (
id SERIAL PRIMARY KEY,
file_path TEXT UNIQUE NOT NULL,
file_name TEXT,
content_type TEXT,
author TEXT,
created_at TIMESTAMPTZ,
modified_at TIMESTAMPTZ,
language TEXT,
page_count INT,
raw_text TEXT,
ingested_at TIMESTAMPTZ DEFAULT NOW()
);
-- Full-text search index (optional but recommended)
CREATE INDEX IF NOT EXISTS idx_documents_fts
ON documents USING gin(to_tsvector('english', COALESCE(raw_text, '')));
Python ingestion script
import os
import requests
import psycopg2
from psycopg2.extras import execute_values
from datetime import datetime
from pathlib import Path
TIKA_URL = "http://localhost:9998"
PG_DSN = "postgresql://user:password@localhost:5432/mydb"
DOCS_DIR = "./documents"
def extract(file_path: str) -> tuple[str, dict]:
with open(file_path, "rb") as f:
data = f.read()
text_resp = requests.put(
f"{TIKA_URL}/tika", data=data,
headers={"Accept": "text/plain"}
)
meta_resp = requests.put(
f"{TIKA_URL}/meta", data=data,
headers={"Accept": "application/json"}
)
text_resp.raise_for_status()
meta_resp.raise_for_status()
return text_resp.text, meta_resp.json()
def parse_date(value) -> datetime | None:
if not value:
return None
if isinstance(value, list):
value = value[0]
try:
return datetime.fromisoformat(value.replace("Z", "+00:00"))
except (ValueError, AttributeError):
return None
def ingest_folder(docs_dir: str):
conn = psycopg2.connect(PG_DSN)
cur = conn.cursor()
pdf_files = list(Path(docs_dir).rglob("*.pdf"))
print(f"Found {len(pdf_files)} PDFs")
rows = []
for pdf in pdf_files:
print(f" Processing {pdf.name} ...", end=" ")
try:
text, meta = extract(str(pdf))
rows.append((
str(pdf),
pdf.name,
meta.get("Content-Type"),
meta.get("dc:creator") or meta.get("Author"),
parse_date(meta.get("dcterms:created") or meta.get("Creation-Date")),
parse_date(meta.get("dcterms:modified") or meta.get("Last-Modified")),
meta.get("language"),
int(meta.get("xmpTPg:NPages", 0) or 0),
text,
))
print("OK")
except Exception as e:
print(f"FAILED: {e}")
# Bulk insert — ON CONFLICT skips already-ingested files
execute_values(cur, """
INSERT INTO documents
(file_path, file_name, content_type, author,
created_at, modified_at, language, page_count, raw_text)
VALUES %s
ON CONFLICT (file_path) DO NOTHING
""", rows)
conn.commit()
cur.close()
conn.close()
print(f"\nInserted up to {len(rows)} rows.")
if __name__ == "__main__":
ingest_folder(DOCS_DIR)
📝 Note on large text fields: PostgreSQL stores TEXT columns as TOAST (The Oversized Attribute Storage Technique) transparently, so you don't need to worry about column size limits. For full-text search, use the GIN index above — it's far faster than
LIKE '%keyword%'on large corpora.
8. Loading Extracted Data into a Vector Database
With text in PostgreSQL, you can do keyword search. For semantic search ("find documents about supply chain risk" even if those exact words don't appear), you need a vector database.
pip install langchain langchain-openai chromadb openai tiktoken
import os
import psycopg2
import chromadb
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
# Alternative: from langchain_community.embeddings import HuggingFaceEmbeddings
PG_DSN = "postgresql://user:password@localhost:5432/mydb"
CHROMA_PATH = "./chroma_db"
COLLECTION = "documents"
OPENAI_API_KEY = os.environ["OPENAI_API_KEY"]
# --- Chunking ---
splitter = RecursiveCharacterTextSplitter(
chunk_size=800, # characters per chunk
chunk_overlap=100, # overlap to preserve context across chunk boundaries
separators=["\n\n", "\n", ". ", " ", ""],
)
# --- Embeddings ---
embeddings = OpenAIEmbeddings(
model="text-embedding-3-small", # cheaper than ada-002, similar quality
openai_api_key=OPENAI_API_KEY,
)
# Local alternative (no API key required):
# embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
# --- ChromaDB client ---
chroma = chromadb.PersistentClient(path=CHROMA_PATH)
collection = chroma.get_or_create_collection(
name=COLLECTION,
metadata={"hnsw:space": "cosine"},
)
def embed_documents():
conn = psycopg2.connect(PG_DSN)
cur = conn.cursor()
cur.execute("""
SELECT id, file_name, raw_text, author, language
FROM documents
WHERE raw_text IS NOT NULL AND raw_text <> ''
""")
rows = cur.fetchall()
cur.close()
conn.close()
total_chunks = 0
for doc_id, file_name, raw_text, author, language in rows:
chunks = splitter.split_text(raw_text)
ids = [f"doc{doc_id}_chunk{i}" for i in range(len(chunks))]
metadatas = [
{
"doc_id": doc_id,
"file_name": file_name or "",
"author": author or "",
"language": language or "",
"chunk_idx": i,
}
for i in range(len(chunks))
]
# Embed in batches of 100 to stay within rate limits
batch_size = 100
for start in range(0, len(chunks), batch_size):
batch_texts = chunks[start:start + batch_size]
batch_ids = ids[start:start + batch_size]
batch_meta = metadatas[start:start + batch_size]
vectors = embeddings.embed_documents(batch_texts)
collection.upsert(
ids=batch_ids,
embeddings=vectors,
documents=batch_texts,
metadatas=batch_meta,
)
total_chunks += len(chunks)
print(f" {file_name}: {len(chunks)} chunks embedded")
print(f"\nTotal chunks in vector DB: {total_chunks}")
if __name__ == "__main__":
embed_documents()
💡 Tip — Pinecone alternative: Replace the ChromaDB block with:
from pinecone import Pinecone, ServerlessSpec pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"]) pc.create_index("documents", dimension=1536, spec=ServerlessSpec(cloud="aws", region="us-east-1")) index = pc.Index("documents") index.upsert(vectors=list(zip(batch_ids, vectors, batch_meta)))
9. Using an LLM to Search and Query the Data
With chunks embedded in ChromaDB, you can now answer natural language questions by retrieving the relevant chunks and passing them as context to an LLM.
pip install openai
import os
import chromadb
from openai import OpenAI
from langchain_openai import OpenAIEmbeddings
CHROMA_PATH = "./chroma_db"
COLLECTION = "documents"
OPENAI_API_KEY = os.environ["OPENAI_API_KEY"]
chroma = chromadb.PersistentClient(path=CHROMA_PATH)
collection = chroma.get_collection(COLLECTION)
embeddings = OpenAIEmbeddings(
model="text-embedding-3-small",
openai_api_key=OPENAI_API_KEY,
)
client = OpenAI(api_key=OPENAI_API_KEY)
# Ollama alternative (local, no API key):
# client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
# model = "llama3"
def answer(question: str, top_k: int = 5, model: str = "gpt-4o") -> str:
# 1. Embed the user's question using the same model
question_vector = embeddings.embed_query(question)
# 2. Retrieve the top-k most similar chunks
results = collection.query(
query_embeddings=[question_vector],
n_results=top_k,
include=["documents", "metadatas", "distances"],
)
chunks = results["documents"][0]
metadatas = results["metadatas"][0]
distances = results["distances"][0]
# 3. Build a context string from retrieved chunks
context_parts = []
for chunk, meta, dist in zip(chunks, metadatas, distances):
source = meta.get("file_name", "unknown")
context_parts.append(f"[Source: {source} | similarity: {1 - dist:.2f}]\n{chunk}")
context = "\n\n---\n\n".join(context_parts)
# 4. Construct the RAG prompt
system_prompt = """You are a precise document analyst.
Answer the user's question using ONLY the context provided below.
If the answer is not present in the context, say "I could not find this in the provided documents."
Do not use any prior knowledge. Cite the source file for each claim."""
user_prompt = f"""Context:
{context}
Question: {question}"""
# 5. Call the LLM
response = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt},
],
temperature=0.1, # low temperature for factual retrieval tasks
max_tokens=1000,
)
return response.choices[0].message.content
# --- Interactive REPL ---
if __name__ == "__main__":
print("Document QA system ready. Type 'quit' to exit.\n")
while True:
question = input("Question: ").strip()
if question.lower() in ("quit", "exit", "q"):
break
if not question:
continue
print("\nAnswer:")
print(answer(question))
print()
Why RAG is more reliable than asking the LLM from memory
A base LLM has no access to your internal documents. It will either hallucinate an answer or refuse to answer. RAG grounds the LLM's response in actual retrieved text from your corpus. Setting temperature=0.1 and instructing the model to cite sources makes hallucination detectable: if a claimed fact does not appear in any retrieved chunk, it did not come from your documents.
💡 Tip — Ollama for local inference: Replace the OpenAI client with:
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama") MODEL = "llama3.1"Run
ollama pull llama3.1first. This gives you a fully air-gapped pipeline with no data leaving your network.
10. End-to-End Pipeline Summary
Documents (PDF, DOCX, XLSX, images, audio …)
│
▼
┌──────────────────┐
│ Apache Tika │ ← single extraction API for all formats
│ REST Server │ runs in Docker, language-agnostic
└──────┬───────────┘
│
text + metadata
│
┌────┴─────────────────────────────────────────┐
│ │
▼ ▼
┌──────────────────┐ ┌────────────────────────┐
│ PostgreSQL │ │ Chunker │
│ │ │ (LangChain splitter) │
│ Structured │ └────────────┬───────────┘
│ metadata + │ │
│ keyword search │ ▼
│ (GIN FTS index) │ ┌────────────────────────┐
└──────────────────┘ │ Embeddings │
│ (OpenAI / local) │
└────────────┬───────────┘
│
▼
┌────────────────────────┐
│ ChromaDB / Pinecone │
│ (Vector store) │
└────────────┬───────────┘
│
semantic retrieval
│
▼
┌────────────────────────┐
│ LLM (GPT-4 / Ollama) │
│ + RAG prompt │
└────────────┬───────────┘
│
▼
Grounded answer
with source citations
Checklist for production
- [ ] Run Tika server with
--forkflag for parser crash isolation - [ ] Use
apache/tika:2.9.1-fullif you need OCR - [ ] Place Tika behind an internal network — no public exposure
- [ ] Set
ON CONFLICT DO NOTHING(orDO UPDATE) in PostgreSQL inserts for idempotent re-runs - [ ] Chunk size 600–1000 characters works well for most technical documents; tune for your domain
- [ ] Use
temperature=0.1for factual RAG queries - [ ] Log retrieval scores alongside answers so you can detect low-confidence retrievals