Context
A large enterprise energy company had accumulated more than 75,000 technical documents, over 300 GB, across PDFs, Word files, PowerPoint decks, Excel sheets, and SAP exports. Engineering manuals ran 1,000 to 6,000 pages, with embedded tables, scanned pages, nested sections, and layouts that varied by document and department. Finding an answer meant an engineer manually searching across multiple repositories and cross-checking sections by hand.
The corpus itself was the constraint. Documents this large and this inconsistently formatted break standard chunking before retrieval even starts, and the domain language compounds it: engineering text is dense with abbreviations, equipment identifiers, and acronyms that don't resemble the plain-language phrasing embedding models are tuned for. Multiple engineering teams also needed to share the same platform without seeing each other's documents, so access control had to sit at the retrieval layer, not just the login screen. A system built on generic parsing, dense-only retrieval, and application-level filtering was never going to hold up against this corpus.
The Hard Problem
Dense semantic retrieval alone hit 65–75% top-k hit rate on an SME-curated evaluation set. The gap wasn't semantic, it was terminological. Engineers searched using shorthand: a query for "PID" needed to match documents that spelled it out as "Piping and Instrumentation Diagram." Embedding models trained on general language don't reliably associate an acronym with its expansion, so the correct document simply never surfaced, regardless of how well the query matched the underlying meaning.
Generic parsers made this worse before retrieval even began. Running standard extraction across mixed PDFs, Office files, and SAP exports flattened tables, lost section hierarchy, and dropped the structural context that made a 6,000-page manual navigable in the first place.
The failure mode that mattered most was an engineer acting on an incorrect or hallucinated answer. In petroleum engineering, wrong technical information can lead to bad operational decisions or safety incidents. That made independent verifiability a hard requirement for every response, not a feature to add later.
How It Works
Ingestion
Multi-parser extraction
Documents are parsed with RAGFlow DeepDoc, PDFPlumber, and custom logic rather than a single extractor, because no one tool preserved table structure, section hierarchy, and metadata across PDFs, Office formats, and SAP exports.
Structural chunk metadata
Each chunk carries document ID, page number, section hierarchy, and paragraph offset. That metadata is what later makes exact citation possible instead of a vague "found in this document" reference.
Synonym expansion at index time
Domain terminology is expanded through a synonym pipeline built for this corpus: "PID" is mapped to "Piping and Instrumentation Diagram" with the abbreviation preserved alongside it, so the expansion is already in the searchable text rather than something the query has to guess.
Query
Authorization scoping
Before any search runs, the request is scoped to the knowledge bases the requesting user is authorized for, so retrieval never touches documents outside that user's access.
Hybrid retrieval
Dense semantic search and sparse lexical search run together, because dense alone missed exact terminology matches and sparse alone missed paraphrased intent.
Chunk-level citation binding
Results from both retrievers are merged before generation, and the response references specific chunk IDs rather than the document as a whole, letting the interface jump to the exact page and paragraph a chunk came from.
Tracing and observability
Every query is traced through Arize Phoenix, capturing retrieval quality, retrieved context, prompt execution, and per-stage latency.
Query intelligence loop
A scheduled offline job clusters production queries to surface recurring topics and retrieval gaps, closing the loop between what engineers are actually asking and what the retrieval and synonym layers still need to cover.
Evaluation
Evaluation methodology
- Test set
- Built by subject matter experts from the client's own petroleum engineering documentation, not synthetic data. For each case, SMEs wrote a representative query, identified the expected supporting context, and provided a reference answer.
- Query types
- Engineering abbreviations and acronyms, equipment and asset documentation, process and operational procedures, technical specifications, cross-document lookups, and general knowledge queries.
- Pass/fail
- Two separate judgments. Retrieval passed only if the SME-identified supporting context appeared in the retrieved chunks, regardless of what the model later generated. Generation passed only if the answer was consistent with the SME reference and grounded in retrieved evidence, with no unsupported claims.
- Feedback loop
- After launch, a structured UAT phase with the client's engineering teams fed production queries, retrieval failures, and user feedback back into the evaluation set, keeping it aligned with real usage rather than freezing it at launch.
Key Decisions
Decision
Dense-only retrieval vs hybrid retrieval with synonym expansion
The choice was between relying on dense semantic search alone or combining it with sparse lexical search and a domain-specific synonym expansion layer. I evaluated three configurations on the SME test set using top-k hit rate: dense-only reached 65–75%, adding sparse retrieval brought it to 75–85%, and adding synonym expansion on top of that reached 90%+. The synonym layer produced the largest single jump, because it directly addressed the abbreviation-to-full-term gap that dense embeddings weren't closing on their own.
Hybrid retrieval means running and fusing two retrieval pipelines instead of one, plus maintaining a synonym mapping as new equipment and terminology enter the corpus. Given the accuracy requirement, the added indexing and query complexity was worth it.
Decision
Generic document parsing vs a custom layout-aware ingestion pipeline
The choice was between using standard parsing libraries or building custom extraction logic for this corpus. Generic parsers lost document hierarchy and table structure on the more complex PDFs and SAP exports, which degraded chunk quality and, downstream, citation accuracy. I combined RAGFlow DeepDoc, PDFPlumber, and custom parsing logic to preserve hierarchy, tables, and section metadata through ingestion.
A custom pipeline costs more to build and maintain than dropping in a library. Retrieval precision across documents running to 6,000 pages depended on structure surviving ingestion, so the maintenance cost was accepted.
Decision
AI-only responses vs citation-grounded verification
The choice was between a conventional chatbot that returns an answer, or a system where every answer is traceable back to its source. During UAT, engineers were reluctant to act on unverified AI output for operational decisions. I built a verification workflow where every response ships with the retrieved chunks in a sidebar, and selecting one opens the source document at the exact page and paragraph it came from.
This requires maintaining a live mapping between every chunk and its document location, page, and paragraph offset, and keeping that mapping in sync as documents are re-ingested. It was the difference between a tool engineers ran queries through and one they trusted enough to act on.
Implementation Detail
Manually reviewing raw query logs to understand what engineers actually needed stopped scaling once query volume picked up. I built a query intelligence pipeline that runs as a scheduled job: it pulls a window of production queries, embeds them, and clusters them with HDBSCAN so semantically related queries group together without predefined categories. For each cluster, GPT-4o Mini generates a representative query describing the topic, and administrators see cluster size, trend over time, and the representative query itself.
This turned retrieval gaps from something someone had to notice by reading logs into something that surfaced on its own, ranked by how many engineers were actually hitting it. Because the whole pipeline runs as an offline batch job, it added no latency to live queries. It is what fed new terms into the synonym expansion layer and pointed at which parts of the corpus needed better coverage next, rather than leaving that discovery to whoever happened to be reading support tickets that week.
Results
90%+
top-k retrieval hit rate
SME-curated evaluation set, top-10 retrieval, hit defined as the expected supporting context appearing in the retrieved chunks. Dense-only baseline was 65–75%.
<2s
latency, P95
End-to-end in production, covering retrieval, reranking, and response generation.
90%+
documents parsed with usable structure
Text, tables, headings, and metadata preserved across the heterogeneous corpus of PDFs, Office files, and SAP exports.
75,000+
documents indexed
300+ GB of engineering documentation, with manuals running 1,000 to 6,000 pages each.
Every retrieval request was scoped to the requesting user's authorized knowledge bases before search ran, so no query ever touched documents outside that user's access, verified across all authenticated users on the platform.
The platform indexed over 75,000 documents and 300+ GB of engineering knowledge, and was adopted as the centralized technical search tool across multiple engineering teams following an on-premises production deployment. Engineers who previously spent hours searching manuals for a single answer could ask a question in plain language and get a citation-grounded response, with direct navigation to the source page and paragraph, in under two seconds.
Reflection
The current system depends on administrators manually uploading documents before they can be indexed, which works for a controlled rollout but adds friction every time documentation changes. With more time, I would build native connectors into systems like SharePoint or network file shares, so a document being added, modified, or removed triggers incremental parsing and re-indexing on its own rather than waiting on a manual upload step.
The tradeoff is real: connector reliability, authentication across multiple storage systems, and change detection are all harder problems than a manual upload form. But for an organization with tens of thousands of documents that keep changing, event-driven sync is what keeps the index actually current instead of slowly drifting out of date behind the source documentation.