RAG over Images and Tables

Charts, diagrams, and tables carry the answers text extraction throws away. Three strategies for retrieving over them.

Real documents are not text. Financial reports are mostly tables. Technical documentation is full of architecture diagrams. Slide decks are pictures with captions.

Extract text from these and you lose the content that mattered. A chunking pipeline over a quarterly report captures the narrative and discards the numbers.

Three strategies, and they differ mainly in where the cost falls.

1 · Describe at index time

Run a vision model over each image, table, or chart and generate a text description. Index the description. Retrieve normally.

[Figure 3: Bar chart comparing quarterly revenue across
2024–2026. Revenue rises steadily, with a pronounced jump
in Q3 2025. Q4 2026 shows the highest value.]

Advantages. Standard text retrieval throughout — no new infrastructure, and hybrid search works unchanged. Descriptions are inspectable, so you can see what was captured.

Limitations. The description is lossy and permanent. Whatever it omits is unretrievable forever. A description mentioning the Q3 jump but not the exact values cannot answer a question about the exact values.

Mitigation: store the description and a reference to the original image, then send the image itself to the model at generation time. Retrieve on text, answer on the original — the same principle as retrieve small, send large.

This is the most practical option for most systems.

2 · Multimodal embeddings

Embed images and text into one shared vector space, then retrieve across both with a single query.

Advantages. No lossy description step. A text query can retrieve an image directly.

Limitations. These spaces are coarse. They capture “this is a bar chart about revenue” far better than the specific values, and they are weak on text rendered inside images — frequently matching on the presence of text-like pixels rather than reading it. Performance also drops on specialized domains that were thin in web caption data.

Best combined with strategy 1 rather than used alone.

3 · Structure extraction for tables

Tables deserve separate treatment because they are the most common case and the most damaged by text extraction.

A table is a 2D structure flattened into a 1D stream, and the flattening destroys the alignment that made it readable. Merged cells, multi-row headers, and borderless tables all break naive extraction.

The approach that works: extract into an actual structured format — JSON, CSV, or markdown — preserving the grid. Then index two things: a natural-language summary of what the table contains, for retrieval; and the structured data itself, for generation.

For numeric questions, the strongest option is to skip retrieval-and-read entirely: load tables into a database and give the model a SQL tool. Aggregation and filtering are what databases do, and asking a language model to sum a column is asking it to do arithmetic — which it does badly.

Extraction quality comes first

Whichever strategy you pick, document extraction sits upstream of all of it, and it is where more failures originate than people expect.

A mangled table produces noise that flows through every later stage. Two-column PDFs read across columns, producing locally fluent and globally scrambled text. Nothing downstream recovers from either.

Check extraction output directly before tuning retrieval. This ordering saves a lot of wasted effort.

Practical notes

Keep references to originals. Store page number, bounding box, and a path to the source image alongside every derived chunk. You need it for citations, for sending originals at generation time, and for debugging.

Cite visually where you can. For document question-answering, showing the user the page region a claim came from is far more trustworthy than a text citation.

Route by question type. Numeric questions to SQL, conceptual questions to text retrieval, visual questions to image retrieval. One pipeline for all three is worse than three routed paths.

Watch the cost. Images consume substantial context — a high-resolution page can exceed several pages of text. Sending five page images per query gets expensive quickly.

Measure per modality. Track retrieval hit rate separately for text, table, and image questions. Aggregate numbers hide which modality is failing.

What to remember

• Text extraction discards the tables, charts, and diagrams that often hold the answer.
• Describe at index time, keep the original — retrieve on generated text, then send the actual image for generation.
• Multimodal embeddings avoid lossy description but are coarse and weak on in-image text; combine rather than substitute.
• Tables need structure extraction; for numeric questions, load into a database and give the model SQL.
• Fix document extraction first — it is upstream of everything and fails quietly.

Next: Evaluating RAG