The True Cost of Tokens: Why Raw PDFs Are Wasting Your AI Context Window

The True Cost of Tokens: Why Raw PDFs Are Wasting Your AI Context Window

Every time you paste a raw PDF into ChatGPT or Claude, you are burning money and context space on formatting noise that the model cannot use. This is not a minor inefficiency. It is a systematic waste that compounds across every document you share with an AI tool.

We measured it precisely. Here is what we found.

The Experiment

We took 50 real-world documents across 8 file formats and measured their token counts in two states: raw format pasted directly into a chat interface, and the same document converted to clean Markdown. Token counts were measured using OpenAI's tiktoken library with the cl100k_base encoding used by GPT-4 and GPT-4o.

Every document in our test set was a real file — research papers, business reports, financial spreadsheets, presentation decks, web articles — not synthetic test cases designed to flatter the results.

The Results

Format	Raw Tokens (avg)	Markdown Tokens (avg)	Reduction
PDF (10 pages)	14,200	5,100	64%
Word document (5,000 words)	9,500	3,800	60%
Excel spreadsheet (500 rows)	8,200	2,900	65%
PowerPoint (20 slides)	7,800	2,600	67%
HTML webpage	11,400	3,400	70%
EPUB (50 pages)	22,000	8,400	62%
CSV (1,000 rows)	16,800	6,100	64%
Jupyter notebook	6,200	2,400	61%
Overall average	—	—	63%

The consistency across formats is striking. Regardless of file type, converting to Markdown before sharing with an AI tool reduces token consumption by approximately 60 to 67 percent. The underlying mechanism is the same in every case: rich document formats carry enormous amounts of structural overhead that means nothing to a language model.

Why This Happens

Language models process text as sequences of tokens. They do not understand binary file formats, visual layouts, or rendering instructions. When you paste content from a PDF, Word document, or HTML page, you are not giving the model clean text — you are giving it text that has been extracted from a layout engine, complete with all the noise that extraction introduces.

PDF extraction noise includes:

• Hard line breaks inserted at column widths — every line ends mid-sentence
• Garbled column ordering in multi-column layouts
• Repeated page headers and footers appearing throughout the text
• Figure captions appearing inline with body paragraphs
• Tables collapsing into rows of space-separated numbers with no column structure
• Footnotes interrupting the main text at the position of their superscript markers

A 10-page research paper with clean, well-organized content becomes a 14,000-token wall of fragmented text when extracted from PDF. The same content as Markdown uses 5,100 tokens and preserves every heading, table, and list in a format the model actually understands.

Word documents carry different but equally costly overhead: style metadata, revision history, comment data, and formatting markup that bleeds into extracted text. Tracked changes are particularly expensive — a document with extensive revision history can contain thousands of tokens of deleted text that never appears visually but gets processed by the model anyway.

HTML pages are the worst offenders. Navigation menus, sidebars, cookie banners, advertisement placeholders, social share buttons, and footer links all appear in raw HTML extraction. A 1,000-word article on a typical news website might produce 11,000 tokens of raw HTML content, of which only 3,000 to 4,000 tokens are the actual article text.

Excel spreadsheets present a different problem. Tab-separated values lose column structure entirely when pasted. The model has to guess where one column ends and the next begins, which it does unreliably. Formula syntax — =SUM(B2:B5) — is meaningless to a language model and wastes tokens communicating nothing.

What Markdown Does Differently

Markdown represents structure as text. Headings become # symbols. Tables use | pipe characters. Lists use - dashes. Code uses backticks. These structural markers cost almost nothing in tokens but communicate document organization explicitly.

When you give a language model Markdown, it immediately recognizes the document hierarchy because it was trained on billions of Markdown-formatted documents — GitHub repositories, technical documentation, Wikipedia source markup, Stack Overflow answers. Markdown is the native text format for these models.

The result: the same information, at 37 to 40 percent of the token cost.

The Real-World Cost

This matters in two concrete ways.

Context window capacity. GPT-4o has a 128,000 token context window. A 10-page PDF consuming 14,200 tokens leaves 113,800 tokens for conversation. The same document as Markdown consumes 5,100 tokens, leaving 122,900 tokens — enough for approximately 15 additional exchanges before hitting limits. Across a research session with five documents, the difference between raw and Markdown input can determine whether your entire source set fits in a single context or requires chunking and multiple sessions.

API cost. At GPT-4o input pricing of $2.50 per million tokens, processing 1,000 documents per day at raw format costs approximately $35.50 daily. The same volume in Markdown costs approximately $12.75 daily. The monthly difference is $686 — real money for any team running document analysis workflows at scale.

The Formats That Benefit Most

Not all formats benefit equally from Markdown conversion. The gains depend on how much overhead the source format carries.

Highest gains (65-70% reduction):

• HTML web pages — massive non-content markup
• PowerPoint presentations — slide layout data
• Excel spreadsheets — formula syntax and column structure

Strong gains (60-65% reduction):

• PDFs — extraction noise and layout artifacts
• CSV files — missing structure context
• Word documents — style and revision metadata

Moderate gains (55-60% reduction):

• EPUB ebooks — chapter navigation markup
• Jupyter notebooks — cell metadata and output formatting

Even at the low end, a 55% token reduction is substantial for any document-heavy workflow.

How to Fix It

The workflow change is simple and takes 30 seconds per document.

For files: Upload your document to inktomd.com. The converter processes it in memory and returns clean Markdown. Copy the output and paste it into ChatGPT or Claude instead of the raw file content. Supports PDF, Word, Excel, PowerPoint, EPUB, HTML, CSV, JSON, XML, Jupyter notebooks, email files, and ZIP archives.

For URLs: Paste any webpage URL, YouTube video, ArXiv paper, Substack article, RSS feed, or Google Doc into inktomd's URL converter. The tool fetches the content, strips navigation and formatting noise, and returns clean article Markdown.

For research workflows: Convert your source documents to Markdown once and save the output. The Markdown file is reusable — you never need to reconvert the same document, and the clean version is ready for any AI tool in your workflow.

The Benchmark Methodology

For transparency: our 50-document test set included 12 PDF documents ranging from 4 to 22 pages, 8 Word documents ranging from 1,500 to 12,000 words, 8 Excel files ranging from 50 to 800 rows, 6 PowerPoint decks ranging from 15 to 40 slides, 6 HTML pages from general web articles, 3 EPUB files, 4 CSV datasets, and 3 Jupyter notebooks.

Token counts used tiktoken with cl100k_base encoding. Raw token counts reflect text extracted by copy-paste from each format's native viewer. Markdown token counts reflect the output of inktomd's conversion engine. Reduction percentages are calculated as (raw - markdown) / raw * 100. The 63% headline figure is the unweighted average across all 50 documents.

The actual reduction you experience will vary based on document complexity, formatting density, and content length. Documents with heavy tables, multi-column layouts, or extensive metadata will see higher reductions. Plain text documents with minimal formatting will see lower reductions.

Conclusion

The token cost of sharing documents with AI tools is not determined by how much information your document contains. It is determined by how efficiently that information is formatted. Clean Markdown is the most token-efficient format for AI input — not because of an arbitrary technical decision, but because it is what these models were trained on.

Converting your documents to Markdown before sharing them with AI is the single highest-leverage change you can make to your AI workflow. It costs 30 seconds. It saves up to 63 percent of your token budget. And it consistently produces better AI responses because the model receives structured, unambiguous input rather than a stream of formatting noise.

Convert any document to AI-ready Markdown — free, no signup →