More and more people are asking generative AI systems like Claude and ChatGPT for information and answers, rather than traditional search engines like Google. How do you position substantial works of long-form content, like reports and toolkits, for maximum visibility in these systems?
Here’s a look at how AI search and answer systems generate their answers.
How do AI systems get information from the web?
“Crawlers” gather information
Automated programs download web pages and PDFs from web sites. They scan pages for text they can read, and discard anything they don’t recognize.
The content gets indexed
Crawlers store the information they find in a database, called an index, that AI agents can search quickly. The content is stored along with information that indicates what the content is about.
AI Agents search for answers
Today’s search and AI engines don’t just match strings of text, they look for ideas and concepts related to a question. This is why it’s critical they have as much context as possible to connect information to those ideas and concepts.
Crawlers read both regular HTML web pages as well as PDF documents. What they understand about them is related to how these two formats were designed.
PDFs convey position on a page.
HTML conveys meaning and importance.
The PDF format was designed to convey exactly how a page should look, regardless of what system it was printed or displayed on.
HTML was designed to convey the structure of a page, what parts mean what, and how they relate to other sections and other pages.
On a web page, a heading is marked as a heading and sections have an explicit order and hierarchy. Headings connect to the content that follows. If two small headings come under a larger heading, the larger heading is understood to pertain to the whole section, while the smaller headings only connect to their own sections. This is exactly how we normally construct large documents, and it naturally conveys the authors’ intent to both human readers and to any automated system trying to understand connections between ideas.
In a PDF, those relationships are often only visual. A heading may simply be text in a different font, color, or size, given a specific position on the page. None of those formatting instructions explicitly says, “This is the heading for that section of text.” Headings can appear in the file separated from, or even after, their content, despite looking correct to human eyes. They can even be stored as an image which gets discarded completely.
Why headings matter
AI engines gather information in part by scanning titles and headings for whatever their user is asking about. Having meaningful headings connected to informative text is one of the best ways to ensure crawlers understand what your content’s about, so it can be used to form answers that:
- communicate your organization’s knowledge and experience on the subject
- reflect your views and positions on the subject
- credit your organization and direct people to your site to learn more.
When AI systems read PDFs, they first have to reconstruct the document’s intended structure from its visual layout. This is a complex process that takes far more resources than reading HTML, so crawlers have an incentive to be as quick about it as they can. And different systems may reconstruct it differently, yielding different understandings of what your content’s about.
While the details of the code search engines and AI systems use to gather information isn’t publicly stated, we’ve studied how different popular PDF code libraries work and how they handle design patterns common to long-form content like reports and toolkits. Reconstructing parts of a document, like headers and their text, in the wrong order is one of the most common mistakes we see. (We’ll be sharing some of these findings in a separate post.)
Images, tables, charts, graphs, process diagrams, and many other types of visual elements are also hard for crawlers to understand. Sometimes, these elements appear in a PDF file as a single image, with no text available for the crawler to read quickly. Sometimes text boxes appear way out of order because they’re positioned visually, but those positions don’t mean anything to the crawler. Sometimes tables and charts all wind up at the end of the file, far from the content they explain.
So what’s the solution?
HTML was designed specifically to convey the kind of context about a document that PDFs lack. Converting a PDF document to HTML web content accomplishes two important things:
- It makes it much easier for crawlers to gather information and meaning from your document.
- It gives you a chance to make adjustments to ensure your content is interpreted correctly. No more hoping the crawlers make the right guesses.
Just converting a PDF into HTML format doesn’t guarantee it will be read and understood properly – by humans or by robots. To make sure you’re maximizing the value of that conversion, see our PDF-to-HTML Checklist. These steps ensure your information carries the context that crawlers are looking for, as well as making your content easier for humans to skim, scan and read.
How does Report Kitchen fit in?
We’ve been refining and testing our conversion process for several years. We extract content from PDFs deliberately and with a substantial set of instructions and clues focused on the kinds of reports and documents we encounter, rather being a general-purpose converter that also has to handle forms, flyers, and all the other things people use PDF for.
But even within this focus, we see PDFs formatted in a wide variety of ways and with significant differences in how they’re structured. Our conversion process can make mistakes, just like the crawlers and AI systems. So how is that any better?
The importance of human review
The difference is that our conversion happens in a controlled process. The document owner can review the result, correct reading-order errors, restore missing context, and confirm that headings, callouts, footnotes, and key findings have been represented accurately before anything is published.
Once that reviewed version is live as structured web content, it becomes a clear, authoritative source for search engines and AI systems. It’s also always connected to the authoring organization instead of found floating alone on some other site that’s republishing PDFs, possibly without permission.
With a PDF alone, every crawler is left to perform its own private interpretation. If it reads a callout in the wrong place, merges two columns, drops headings, or mistakes formatting for structure, the document owner will never know, and will never have a chance to make corrections.
Under the hood, PDFs can get ugly, and our process doesn’t always get everything right. The difference is, we add a human review step where any interpretation errors can be caught and fixed once, instead of allowing countless automated systems to make their own guesses.
