PDF to Text, a challenging problem

PDFs inherently are a markup / xml format, the standard is available to learn from.

It's possible to create the same PDF in many, many, many ways.

Some might lean towards exporting a layout containing text and graphics from a graphics suite.

Others might lean towards exporting text and graphics from a word processor, which is words first.

The lens of how the creating app deals with information is often something that has input on how the PDF is output.

If you're looking for an off the shelf utility that is surprisingly decent at pulling structured data from PDFs, tools like cisdem have already solved enough of it for local users. Lots of tools like this out there, many do promise structured data support but it needs to match what you're up to.

> PDFs inherently are a markup / xml format

This is false. PDFs are an object graph containing imperative-style drawing instructions (among many other things). There’s a way to add structural information on top (akin to an HTML document structure), but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format.

Thanks for your comment.

Indeed. Therein lies the rub.

Why?

Because no matter the fact that I've spent several years of my latent career crawling and parsing and outputting PDF data, I see now that pointing my LLLM stack at a directory of *.pdf just makes the invisible encoding of the object graph visible. It's a skeptical science.

The key transclusion may be to move from imperative to declarative tools or conditional to probabilistic tools, as many areas have in the last couple decades.

I've been following John Sterling's ocaml work for a while on related topics and the ideas floating around have been a good influence on me in forests and their forester which I found resonant given my own experience:

https://www.jonmsterling.com/index/index.xml

https://github.com/jonsterling/forest

I was gonna email john and ask whether it's still being worked on as I hope so, but I brought it up this morning as a way out of the noise that imperative programming PDF has been for a decade or more where turtles all the way down to the low-level root cause libraries mean that the high level imperative languages often display the exact same bugs despite significant differences as to what's being intended in the small on top of the stack vs the large on the bottom of the stack. It would help if "fitness for a particular purpose" decisions were thoughtful as to publishing and distribution but as the CFO likes to say, "Dave, that ship has already sailed." Sigh.

¯\_(ツ)_/¯

I appreciate the clarification. Should have been more precise with my terminology.

That being said, I think I'm talking about the forest of PDFs.

When I said PDFs have a "markup-like structure," I was talking from my experience manually writing PDFs from scratch using Adobe's spec.

PDFs definitely have a structured, hierarchical format with nested elements that looks a lot like markup languages conceptually.

The objects have a structure comparable to DOM-like structures - there's clear parent-child relationships just like in markup languages. Working with tags like "<<" and ">>" feels similar to markup tags when hand coding them.

This is an article that highlights what I have seen (much cleaner PDF code): "The Structure of a PDF File" (https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...) which says:

"There are several types of objects. If you are familiar with JSON, YAML, or the object model in any reasonably modern programming language, this will seem very familiar to you... A PDF object may have one of the following types: String, Number, Boolean, Null, Name, Array, Dictionary..."

This structure with dictionaries in "<<" and ">>" and arrays in brackets really gave me markup vibes when coding to the spec (https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandard...).

While PDFs are an object graph with drawing instructions like you said, the structure itself looks a lot like markup formats.

Might be just a difference in choosing to focus on the forest vs the trees.

That hierarchical structure is why different PDF creation methods can make such varied document structures, which is exactly why text extraction is so tricky.

Learning to hand code PDFs in many ways, lets you learn to read and unravel them a little differently, maybe even a bit easier.

> but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format.

This is what I kind of suspected but, as I said in my original comment, I'm not an expert and for the PDFs I'm reading I didn't need to delve further because that metadata simply isn't in there (although, boy do I wish it was) so I needed to use a different approach. As soon as I realised what I had was purely presentation I knew it was going to be a bit grim.

Markup is only indirectly related to hierarchical structure. “Markup” means that there is text that is being “marked up” with additional attributes (styling, structure information, metadata, …). This is how HTML and XML work, and also languages like TeX, Troff, and Markdown. For example, in the text “this is some text”, you can mark up the word “some” as being emphasized, as in “this is <em>some</em> text”.

The general principle is that the base content is plain text, which is augmented with markup information, which may or may not have hierarchical aspects. You can simply strip away the markup again and recover just the text. That’s not at all how PDF works, however.

You cite a comparison to JSON and YAML. Those are not markup languages (despite what YAML originally was an abbreviation for, see [0]). (HTML also isn’t DOM.)

[0] https://stackoverflow.com/a/18928199

uh. There is very little XML and the spec is a thousand pages long.

I was quoting the article there about JSON/YAML, not making that claim myself.

Did you take a look at the article I linked? It shows visual examples of hand-coded PDFs that demonstrate the structural similarities I am talking about.

Thanks for the clarification on terminology. I could have been clearer and more precise. I referred to "DOM-like structures" as an analogy for the hierarchical nature of PDF objects, not to claim HTML is DOM.

My core point wasn't about the technical definition of markup languages, but about the structural similarity between PDF's object model and hierarchical formats.

When coding a PDF document by hand, you work with nested structures using delimiters like "<<" and ">>" that create hierarchical relationships between objects - which has practical parallels to working with nested elements in other formats.

The forest vs. trees metaphor was to acknowledge that while PDFs aren't primarily markup formats (the trees), they do share structural characteristics with hierarchical formats (the forest) based on my hands-on experience with manual PDF creation.

Hope that helps clarify things a bit.

Clarified above - referring to the visual side of coding PDFs by hand.

https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...