←back to thread

PDF to Text, a challenging problem

(www.marginalia.nu)
357 points ingve | 1 comments | 13 May 25 15:01 UTC | HN request time: 0.203s | source
Show context
wrs ◴[13 May 25 15:57 UTC] No.43974336[source]▶
Since these are statistical classification problems, it seems like it would be worth trying some old-school machine learning (not an LLM, just an NN) to see how it compares with these manual heuristics.
replies(1): >>43974445 #
marginalia_nu ◴[13 May 25 16:06 UTC] No.43974445[source]▶
I imagine that would work pretty well given an adequate and representative body of annotated sample data. Though that is also not easy to come by.
replies(2): >>43975438 #>>43980893 #
1. ted_dunning ◴[13 May 25 17:28 UTC] No.43975438[source]▶
Actually, it is easy to come up with reasonably decent heuristics that can auto-tag a corpus. From that you can look for anomalies and adjust your tagging system.

The problem of getting a representative body is (surprisingly) much harder than the annotation. I know. I spent quite some time years ago doing this.