←back to thread

Strategies for Fast Lexers

(xnacly.me)
180 points xnacly | 1 comments | 14 Jul 25 14:42 UTC | HN request time: 0.209s | source
Show context
norir ◴[14 Jul 25 16:43 UTC] No.44562212[source]▶
Lexing being the major performance bottleneck in a compiler is a great problem to have.
replies(3): >>44563135 #>>44568294 #>>44568430 #
norskeld ◴[14 Jul 25 17:53 UTC] No.44563135[source]▶
Is lexing ever a bottleneck though? Even if you push for lexing and parsing 10M lines/second [1], I'd argue that semantic analysis and codegen (for AOT-compiled languages) will dominate the timings.

That said, there's no reason not to squeeze every bit of performance out of it!

[1]: In this talk about the Carbon language, Chandler Carruth shows and explains some goals/challenges regarding performance: https://youtu.be/ZI198eFghJk?t=1462

replies(3): >>44563278 #>>44564311 #>>44568469 #
1. burnt-resistor ◴[15 Jul 25 06:37 UTC] No.44568469[source]▶
Extending beyond the capabilities of PCHs, but there used to be incremental (C?) compilers/IDEs (maybe there still are?) that cached ASTs and were smart enough to invalidate and reparse just those portions of the local AST that changed based on editor updates. This was back when storage was very, very slow.