←back to thread

Parse, Don't Validate (2019)

(lexi-lambda.github.io)
389 points melse | 4 comments | 26 Jun 21 07:23 UTC | HN request time: 0.632s | source
Show context
kortex ◴[26 Jun 21 14:15 UTC] No.27642049[source]▶
This principle is how pydantic[0] utterly revolutionized my python development experience. I went from constantly having to test functions in repls, writing tons of validation boilerplate, and still getting TypeErrors and NoneTypeErrors and AttributeErrors left and right to like...just writing code. And it working! Like one time I wrote a few hundred lines of python over the course of a day and then just ran it... and it worked. I just sat there shocked, waiting for the inevitable crash and traceback to dive in and fix something, but it never came. In Python! Incredible.

[0] https://pydantic-docs.helpmanual.io/

replies(8): >>27642308 #>>27642664 #>>27643276 #>>27643474 #>>27644758 #>>27645737 #>>27646367 #>>27647141 #
jimmaswell ◴[26 Jun 21 15:23 UTC] No.27642664[source]▶
I've found this to be simply a matter of experience, not tooling. As the years go by I find the majority of my code just working right - never touched anything like pydantic or validation boilerplate for my own code, besides having to write unit tests as an afterthought at work to keep the coverage metric up.
replies(5): >>27642800 #>>27642869 #>>27643369 #>>27643588 #>>27644097 #
1. kortex ◴[26 Jun 21 17:47 UTC] No.27644097[source]▶
No this was like over a week, and 100% due to the tooling. Pydantic, pycharm, black, mypy, and flake8. Pretty much went from "type hints here and there" to "what happens if I try writing python as if it were (95%) statically typed." I'd been testing well before this point but it's not the same as writing test.

The development process is totally different when you write structured types first and then write your logic. 10/10 would recommend.

Usual caveat: this is what makes sense to me and my brain. Your experience may be different based on neurotype.

replies(1): >>27644562 #
2. Scarbutt ◴[26 Jun 21 18:33 UTC] No.27644562[source]▶
The development process is totally different when you write structured types first and then write your logic. 10/10 would recommend.

Unless you were writing very small throwaway scripts, in what world where you writing your logic first and thinking about your data structures later?

replies(2): >>27648867 #>>27652769 #
3. Too ◴[27 Jun 21 05:31 UTC] No.27648867[source]▶
Even having type hints on basic functions like def foo(bar: str) in a throwaway-script helps because it gives me reliable autocompletion on bar. I might be getting old or toggle between so many different languages nowadays that even basic stuff i don't want to remember, like whether it is bar.lower(), bar.lowerCase(), bar.toLower(), bar.toLocaleLowerCase().

Defining a data structure up front doesn't require a lot of boilerplate as Java incorrectly have taught all of us. Writing a statically typed typing.NamedTuple or @dataclass is literally a one-liner.

4. kortex ◴[27 Jun 21 16:16 UTC] No.27652769[source]▶
The world of data science, ML and computer vision research. It's very academic-heavy, which has two effects. There's an insulation between commercial software dev and it, which results in a lot of NIH, a lot of reinforcement of bad habits, and a lag in propagation of best practices. Second, and related to this, there is a tendency to just piecemeal hack towards the solution, rather than architect the system from the ground up.

It's not zero consideration of data structures, it's mostly a focus on the main data type (arrays and data frames) and not really thinking about typed records, data models and such. The majority of types are float, str, dict, np.ndarray, pd.DataFrame. No dataclasses, minimal classes, and when classes are used, it's Java101 style "all the bad parts of OOP" programming. Sadly, I've spent years in this space before learning better.