Parse, Don't Validate (2019)

I’d argue that the more experience you get the more you write code for other people which involves adding lots of tooling, tests, etc. Even if the code works the first time, a more senior dev will make sure others have a “pit of success” they can fall into. This involves a lot more than just some “unit tests as an afterthought to keep the coverage up.”

Keeping the code simple, finding the right abstractions, untangling coupling, gets the most bang for the buck. See the “beyond pep8” talk for a enlightened perspective.

That said, lightweight testing and tools like pyflakes to prevent egregious errors helps an experienced dev write very productively. Typing helps the most with large, venerable projects with numerous devs of differing experience levels.

I hated working with those coders because they weren't really very good and their code was always the worst to maintain. They are the equivalent of a carpenter who brags about how quickly they can bang nails but can't build a stable structure to save their life.

It immediately tells me that they've never worked on large software projects, and if they have they haven't worked on ones that lasted more than a few months.

I apologize to folks reading this for my rather aggressive tone but I've been writing software for a long time in numerous languages, and people with the unit tests as an afterthought attitude are typically rather arrogant in fool hardy.

The most recent incarnation I've encountered is the hotshot data scientist who did okay in a few Kaggle competitions using Jupyter notebooks, and thinks they can just write software the way they did for the competitions with no test of any kind.

I had one of these on my team recently and naturally I had to do 95% of the work to turn anything he produced into a remotely decent product. I couldn't even get the guy to use nbdev, which would have allowed him to use Jupyter to write tested, documented, maintainable code.

The best argument I've heard for doing type annotation is for documentation purposes to help future devs. But I don't completely buy this either. I touch new codebases all the time and I rarely spend much time thinking about what types will be passed. I can only assume it comes with experience.

Type annotation actually ends up taking a hell of a long time to do and is of questionable benefit if some of the codebase is not annotated. People spend sometimes hours just trying to get the type checker to say OK for code that actually works just fine!

The development process is totally different when you write structured types first and then write your logic. 10/10 would recommend.

Usual caveat: this is what makes sense to me and my brain. Your experience may be different based on neurotype.

I disagree. I've started using types from the ground up and it helps almost equally at every stage of the game. Also I aggressively rely on autocomplete for methods. It's faster this way than usual "dynamic" or "pythonic" python.

Part of it might be exactly because writing my datatypes first helps me think about the right abstractions.

The big win with python is maybe 2-10% of functions, I just want to punt and use a dict. But I have shifted >80% of what used to be dicts to Models/dataclasses and it's so much faster to write and easier to debug.

On certain backend code where I am able to do unit tests, they do catch the occasional edge case logic error but not at a rate that makes me concerned about only checking them in some time after the original code, which I'll have already tested myself in real use as I went along.

Unless you were writing very small throwaway scripts, in what world where you writing your logic first and thinking about your data structures later?

Also, what makes you think I’m not aware of datatypes? Currently working eight hours a day on Django models.

In short, there are choices besides, “I alone have to do all the hard work.”

I was more experienced with predictive algorithms and deep learning than any of the data scientists at the company but because they were brought in from an acquisition of a company that had an undeserved reputation due to a loose affiliation with MIT, they were treated like magicians while the rest of us were treated like blacksmiths.

I had the choice and I made the choice to leave. And of course I raised hell with the bosses about them not writing remotely production quality code that required extensive refactoring.

And yes I was paid to do the work but the work occupied time that I could have spent working on the other projects I had that were more commercially successful but less sexy to Silicon Valley VCs who look at valuations based on other companies' newest hottest product.

Tests should prove a desired behaviour. Sometimes it‘s not possible to fully run code until late in some staging, just because there are a lot of dependencies and conplexity. That‘s what tests are for (on various lebels of abstraction).

I was specifically responding to the commenter I replied to, who said they didn’t need tests because their code just worked the first time after he wrote it.

You write automated tests so that you can keep running the tests later such that the behaviour is maintained through refactor and non-breaking changes.

Defining a data structure up front doesn't require a lot of boilerplate as Java incorrectly have taught all of us. Writing a statically typed typing.NamedTuple or @dataclass is literally a one-liner.

I don't deny it. Join the cult of Static Python. We have cookies! And lower stress levels!

I usually wrap that spiel with my caveat "this depends greatly on your neurotype, style, environment, and other things." I have ADHD and my brain struggles with keeping bits of state in memory, so having to remember the type of every variable without my IDE tracking it for me is a huge performance drain.

However, I would contend even if your neurotype supported that mental workflow... it isn't actually better. Humans on average can handle 7 +/- 2 "pieces" of information in focus. Why spend any of your precious half-dozen pieces of salient consciousness on something a machine is really good at doing?

It's not zero consideration of data structures, it's mostly a focus on the main data type (arrays and data frames) and not really thinking about typed records, data models and such. The majority of types are float, str, dict, np.ndarray, pd.DataFrame. No dataclasses, minimal classes, and when classes are used, it's Java101 style "all the bad parts of OOP" programming. Sadly, I've spent years in this space before learning better.

Opinion: This is actually a symptom of what is (imho) a pervasive problem lodged deep in the collective consciousness of software dev: OOP with fine-grained objects. I blame (early) Java in large part for exacerbating this mentality. Encapsulation of state with mutator methods in particular. It sprays state all over the application, encourages mutation in place over immutability, coupling, validating-not-parsing, and makes it nigh-well impossible to write good tests.

It's really hard to write objects that enforce all invariants under every mutation. And when you have state strewn everywhere, it's impossible to test every nook and cranny. The combinatorial space explodes.

Objects are helpful for encapsulating state when they are course-grained, mutations are atomic, coupling occurs in one place, state changes are auditable, and the entire state can be replayed/set at once, to enable mock tests and subsystem integration tests. AKA, things like databases, reactors, and persistent data structures.

Also since the tools are immature and bolted on afterward in Python, I think it's even a bit worse than it would be in something decent like C#.