Software Engineers: Parse, don't validate.
Mathematicians: Parsing is validation
replies(3):
Mathematicians: Parsing is validation
This paper.... uh.... what exactly is it good for?
I suppose it could be kind of nice as some kind of undergraduate paper writing project kind of thing but it looks too professional for that.... I am kind of at a loss why this was written. Maybe it is some strange kind of satire....
Are you talking about a bijective mapping or are you saying it's a synonym for identical?
Because the former doesn't make any sense here and the latter is not true.
Red is a color does not imply that all colors are red.
basically ist’s just functions that take a value of one type and return a other one
"Red is a color" can be formalized as "Red is a type of color" or "Red is member of set Colors".
You can't formalize "Color is red" because it doesn't mean anything.
When I say "Parsing is validation" I am using the verb "is" to mean an isomorphism.
[1]: http://www.sigbovik.org/ [2]: http://www.sigbovik.org/2021/proceedings.pdf
"A square is a rectangle" means "A square is a TYPE of rectangle" (at least, that is what I am parsing it as).
"Parsing is Validation" means Parsing is isomorphic to Validation.
How do I know? Because that is how I want you to parse it.
parse verb. resolve (a sentence) into its component parts and describe their syntactic roles.
In computer science what we do is precisely syntax analysis. Determining the meaning of operators.
Mathematicians have the exact same problem with respect to the equality operator.
Does my thing have a different name? Where can I read up on how to do that best?
The essential point of this blog post is to avoid "shotgun parsing", where parsing/validating is done just from a procedural standpoint, where it matters when exactly it happens. In the paper "Out of the Tar Pit" it is asserted that this leads to "accidental complexity" (AKA "pain and anxiety"), which is something every programmer has experienced before, possibly many times.
I've become a fan of declarative schema to (json-schema/OpenApi, clojure spec etc.) to express this kind of thing. Usually this is used at the boundaries of an application (configuration, web requests etc.) but there are many more applications for this within the flow of data transformations. If you apply the "parse don't validate" principle you turn schema-validated (sic!) data into a new thing. Whether that is a "ValidatedData" type or meta data, a pre-condition or runtime check says more about the environment you program rather than the principle in discussion. The benefit however is clear: Your code asserts that it requires parsed/validated data where it is needed, instead of when it should happen.
- don't drop the info gathered from checks while validating, but keep track of it
- if you do this, you'll effectively be parsing
- parsing is more powerful that validating
"Extra steps" would be keeping track of info gathered from checks.
import * as $ from '@appliedblockchain/assert-combinators'
const validateFooBar = (
$.object({
foo: $.string,
bar: $.boolean
})
)
// probably roughly equivalent to
/*
const validateFooBar = (x) => {
console.assert(
typeof x === 'object' &&
typeof x.foo === 'string' &&
typeof x.bar === 'boolean'
)
return x
}
*/
const test1 = { foo: "abc", bar: false }
const test2 = { foo: 0, quux: true }
const { foo, bar } = validateFooBar(test1) // ok
const oops = validateFooBar(test2) // throws error
https://github.com/appliedblockchain/assert-combinators/blob...
https://github.com/appliedblockchain/assert-combinators/blob...
So is the joke on Computer Scientists or Mathematicians? You decide ;)
Beware of bugs in the above code; I have only proved it correct, not tried it --Donald Knuth
There's beautiful clarity in the articulation, and the essence is easy to grasp yet powerful. It reminds me a bit of Scott Wlaschin's Railway Oriented Programming (ROP) [0]. As a technique, ROP nicely complements "parse don't validate". As an explanation, it's similarly simple yet wonderfully effective.
I've a real admiration for people who can explain and present things so clearly. With ROP, for example, the reader learns the basics of monads without even realising it.
For example let's say that I have written an HTTP API that accepts application/x-www-form-urlencoded data to one of its endpoints. Let's say `POST /users`, and this is where the client-side application posts data to.
Now I can implement this in many ways. I can for example define
pub struct Person {
name: String,
phone_number: String,
}
And of course I could go further and define types for the name and the phone number but in this case lets say that I have decided that strings are the proper representation in this case.
If the fields of my structs were directly accessible
pub struct Person {
pub name: String,
pub phone_number: String,
}
// ...
let name = post_data.name;
let phone_number = post_data.phone_number;
let norwegian_phone_number_format = Regex::new(r"^(\+47|0047)?\d{8}$").unwrap();
// ...
The point is that here I am doing some rudimentary validation on the phone number, requiring it to be in Norwegian format. But I am enforcing this in the implementation of the handler for the HTTP API endpoint, rather than in the data type itself.
Whereas if I was instead doing
pub struct Person {
name: String,
phone_number: String,
}
impl Person {
pub fn try_new (name: String, phone_number: String) -> std::result::Result<Self, PersonDataError> {
// ...
let norwegian_phone_number_format = Regex::new(r"^(\+47|0047)?\d{8}$").unwrap();
// ...
}
}
And in this manner, even though my validation is still rudimentary, and a type purist might find the type insufficiently constrained, I have indeed in my own book gone from just validation to actual parsing. Because I have made it so that the construction of the type enforces the constraints on the data.
I personally use myzod as its fast it parsing, zero dependancies and you can infre types from your schemas.
General case: Validating random data as input into some program.
Particular case: Validating random source code (data) as input into some compiler (program).
Do compilers parse or validate?
"parsing is validation, but validation is not parsing" if that were true then you should be able to give an example of a compiler doing some sort of validation on the random source code (data) that is not parsing.
The very thing which determines the validity of random source code is the compiler's ability to parse it.
And it's getting some wide adoption, for instance FastAPI which uses it for request validations.
I feel that we don't put enough value, these days, on the ability to write clear, articulate exposition. Also, I believe that many people are not willing to read articles, books, or papers, of any meaningful length.
Everything needs to be boiled down to <10 min. read time, or <18 min. TED talks.
The argument is that if you need to interact with or operate on some data you shouldn't be designing functions to validate the data but rather to render it into a useful output with well defined behaviour.
> IME, people in dynamic languages almost never program this way, though—they prefer to use validation and some form of shotgun parsing. My guess as to why? Writing that kind of code in dynamically-typed languages is often a lot more boilerplate than it is in statically-typed ones!
I feel that once you've got experience working in (usually functional) programming languages with strong static type checking, flakey dynamic code that relies on runtime checks and just being careful to avoid runtime errors makes your skin crawl, and you'll intuitively gravitate towards designs that takes advantage of strong static type checks.
When all you know is dynamic languages, the design guidance you get from strong static type checking is lost so there's more bad design paths you can go down. Patching up flakey code with ad-hoc runtime checks and debugging runtime errors becomes the norm because you just don't know any better and the type system isn't going to teach you.
More general advice would be "prefer strong static type checking over runtime checks" as it makes a lot of design and robustness problems go away.
Even if you can't use e.g. Haskell or OCaml in your daily work, a few weeks or just of few days of trying to learn them will open your eyes and make you a better coder elsewhere. Map/filter/reduce, immutable data structures, non-nullable types etc. have been in other languages for over 30 years before these ideas became more mainstream best practices for example (I'm still waiting for pattern matching + algebraic data types).
It's weird how long it's taking for people to rediscover why strong static types were a good idea.
Maat was created before dataclasses existed. For validation Maat offers the same. But it also allows for some really neat features such as validation on encrypted data. https://github.com/Attumm/Maat/blob/main/tests/test_validati...
Since validation is written as dictionaries its possible to store the validations in caching db such as Redis.
And since its simple its easy to extend for anyone use case. And there are no other dependencies.
Benchmarks of pydantic has Maat around twice as Pydantic.
And of course I agree 100% about tradoffs in engineering. However I usually advise against using 2 dependencies doing mainly the same thing if possible within the same project.
Anyway, good catch, thanks for enlightening me!
General case: Validating random data as input into some program.
Particular case: Validating random source code (data) as input into some compiler (program).
Do compilers parse or validate?
> "the converse of ‘parsing is validation’ is not true."
If that were the case then you should be able to give an example of a compiler validating random source code (data) but not parsing it.
What determines the validity of random input is precisely a compiler's ability to parse it.
"Bonkers" premises. Iterate, iterate, iterate. "Bonkers" conclusions. GIGO.
And yet the result is reified, exists and speaks for itself. So what is so "absurd" and "bonkers" about a result that is right before your eyes?
https://en.wikipedia.org/wiki/Reification_(computer_science)
But even if these were the same thing and we want to be a bit pendantic since this is HN after all, structural type systems often support some kind of subtyping or row polymorphism, but it's not a strict requirement for a type system to be "structural". You could have a structural type system that doesn't allow
{ a: int, b: int }
{ a: int }
I don't think it's weird. Most of those languages were not popular in industry for various reasons, and the ones that were (especially in say, the 90s) did not have particularly capable static type systems. The boilerplate/benefit ratio was all off.
The way I describe this dichotomy personally is, I would rather use Ruby than Java 1.5. I would rather use Rust than Ruby. (Java 1.5 is the last version of Java I have significant development experience in, and they've made the type system much more capable since those days.)
TypeScript doesn't have this check (https://github.com/Microsoft/TypeScript/issues/12936) and I've found it can be really error prone when you're wanting to return a copy of an object with some fields updated. Spot the bug in this example: https://www.typescriptlang.org/play?#code/C4TwDgpgBAKlC8UDeU...
For example, one good reason why strong static types are a bad idea... they prevent you from implementing dynamic dispatch.
Routers. You can't have routers.
With the right amount of indirection/abstraction you can implement everything in Assembly.
But you don't. Because you like all the heavy lifting the language does for you.
First Class citizens is what we are actually interested in when we talk about programming language paradigm-choices.
One trick I've found very useful is to realise that Maybe (AKA Option) can be though of as "a list with at most one element". Dynamic languages usually have some notion of list/array, which we can use as if it were a Maybe/Option type; e.g. we can follow a 'parse, don't validate' approach by wrapping a "parsed" result in a list, and returning an empty list otherwise. This allows us to use their existing 'map', 'filter', etc. too ;)
(This is explored in more detail, including links to logic programming, in https://link.springer.com/chapter/10.1007%2F3-540-15975-4_33 )
If we want to keep track of useful error messages, I've found Scala's "Try" type to be useful ('Try[T]' is isomorphic to 'Either Throwable T'). Annoyingly, built-in sum type; the closest thing is usually a tagged pair like '[true, myFoo]'/'[false, myException]', which is pretty naff.
I've found scala or even LINQ to really hammer down this point, even to those who aren't into FP very much. Doing that map/flatmap makes it click for just about anyone
function rowPolymorphic<R extends { a: number }>(record: R): R & { a: string } {
return {
...record,
a: record.a.toString(),
}
}
const rec = rowPolymorphic({
a: 123,
b: "string",
})
console.log(rec.b)And I explicitly said "escape hatch" meaning language feature. You don't need that much indirection to get routers in Haskell, Rust, Go, C, C++... like I fail to see how implementing routers are a barrier in strict type system languages.
Is it easier in python or js? Sure. can't? hardly.
E: here's some vtable dispatch (unless that doesn't count as "dynamic dispatch") in Rust. Looks really straightforward.
I do believe that, for long lasting, larger projects, static typing tends to make the code easier to maintain as time goes on. But not every project is like that. In fact, not every project uses a single language. Some use statically typed languages for some parts, and dynamically typed for others (this is common in web dev).
Maybe because you said:
> they prevent you from implementing dynamic dispatch.
and
> Routers. You can't have routers.
Which just isn't true. You can implement dynamic dispatch and you can have routers, but they come at a cost (either of complex code or of giving up compile-time type safety, but in a dynamic language you don't have the latter anyway, so with a static language you can at least choose when you're willing to pay the price).
> First Class citizens is what we are actually interested in when we talk about programming language paradigm-choices.
But that's not what you said in your other comment. You just said you can't have these things, not they're not first class citizens. Besides, some static languages do have first class support for more dynamic features. C++ has tools like std::variant and std::any in its standard library for times you want some more dynamism and are willing to pay the tradeoffs. In Java you have Object. In other static languages, you have other built-in tools.
No offense, but your claim sounds like you're confused to me, but maybe I am the one confused. AFAIK I do dynamic dispatch all the time in strongly typed languages. Can you show an example that a strongly typed language can't accomplish?
That is what “trade offs” means.
You can have any feature in any language once you undermine the default constraints of your language. You can implement Scala in Brainfuck. Turing completeness guarantees it!
But this is not the sort of discourse we care about in practice.
Neither C++ nor Rust give you static type safety AND dynamic dispatch because all of the safety checks for C++ and Rust happen at compile time. Not runtime.
You said "you can't", kortex said "you can" and then you moved the goal posts to "you can because of turing completeness, but its bad, Why do you even bother making the point?" to which I replied "because its a valid response to you're `you can't`" now you moved them again to "everything comes at a cost" (which... I also said?).
Of course everything comes at a cost and yes, that's what "trade off" means. Dynamic languages come at a cost too (type checking is deferred to run time). So, this time, let me ask you: Why do you even bother making the point?
There's a follow-up article by the same author (that I unfortunately can't find), in which she explains this point.
As an example, returning a NonZero newtype over Int is not as type safe as using an ADT that lacks a zero value altogether. Using a NonEmpty newtype over List is not as type safe as using the NonEmpty ADT that has an element as part of its structure.
Basically newtype still has use, but it is not as airtight as a well-designed ADT.
Every time I read something like this my mind translates it to "after building an ad-hoc compiler you can do all the things a compiler can do. Just not as well, but you can do it." -- Same with "I don't need a compiler, my tests stop all this kind of bugs"
At the end of parsing, you have a structure with a type. After validation, you may or may not have a structure with that type, depending on how you chose to validate.
But I think the big win is, parsers are usually much easier to compose (since they themselves are structured functions) and so if you start with the type first, you often get the "validation" behavior aspect of parsing for "free" (usually from a library). Maybe title should have been "Parse, don't manually validate."
But if your type doesn't catch all your invariants, yeah it does feel kinda just like validation.
C++ and Rust let you have compile-time safety, until you choose to give it up and have runtime checks instead. Dynamic languages only allow the latter. Static languages let you choose, dynamic languages chose the latter for you in all cases. Both can have dynamic dispatch.
Besides, static languages can have compile-time type safe dynamic dispatch, if you constrain the dispatch to compile-time-known types (eg std::variant). You only lose that if you want fully unconstrained dynamism, in which case you defer type checking to runtime. Which is what dynamic languages always have.
So both C++ and Rust DO have dynamic dispatch and the programmer gets to choose what level of the dynamism/safety trade off they want. And yes, these features ARE first class features of the languages.
You don't grok the difference.
You can implement EVERYTHING in Brainfuck. Tractability is the reason you don't.
The goalposts are exactly where I set them. With my first comment.
"Every programming paradigm is a good idea if the respective trade-offs are acceptable to you."
Not to steal vvillena's thunder, but that's pretty much the dictionary definition of "parsing"
> analyze (a string or text) into logical syntactic components, typically in order to test conformability to a logical grammar.
Parsing is taking some collection of symbols, and emitting some other structure that obeys certain rules. Those symbols need not be text, they can be any abstract "thing". A symbol could be a full-blown data structure - you can parse a List into a NotEmptyList, where there's some associated grammar with the NEL that's a stricter version of the List grammar.
I also find this a confusing use of the word "parse", and it's not explained in the post, and I think "parse, don't validate" is a poor slogan as a result. The traditional slogan is "make illegal states unrepresentable", though that's a bit narrower of a concept.
PRECISELY
You have to give up the safety to get the feature.
So you "want type-safety". Until you don't.
>static languages can have compile-time type safe dynamic dispatch
"Compile-time dynamic dispatch" is an oxymoron. Dynamic dispatch happens at runtime.
Anyways, I definitely agree.
This was more aimed at people who are new to the idea of parsing over validating. In a strong statically typed language, the type system would naturally guide you to use this approach so if this isn't natural to you then time in other languages would probably be worthwhile.
> "Every programming paradigm is a good idea if the respective trade-offs are acceptable to you."
That's not what we are responding to. Nobody here is arguing over this statement. We are responding to you assertion that static typed compile-time checked languages _prevent_ you from having dynamic dispatch and that you _can't have_ routers because of that. Neither of which are true.
Dynamic languages prevent you from having compile time checks. Does that make them bad? Static languages give you compile time safety, but if you're willing to forego that [1], then you can get the EXACT SAME behavior as dynamic languages give you.
You literally said:
For example, one good reason why strong static types are a bad idea... they prevent you from implementing dynamic dispatch.
Routers. You can't have routers.
Just because a hammer can't screw in screws doesn't mean its a bad idea, it just means that you can't use it for all use cases. This is the same. You can use static types and for the few cases where you need runtime dynamism, then you use that. That doesn't make the static types in the rest of your code bad. It just gives you additional tools that dynamic types alone don't have.
[1] to various degrees, its not all or nothing like you seem to be implying, there are levels of middle ground, like std::variant which maintains safety but you need to enumerate all possible types, or std::any which is fully dynamic but you give up compile time checks
There's a field of study called "parsing", which studies "parsers". Hundreds of papers. Very well defined problem: turning a list of symbols into a tree shaped parse tree (or data structure). The defining aspect of parsing, that makes it difficult and an interesting thing to study, is that you're starting with a list and ending with a tree. If you're converting tree to tree (that is, a typical data structure to a typical data structure), all the problems vanish (or change drastically) and all the parsing techniques are inapplicable.
I'm kind of annoyed that people are starting to use the word "parse" metaphorically. Bit by bit, precise words turn fuzzy. Alas, it will be a lost battle.
Static typing really appeals to me on a personal level. I enjoy the process of analysis it requires. I love the notion of eliminating whole classes of bugs. It feels way more tidy. I took Odersky's Scala class for fun and loved it.
But in practice, they're just a bad match for projects where the defining characteristic is unstable ground. They force artificial clarity when the reality is murky. And they impose costs that pay off in the long run, which only matters if the project has a long run. If I'm building something where we don't know where we're going, I'll reach for something like Python or Ruby to start.
This has been brought home to me by doing interviews recently. I have a sample problem that we pair on for an hour or so; there are 4 user stories. It involves a back end and a web front end. People can use any tools they want. My goal isn't to get particular things done; it's to see them at their best.
After doing a couple dozen, I'm seeing a pattern: developers using static tooling (e.g., Java, TypeScript) get circa half as much done as people using dynamic tooling (Python, plain JS). In the time when people in static contexts are still defining interfaces and types, people using dynamic tools are putting useful things on the page. Making a change in the static code often requires multiple tweaks in situations where it's one change in the dynamic code. It makes the extra costs of static tooling really obvious.
That doesn't harm the static-language interviewees, I should underline. The goal is to see how they work. But it was interesting to see that it wasn't just me feeling the extra costs. And those costs are only worth paying when they create payoffs down the road.
I found myself replacing the configuration parsing code in a C++ project that was littered with exactly the validation issues described, and converted it to that which the author advocates. The result was a vastly more readable and maintainable codebase, and it was faster and less buggy to boot.
Another nice advantage is that the types are providing free/self- documentation, too.
I believe semantics is getting in our way of communicating.
You don't do dynamic dispatch ___ALL___ the time. You only do it at runtime. And you only do static type safety at compile time. Those are different times.
You can't have both of those features at the __SAME TME__, therefore you can't have both features ALL the time. They are mutually exclusive.
So let me restate what I think you meant. Maybe I’m wrong.
> a limitation of compile time static types is that they cannot do dynamic dispatch, which is relies on information that isn’t known at compile time.
I think we can all agree on this?
Note that I said it’s a limitation, not that it’s bad. Limitations don’t make something bad in themselves, they only constrain when it is an appropriate tool. You wouldn’t say that a submarine is bad because it can’t go on land? It’s a limitation but that doesn’t make it bad.
Then you said that because of this limitation, “you can’t have routers”.
But you can have routers in static languages like C++, Rust or even Haskell.
You have a choice (trade off) to make:
1. either you give up on some dynamism to keep compile time checks by enumerating the types you can dispatch on (as std::variant does)
2. or you choose to give up static checking (for this part of your code only!) and move it to runtime first full dynamic logic (as std::any does). When giving up compile time safety for one part of your code, you do not give it up for all of your code, because you can runtime validate then on the boundaries going back into statically checked code paths, so the compiler cab assume that they are valid if they get back in (as the runtime checks would reject bad types)
Neither case is “you can’t have routers”, but sure you can’t have fully dynamic routers purely at compile time.
Also, both options are perfectly tractable and both cases are typically first class language features (at least in the static types languages I’ve used). In no case are the “bad” options, despite each option having different limitations.
In a dynamic-types-only language, you don’t get to choose the trade offs at all, you only get “fully dynamic no compile time checking, runtime checks only”.
Note also that in real life, few things are truly fully dynamic. You are always constrained by the operations that are expected to be carried out on the dynamic data and while you might not know at runtime what days you could get, you DO know what operations you expect to run on the data. And you can perform compile time checks around that.
So for a router, so you really need it to be fully dynamic? Or can you live with it being generic (ie it’s a library that does routing and it can support any type, but you constrain it to known types when you actually use it). If so, you have options that maintain type safety: you can use OOP inheritance, you can use enumerated types like std::variant, you can use generics/templates. The library works for any types but the user chooses what types to implement based on the operations that will be performed on the data. Even dynamic types do this, they just defer it to runtime.
Or you can have the router operate on purely dynamic types but the handlers that are routed to are statically typed (eg if in C++ the router uses std::any and the handler registers what types it can accept and the router checks validity before handing the data off).
Please read about it on Wikipedia
https://en.m.wikipedia.org/wiki/Dynamic_dispatch
And for the future to not litter HN with comments like these, next time 10 different people in thread are all explaining to you why you're mistaken, take a moment to try to listen and think through what they're saying instead of just digging deeper.
Having an open mind to learning something new, not just arguing a point is a great approach to life.
They don’t go into deep category theory, you won’t find monads and friends, they are first level, straightforward any typescript developer can pick up in minutes - this is by design. It stops at combinators in typescript to solve very specific problem and nothing more. Haskell in ts is not the goal of this npm.
At that point, you could have corrected us and revised your wording for clarity, but you did not, you dug your heels in, you moved the goal posts, you claimed we don’t understand various things that weren’t even really related to the comment we were responding to and you brought in irrelevant points like Turing completeness. You didn’t “get lynched“ right away, you could have reworded or clarified or asked what we didn’t understand about your statement.
Also, YOU didn't practice the principle of charity!
When people responded to what you wrote, you dug in and claimed we didn't understand compilers or tractability vs possibility and various other things, rather than thinking "maybe they didn't understand my point, I should clarify". So its on you, not us.
I’m still not sure if I understand what you were trying to say, I am assuming that you meant what I wrote when I restated your comment, by piecing all of your different comments together. I’m still not sure if you actually meant static types are bad period (vs being bad at certain things and having certain limitations). And I still don’t agree with “you can’t have routers”.
Anyway, I’m done, have a nice day.
EDIT: I know I said I'm done, but your reply: "parse better", I gave you an out and still you blame everyone else and don't accept that you might have made a mistake. You're so sure that you are right and everyone else is wrong that you don't even entertain the possibility that you might have made a mistake (either in your reasoning or your explanation thereof). You appear to have an ego problem. You should take some time out and reflect on your life a bit.
Focusing exclusively on what happens at compile-time is to miss the whole reason static type systems are useful in the first place: they allow compile-time reasoning about runtime behavior. Just as we can use a static type system to make predictions about programs that pass around first-class functions via static dispatch, we can also use them to make predictions about programs that use vtables or other constructions to perform dynamic dispatch. (Note that the difference between those two things isn’t even particularly well-defined; a call to a first-class function passed as an argument is a form of unknown call, and it is arguably a form of dynamic dispatch.)
Lots of statically typed languages provide dynamic dispatch. In fact, essentially all mainstream ones do: C++, Java, C#, Rust, TypeScript, even modern Fortran. None of these implementations require sacrificing static type safety in any way; rather, type systems are designed to ensure such dispatch sites are well-formed in other ways, without restricting their dynamic nature. And this is entirely in line with the OP, as there is no tension whatsoever between the techniques it describes and dynamic dispatch.
https://www.geeksforgeeks.org/dynamic-method-dispatch-runtim...
And using it doesn't give up any of Java's type safety guarantees. The arguments and return type of the method you call (which will be invoked with dynamic dispatch) are type checked.
I’d argue that the more experience you get the more you write code for other people which involves adding lots of tooling, tests, etc. Even if the code works the first time, a more senior dev will make sure others have a “pit of success” they can fall into. This involves a lot more than just some “unit tests as an afterthought to keep the coverage up.”
I see they’ve raised a lot of money. Does anyone know what their revenue model is?
Here's validating a CSV in Python (which I'm using because it's a language that's, well, less excited about types than the author's choice of Haskell, to show that the principle still applies):
def validate_data(filename):
reader = csv.DictReader(open(filename))
for row in reader:
try:
date = datetime.datetime.fromisoformat(row["date"])
except ValueError:
print("ERROR: Invalid date", row)
if date < datetime.datetime(2021, 1, 1):
print("ERROR: Last year's data", row))
# etc.
return errors
def actually_work_with_data(filename):
reader = csv.DictReader(open(filename))
for row in reader:
try:
date = datetime.datetime.fromisoformat(row["date"])
except ValueError:
raise Exception("Wait, didn't you validate this already???")
# etc.
The insight is that the work of reporting errors in the data is exactly the same as the work of getting non-erroneous data into a usable form. If a row of data doesn't have an error, that means it's usable; if you can't turn it into a directly usable format, that necessarily means it has some sort of error.
So what you want is a function that takes the data and does both of these at the same time, because it's actually just a single task.
In a language like Haskell or Rust, there's a built-in type for "either a result or an error", and the convention is to pass errors back as data. In a language like Python, there isn't a similar concept and the convention is to pass errors as exceptions. Since you want to accumulate all the errors, I'd probably just put them into a separate list:
@attr.s # or @dataclasses.dataclass, whichever
class Order:
name: str
date: datetime.datetime
...
def parse(filename):
data = []
errors = []
reader = csv.DictReader(open(filename))
for row in reader:
try:
date = datetime.datetime.fromisoformat(row["date"])
except ValueError:
errors.append(("Invalid date", row))
continue
if date < datetime.datetime(2021, 1, 1):
errors.append(("Last year's data", row))
continue
# etc.
data.append(Order(name=row["name"], date=date, ...))
return data, errors
And if your file format changes, you make the change in one spot; you've kept the code DRY.
Obviously static type systems are useful. I don't even think my point is contrary to anything you are saying. This is not being said as way of undermining any particular paradigm because computation is universal - the models of computation on the other hand (programming languages) are not all “the same”. There are qualitative differences.
Every single programming paradigm is a self-imposed restriction of some sort. It is precisely this restriction that we deem useful because they prevent us from shooting off our feet with shotguns. And we also prevent ourselves from being able to express certain patterns (of course we can deliberately/explicitly turn off the self-imposed restriction! ).
Like the restriction you are posing on your self is explicit in "type systems are carefully designed so that the compile-time reasoning says something useful about what actually occurs at runtime"
If you could completely determine everything that happens at runtime you wouldn't need exception/error handling!
All software would be 100% deterministic.
And it isn't.
I can say nothing of the structure of random bitstreams from unknown sources. I only know what I EXPECT them to be. Not what they actually are.
In this context parsing untrusted data IS runtime type-checking.
Static types are awesome for local reasoning, but they are not that helpful in the context of the larger system (this already starts at the database, see idempotency mismatch).
Code with static types is sometimes larger and more complex than the problem its trying to solve
They tightly couple data to a type system, which (can) introduce incidental complexity >(I'm still waiting for pattern matching + algebraic data types) This is a good example, if you pattern match to a specific structure (e.g. position of fields in your algebraic data type), you tightly coupled your program to this particular structure. If the structure change, you may have to change all the code which pattern matches this structure.
On the other hand, as the initial code grows and grows, the cost of moving it all to a saner language grows too... discouraging a rewrite. So we end up with very complex production software that started as dynamic and is still dynamic.
Keeping the code simple, finding the right abstractions, untangling coupling, gets the most bang for the buck. See the “beyond pep8” talk for a enlightened perspective.
That said, lightweight testing and tools like pyflakes to prevent egregious errors helps an experienced dev write very productively. Typing helps the most with large, venerable projects with numerous devs of differing experience levels.
Your statement is verging on the nonsensical, like saying to someone "You don't use integers all the time, sometimes you use strings, they're different things." Well, duh?
EDIT: Also, it's discouraged on this site in general to use all caps for emphasis. *phrase* produces an italicized/emphasized form with phrase.
Yes, of course dynamic dispatch is a runtime phenomenom, that's the dynamic part of it. But there's nothing stopping the code that performs dynamic dispatch from being strongly typed. Strong types are instructions used to prove that the code holds certain properties, they are a separate program from the final binary the compiler gives you. Do you also think that code that gets unit tested can't perform dynamic dispatch?
If your point is that "types don't exist at runtime anyway" (reflection aside) then you don't understand what the purpose of a type system is, nor what strongly typed code means.
match event.get():
case Click((x, y)):
handle_click_at(x, y)
In both Python and statically typed languages you can avoid this by matching against field names rather than positions, or using some other interface to access data. This is an important design aspect to consider when writing code, but does not have anything to do with dynamic programming. The only difference static typing makes is that when you do change the type in a way that breaks existing patterns, you can know statically rather than needing failing tests or runtime errors.
The same is true for the rest of the things you've mentioned: none are specific to static typing! My experience with a lot of Haskell, Python, JavaScript and other languages is that Haskell code for the same task tends to be shorter and simpler, albeit by relying on a set of higher-level abstractions you have to learn. I don't think much of that would change for a hypothetical dynamically typed variant of Haskell either!
[1]: https://www.python.org/dev/peps/pep-0636/#matching-sequences
https://en.wikipedia.org/wiki/Object%E2%80%93relational_impe...
I hated working with those coders because they weren't really very good and their code was always the worst to maintain. They are the equivalent of a carpenter who brags about how quickly they can bang nails but can't build a stable structure to save their life.
It immediately tells me that they've never worked on large software projects, and if they have they haven't worked on ones that lasted more than a few months.
I apologize to folks reading this for my rather aggressive tone but I've been writing software for a long time in numerous languages, and people with the unit tests as an afterthought attitude are typically rather arrogant in fool hardy.
The most recent incarnation I've encountered is the hotshot data scientist who did okay in a few Kaggle competitions using Jupyter notebooks, and thinks they can just write software the way they did for the competitions with no test of any kind.
I had one of these on my team recently and naturally I had to do 95% of the work to turn anything he produced into a remotely decent product. I couldn't even get the guy to use nbdev, which would have allowed him to use Jupyter to write tested, documented, maintainable code.
Say I have a string that’s supposed to represent an integer. To me, “Validate” means using a regex to ensure it contains only digits (raising an error if it doesn’t) but then continuing to work with it as a string. “Parse” means using “atoi” to obtain an integer value (but what if the string’s malformed?) and then working with that.
I first thought this article was recommending doing the latter instead of the former, but the actual recommendation (and I believe best practice) is to do both.
Ha yeah nice catch, that's why I added that in there. In this case the dictionary is slightly wrong.
> The defining aspect of parsing, that makes it difficult and an interesting thing to study, is that you're starting with a list and ending with a tree.
Ah, I didn't know that! Great bit to learn.
In that case, I will say that the "increase a data structure's rules" is a bit ambiguous.
I think my statement is still correct in that "a symbol could be a data structure," right? Like you could take a list of dicts and emit a tree of dicts.
But wait, a list is a kind of tree, or rather, there is a parse tree of recursive head/tail branches. So I think you could still argue List->NotEmptyList is a Parse because NEL requires a nonzero "head" and zero or one NEL as "tail."
When the problem space is less well defined the type-related boiler plate adds a lot of friction. It's not impossible to overcome that friction but it slows down progress. When you're under a tight deadline development velocity is often more valuable than absolute correctness or even overall runtime efficiency.
An delivered product that works is usually more valuable than an undelivered product that's more "correct" or efficient. A development project is just a cost (for various values of cost) until it ships.
The best argument I've heard for doing type annotation is for documentation purposes to help future devs. But I don't completely buy this either. I touch new codebases all the time and I rarely spend much time thinking about what types will be passed. I can only assume it comes with experience.
Type annotation actually ends up taking a hell of a long time to do and is of questionable benefit if some of the codebase is not annotated. People spend sometimes hours just trying to get the type checker to say OK for code that actually works just fine!
C++ the language very much does tell you what to expect at runtime, though perhaps not everything you could ever want. I mean, it's not Haskell or Idris with their much richer type systems.
Proof-checking happens only at compile time.
The implementation that you want to prove things about is only available at runtime!
Non-availability (incompleteness) of information is what is preventing you…
Compile time is not runtime.
The Java compiler is not the JVM.
The compiler does type checking. The JVM does the dynamic dispatching.
Neither does both.
Yeah I guess. Text combinator libraries like Haskell's Parsec and Rust's Nom are typically parametric over the type of "characters". Realistically I don't think I've ever seen anyone use one of those libraries for an input that wasn't text-like, though; do you have a use case in mind?
> But wait, a list is a kind of tree, or rather, there is a parse tree of recursive head/tail branches.
Yes, so you can run into parsing problems when working with trees, if you work really hard at it. But if you do the correct action is "reconsider your life choices" and not "use parsing theory".
Consider addition. The compiler does type checking, and the JVM actually adds the numbers. Nonetheless, the addition is type checked, and does not represent a weakness of static type checking. How is dynamic dispatch different than this?
You can’t have both static type safety AND dynamic dispatch at the same time and in the same context about the same data.
Choose one. Give up the other. Make a conscious trade off.
The language that you are using is making such trade offs for you - they are implicit in the language design. Best you know what they are because they are meaningful in principle and in practice.
The development process is totally different when you write structured types first and then write your logic. 10/10 would recommend.
Usual caveat: this is what makes sense to me and my brain. Your experience may be different based on neurotype.
I disagree. I've started using types from the ground up and it helps almost equally at every stage of the game. Also I aggressively rely on autocomplete for methods. It's faster this way than usual "dynamic" or "pythonic" python.
Part of it might be exactly because writing my datatypes first helps me think about the right abstractions.
The big win with python is maybe 2-10% of functions, I just want to punt and use a dict. But I have shifted >80% of what used to be dicts to Models/dataclasses and it's so much faster to write and easier to debug.
The input-data to the compiler can’t be handled by the JVM and vice versa.
The JVM handles byte code as input. The compiler handles source code as input.
That is two different functions with two different data domains.
They literally have different types!
Which one of the two functions is the thing you call “Java”?
Apparently you do know what dynamic dispatch is, you're just wrong that it can't be type checked.
In Java, say you have an interface called `Foo` with a method `String foo()`, and two classes A and B that implement that method. Then you can write this code (apologies if the syntax isn't quite right, it's been a while since I've written Java):
Foo foo = null;
if (random_boolean()) {
foo = new A();
} else {
foo = new B();
}
// This uses dynamic dispatch
System.out.println(foo.foo())
On certain backend code where I am able to do unit tests, they do catch the occasional edge case logic error but not at a rate that makes me concerned about only checking them in some time after the original code, which I'll have already tested myself in real use as I went along.
for all expressions e:
if e type checks with type t, then one of the following holds:
- e evaluates to a value v of type t; or
- e does not halt; or
- e hits an "unavoidable error"
like division by 0 or null deref
(what counts as "unavoidable" varies from lang to lang)
> The same is true for the rest of the things you've mentioned: none are specific to static typing!
Sure, I could be wrong here. I frequently am. But could you point out why do you think that?
I don't think it's worthwhile for anyone else to argue with you further. C++ and Java both support dynamic dispatch although you deny it.
You've taken up almost a full page of HN arguing with everyone trying to explain it to you. People have pointed you to wikipedia showing you that you're wrong. [1]
ISOCPP of which Bjarne is a director [2] says that C++ supports dynamic dispatch. [3]
And you continue to attempt to argue that everyone on HN, Wikipedia and the creator of the C++ language are all wrong and don't know what dynamic dispatch is.
Your continued insistence is both wrong and a negative impact at this point on hn. Please stop arguing something that numerous people have taken lots of patience and charity in trying every way possible to explain to you and what is clearly factually wrong.
If you're going to reply, please explain why an organization that Bjarne Stroustrup is a director of believes that C++ supports dynamic dispatch.
1. https://en.wikipedia.org/wiki/Dynamic_dispatch#C++_implement...
Unless you were writing very small throwaway scripts, in what world where you writing your logic first and thinking about your data structures later?
Different implementations of eval() a.k.a different programming languages have different semantics.
Which implicit eval() implementation does the above hold?
What type system do you have in mind such that the above holds for ALL expressions. Type-checking itself is not always decidable.
Now, even in the rare case where I write some Python, JS or PHP, I write it in a very statically typed style, immediately parsing input into well-thought-out domain classes. And for backend services, I almost always go with 3 layers of models:
1) Data Transfer Objects. Map directly to the wire format, e.g. JSON or Protobuf. Generally auto-generated from API specs, e.g. using Open API Generator or protoc. A good API spec + code gen handles most input validation well
2) Domain Objects. Hand written, purely internal to the backend service, faithfully represent the domain. The domain layer of my code works exclusively with these. Sometimes there’s a little more validation when transforming a DTO into a domain model
3) Data Access Objects. Basically a representation of DB tables. Generally auto-generated from DB schemas, e.g. using libs like Prisma for TS or SQLBoiler for Go
Can’t imagine going back to the “everything is a dictionary” style for any decent sized project, it becomes such a mess so quickly. This style is a little more work up front, when you first write the code, but WAYYYYYY easier to maintain over time, fewer bugs and easier to modify quickly and confidently, with no nasty coupling of your domain models to either DB or wire format concerns. And code gen for the DTO and DAO layers makes it barely more up-front work.
Because the implementation details of Foo are actually know at compile time. Which is why you are able to type-check it.
You have literally declared all allowed (but not all possible) implementations of Foo.
What happens when Foo() is a remote/network call?
If you can type-check your dispatcher at compile time then there is nothing dynamic about it.
Decidable (ahead of time) means your function is fully determined. Something that is fully determined is not dynamic.
It is the conception computer scientists use.
I thought you were just mistaken about something, but no, instead you've redefined a well understood term without telling anyone, then aggressively refused to clarify what you meant by it and argued for hours with people, while saying they were all wrong when they used the well established term to mean its well established meaning.
The thing you're talking about is an interesting concept, but it's not called dynamic dispatch, and you will confuse everyone you talk to if you call it that. I don't know if there's a term for it.
Also, what makes you think I’m not aware of datatypes? Currently working eight hours a day on Django models.
Wikipedia, every textbook you can find, the top dozen search results for "dynamic dispatch", me who has a PhD in computer science plus all the other CS PhD people I know, everyone in my office who knows the term (who are industry people, not academia people), every blog post I have ever read that uses the term, and all the other HN commenters except you. I'm really not exaggerating; a lot of CS terms have disputed meanings but not this one.
EDIT: Sorry all for engaging the troll. I thought there might have been some legitimate confusion. My bad.
In short, there are choices besides, “I alone have to do all the hard work.”
Voila, now your 'string' is 'binary data' not 'text'.
Parsing binary data is my bread and butter, so I might be biased but: it works fine.
Anything which comes over the wire is a string, anything which comes out of store is a string. If you're using something like protobufs, that's great, because having to marshal/serialize/parse along every process boundary is expensive and probably unnecessary.
But at some point, and anywhere on the 'surface' of the system, data has to be un-flattened into a shape. That's parsing.
Oh, that's recursive! Which is Computer Science's domain of expertise, not the public domain.
We are talking about formal semantics here. What do programs (and computer languages are themselves programs) mean?
Point 0 of Wadler's law.
https://en.wikipedia.org/wiki/Semantics_(computer_science)
If you can type-check it at compile time then it is NOT dynamic dispatch. It's a contextual confusion.
Registering pointers to new implementations at runtime (adding entries to the dispatch table) Unregistering pointers to old implementations at runtime (removing entries from the dispatch table).
If your dispatch table is immutable (e.g static!), there's nothing dynamic about your dispatch!
I was more experienced with predictive algorithms and deep learning than any of the data scientists at the company but because they were brought in from an acquisition of a company that had an undeserved reputation due to a loose affiliation with MIT, they were treated like magicians while the rest of us were treated like blacksmiths.
I had the choice and I made the choice to leave. And of course I raised hell with the bosses about them not writing remotely production quality code that required extensive refactoring.
And yes I was paid to do the work but the work occupied time that I could have spent working on the other projects I had that were more commercially successful but less sexy to Silicon Valley VCs who look at valuations based on other companies' newest hottest product.
Quoting from "http://math.andrej.com/2006/03/27/sometimes-all-functions-ar..."
"The lesson is for those “experts” who “know” that all reasonable models of computation are equivalent to Turing machines. This is true if one looks just at functions from N to N. However, at higher types, questions of representation become important, and it does matter which model of computation is used."
Tests should prove a desired behaviour. Sometimes it‘s not possible to fully run code until late in some staging, just because there are a lot of dependencies and conplexity. That‘s what tests are for (on various lebels of abstraction).
Because the meaning of "dynamic" is ambigous!
Since you are pointing me to wikipedia I'll point you right back...
https://en.wikipedia.org/wiki/Virtual_method_table#Compariso...
"Virtual method tables also only work if dispatching is constrained to a known set of methods, so they can be placed in a simple array built at compile time."
If the vtable is generated at compile and is constrained to a known set of methods then that array is immutable! Calling that "dynamic" is an obvious misnomer!
You are neither charitable nor patient. You are committing the bandwagon fallacy as we speak. You and the other 120 (and counting) angry downvoters ;)
I am using the word "dynamic" to actually mean dynamic! I am not going to define it. Use your judgment. Dynamic is NOT static. I am not asking you to "educate me", or to tell me I am right; or wrong. I am asking you to understand the sort of programming language design I have in mind!
Either you understand; or you don't.
I was specifically responding to the commenter I replied to, who said they didn’t need tests because their code just worked the first time after he wrote it.
That phrase makes me sad. Mainstream languages have a lot of scope for improvement in their type systems.
When using a data structure, I know what set of fields I expect it to have. In TypeScript, I can ask the compiler to check that my function's callers always provide data that meets my expectations. In JavaScript, I can check for these expectations at runtime or just let my function have undefined behavior.
Either way, if my function's assumptions about the data's shape don't turn out to be correct, it will break, whether or not I use a dynamic language.
It seems that most of the people who make this argument against static typing are actually arguing against violations of the Robustness Principle[0]: "be conservative in what you send, be liberal in what you accept".
A statically typed function that is as generous as possible should be no more brittle against outside change than an equally-generous dynamically typed function. The main difference is that the statically typed function is explicit about what inputs it has well-defined behavior for.
However, it does have a strongly opinionated approach to casting that can sometimes yield non-obvious results. This behavior is documented and I would suggest new potential adopters of the library to explore this casting/coerce feature in the context of your product/app requirements.
For the most part, it's not an huge issue, but I've run into a few surprising cases. For example, sys.maxint, 0, '-7', 'inf', float('-inf') are all valid datetime formats.
- https://pydantic-docs.helpmanual.io/usage/models/#data-conve... - https://gist.github.com/mpkocher/30569c53dc3552bc5ad73e09b48...
You write automated tests so that you can keep running the tests later such that the behaviour is maintained through refactor and non-breaking changes.
Terms in computing are so overloaded that these days I try[0] and never correct anyone on how they use a term. Instead I ask them to define it, and debate off of that definition.
So instead of downvoting this guy for using different terminology - we can ask him what he means and just have a discussion.
[0] alright I don't always succeed but it's an ideal to strive for
Computation is a general, abstract and incredibly useful idea disconnected from any particular model of computation (programming language).
Different languages are just different models of computation and have different (desirable, or undesirable) semantic properties independent from their (trivial) syntactic properties.
It's this sort of angry dogmatism which prevents people from talking about programming language design.
Not for a second do they pause to think their own understanding may be limited.
I’ve run across DSLs that have three or more layers of parsing and validation. Embedding different languages within each other (eg: JSON snippets within your own DSL) definitely leads to the issues the article talks about.
Also, growing your own parser without understanding standard lexer/parser basics seems far more common than it ought to be. I’m not talking brilliant design, rather the extremely naive one-character-at-a-time-in-a-really-complex-loop variety of design.
The better level of bad is, “I know what lexers/parsers are, now I’ll write something to basically implement a type-checking parser with the lexed+parsed tree as input.”
This article is basically stating, “Why not just get your parser to do it all for you in one swell foop?” When I have refactored code to follow this kind of design, I have never regretted the outcome.
The dogmatism demonstrated today was in your comments, ukj. Your inability to understand that your non-standard use of terms makes it impossible for others to communicate with you in any effective way made this a remarkable farce of a conversation or debate.
There is only the model you (and your tribe) believe is "standard" implicitly. Do you actually understand that?
I kinda thought navigating the inherent ambiguity of all language (programming or natural) is a fundamental skill for software engineers.
Communication is indeed impossible when you think you possess the "right" meaning of words.
Oh... by the way, somebody capable of understanding/listening has just informed me that what I am actually talking about is closer to multiple dispatch than dynamic dispatch.
Multiple dispatch is, in fact, more dynamic than "dynamic dispatch". Hilarity ensues.
The problem in this discussion is that ukj has come to the belief (but communicated it poorly) that dynamic dispatch is somehow incompatible with static typing. And for some reason this also matters.
Static typing does not preclude dynamic dispatch, and despite being pointed to several mainstream languages that have both features, ukj decided to ignore reality or the common understanding of the phrase "dynamic dispatch" and produced this grotesque example of trying to communicate with an individual who is, apparently, just a troll. Feeding the troll, ukj, is probably the dumbest thing I did today, but I'll blame that on the insomnia reducing my ability to detect trolls.
According to the above definition C++ does not have dynamic dispatch!
The vtable in C++ is generated at compile time and used at runtime. It's immutable at runtime. That means it is NOT dynamic!
https://en.wikipedia.org/wiki/Virtual_method_table#Compariso...
"Virtual method tables also only work if dispatching is constrained to a known set of methods, so they can be placed in a simple array built at compile time."
Like, I don't care how you use words, but you (and everyone) are using "dynamic" to speak about a statically-behaving system!
What I am talking about when I say "dynamic dispatch" is the sort of dispatching done by R, LISP and Julia (and not by C++ or Java). Now, we can bicker about labels and you can insist that it's actually called "multiple dispatch" and not "dynamic dispatch", but you can't bicker about the semantic fact that "multiple dispatch" is actually more dynamic than "dynamic dispatch".
And this semantic point would've been trivial to unpack if you weren't try to win an argument.
Indeed, sorry for engaging the trolls. 20; or 30 of you. Lost count.
And sadly Cursors haven't been generalized to all associative iterable structures in Ada (arrays for example).
Not being a big fan of method chaining, a null saavy foreach would probably eliminate most of my null checks, need for Optional.
If you derive a C++ compiler that accepts file A as valid C++. And I derive a C++ compiler that rejects file A as valid C++, then from the lens of type theory the two compilers have different type-signatures!
They are not the same formal language. One, or both compilers implement a language that is not C++.
If there was a "standard use" languages wouldn't evolve and we wouldn't have so many of them. 8 billion people on Earth would be speaking The Standard Language!
Perhaps the thing you don't understand (over and above how to communicate) is this comic?
My cons are: - Uses some dsl to define types - Doesn’t marshal to model objects by default, but from DICT to DICT
My pro is: - much more configurable and powerful
Recently, I spent 3 years on Scala then switched jobs and spent 3 years in Ruby.
It was a shock to go back to dynamic languages, but after 3 months, I honestly couldn't tell which felt more productive or led to more stable high quality product.
In Ruby, we had all the issues people point out about dynamic languages, but the product didn't lean heavily on complex data structures or algorithms. We embraced complexity and failure and get good processes, designs and practices to deal with this.
In Scala, we had more rigour, but I also know I spent a lot of time on type design. Once things were sorted there was a lot of confidence in it, but generally, it took a lot longer to get there.
For certain systems that is absolutely worth it, for others (and in my case) it did feel like the evolution of the product meant this effort never really paid off.
0. Dynamic dispatch (as you are using it) 1. Double dispatch 2. Multiple dispatch 3. Predicate dispatch 4. All of the above collectively.
What you thought was a confusing re-definition on my part might have been an ignorance of alternative uses on yours ;)
Java doesn't do 1, 2 or 3.
Defining a data structure up front doesn't require a lot of boilerplate as Java incorrectly have taught all of us. Writing a statically typed typing.NamedTuple or @dataclass is literally a one-liner.
The rebuttal to this is provided by another post from Alexis King: https://lexi-lambda.github.io/blog/2020/01/19/no-dynamic-typ...
> This story sounds compelling, but it isn’t true. The flaw is in the premise: static types are not about “classifying the world” or pinning down the structure of every value in a system. The reality is that static type systems allow specifying exactly how much a component needs to know about the structure of its inputs, and conversely, how much it doesn’t.
Static typing helps you to safely model the parts of the system that you actually do know about, while allowing you to leave the unknown parts loosely defined. And when business requirements change, the compiler helps you to make those changes in a safe, guided way. This is why people often report the experience of doing large refactorings driven by compiler error messages, then running and finding that the change works correctly on the first try. A confidence few report in dynamically-typed languages.
> After doing a couple dozen, I'm seeing a pattern: developers using static tooling (e.g., Java, TypeScript) get circa half as much done as people using dynamic tooling (Python, plain JS).
I hope you understand and accept the fact that interview code challenges are not very representative of real-world software engineering. On-the-spot interview code is throwaway; real production code is not.
Static types can absolutely help with more than local reasoning. One of the main reasons I like static types is that they give me a place in the code to reflect aspects of the architecture that otherwise go implicit. Databases are a great example: in both static and dynamic languages, the behavior of your code is going to depend on the database schemas you're using, and static types give you a mechanism to declare and manage this dependency explicitly. In a general sense, the key in architecture is how a system is split up into parts and what interfaces the parts have, and a static type system is a way to make those interfaces first-class citizens of a language subject to abstraction and automation.
Code with static types might be larger than the problem it's trying to solve, but so might code with dynamic types—that's more a function of the sort of abstractions and complexity that go into the code. I've written some code to do very similar things in Haskell and Python, and my experience is that the Haskell code tends to be noticeably more compact than the Python code, even though I make liberal use of types and write out all the top-level type signatures. While some of this comes down to other differences between the languages, part of Haskell's expressiveness absolutely comes down to static types—static types make it easier to write very general code against high-level abstractions (eg monads) that would need to be narrower in dynamic languages (eg promises).
And sure, you can couple code and data in static languages, but at least the types will help you when it comes time to change. I've worked with dynamically typed programs where functions rely on some mixture of nested tuples, hashmaps, lists... etc, and it's hard to understand exactly how concepts are represented in code, much less change that representation. If you represent some central notion in your code as some nested primitive type (a hashmap of tuples of ...) and you want to change that representation, you'll still have to update the places it's used in the code, but without a static type system you won't get a list of those places, and you won't have assurance that updated everywhere that mattered. I'm not sure I'm explaining this pattern well, but I've worked in JavaScript codebases where making breaking changes to core data components was basically impossible.
Point being, all the problems you mention come up in both static an dynamic languages. There might be a case that they're more common in one or the other, but it's not obvious either way, and it's going to depend a lot on which specific static type system you're thinking of.
I think this is it: https://lexi-lambda.github.io/blog/2020/11/01/names-are-not-...
When a router does lookups from a static table it's "static routing".
When Java does lookups from a static table it's "dynamic dispatch".
The same type of computation is being characterised as both "static" and "dynamic".
When a router does lookups from a dynamic table it's "dynamic routing" - there is no equivalent in Java because making the dispatch table reflexive/mutable is precisely what violates type-safety!
What routers call "dynamic routing" is having the lookup table mutable at runtime.
There can be no equivalent to that in type-safe languages because when you mutate the dispatch table you lose type-safety.
Even a lot of JSON or XML parsing, you throw it into a parser and take what you need; if an unrelated field isnt what you expected, just move on with things rather than stop everything because the library author forgot about extension or openness possibilities (ie: an xs:any in a schema).
This attitude comes from the assumption not that types are unhelpful; just the chances we've modelled every outcome into our view of the world and gotten that right is unlikely.
When you get to complex systems with state changes to data and strict, well defined policies, rules engines, etc? Thats where dynamic languages often start adding all of that validation layer, to assert for right now you should act more like a type system and it's important - it probably has financial or security or other risks.
I disagree that "real production code" is not throwaway. In situations of high requirements volatility, a great deal of production code ends up getting thrown away, because either a) it's intended to be a short-lived way to learn about the domain, or b) it's built on an assumption that turns out to be incorrect. And until the project hits economic sustainability, a lot of assumptions necessarily get made. It's only once the project is economically sustainable that we have some confidence that the code will be around for the long haul.
If your point is that most people are using statically typed languages wrong for high-volatility situations, I'll take your word for it. There were definitely times Typescript users seemed to be unnecessarily constraining things, for example. But I'm not sold that the more elaborate approach of being thoughtfully and explicitly untyped for a project's early stages for the sake of prototyping in a statically typed language would yield much practical gain.
I haven't used pydantic's ORM integration, but I don't hesitate to use pydantic models everywhere as business logic classes unless I need ludicrous speed.
That's all opinion, but I'd definitely give pydantic a swing.
> There can be no equivalent to that in type-safe languages because when you mutate the dispatch table you lose type-safety.
That's exactly what I mean by escape hatches. Rust has unsafe. You can write a WTF FastInverseSqrt in Rust. You shouldn't 99% of the time. But you can.
Rust also has Box, Rc, Arc, and other tools. I'm not fluent enough in rust to know how to, but I'm quite confident and will eat my hat if you can't accomplish what is effectively a mutable vtable dispatch in Rust.
But also...why? Yeeting a function call into the void without any type knowledge seems way more bug-prone than interface/trait-based dispatch, to what benefit? Save developer time? I've spent countless dev-days of my life I won't get back debugging exactly this kind of dynamic-dispatched, json-dsl, frankly loosey-goosey untyped bullshit. There's so many better ways.
I know reaction comments are discouraged on HN, but this had me in stitches. Top-tier gourmet dig right there.
I don't deny it. Join the cult of Static Python. We have cookies! And lower stress levels!
I usually wrap that spiel with my caveat "this depends greatly on your neurotype, style, environment, and other things." I have ADHD and my brain struggles with keeping bits of state in memory, so having to remember the type of every variable without my IDE tracking it for me is a huge performance drain.
However, I would contend even if your neurotype supported that mental workflow... it isn't actually better. Humans on average can handle 7 +/- 2 "pieces" of information in focus. Why spend any of your precious half-dozen pieces of salient consciousness on something a machine is really good at doing?
It's not zero consideration of data structures, it's mostly a focus on the main data type (arrays and data frames) and not really thinking about typed records, data models and such. The majority of types are float, str, dict, np.ndarray, pd.DataFrame. No dataclasses, minimal classes, and when classes are used, it's Java101 style "all the bad parts of OOP" programming. Sadly, I've spent years in this space before learning better.
Opinion: This is actually a symptom of what is (imho) a pervasive problem lodged deep in the collective consciousness of software dev: OOP with fine-grained objects. I blame (early) Java in large part for exacerbating this mentality. Encapsulation of state with mutator methods in particular. It sprays state all over the application, encourages mutation in place over immutability, coupling, validating-not-parsing, and makes it nigh-well impossible to write good tests.
It's really hard to write objects that enforce all invariants under every mutation. And when you have state strewn everywhere, it's impossible to test every nook and cranny. The combinatorial space explodes.
Objects are helpful for encapsulating state when they are course-grained, mutations are atomic, coupling occurs in one place, state changes are auditable, and the entire state can be replayed/set at once, to enable mock tests and subsystem integration tests. AKA, things like databases, reactors, and persistent data structures.
The reason that "dynamic dispatch" in Java, etc. is called that is that the instance of the dispatch table to use is chosen dynamically, rather than being fixed at the callsite. It's true that Java doesn't let the shape of the dispatch table change at runtime, but that's not what dynamic vs static refers to conventionally in this context. The ability to dynamically add and remove methods from a class is something which you typically only get in dynamically typed languages but dynamic dispatch and dynamic typing are not the same thing.
In particular, while a full-featured routing information base implementation will usually use some form of dynamic dispatch to customize the behavior of routes originated through different protocols, it's very uncommon for an implementation to rely on dynamic typing which adds or mutates the methods associated with different entities. That's simply a different kind of tool used for different purposes. It's something which can be helpful in object-relational mapping, for instance, because you can create methods based on a dynamic database schema. The RIB is not going to have a schema like that which changes at runtime.
That's precisely the point. You can specify part of the routing table at compile/configure time - the rest gets generated at runtime.
The data/control plane distinction is conceptual. It doesn't hold in memory when the router is handling its own network traffic - it has a single routing table/world-view.
My own routing table is shared by the data plane AND control plane.
At some point you will receive an external data (routing update) which requires runtime validation, you will do reflection and update your own routing table (ring 0 address space) based on external events.
>The RIB is not going to have a schema like that which changes at runtime.
The schema need not change. The entries/relations between objects changing is sufficient to violate type-safety.
Route add( *str1, *str2) to Number.add().
Good luck with that.
Also since the tools are immature and bolted on afterward in Python, I think it's even a bit worse than it would be in something decent like C#.
That's exactly my point–code written under pressure during an hour-long code challenge will never be representative of code written during normal day-to-day coding. Even if the problem domain is similar to a real-world problem.
> I disagree that "real production code" is not throwaway. In situations of high requirements volatility, a great deal of production code ends up getting thrown away
But it's not written to be thrown away an hour later. That's not really the kind of 'production code' that any reasonable person would be talking about in this thread. If you're talking about exploratory data analysis, data science or whatever, sure, you can consider it 'production code', but it's not the product of the craft of software engineering, it's more about running quick experiments.
> I'm not sold that the more elaborate approach of being thoughtfully and explicitly untyped for a project's early stages for the sake of prototyping in a statically typed language would yield much practical gain.
There are many different kinds of projects. And people who work with good statically typed languages can prototype quite well using types to drive their domain modelling. It's a different way of working. You should read some of what Scott Wlaschin has written to get an idea.
It's not clear to me if you are suggesting that a dynamic routing table with different kinds of routes cannot be implemented in a type-safe manner in a statically typed language, or if you're working with an analogy where a routing table is like a dynamic programming language at runtime, in that a static set of entities and relations are known ahead of time and those are modified by runtime input. If it's the latter I'm not really sure how the analogy works—if programming languages are to routers as types are to routing entries, what in a router is analogous to a value of a given type?
I can speak more to the former possibility; here's a rough sketch of how one could implement a routing table using the tools available in a statically typed environment (and in a type-safe way). One way to do it (I believe the common way, and certainly the only one I've seen in commercial router implementations) is to treat statically populated and dynamically learned routes more or less uniformly in the data structures used to perform data-plane lookups. Each such route entry has the same fields and gets inserted into a data structure with a predefined shape. Where special behavior is needed for routes of different kinds, that behavior can be implemented by using dynamic dispatch in the sense it's usually used in C++, Java, Rust, etc. to call a method associated with a route entry, or using other techniques common to statically compiled languages—there is a fixed set of such operations defined up front. Adding and removing entries from the routing table at runtime does not typically implicate type safety because the types used to describe the table describe all of its possible valid states. For instance, the type for a node in a radix trie might describe how it can either be leaf node or contain subnodes, etc.
> The schema need not change. The entries/relations between objects changing is sufficient to violate type-safety. > > Route add( str1, str2) to Number.add().
It's obviously not always true that entries or relations changing will validate type safety; any non-trivial system will let you perform some kinds of data manipulation at runtime. Conventional static type systems will allow some kinds of mutations (like changing around pointers in a radix trie to insert a new node) but will not have the flexibility to support some others (like changing the shape of a dispatch table at runtime).
One kind of call pattern which is incompatible with statically compiled dynamic dispatch is where the types of parameters change along with the base type which owns the dispatch table; I think this is what your add() example is getting at—you need the type of the second parameter to match the first, which you can't validate without runtime checks if you don't know what concrete implementations will be in use at runtime. In the case of a routing table I don't think this kind of polymorphism is needed though; I can't think of an instance where an operation would fundamentally require a fixed relation in the concrete types of different routes. For instance, when routes overlap you can derive a priority value for each one to decide which one to use, rather than directly implementing some kind of function whichIsBetter(a, b) which relies on knowing what concrete route kinds a and b are.
Its weird how long its taken for languages with static typing and type systems designed for correctness (and designed well for that end) rather than princupally for convenience of compilation to be available that are generally usable (considering licensing model, features, ecosystem, etc.)
Programming is about explaining desires to computers. It is a complex, error-prone activity and we depend on our automated tooling to save us from ourselves.
Obviously, in theory you can build the exact same software in Assembly as you can with Java. But not in practice because the programming language is a human-computer interface and a symbiosis with useful feedback loops emerges at higher levels of abstraction.
The design choices of your language impose certain limits/discipline on your expressive power.
Some design patterns become easy to express; where others become difficult to express and your paradigm’s features become obstacles.
At which point it is your choice to switch off the guard rails.
Like the chap somewhere down below telling me he will use Rust “unsafe” so he can have direct control of his memory. Yes! You can! You can also read/write to /dev/mem directly!
Translated in English “I will turn off the compiler’s safety checks because it is getting in my way”. There are many ways to shoot yourself in the foot with this level of power.
He is agreeing with me, but this is HN and opposition must be maintained. Dung must be flung. For reasons.
Sure, for certain kinds of protocols this kind of polymorphism is not always needed. But I am merely pointing out the threshold where the type-safe paradigm begins to falter. The corner cases where the pros don’t outweigh the cons. Where the “compile time” and “runtime” distinction becomes a hindrance rather than a useful separation of concerns.
You can go further. Software Defined Networks. Application router.
In general, contexts in which it would be useful to know the shape of your data at runtime!
As a spontaneous aside… if your entire information-processing system is well-typed (top to bottom) - congratulations. You are well on your way to understanding Category Theory.
The entire data-schema of your stack is what Mathemaricians call a (small) “Category”.
Sure? This seems pretty no-true-Scotsman to me. If you want to define some the things I need to do as not real software, feel free. But they're still things my team has to get done. To me there's a spectrum of domain stability on which the code rests. It's frequently not possible to know in advance which bits will be long lived and which won't be. So whether or not one of htose bits is the true "craft of software engineering" in somebody's eyes has very little interest to me. Especially when it's an anonymous rando.
> people who work with good statically typed languages can prototype quite well using types to drive their domain modelling.
I can certainly believe that's possible. Hopefully some of them will interview with me so I can see how they work. I'm just reporting the data I have.
> that's not really the kind of 'production code' that any reasonable person
In startup contexts I will quite often try an experiment that is quick to build. A few hours is not at all uncommon. That you consider me unreasonable is a sign I can better spend my time elsewhere.
Any routing logic which makes routing decisions based on information nested arbitrarily deep into your packets.
Is this an IP packet containing data; or an IP packet containing an HTTP GET request to www.google.com?
You have to do arbitrarily deep type inference in real time.
The the data has no semantic content until you infer some! Is just a bitstream!
It's not an analogy. Programming languages and routers are particular instances of computable functions. "dispatching" and "routing" are just another example of us using different English words to describe the exact same computation: M:N mapping function.
Whether the input is mapped to an IP address or a memory address - boring implementation details.
Nothing in a router is analogous to a value of a type because there is no such thing as "types" at runtime unless you infer them! Types exist only at compile time. Types are semantic annotations of data. You are helping the compiler help you by telling it what you know about the data you are handling.
This blob encodes a Number. That blob encodes a String.
If you don't want any help from your compiler, you don't have to tell it the types of anything - just manipulate the data directly!
That's precisely what an Assembly language do. Everything is untyped.
How? A routing function is precisely a M:N mapping with untyped input.
It's just some data somebody sent you! Route it.
Unless you can infer more about the meaning of those bits the only strategies possible are static(M), round-robin(M) or random(M).
In our app, django essentially sits between a compute cluster and a front end. The pydantic objects are used to define the work to be done on the compute cluster.
This is a false statement, considering that the cases where something is validation but not parsing may very well lie in the complement of the set of compilers to the set of all computable functions. The converse statement that if this were true about the smaller set (of compilers), it would also be true about the larger set (of all programs) would on the other hand be correct.