For micrograd: is there more documentation available than just the source code in the Github repo?
replies(1):
https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...
Let me share some of my favourites not listed here, off the top of my head:
- Ian Piumarta’s “Open, Extensible Object Models” (https://www.piumarta.com/software/id-objmodel/objmodel2.pdf) is about creating the most minimal object-oriented metaobject system that allows the maximum amount of freedom for the programmer. It basically only defines a message send operation, everything else can be changed at runtime. The practical counterpart to the dense “Art of the Metaobject Protocol” book.
- John Ousterhout “Scripting: Higher-Level Programming for the 21st Century” (https://web.stanford.edu/~ouster/cgi-bin/papers/scripting.pd...) - not really a paper, but an article from the creator of Tcl about the dichotomy between systems programming languages and scripting languages. Obvious at first sight, the lessons therein have wide ramifications IMO. We always seek the perfect multi-paradigm language that can do anything at high performance with the most productivity, while perhaps it is best to have compiled, fast, clunky systems languages paired with ergonomic, flexible interpreted frontend. Often all you need is C+Tcl in the same app. A must-read for anyone writing yet another programming language.
- Niklaus Wirth's Project Oberon (https://people.inf.ethz.ch/wirth/ProjectOberon/) is the implementation of an entire computer system, from the high-level UI down to kernel, compiler, and a RISC-like CPU architecture. He wrote the seminal "plea for lean software" and actually walked the walk. A long lost art in the era of dependency hell and towering abstractions from mediocre coders.
(Joking of course. I much prefer “Hammock driven development” but it’s not very corporate friendly)
First, my understanding of his points - a language is either a systems language like C or a scripting language like TCL or python. Systems language have "strong" types and are for data structures/algorithms, scripting languages are "typeless" and are for "gluing things together".
The main claim is that scripting languages are more concise and allow for faster development when gluing due to their 'typeless' nature.
In his example, he creates a button in Tcl.
button .b -text Hello! -font {Times 16} -command {puts hello}
He goes on to say:
With C++ and Microsoft Foundation Classes (MFC), it requires about 25 lines of code in three procedures. 1 Just set- ting the font requires several lines of code in MFC: CFont *fontPtr = new CFont(); fontPtr->CreateFont(16, 0, 0, 0, 700, 0, 0, 0, ANSI_CHARSET, OUT_DEFAULT_PRECIS, CLIP_DEFAULT_PRECIS, DEFAULT_QUALITY, DEFAULT_PITCH|FF_DONTCARE, “Times New Roman”); buttonPtr->SetFont(fontPtr);
Much of this code is a consequence of the strong typ- ing[...] In Tcl, the essential characteristics of the font (typeface Times, size 16 points) can be used immediately with no declarations or conversions. Furthermore, Tcl allows the button’s behavior to be included directly in the command that cre- ates the button, while C++ and Java require it to be placed in a separately declared method.
End quote.
We've come a long way, and examples have made clear that this dichotomy is a false one. Ousterhout's view was colored by his limited experience, resulting in him misunderstanding what he actually likes about Tcl.
Let's talk about syntax. His exact Tcl code as presented could be parsed and compiled by a language that does static type analysis. It's not, because he's running it in an interpreter that only checks types at runtime. But the point is that whether code is compiled or interpreted is an implementation detail that has very little to do with the syntax. He likes his syntax because his syntax is obviously better than C++, nothing more.
And types: he claims that 'typeless' languages allow for faster development because of fewer restrictions. This is, ofc, nonsense. The amount of restrictions you have is a function of your problem, not your language. If it feels like dynamic typing is letting you develop faster, that's because you're putting off encountering those restrictions til later.
It's better to guarantee we encounter all type errors before the program even runs. But since you can do static type analysis for any language, why don't all languages do that?
Because it's hard. Type theory is complicated. Not all type systems are decidable, we need to pick one that is for practical reasons. Ones that are may require annotations or complex algorithms/semantics restrictions like Hindley-Milner.
As a PL designer, it's a whole lot easier to just give up and only check types at runtime. And that makes a whole lot of sense if your priority is just embedding a usable language ASAP. But let's not pretend that it's because it's actually better.
This is only valid if you either are writing mission-critical software, or have infinite time.
Your argument doesn’t consider the case that you have a deadline and need to ship, so optimising for productivity, rather than asymptotic typing perfection, is paramount. There is a reason even performance-critical environments, where scripting languages are not very well suited, somehow still lean on them for productivity.
Case in point, game dev (Unreal Engine with its blueprint system, Godot with GDScript, the myriad of game engines in C++ paired with Lua.) Of course in a vacuum game devs would like to write ideal data structures with strong typing so that the game doesn’t crash, but their goal is to ship a game within the decade, so limit the strong typing to performance critical engine and can focus on building and iterating on the core product without consulting type theory books.
The point of Mr. Ousterhout’s argument is that there are only two choices: either we invent the panacea, the mythical productive strong-typed language that gives 100% safety yet enables experimentation, optional typing and compiles to C++ speed native code, or we accept that this is an impossible pipe dream, and we need to use the correct tool for the problem. Again, obvious on the surface, but still a contentious point to this day.
Unless you're implementing something that's already known, you're effectively carving out your problem when you're in the process of writing code. There are more unknowns than there are knowns. A strongly typed language forces you to focus on satisfying the types before you can do anything, at a point where you don't exactly yet know what you want to do. The right time to cast your data rigidly into types is gradually when your data structures settle in their own place one by one; in other words when you're pretty sure things won't change too easily anymore. This allows writing code that works first and then, depending on your needs, gets next correct and then fast, or next fast and then correct. You use the weakly typed or "typeless" style as a prototyping tool that generates snippets of solid code that you can then typefy and write down for good.
I often think about this quote from TAPL. This framing of “safety” changed how I design systems.
> Informally, though, safe languages can be defined as ones that make it impossible to shoot yourself in the foot while programming.
> Refining this intuition a little, we could say that a safe language _is one that protects its own abstractions_.
> Safety refers to the language's ability to guarantee the integrity of these abstractions and of higher-level abstractions introduced by the programmer using the definitional facilities of the language. For example, a language may provide arrays, with access and update operations, as an abstraction of the underlying memory. A programmer using this language then expects that an array can be changed only by using the update operation on it explicitly—and not, for example, by writing past the end of some other data structure.
https://www.cosmopolitan.com/lifestyle/a7664693/mandela-effe...
How often are you passing around data that you don't fully understand?
Also, people use types and then end up reaching for reflection to perform pattern matching on types at which point you've just moved the typing from the user level to a type system. Not much gained imo.
Ultimately, it all compiles down to assembly (which is executed), or an interpreter (which is executed). Assembly all the way. The real choice here is: turn into assembly ahead of time (programmer's code becomes assembly), or do so at runtime (assembly interprets programmer's code). Or some in-between like JIT compilation. And: what guardrails you put in place.
So you could say that higher-level languages are just a way to make tedious/nasty assembly coding more palatable. Eg. by producing error reports vs. just crashing a machine. Or provide a sandbox. Or provide a model of computation that fits well with programmer's thinking.
Between those, you simply need some useful abstractions. Preferably ones that are simple yet versatile/universal.
Eg. a (bitmap) "screen" could just be a flat memory space (framebuffer). Or a 2-D array of pixels, potentially with "pixel" defined elsewhere.
Programmer doesn't care how GPU is coerced into displaying those pixels, low-level code doesn't care what higher-level software does to produce them.
And then have as few as those abstractions as possible (but no less!). Tcl: "everything is a string". Lisp: "everything is a list". Unix(-like): "everything is a file". Etc.
Personally, I have a soft spot for languages that punch above their weight for expressiveness / few but useful abstractions vs. their implementation size. Things like Forth, Tcl or Lua come to mind.
But hey that's just me. Developers & organisations they're in make their own choices.
Those language restrictions slow down your development as you have to design and code around them. People see the extra design\code as a good trade off because it improves safety, but it slows you down if you are just messing around.
I wouldn't be so sure. Even Javascript, when is added types in the form of Typescript, looks more like the convoluted Windows API example and less like the TCL example.
>Let's talk about syntax. His exact Tcl code as presented could be parsed and compiled by a language that does static type analysis. It's not, because he's running it in an interpreter that only checks types at runtime. But the point is that whether code is compiled or interpreted is an implementation detail that has very little to do with the syntax.
Pedantically true. You can of course compile anything, including BASIC and TCL, or intepret C++ if you wished so.
But to really take advantage of compilation, languages tend to add types and other annotations or semantic constructs (like Haskell does). And so compiled languages tend to a certain syntax/semantics, and interpreted languages to another.
>And types: he claims that 'typeless' languages allow for faster development because of fewer restrictions. This is, ofc, nonsense. The amount of restrictions you have is a function of your problem, not your language. If it feels like dynamic typing is letting you develop faster, that's because you're putting off encountering those restrictions til later.
So it's not "nonsense", but rather a tradeoff, just like Ousterhout puts it.
For many, it is much faster to "putting off encountering those restrictions til later", as that's a large part of exploring the problem space.
See, while "the amount of restrictions you have is a function of your problem", you don't always know your exact problem, its boundaries, and the form you'll use to solve it, until you've experimented and played with early prototype solutions and changed them, etc. While doing so, having to tackle hard restrictions based on assumptions that can and will change or be discarded, slows you down.
>It's better to guarantee we encounter all type errors before the program even runs.
Only if it doesn't entail other issues, like it making our exploratory programming to end up with a design for our program slower and more tedious.
Like all the time. As a program grows, the full extend of what is passed where, explodes.
>Also, people use types and then end up reaching for reflection to perform pattern matching
As a tool in their disposal, with the default being the opposite.
https://news.ycombinator.com/item?id=43595283
I hacked up a toy brainfuck interpreter using that technique and it was pretty fast. Not sure I'd get the chance to use it elsewhere, but experimenting with it was useful regardless:
Well, the point of scripting languages is mostly dynamic typing and dynamic binding, which means resolving bindings at runtime, which means slower in execution. There is no silver bullet: you want speed, you need a clunky language that compiles to fast machine code. If you want ease of use and less ceremony, you’ll get a slower language.
I wish OCR technology got so good that I could only write by hand and have perfect storage and searchability.
For a related example, the BASIC programming language is great for programs that fit in 24 lines (which is how many lines a glass tty displayed).
They are not necessarily easier to use for statically-knowable problems.
Static languages get benefits in correctness and speed.
Key word is dynamic.
https://www.stackage.org/haddock/lts-23.22/base-4.19.2.0/Dat...
It seems like all of the dynamically typed programming languages are pursuing (optional/gradual) static typing systems, and we've mostly moved on past "pure" dynamically typed PLs at this point. It's more about how-much and what-kind of static typing. And perhaps this depends upon your domain (e.g. Rust and its focus on memory safety while eschewing GC, which necessitates lifetime annotations).
0) I don't think type inference is a panacea. For example, it's not super helpful when you have a complex type system inferring crazy types for you, and you're stuck debugging them when the inferrer has failed.
Dynamic typing has no such constraints to help you catch errors at compile time. Dynamic typing allows you to build organic structures that more fluidly change. If you change a function signature, you don't have to change every piece of code that calls the function. You can change the function and test it in a running application without restarting the application. Doing this takes seconds. Dynamic typing allows you to iterate 100x faster. Being able to rapidly iterate often means better tested code and more reliable code.
Both are valuable. Dynamic typing is valuable during development when you need to iterate rapidly and build a prototype when requirements are often changing. Static typing is valuable when code need is more mature, changing less frequently, and code needs to be put in production.
Ideally you have both. More mature parts of the program are statically typed and more fluid parts of the program are dynamically typed.
A language's preferred compilation/interpretation model is, roughly speaking, orthogonal to its type system and to whether it emphasises speed of development, correctness, or performance. Also, many languages can practically be implemented using various compilation/interpretation models. There are interpreters for C, for instance, and ahead-of-time machine-code compilers for Java.
I strongly disagree that this sort of refactoring is "quicker" with dynamically typed languages, particularly with non-trivial (i.e. large) code bases.
You have zero feedback and no indication of how disruptive such a change is when working with a dynamically typed language. In fact, you have no feedback on whether you've actually finished the change at all.
While with static type checking, you have immediate and valuable feedback guiding you through such a change. Each line of code which requires updating is identified for you and communicated to you.
This is not my experience at all. Virgil is a fully statically-typed language and compiles fast. When working on the Virgil compiler (approximately 50KLOC), my typical workflow is to run the compiler in the interpreter in my debug, edit, run loop. The interpreter can load the entire compiler, typecheck it, and run it with my changes in 90 milliseconds. If we throw a complete optimized compile in the loop, it's 400 milliseconds.
It takes me longer to switch between my two terminal tabs.
It's not static typing versus dynamic typing. It's ridiculously inefficient languages/build systems versus efficient ones.
Maybe in a larger code base you could argue that the effort involved in satisfying typechecks meaningfully slows down iteration, but even then it's clear that being able to compute the list of type errors you introduced elsewhere (via static typing semantics/analysis) is valuable.
Modern BASICs are suitable for writing larger programs.
Also seen today on HN:
"I Miss Visual Basic" https://news.ycombinator.com/item?id=43988853
"The history and legacy of Visual Basic" https://news.ycombinator.com/item?id=43949855
For me, the most important reason is that static typing makes the code much more self-documenting.
I can see what that function is returning, and if it's something unknown I can go to the type declaration and see what I can do with that type.
This is entirely unlike dynamic typing where you have to pray the docs are very good and up to date, which frequentlyis not the case.
The 24 line thing was a limitation of the glass tty, not BASIC.
For example, a garbage collector is the compiler/runtime taking on more work to avoid the restrictions of proper memory management.
This is in some sense analogous to the C++ vs Rust debate. Borrow checking allows the compiler to statically verify whether the program is memory safe. Like statically verified types, it turns a class of undesirable runtime behavior into a compilation failure. Ofc, this is a restriction that may be undesirable, which is why you can turn it off with 'unsafe'.
Python and Java though... do you really have that much more to worry about in Java?
In my eyes, you've got your problem. Regardless of your language choice, you can write a pseudocode algorithm for it. Terms in your algorithm have types already.
When we translate this algorithm into python/Java, the types of the program need to match the types of our pseudocode. Regardless of whether we get feedback on errors at runtime or compile time, we just need to get the types right. I'd prefer my error at compile time. The benefit is tiny for short scripts, but becomes massive when working on a large code base.
What about Java (being a "systems" language) makes you worry about so much more than Python (the "scripting" language)? There are definitely differences, but I don't think they stem from this systems/scripting dichotomy that's intrinsically linked to whether you check your types at compile or run time.
Of course Python has some restrictions due to the garbage collector. Good catch on that as it is if course true and limits you in other areas I don't care about. I was mainly referring to scripting needs where some languages really do shine above others. There's a reason Python is ubiquitous in that area and Java is not. Just like there's a reason Java and C# are ubiquitous in large enterprise software and Python is not.
To summarize though, I think I understand your point, but we're getting lost in semantics haha.
There are many statically typed languages that can compile in under a second. But when you change any function signature, you still have to ensure every caller is fixed. You then have to recompile, restart the application and rebuild all the data structures. Some static languages allow for hot reloading in interpretive mode, but the types of changes allowed are limited.
In dynamic languages like Common Lisp and Smalltalk you can update the program while it is running. The speed of iteration is 100x faster, because you don't have to restart the application and rebuild all the data structures to get to the point where you can test your change.
I learnt a lot about bytecode interpreters from working on it, and it helped me understand the cpython source code a lot more easily from having played with a python translation of it first.
My point was that printing, teletype-style terminals don't have an intrinsic vertical space limit like video display terminals (and eventually the printouts can get quite long).
But now that I think about it, perhaps the designers of BASIC might have been thinking of making it easy for a beginner to spend an afternoon or two learning how to write something comparable to a short FORTRAN program that might fit on a small deck of 80 column (72 with sequence number) punched cards.