RustGPT: A pure-Rust transformer LLM built from scratch

> ndarray = "0.16.1" rand = "0.9.0" rand_distr = "0.5.0"

Looking good!

Absolutely love how readable the entire project is

I was slightly curious: cargo tree llm v0.1.0 (RustGPT) ├── ndarray v0.16.1 │ ├── matrixmultiply v0.3.9 │ │ └── rawpointer v0.2.1 │ │ [build-dependencies] │ │ └── autocfg v1.4.0 │ ├── num-complex v0.4.6 │ │ └── num-traits v0.2.19 │ │ └── libm v0.2.15 │ │ [build-dependencies] │ │ └── autocfg v1.4.0 │ ├── num-integer v0.1.46 │ │ └── num-traits v0.2.19 () │ ├── num-traits v0.2.19 () │ └── rawpointer v0.2.1 ├── rand v0.9.0 │ ├── rand_chacha v0.9.0 │ │ ├── ppv-lite86 v0.2.20 │ │ │ └── zerocopy v0.7.35 │ │ │ ├── byteorder v1.5.0 │ │ │ └── zerocopy-derive v0.7.35 (proc-macro) │ │ │ ├── proc-macro2 v1.0.94 │ │ │ │ └── unicode-ident v1.0.18 │ │ │ ├── quote v1.0.39 │ │ │ │ └── proc-macro2 v1.0.94 () │ │ │ └── syn v2.0.99 │ │ │ ├── proc-macro2 v1.0.94 () │ │ │ ├── quote v1.0.39 () │ │ │ └── unicode-ident v1.0.18 │ │ └── rand_core v0.9.3 │ │ └── getrandom v0.3.1 │ │ ├── cfg-if v1.0.0 │ │ └── libc v0.2.170 │ ├── rand_core v0.9.3 () │ └── zerocopy v0.8.23 └── rand_distr v0.5.1 ├── num-traits v0.2.19 () └── rand v0.9.0 ()

yep, still looks relatively good.

Never knew Rust could be that readable. Makes me think other Rust engineers are stuck in a masochistic ego driven contest, which would explain everything else I've encountered about the Rust community and recruiting on that side.

I’m curious where you got your training data? I will look myself, but saw this and thought I’d ask. I have a CPU-first, no-backprop architecture that works very well on classification datasets. It can do single‑example incremental updates which might be useful for continuous learning. I made a toy demo to train on tiny.txt and it can predict next characters, but I’ve never tried to make an LLM before. I think my architecture might work well as an on-device assistant or for on-premises needs, but I want to work with it more before I embarrass myself. Any open-source LLM training datasets you would recommend?

This looks rather similar to when I asked an AI to implement a basic xor problem solver I guess fundementally there's really only a very limited amount of ways to implement this.

huggingface has plenty of openai and antrophic user to assistant chains, beware there are dragons (hallucinations), but good enough for instruction training. I actually recommend distilling kimi k2 instead for instruction following capabilities.

is this satire or does I must know context behind this comment???

Not sure what you’re alluding to but that’s just ordinary Rust without performance or async IO concerns.

These are a few well-chosen dependencies for a serious project.

Rust projects can really go bananas on dependencies, partly because it's so easy to include them

Nice. Mind to put a license on that?

It is very procedural/object oriented. This is not considered good Rust practice. Iterators make it more functional, which is better, more succinct that is, and enums more algebraic. But it's totally fine for a thought experiment.

The project only has 3 dependencies which i interpret as a sign of quality

As someone who has spent days wrestling with Python dependency hell just to get a model running, a simple cargo run feels like a dream. But I'm wondering, what was the most painful part of NOT having a framework? I'm betting my coffee money it was debugging the backpropagation logic.

I did this [0] (gpt in rust) with picogpt, following the great blog by jaykmody [1].

[0]: https://github.com/enricozb/picogpt-rust [1]: https://jaykmody.com/blog/gpt-from-scratch/

nice

lowkey ppl who praise cargo seem to have no idea of the tradeoffs involved in dependency management

the difficulty of including a dependency should be proportional to the risk you're taking on, meaning it shouldn't be as difficult as it in, say, C where every other library is continually reinventing the same 5 utilities, but also not as easy as it is with npm or cargo, because you get insane dependency clutter, and all the related issues like security, build times, etc

how good a build system isn't equivalent of how easy it is include a dependency, while modern languages should have a consistent build system, but having a centralised package repository that anyone freely pull to/from, and having those dependencies freely take on any number of other dependencies is a bad way to handle dependencies

This is great ! Congratulations. I really like your project, especially I like how easily it is to peak at.

Do you plan on moving forward with this project ? I seem to understand that all the training is done on the CPU, and that you have next steps regarding optimizing that. Do you consider GPU accelerations ?

Also, do you have any benchmarks on known hardware ? Eg, how long would it take to train on a macbook latest gen or your own computer ?

It's AI generated

linking both rand-core 0.9.0 and rand-core 0.9.3 which the project could maybe avoid by just specifying 0.9 for its own dep on it

I guess, resource utilization like GPU, etc

How do you know? The over-commenting?

Cool stuff! I can see some GPT comments that can be removed

// Increased for better learning

this doesn't tell me anything

// Use the constants from lib.rs

const MAX_SEQ_LEN: usize = 80;

const EMBEDDING_DIM: usize = 128;

const HIDDEN_DIM: usize = 256;

these are already defined in lib.rs, why not use them (as the comment suggests)

I know because this is how an AI generated project looks. Clearly AI generated README, "clean" code, the way files are named, etc.

What tool or ecosystem does this well, in your opinion?

To me it looks like LLM generated README, but not necessarily the source (or at least not all of it).

Or there's been a cleaning pass done over it.

Security is another problem, and should be tackled systematically. Artificially making dependency inclusion hard is not it and is detrimental to the more casual use cases.

Have you tried uv [1]? It has removed 90% of the pain of running python projects for me.

[1] https://github.com/astral-sh/uv

Some commentary from the author here: https://www.reddit.com/r/rust/comments/1nguv1a/i_built_an_ll...

The repeated Impls are strange.

Congrats - there is a very small problem with the LLM - its reusing transformer blocks and you want to use different instances of them.

Its a very cool excercise, I did the same with Zig and MLX a while back, so I can get a nice foundation, but since then as I got hooked and kept adding stuff to it, switched to Pytorch/Transformers.

I think pretty clearly the source is also at least partially generated. None the less, just a README like that already sends a strong signal to stop looking and not trust anything written there.

Not sure myself. Commit messages look pretty human. But the emojis in readme and comments like "// Re-export key structs for easier access", "# Add any test-specific dependencies here if needed" are sus indeed.

Most Rust code looks like this - only generic library code goes crazy with all the generics and lifetimes, due to the need to avoid unnecessary mallocs and also provide a flexible API to users.

But most people aren't writing libraries.

To my untrained eye, this looks more like an instruct dataset.

For just plain text, I really like this one - https://huggingface.co/datasets/roneneldan/TinyStories

correction: It's a cool exercise if you write it yourself and not use GPT

Where? Don't see any on latest main (685467e).

well, hopefully the author did learn something or at least enjoyed the process :)

(the code looks like a very junior or a non-dev wrote it tbh).

License added! Good catch

Is your argument that python's package management & ecosystem is bad by design - to increase security?

In my experience it's just bugs and poor decision making on the maintainers (eg. pytorch dropping support for intel mac, leftpad in node) or on the language and package manager developers side (py2->3, commonjs, esm, go not having a package manager, etc).

Cargo has less friction than pypi and npm. npm has less friction than pypi.

And yet, you just need to compromise one lone, unpaid maintainer to wreck the security of the ecosystem.

> lowkey ppl who praise cargo seem to have no idea

Way to go on insulting people on HN. Cargo is literally the reason why people coming to Rust from languages like C++ where the lack of standardized tooling is giant glaring bomb crater that poses burden on people every single time they need to do some basic things (like for example version upgrades).

Example:

https://github.com/facebook/folly/blob/main/build.sh

This is the weirdest excuse for Python's terrible tooling that I've ever heard.

"It's deliberately shit so that people won't use it unless they really have to."

uv is great, but I think the real fix is just abandoning Python.

The culture that language maintains is rather hostile to maintainable development, easier to just switch to Rust and just write better code by default.

There's not really another game in town if you want to do fast ML development :/

Every tool for the right job. If you are doing tons of scripting (for e.g. tests on platforms different than Rust), Python can be a solid valid alternative.

Also, tons of CAE platforms have Python bindings, so you are "forced" to work on Python. Sometimes the solution is not just "abandoning a language".

If it fits your purpose, knock yourself out, for others that may be reading: uv is great for Python dependency management on development, I still have to test it for deployment :)

> spent days wrestling with Python dependency hell

I mean I would understand that comment in 2010, but in 2025 it's grossly ridiculous.

`llm.rs` has many `impl LLM` blocks