So why do we need Rust at all? What's the use case for it?
Anything that I'm missing?
C# is garbage collected. This is a no-go in many/most embedded software applications.
C# also grants you poor explicit control over heap/stack allocation: this is essential for embedded development.
I think that most modern embedded systems are nowadays more powerful than my first desktop computer — is it really still worth for the majority of embedded projects to count every byte at the expense of developers' productivity (and overall project success, as a result)?
Yes, you can run a VM on one of those processors in which you run a language like C# or Java. Look at MicroPython as a successful example. But the code which runs that VM has to be written in something. Typically that has been C and C++ - but Rust can also do it.
You're right though - a lot of what we call "embedded computing" could be done using a modern VM interpreted language. There are some languages out there - MicroPython and Squirrel come to mind - which can run on the memory and storage constrained environment of a microcontroller. The mainstream implementations of Java and C# use way to much memory - they've been optimised for desktop environments. Microcontrollers often have 64 - 1024 kB of RAM.
You could ship products with application processors like the Raspberry Pi, running an OS, and write your application in whatever language you like. But that costs 1-2 orders of magnitude more money. A cheap microcontroller is $1. By the time you've added the RAM, flash and supporting power rails, a cheap application processor is $10+. Then you have to maintain an OS - that's a lot of engineering time. Sometimes it's worth it, sometimes it's not - the tradeoff depends on the product.