GGML – AI at the Edge

This guy is damned good. I sponsored him on Github because his software is dope. I also like how when some controversy erupted on the project he just ejected the controversial people and moved on. Good stewardship. Great code.

I recall something like when he first ported it and it worked on my M1 Max he hadn't even yet tested it on Apple Silicon since he didn't have the hardware.

Honestly, with this and whisper, I am a huge fan. Good luck to him and the new company.

Another important detail about the ejections that I think is particularly classy is that the people he ejected are broadly considered to have world-class technical skills. In other words, he was very explicitly prioritizing collaborative potential > technical skill. Maybe a future BDFL[1]!

[1] https://en.wikipedia.org/wiki/Benevolent_dictator_for_life

What was the controversy?

he has been amazing to watch and has even helped me out with my app that uses his whisper.cpp project (https://superwhisper.com)

Excited to see how his venture goes!

> I also like how when some controversy erupted on the project he just ejected the controversial people and moved on. Good stewardship

Do you have more info on the controversy? I'm not sure ejecting developers just because of controversy is honestly good stewardship.

Right. More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). TikTok was using this method for a while by ejecting both bullies and victims, and it "solved" the problem but most people see the injustice there.

I'm not saying it was bad stewardship, I honestly don't know. I just agree that we shouldn't make a judgment without more information.

> More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily).

Man, nobody has time for this shit. Leave the games and the drama for the social justice warriors and the furries. People building shit ain't got time for this - ejecting trouble makers is the right way to go regardless of which "side" they're on.

I would agree that there needs to be a balance because wasting time babysitting adults is dumb, but what if one person is a good and loved contributor, and the other is a social justice warrior new to the project that is picking fights with the contributor? Your philosophy makes not only bad stewardship but an injustice. I'm not suggesting this is the only scenario, just merely a hypothetical that I think illustrates my position.

And what do you do when every contributor to the project, including the founder, has been labeled a troublemaker?

> and the furries

Um, what?

>justice

For an individual running a small open source project, there's time enough for coding or detailed justice, but not both. When two parties start pointing fingers and raising hell and its not immediately clear who is in the right, ban both and let them fork it.

Pick the fork that has devs who are focused on contributing code and not pursuing drama.

> More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). TikTok was using this method for a while by ejecting both bullies and victims,

This is SOP for American schools. It's laziness there, since education is supposed to be compulsory. They can't be bothered to investigate (and with today's hostile climate, I don't blame them) so they consign both parties to independent-study programs.

For volunteer projects, throwing both overboard is unfortunate but necessary stewardship. The drama either attracts destabilizes the entire project, which only exists as long as it remains fun for the maintainer. It's tragic, but victims who can't recover gracefully are as toxic as their abusers.

The code is MIT licensed. If you don't agree with the direction the project is taking you can fork it and add whatever you want.

I don't understand why this is so difficult for software developers with GitHub accounts to understand.

You've missed the point here more than I've seen anyone miss the point in a long time.

If you know, you know

Gerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run a collaborative environment that's open when you pander to hate, because hate subverts communities; it's impossible to compromise with anonymous trolls who harass a public figure over physical traits about her body she can't change.

You don't have to take my word on it. Here are some archives of the 4chan threads where they coordinated the raid. It went on for like a month. https://archive.is/EX7Fq https://archive.is/enjpf https://archive.is/Kbjtt https://archive.is/HGwZm https://archive.is/pijMv https://archive.is/M7hLJ https://archive.is/4UxKP https://archive.is/IB9bv https://archive.is/p6Q2q https://archive.is/phCGN https://archive.is/M6AF1 https://archive.is/mXoBs https://archive.is/68Ayg https://archive.is/DamPp https://archive.is/DiQC2 https://archive.is/DeX8Z https://archive.is/gStQ1

If you read these threads and see how nasty these little monsters are, you can probably imagine how Gerganov must have felt. He was probably scared they'd harass him too, since 4chan acts like he's their boy. I wouldn't even be surprised if he's one of them. Plus it was weak leadership on his part to disappear for days, suddenly show up again to neutral knight the situation (https://justine.lol/neutral-knight.png) by telling his team members they're no longer welcome, and then going back and deleting his comment later. It goes to show that no matter how brilliant you are at hard technical skills, you can still be totally clueless about people.

I didn't want to not reply but I also didn't want to be swept into a potentially fraught internet argument. So, I tried to edit my comment as a middle ground, but it looks like I can't, I guess there must be a timeout. If I could edit it, I'd add the following:

"I should point out that I wasn't personally involved, haven't looked into it in detail, and that there are many different perspectives that should be considered."

Really curious why you tried to rename the file format magic string to have your initials? Going from GGML (see Title of this post) to GGJT with JT being Justine Tunney? Seems quite unnecessary and bound to have rubbed a lot of people the wrong way.

Here is the official commit undoing the change:

https://github.com/ggerganov/llama.cpp/pull/711/files#diff-7...

Because the previous changes to the file format were done by changing the last two initials of the magic. Someone commented on the pull request suggesting using a version field instead, which hadn't been documented, but Gerganov was so happy with the mmap() contribution that he asked me to keep the initials. So you should ask him why he wanted my initials to be there. I didn't see anyone else raise concerns until later on when the 4chan raid happened. I guess I failed to consider that folks who hate trans women would feel uncomfortable needing to mark their files with the initials of one. Here's the pull request: https://github.com/ggerganov/llama.cpp/pull/613

Strange comment. This doesn't sound like a legitimate criticism because of "Here is the official commit undoing the change:" being not a link to a commit to start with, and secondly being a declined pull request, and thirdly for the reality that `master` writes `ggjt` header.

Really looks like some axe-grinding here, if I'm being honest. Especially because it takes very little effort to find out what the present header is by someone who can write software.

I get that a hateful mob jumped all over this widely-publicized PR and that's really, really not ok, but it doesn't make you automatically in the right. Sometimes our egos get the better of us, mistakes are made. Subsequently the only choice you have is between being someone who escalates drama and someone who defuses it. I promise you that being the latter is the better choice, even if it doesn't come with the ego-boosting joy of being "right". The person who can rise above it all is the one who ends up winning respect in the long-run, but it requires acknowledging one's own fallibility in the short-term.

Are you talking about Slaren? I wrote a blog post promoting him a while back. You can read it here: https://justine.lol/mmap/ It also spotlights other unsung heroes who played important roles in helping me bring mmap() to the machine learning community. As for me being trans, the problem is that 4chan does care. The /g/ local model discussion board was consumed with discussing my gender status for more than a month. Stirring up hate is how they rally anons to venture out on raids which have damaged my professional relationships.

Right, I for one would also prefer it if people who face harassment and hate would just accept it gracefully and move on. I mean, get over yourself, amiright?

Just because someone can't wring your hand doesn't mean they don't deserve to influence a project. Releasing work under a license like MIT is an act of benevolence since it's giving up leverage and relying on de facto leadership. It's the hardest kind of leader to be, since de facto leaders are only followed on merit. It's not unusual at all for folks who started projects to end up caving to political pressures and getting replaced by the sorts of people you're talking about. But every now and again a Medici comes along.

I'm a dev and know how to use git. Honestly that's just the PR I could find whilst catching up on this story and was curious what this "magic string" was. The change is there and got reverted 100%, see the other commenter who was the one that made it. If there is a better link you're welcome to post it.

>I wrote a blog post promoting him a while back.

That's about 2 weeks after the drama around PR 613, which you factually touted as "your work" in several different places.

I'm confused about the scenario you're describing here.

Look, my message is simple and clear: keep the politics and drama out of it. If you partake in politics and drama, you'll be ejected from the project. I don't have the time or the energy to police or play games with people. We're here to build things, not to partake in social activism or sling crap at each other over codes of conduct, pronouns, hair color or magic strings. If you're hurt - fork the project (as long as the license allows for it) and have fun playing somewhere else.

Just wanted to share I think you're a superstar engineer.

Also from the links you shared it looked like some users on 4chan decided to go out and harass you. If they didn't know you are a trans woman, I'm sure they would've defaulted to calling you a n***** f***** instead. But they were going to harass you nonetheless.

It was very sad to see how things developed over a small issue. I'm sure this could've gotten resolved civilly since I believe you and everyone else involved in the project had good intentions and were doing everything out of love.

The scenario is a mob of trolls attacking a contributor in bad faith. If you kick out the contributor, what's to stop the mob from picking off someone else?

Not until I'm made whole. I donated a lot of resources to the llama.cpp project. I volunteered and successfully contributed one of its most impactful features. I was rewarded with harassment and being publicly humiliated by its leader, for no reason at all. They also reneged on promises they made me. I'm owed a lot more than apology, but I haven't even received that.

If a bunch of random strangers (external to the project) are "attacking" your project somewhere on the internet (for example, on Twitter) - just ignore them and move on with your day. They don't have any power over your project and their opinions don't matter. Go on with your life and continue building.

If a bunch of random strangers (external to the project) are messing with your tools and workflows (stirring things up in the issue tracker, creating drama and playing games with silly Pull Requests and comments) - lock down your tools such that they can only be used by trusted members of your team. Close down and remove all bullshit conversations without spending any further time or energy on any of it. Platforms like GitHub blur the lines between "a suite of productivity tools for software development" and "a social network" - so make sure to lock down and limit the "social networks" aspects whilst optimizing for the "software development productivity" aspect. Go on with your life and continue building.

If the "attacks" happens internally within the project (between two or more members of the team) - eject all parties involved because they're clearly not here to build stuff. Go on with your life and continue building.

Your goal should be to spend your energy on building and creating, and collaborating with like-minded people on building and creating. Not on policing, moderating, or playing games with people.

Liar. https://news.ycombinator.com/item?id=35455930#35458068

This user stole another user's code, closed his PR, and opened a new one where she started using words like "my work," "I'm the author," "author here," etc., and trying to cozy up to the project lead.

Gerganov figured out what was happening and actually banned her from all further contributions. The user whose code was stolen, Slaren, is still contributing.

You didn't write that feature. Slaren did. You closed his PR and made minor changes, then gradually shifted from "our feature" to "my feature."

----

That's not the original PR. jart was working on a malloc() approach that didn't work and slaren wrote all the code actually doing mmap, which jart then rebased in a random new PR, changed to support an unnecessary version change, magic numbers, a conversion tool, and WIN32 support when that was already working in the draft PR. https://archive.ph/Uva8c

This is the original PR: https://github.com/ggerganov/llama.cpp/pull/586.

Jart's archived comments:

"my changes"

"Here's how folks in the community have been reacting to my work."

"I just wrote a change that's going to let your LLaMA models load instantly..."

https://archive.ph/PyPFZ

"I'm the author"

https://archive.ph/qFrcY

"Author here..."

"Tragedy of the commons...We're talking to a group of people who live inside scientific papers and jupyer notebooks."

"My change helps inference go faster."

"The point of my change..."

"I stated my change offered a 2x improvement in memory usage."

https://archive.ph/k34V2

"I can only take credit for a 2x recrease in RAM usage."

https://archive.ph/MBPN0

"I just wrote a change that's going to let your LLaMA models load instantly, thanks to custom malloc() and the power of mmap()"

https://archive.ph/yrMwh

slaren replied to jart on HN asking her why she was doing and saying those things, and she didn't bother to reply to him, despite replying to others in that subthread within minutes. https://archive.ph/zCfiJ

----

You didn't make whole the people you damaged or the project you attempted to harm with plagiarism and pathological levels of manipulation and lying.

Good artists copy and great artists steal.

>Good artists copy and great artists steal.

This user claims Gerganov publicly humiliated her, but she does it to herself.