←back to thread

Francois Chollet is leaving Google

(developers.googleblog.com)
377 points xnx | 5 comments | 13 Nov 24 22:28 UTC | HN request time: 0.001s | source
Show context
fchollet ◴[14 Nov 24 07:27 UTC] No.42133844[source]▶
Hi HN, Francois here. Happy to answer any questions!

Here's a start --

"Did you get poached by Anthropic/etc": No, I am starting a new company with a friend. We will announce more about it in due time!

"Who uses Keras in production": Off the top of my head the current list includes Midjourney, YouTube, Waymo, Google across many products (even Ads started moving to Keras recently!), Netflix, Spotify, Snap, GrubHub, Square/Block, X/Twitter, and many non-tech companies like United, JPM, Orange, Walmart, etc. In total Keras has ~2M developers and powers ML at many companies big and small. This isn't all TF -- many of our users have started running Keras on JAX or PyTorch.

"Why did you decide to merge Keras into TensorFlow in 2019": I didn't! The decision was made in 2018 by the TF leads -- I was a L5 IC at the time and that was an L8 decision. The TF team was huge at the time, 50+ people, while Keras was just me and the open-source community. In retrospect I think Keras would have been better off as an independent multi-backend framework -- but that would have required me quitting Google back then. Making Keras multi-backend again in 2023 has been one of my favorite projects to work on, both from the engineering & architecture side of things but also because the product is truly great (also, I love JAX)!

replies(20): >>42133884 #>>42133989 #>>42134014 #>>42134046 #>>42134074 #>>42134092 #>>42134212 #>>42134240 #>>42134249 #>>42134580 #>>42134819 #>>42134892 #>>42134936 #>>42134946 #>>42135297 #>>42135510 #>>42135776 #>>42135839 #>>42136118 #>>42136329 #
1. harisec ◴[14 Nov 24 13:51 UTC] No.42136118[source]▶
Congrats, good luck with your new company!

I have one question regarding your ARC Prize competition: The current leader from the leaderboard (MindsAI) seems not to be following the original intention of the competition (fine tune a model with millions of tasks similar with the ARC tasks). IMO this is against the goal/intention of the competition, the goal being to find a novel way to get neural networks to generalize from a few samples. You can solve almost anything by brute-forcing it (fine tunning on millions of samples). If you agree with me, why is the MindsAI solution accepted?

replies(2): >>42142665 #>>42145249 #
2. versteegen ◴[15 Nov 24 00:04 UTC] No.42142665[source]▶
> the goal being to find a novel way to get neural networks to generalize from a few samples

Remove "neural networks". Most ARC competitors aren't using NNs or even machine learning. I'm fairly sure NNs aren't needed here.

> why is the MindsAI solution accepted?

I hope you're not serious. They obviously haven't broken any rule.

ARC is a benchmark. The point of a benchmark is to compare differing approaches. It's not rigged.

replies(1): >>42144913 #
3. Borchy ◴[15 Nov 24 08:24 UTC] No.42144913[source]▶
I also don't understand why MindsAI is included. ARC is supposed to grade LLMs on their ability to generalize i.e. the higher score the more useful they are. If MindsAI scores x2 than the current SOTA then why are we wasting our $20 on inferior LLMs like ChatGPT adn Claude when we could be using the one-true-god MindsAI? If the answer is "because it's not a general-purpose LLM" then why is ARC marketed as the ultimate benchmark, the litmus test for AGI (I know I know, passing ARC doesn't mean AGI, but the opposite is true, I know)?
replies(1): >>42145253 #
4. fchollet ◴[15 Nov 24 09:28 UTC] No.42145249[source]▶
I believe the MindsAI solution does feature novel ideas that do indeed lead to better generalization (test-time fine-tuning). So it's definitely the kind of research that ARC was supposed to incentivize -- things are working as intended. It's not a "hack" of the benchmark.

And yes, they do use a lot of synthetic pretraining data, which is much less interesting research-wise (no progress on generalization that way...) but ultimately it's on us to make a robust benchmark. MindsAI is playing by the rules.

5. fchollet ◴[15 Nov 24 09:29 UTC] No.42145253{3}[source]▶
ARC was never supposed to grade LLMs! I designed the ARC format back when LLMs weren't a thing at all. It's a test of AI systems' ability to generalize to novel tasks.