←back to thread

The deep learning boom caught almost everyone by surprise

(www.understandingai.org)
251 points slyall | 8 comments | 06 Nov 24 04:05 UTC | HN request time: 0.001s | source | bottom
Show context
kleiba ◴[06 Nov 24 12:23 UTC] No.42061089[source]▶
> “Pre-ImageNet, people did not believe in data,” Li said in a September interview at the Computer History Museum. “Everyone was working on completely different paradigms in AI with a tiny bit of data.”

That's baloney. The old ML adage "there's no data like more data" is as old as mankind itself.

replies(6): >>42061617 #>>42061818 #>>42061987 #>>42063019 #>>42063076 #>>42064875 #
FrustratedMonky ◴[06 Nov 24 13:01 UTC] No.42061617[source]▶
Not really. This is referring back to the 80's. People weren't even doing 'ML'. And back then people were more focused on teasing out 'laws' in as few data points as possible. The focus was more on formulas and symbols, and finding relationships between individual data points. Not the broad patterns we take for granted today.
replies(2): >>42062250 #>>42063993 #
1. criddell ◴[06 Nov 24 13:50 UTC] No.42062250[source]▶
I would say using backpropagation to train multi-layer neural networks would qualify as ML and we were definitely doing that in 80's.
replies(1): >>42062594 #
2. UltraSane ◴[06 Nov 24 14:14 UTC] No.42062594[source]▶
Just with tiny amounts of data.
replies(1): >>42062627 #
3. jensgk ◴[06 Nov 24 14:17 UTC] No.42062627[source]▶
Compared to today. We thought we used large amounts of data at the time.
replies(1): >>42062803 #
4. UltraSane ◴[06 Nov 24 14:29 UTC] No.42062803{3}[source]▶
"We thought we used large amounts of data at the time."

Really? Did it take at least an entire rack to store?

replies(1): >>42063257 #
5. jensgk ◴[06 Nov 24 14:56 UTC] No.42063257{4}[source]▶
We didn't measure data size that way. At some point in the future someone would find this dialog, and think that we dont't have large amounts of data now, because we are not using entire solar systems for storage.
replies(1): >>42065235 #
6. UltraSane ◴[06 Nov 24 16:57 UTC] No.42065235{5}[source]▶
Why can't you use a rack as a unit of storage at the time? Were 19" server racks not in common use yet? The storage capacity of a rack will grow over time.

my storage hierarchy goes 1) 1 storage drive 2) 1 server maxed out with the biggest storage drives available 3) 1 rack filled with servers from 2 4) 1 data center filled with racks from 3

replies(1): >>42066284 #
7. fragmede ◴[06 Nov 24 17:54 UTC] No.42066284{6}[source]▶
How big is a rack in VW beetles though?

It's a terrible measurement because it's an irrelevant detail about how their data is stored that no one actually knows if your data is being stored in a proprietary cloud except for people that work there on that team.

So while someone could say they used a 10 TiB data set, or 10T parameters, how many "racks" of AWS S3 that is, is not known outside of Amazon.

replies(1): >>42072934 #
8. UltraSane ◴[07 Nov 24 03:06 UTC] No.42072934{7}[source]▶
a 42U 19" inch rack is an industry standard. If you actually work on the physical infrastructure of data centers it is most CERTAINLY NOT an irrelevant detail.

And whether your data can fit on a single server, single rack, or many racks will drastically affect how you design the infrastructure.