←back to thread

Multi-Token Attention

(arxiv.org)
152 points fzliu | 1 comments | 02 Apr 25 22:20 UTC | HN request time: 0.918s | source
Show context
bob1029 ◴[02 Apr 25 23:14 UTC] No.43562889[source]▶
So, we're proposing a multiplicative increase of something that already scales quadratically with the context size?

I think we've already got a bit of a bottleneck in terms of memory bandwidth utilization.

replies(4): >>43563169 #>>43563334 #>>43563390 #>>43563970 #
1. jgalt212 ◴[02 Apr 25 23:48 UTC] No.43563169[source]▶
Maybe Sam was right about needing one trillion dollars!