Embeddings are underrated (2024)

> I don’t know. After the model has been created (trained), I’m pretty sure that generating embeddings is much less computationally intensive than generating text.

An embedding is generated after a single pass through the model, so functionally it's the equivalent of generating a single token from an text generation model.

I might be wrong but aren't embedding models usually bidirectional and not causal, so the attention mechanism itself is more expensive.