Modern Large Language Models (LLMs) represent token positions using either absolute positional embeddings, which add a unique vector to each token embedding, or relative positional embeddings, which encode relative positions directly into the attention calculation. The article discusses the classic and contemporary methods of encoding positional information, including learned absolute position embeddings, fixed sinusoidal position encodings, and Rotary Position Embedding (RoPE), which introduces relative position information into the attention calculation. AI summary
Firehose
Filtered to People, tagged “token representations” · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives