The KV cache is a shared memory-mapped pool that stores key-value pairs for the Llama model, with a capacity of 128 MiB by default. It is used to cache query and key values during the inference process, allowing for efficient access and reuse of these values across multiple requests. AI summary
Firehose
Filtered to Hacker News, tagged “large language models” · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives