Claude Fable 5.2, Inference Performance Index: changing the way we measure inference capacity, a paper on Tools Need Not Speak in Text: Latent Context Compression for Agents, and many more!
Firehose
Filtered to tagged “inference” · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives
Browse by tag
artificial intelligence 63continual learning 22AI 19agentic coding 17open-weight models 14reinforcement learning 12AI agents 11AI safety 8datasette 7machine learning 7natural language processing 7benchmarks 6language models 6Nvidia 6Reinforcement learning 6robotics 6deep learning 5Diffusion models 5large language models 5open-source 5Agentic AI 4AI ethics 4AI Gateway 4Databricks 4decision models 4Gemini 4general 4Jensen Huang 4Recursive self-improvement 4software development 4
The KV cache is a shared memory-mapped pool that stores key-value pairs for the Llama model, with a capacity of 128 MiB by default. It is used to cache query and key values during the inference process, allowing for efficient access and reuse of these values across multiple requests. AI summary