This paper improves the speed of generating text and images in vision-language models by adapting the way the model proposes tokens, and it could be useful to developers who want to build faster and more accurate models for applications like image captioning or text-to-image synthesis.
Firehose
Filtered to tagged “speculative decoding” · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives
Browse by tag
artificial intelligence 63continual learning 22AI 19agentic coding 17open-weight models 14reinforcement learning 12AI agents 11AI safety 8datasette 7machine learning 7natural language processing 7benchmarks 6language models 6Nvidia 6Reinforcement learning 6robotics 6deep learning 5Diffusion models 5large language models 5open-source 5Agentic AI 4AI ethics 4AI Gateway 4Databricks 4decision models 4Gemini 4general 4Jensen Huang 4Recursive self-improvement 4software development 4
In this episode, Philip Kiely and Ali Taha from Baseten discuss the complexities and innovations in inference engineering for large AI models. They cover topics including model deployment, speculative decoding, quantization, hardware optimi…
Inference engineeringSpeculative decodingQuantizationModel deploymentTool callingKV cacheTensor parallelismExpert parallelismGPU hardwareRubin GPUVideo diffusionAutoregressive modelsDiffusion modelsTraining-inference convergenceContinual learningOpen source modelsInference infrastructureModel optimizationLatency vs throughputMulti-modal models
In this episode, Akshat Bubna, CTO of Modal, discusses the evolution of AI infrastructure tailored for agent experience, highlighting Modal's journey from a runtime platform to a specialized cloud for AI workloads. They explore challenges i…