VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders
This paper develops a method to transform video foundation models' representations into compact, reconstruction-capable, and generation-friendly video latents, which can be used in various generative modeling tasks. Practitioners can use VideoRAE to improve the performance of their models by leveraging the semantic and spatio-temporal structure captured by the frozen video foundation encoder.