AI NEWS.ccSelf-Forcing
Self Forcing trains autoregressive video diffusion models by **simulating the inference process during training**, performing autoregressive rollout with KV caching. It resolves the train-test distribution mismatch and enables **real-time, streaming video generation on a single RTX 4090** while matching the quality of state-of-the-art diffusion models. We tested this repo on the following setup: * Nvidia GPU with at least 24 GB memory (RTX 4090, A100, and H100 are tested). * Linux operating system. * 64 GB RAM. Note: * **Our model works better with long, detailed prompts** since it's trained with such prompts. We will integrate prompt extension into the codebase (similar to Wan2.1) in the future. For now, it is recommended to use third-party LLMs (such as GPT-4o) to extend your prompt before providing to the model. * You may want to adjust FPS so it plays smoothly on your device. * The speed can be improved by enabling `torch.compile`, TAEHV-VAE, or using FP8 Linear layers, although the latter two options may sacrifice quality. It is recommended to use `torch.compile` if possible and enable TAEHV-VAE if further speedup is needed.
Self Forcing trains autoregressive video diffusion models by **simulating the inference process during training**, performing autoregressive rollout with KV caching. It resolves the train-test distribution mismatch and enables **real-time, streaming video generation on a single RTX 4090** while matching the quality of state-of-the-art diffusion models. We tested this repo on the following setup: * Nvidia GPU with at least 24 GB memory (RTX 4090, A100, and H100 are tested). * Linux operating system. * 64 GB RAM. Note: * **Our model works better with long, detailed prompts** since it's trained with such prompts. We will integrate prompt extension into the codebase (similar to Wan2.1) in the future. For now, it is recommended to use third-party LLMs (such as GPT-4o) to extend your prompt before providing to the model. * You may want to adjust FPS so it plays smoothly on your device. * The speed can be improved by enabling `torch.compile`, TAEHV-VAE, or using FP8 Linear layers, although the latter two options may sacrifice quality. It is recommended to use `torch.compile` if possible and enable TAEHV-VAE if further speedup is needed.