`nvidia/audio-flamingo-next-captioner-hf` is the long-form captioning checkpoint in the Audio Flamingo Next family. It is designed for rich, descriptive outputs over long and complex audio, including speech-heavy recordings, environmental sound scenes, and music. Audio Flamingo Next (AF-Next) is the next-generation open audio-language model in the Audio Flamingo series, built for speech, environmental sound, and music understanding with audio inputs up to `30` minutes. This checkpoint corresponds to **AF-Next-Captioner**, the model obtained at the end of AF-Next mid-training after the training mixture is expanded with newly collected long-audio captioning and QA datasets. It is the best AF-Next variant when you want:
Audio2Face-3D-v3.0 generates 3D facial animations from audio inputs, for use in applications such as video conferencing, virtual reality, and digital content creation. This model is ready for commercial/non-commercial use. For source code, documentation, helper scripts, packaged builds, and links to all components in the Audio2Face-3D technology stack, visit the Audio2Face-3D GitHub repository Use of this model is governed by the NVIDIA Open Model License Audio2Face-3D-v3.0 is designed for developers and researchers working on audio-driven animation and emotion detection applications, such as virtual assistants, chatbots, and affective computing systems.
AutoGaze is a ultra light-weight model that automatically removes redundant patches in a video before passing it to any Vision Transformer (ViT) or Multi-modal Large Language Model (MLLM). Speficially, AutoGaze perceives each frame and autoregressively selects ("gazing") a minimal set of patches that can reconstruct the original video (i.e., non-redundant patches) up to a reconstruction loss threhold provided by the user. AutoGaze can self-decide when to stop gazing for each frame based on user's request on the acceptable maximum reconstruction loss. Empircally, AutoGaze can reduce #tokens in ViTs/MLLMs by up to 100x, reducing their latency by up to 19x/10x. This enables efficiently scaling MLLMs to 4K-resolution, 1K-frame videos, improving performance on benchmarks such as VideoMME. Especially, it improves performance by 14% on HLVid, a high-resolution long-form video benchmark proposed in this work as well.
- **Jul 2024 (v2.3):** - General refactor and code improvements for improved readability. - Fully fused CUDA kernel of anti-alised activation (upsampling + activation + downsampling) with inference speed benchmark. - **Jul 2024 (v2.2):** The repository now includes an interactive local demo using gradio. - **Jul 2024 (v2.1):** BigVGAN is now integrated with 🤗 Hugging Face Hub with easy access to inference using pretrained checkpoints. We also provide an interactive demo on Hugging Face Spaces.
- **Jul 2024 (v2.3):** - General refactor and code improvements for improved readability. - Fully fused CUDA kernel of anti-alised activation (upsampling + activation + downsampling) with inference speed benchmark. - **Jul 2024 (v2.2):** The repository now includes an interactive local demo using gradio. - **Jul 2024 (v2.1):** BigVGAN is now integrated with 🤗 Hugging Face Hub with easy access to inference using pretrained checkpoints. We also provide an interactive demo on Hugging Face Spaces.
- **Jul 2024 (v2.3):** - General refactor and code improvements for improved readability. - Fully fused CUDA kernel of anti-alised activation (upsampling + activation + downsampling) with inference speed benchmark. - **Jul 2024 (v2.2):** The repository now includes an interactive local demo using gradio. - **Jul 2024 (v2.1):** BigVGAN is now integrated with 🤗 Hugging Face Hub with easy access to inference using pretrained checkpoints. We also provide an interactive demo on Hugging Face Spaces.
This model performs visual feature extraction. For instance, RADIO generates image embeddings that can be used by a downstream model to classify images. **Input Type(s):** Image **Input Format(s):** Red, Green, Blue (RGB) pixel values in [0, 1] range. **Input Parameters:** Two Dimensional (2D) **Other Properties Related to Input:** Image resolutions up to 2048x2028 in increments of 16 pixels **Output Type(s):** Embeddings **Output Format:** Tensor **Output Parameters:** 2D **Other Properties Related to Output:** Downstream model required to leverage image features
