NVIDIA Nemotron Parse v1.2 is designed to understand document semantics and extract text and tables elements with spatial grounding. Given an image, NVIDIA Nemotron Parse v1.2 produces structured annotations, including formatted text, bounding-boxes and the corresponding semantic classes, ordered according to the document's reading flow. It overcomes the shortcomings of traditional OCR technologies that struggle with complex document layouts with structural variability, and helps transform unstructured documents into actionable and machine-usable representations. This has several downstream benefits such as increasing the availability of training-data for Large Language Models (LLMs), improving the accuracy of extractor, curator, retriever and AI agentic applications, and enhancing document understanding pipelines. NVIDIA-Nemotron-Parse v1.2 extends the prompt interface with a fourth prompt token category — **Text-in-picture prompts** — controlling whether the model extracts text from embedded images (e.g. photographs, figures): - ` `: extract text found inside pictures. - ` `: skip text extraction from pictures, reducing generation time by avoiding unnecessary tokens.
NVIDIA NemotronLabs RE-USE is a 9.6-million-parameter **multilingual universal speech enhancement (USE)** model designed to restore speech quality across a wide range of real-world degradations while preserving speech fidelity. It has the following desirable properties: - Robustness to **diverse degradations**, including additive noise, reverberation, clipping, bandwidth limitation, codec artifacts, packet loss and low-quality mics. - Support for **multiple input sampling rates**, including 8, 16, 22.05, 24, 32, 44.1, and 48 kHz. - Strong **language-agnostic** capability, enabling effective performance across different languages. - If better laughter preservation is desired, please use: NVIDIA-NemotronLabs-RE-USE-laughter. (Note: For overlapping speech, this model is trained to preserve the main speaker while suppressing other speakers.) Comparison with other open-source USE models on the (subsets) of the URGENT 2025 non-blind test set, referenced against original anechoic clean speech.
NVIDIA NemotronLabs RE-USE-laughter is a 9.6-million-parameter **multilingual universal speech enhancement (USE)** model designed to restore speech quality across a wide range of real-world degradations while preserving speech fidelity. It has the following desirable properties: - **This model better preserves laughter while restoring the quality of the input speech.** - Robustness to **diverse degradations**, including additive noise, reverberation, clipping, bandwidth limitation, codec artifacts, packet loss and low-quality mics. - Support for **multiple input sampling rates**, including 8, 16, 22.05, 24, 32, 44.1, and 48 kHz. - Strong **language-agnostic** capability, enabling effective performance across different languages. (Note: For overlapping speech, this model is trained to preserve the main speaker while suppressing other speakers.) Comparison with other open-source USE models on the (subsets) of the URGENT 2025 non-blind test set, referenced against original anechoic clean speech.
NVIDIA NemotronLabs VoiceChat is a **11B end-to-end, real-time speech full duplex (FD)** model for conversational AI that jointly performs streaming speech understanding and speech generation [1, 2, 3]. Unlike traditional cascaded stacks (ASR → LLM → TTS), this model achieves full duplex, real-time, seamless voice interaction in one unified architecture, eliminating the need for multiple models or API handoffs, thus reducing end-to-end latency. It sets new benchmarks by bringing open, robust, and highly natural conversation capabilities. Moreover, NVIDIA NemotronLabs VoiceChat is **the first open full-duplex model to support tool calling** while maintaining a natural conversation flow during tool execution. For each tool, a specific “on-hold” message can be defined that will be spoken by the agent as soon as the LLM generates the text that will trigger the tool call and response. The model operates on audio signals, which are encoded using a fast conformer module. The resulting audio tokens are inputted into a Nemotron Nano V2 9B LLM backbone to predict text tokens, which are fed to a TTS decoder to predict audio codes for generating the agent's speech. A separate output channel is used to predict tool calling scripts. NemotronLabs VoiceChat offers an unprecedented trade-off between intelligence and latency in the space of open-source voice agents, as highlighted by our benchmarking results below. NemotronLabs VoiceChat is targeted for researchers, developers, and professionals in the field of natural language processing (NLP) and speech technology for purposes such as automatic speech recognition (ASR), text-to-speech synthesis (TTS), and voice assistant development.
NVILA-HD-Video is a Multi-modal Large Language Model with 8B parameters that understands and answers questions about videos with up to 4K resolution and 1K frames. Specifically, NVILA-HD-Video uses AutoGaze to reduce redundant patches in a video before running the ViT or LLM. Empirically, AutoGaze can reduce #tokens in in a video by up to 100x, reducing the latency of ViT/LLM by up to 19x/10x. This enables NVILA-HD-Video to efficiently scale to 4K-resolution, 1K-frame videos and achieve improved performance on benchmarks such as VideoMME and state-of-the-art performance on HLVid, a high-resolution long-form video benchmark proposed in this work as well. For more details, see the VILA github repo.
This family of models performs vision-language and text-only tasks including optical character recognition, multimodal reasoning, localization, common sense reasoning, world knowledge utilization, and coding. Additional Information: LICENSE · Qwen/Qwen2-72B-Instruct at main for Qwen2-72B-Instruct and The MIT License – Open Source Initiative for InternViT-6B-448px-V1-2. Today (September 17th, 2024), we introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training.
