Llama3-MS-CLIP is the first Vision-Language Model in the CLIP family that understands multispectral imagery. It is trained on one million image-text pairs from the SSL4EO-S12-v1.1 dataset with generated captions. The CLIP model consists of two encoders for text and images. We extended the RGB patch embeddings to multispectral input and initialized the weights of the additional input channels with zeros. During the continual pre-training, the images and texts of each batch are encoded and combined. The loss increases the similarity of matching pairs while decreasing other combinations. We evaluated Llama3-MS-CLIP with zero-shot classification and text-to-image retrieval results, measured in accuracy (%) ↑ and mAP@100 (%) ↑, respectively. The following Figure compares our model with the OpenCLIP baselines and other EO-VLMs. We applied a smoothed min-max scaling and annotated the lowest and highest scores. Our multispectral CLIP model is outperforming other RGB-based models on most benchmarks.
翻訳待ち・原文を表示
分類準備中
入力 未確認→ 出力 未確認
TerraMind is the first multimodal any-to-any generative foundation model for Earth Observation jointly developed by IBM, ESA, and Forschungszentrum Jülich. TerraMind uses a dual-scale transformer-based encoder-decoder architecture, simultaneously processing pixel-level and token-level data. The model was pre-trained on 500B tokens from 9M spatiotemporally aligned multimodal samples from the TerraMesh dataset. Modality-specific patch embeddings allow direct processing of raw inputs, while modality-specific FSQ-VAEs are used for image tokenization. For sequence-like modalities such as coordinates, an adapted WordPiece tokenizer is employed. During pre-training, TerraMind leverages masked token reconstruction, learning complex cross-modal correlations to generate high-quality latent representations.
翻訳待ち・原文を表示
分類準備中
入力 未確認→ 出力 未確認
TerraMind is the first multimodal any-to-any generative foundation model for Earth Observation jointly developed by IBM, ESA, and Forschungszentrum Jülich. TerraMind uses a dual-scale transformer-based encoder-decoder architecture, simultaneously processing pixel-level and token-level data. The model was pre-trained on 500B tokens from 9M spatiotemporally aligned multimodal samples from the TerraMesh dataset. Modality-specific patch embeddings allow direct processing of raw inputs, while modality-specific FSQ-VAEs are used for image tokenization. For sequence-like modalities such as coordinates, an adapted WordPiece tokenizer is employed. During pre-training, TerraMind leverages masked token reconstruction, learning complex cross-modal correlations to generate high-quality latent representations.
翻訳待ち・原文を表示
分類準備中
入力 未確認→ 出力 未確認
基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。
分類準備中
入力 未確認→ 出力 未確認
TerraMind is the first multimodal any-to-any generative foundation model for Earth Observation jointly developed by IBM, ESA, and Forschungszentrum Jülich. TerraMind uses a dual-scale transformer-based encoder-decoder architecture, simultaneously processing pixel-level and token-level data. The model was pre-trained on 500B tokens from 9M spatiotemporally aligned multimodal samples from the TerraMesh dataset. Modality-specific patch embeddings allow direct processing of raw inputs, while modality-specific FSQ-VAEs are used for image tokenization. For sequence-like modalities such as coordinates, an adapted WordPiece tokenizer is employed. During pre-training, TerraMind leverages masked token reconstruction, learning complex cross-modal correlations to generate high-quality latent representations.
翻訳待ち・原文を表示
分類準備中
入力 未確認→ 出力 未確認
基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。
分類準備中
入力 未確認→ 出力 未確認
基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。
分類準備中
入力 未確認→ 出力 未確認
TerraMind is the first multimodal any-to-any generative foundation model for Earth Observation jointly developed by IBM, ESA, and Forschungszentrum Jülich. The model is pre-trained using FSQ-VAE tokens as targets. This tokenizer encodes and decodes land-use land-cover (LULC) maps for the TerraMind model. The tokenizer uses FSQ with five dimensions and a codebook size of 4'375 tokens. The model was pre-trained for 20 epochs on nine million LULC images from the TerraMesh dataset which are sourced from ESRI. The maps include nine classes and a 10th no-data class: No data, water, trees, flooded vegetation, crops, built area, bare ground, snow/ice, clouds, rangeland. The tokenizer is fully integrated into the fine-tuning toolkit TerraTorch. You can initialize the pre-trained tokenizer with:
翻訳待ち・原文を表示
分類準備中
入力 未確認→ 出力 未確認
基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。
基本情報を収録・詳細は順次追加
分類準備中
入力 未確認→ 出力 未確認
基础档案已收录，详细介绍后续补充。可先查看厂商、分类与来源。
分類準備中
入力 未確認→ 出力 未確認
TerraMind is the first multimodal any-to-any generative foundation model for Earth Observation jointly developed by IBM, ESA, and Forschungszentrum Jülich. The model is pre-trained using FSQ-VAE tokens as targets. This tokenizer encodes and decodes Sentinel-2 L2A satellite images for the TerraMind model. The tokenizer uses FSQ with five dimensions and a codebook size of 15'360 tokens. The decoding process uses diffusion steps for the reconstruction. The model was pre-trained for 100 epochs on nine million S-2 L2A images from the TerraMesh dataset. The tokenizer is fully integrated into the fine-tuning toolkit TerraTorch. You can initialize the pre-trained tokenizer with:
翻訳待ち・原文を表示
分類準備中
入力 未確認→ 出力 未確認
