AutoCompressor-Llama-2-7b-6k
princeton-nlp基础档案已收录,详细介绍后续补充。可先查看厂商、分类与来源。
Basic profile indexed · More details to follow
From language to images, open weights to APIs. Understand capabilities and compare real costs.
Find the right engine for your next idea.
基础档案已收录,详细介绍后续补充。可先查看厂商、分类与来源。
Basic profile indexed · More details to follow
基础档案已收录,详细介绍后续补充。可先查看厂商、分类与来源。
Basic profile indexed · More details to follow
**ProLong** ( Pr incet o n long -context language models) is a family of long-context models that are continued trained and supervised fine-tuned from Llama-3-8B, with a maximum context window of 512K tokens. Our main ProLong model is one of the best-performing long-context models at the 10B scale (evaluated by HELMET). To train this strong long-context model, we conduct thorough ablations on the long-context pre-training data, SFT data, and numerous other design choices. We demonstrate our findings in our paper, How to Train Long-Context Language Models (Effectively). Here are some quick facts about our main ProLong model: princeton-nlp/Llama-3-8B-ProLong-512k-Instruct. * Base model: meta-llama/Meta-Llama-3-8B-Instruct * Long-context continued training: 20B tokens on 64K training data (princeton-nlp/prolong-data-64K), and 20B tokens on 512K training data (princeton-nlp/prolong-data-512K) * Supervised fine-tuning (SFT): UltraChat * Maximum context window: 512K tokens
**ProLong** ( Pr incet o n long -context language models) is a family of long-context models that are continued trained and supervised fine-tuned from Llama-3-8B, with a maximum context window of 512K tokens. Our main ProLong model is one of the best-performing long-context models at the 10B scale (evaluated by HELMET). To train this strong long-context model, we conduct thorough ablations on the long-context pre-training data, SFT data, and numerous other design choices. We demonstrate our findings in our paper, How to Train Long-Context Language Models (Effectively). Here are some quick facts about our main ProLong model: princeton-nlp/Llama-3-8B-ProLong-512k-Instruct. * Base model: meta-llama/Meta-Llama-3-8B-Instruct * Long-context continued training: 20B tokens on 64K training data (princeton-nlp/prolong-data-64K), and 20B tokens on 512K training data (princeton-nlp/prolong-data-512K) * Supervised fine-tuning (SFT): UltraChat * Maximum context window: 512K tokens
**ProLong** ( Pr incet o n long -context language models) is a family of long-context models that are continued trained and supervised fine-tuned from Llama-3-8B, with a maximum context window of 512K tokens. Our main ProLong model is one of the best-performing long-context models at the 10B scale (evaluated by HELMET). To train this strong long-context model, we conduct thorough ablations on the long-context pre-training data, SFT data, and numerous other design choices. We demonstrate our findings in our paper, How to Train Long-Context Language Models (Effectively). Here are some quick facts about our main ProLong model: princeton-nlp/Llama-3-8B-ProLong-512k-Instruct. * Base model: meta-llama/Meta-Llama-3-8B-Instruct * Long-context continued training: 20B tokens on 64K training data (princeton-nlp/prolong-data-64K), and 20B tokens on 512K training data (princeton-nlp/prolong-data-512K) * Supervised fine-tuning (SFT): UltraChat * Maximum context window: 512K tokens
**ProLong** ( Pr incet o n long -context language models) is a family of long-context models that are continued trained and supervised fine-tuned from Llama-3-8B, with a maximum context window of 512K tokens. Our main ProLong model is one of the best-performing long-context models at the 10B scale (evaluated by HELMET). To train this strong long-context model, we conduct thorough ablations on the long-context pre-training data, SFT data, and numerous other design choices. We demonstrate our findings in our paper, How to Train Long-Context Language Models (Effectively). Here are some quick facts about our main ProLong model: princeton-nlp/Llama-3-8B-ProLong-512k-Instruct. * Base model: meta-llama/Meta-Llama-3-8B-Instruct * Long-context continued training: 20B tokens on 64K training data (princeton-nlp/prolong-data-64K), and 20B tokens on 512K training data (princeton-nlp/prolong-data-512K) * Supervised fine-tuning (SFT): UltraChat * Maximum context window: 512K tokens
基础档案已收录,详细介绍后续补充。可先查看厂商、分类与来源。
Basic profile indexed · More details to follow
The model is a sequence classification model that predicts quality ratings across 4 critera as outputs. The quality ratings (which are unnormalized) can be accessed via the 4 logits in the output of the model. They are in the following order: * Logit 0: Writing Style * Logit 1: Required Expertise * Logit 2: Facts and Trivia * Logit 3: Educational Value **Note:** The model was only fine-tuned on sequences of up to 512 tokens, and should not be used for longer documents. Instead, compute the quality ratings for windows of up to 512 token and average them weighted by the window length. In the paper, we document various types of bias that are present in the quality ratings from the QuRater model (biases related to domains, topics, social roles, regions and languages - see Section 6 of the paper). Hence, be aware that data selection with QuRating could have unintended and harmful effects on the language model that is being trained. We strongly recommend a comprehensive evaluation of the language model for these and other types of bias, particularly before real-world deployment. We hope that releasing the data/models can facilitate future research aimed at uncovering and mitigating such biases. Note that the quality ratings do not measure the social or literary value of a text and should *not* be used for textual or demographic studies.
**Paper**: https://arxiv.org/pdf/2310.06694.pdf **Code**: https://github.com/princeton-nlp/LLM-Shearing **Models**: Sheared-LLaMA-1.3B, Sheared-LLaMA-2.7B **Pruned Models without Continued Pre-training**: Sheared-LLaMA-1.3B-Pruned, Sheared-LLaMA-2.7B-Pruned **Instruction-tuned Models**: Sheared-LLaMA-1.3B-ShareGPT, Sheared-LLaMA-2.7B-ShareGPT Sheared-LLaMA-1.3B is a model pruned and further pre-trained from meta-llama/Llama-2-7b-hf. We dynamically load data from different domains in the RedPajama dataset to prune and contune pre-train the model. We use 0.4B tokens for pruning and 50B tokens for continued pre-training the pruned model. This model can be loaded with HuggingFace via We evaluate on an extensive set of downstream tasks including reasoning, reading comprehension, language modeling and knowledge intensive tasks. Our Sheared-LLaMA models outperform existing large language models.
基础档案已收录,详细介绍后续补充。可先查看厂商、分类与来源。
Basic profile indexed · More details to follow
基础档案已收录,详细介绍后续补充。可先查看厂商、分类与来源。
Basic profile indexed · More details to follow
基础档案已收录,详细介绍后续补充。可先查看厂商、分类与来源。
基础档案已收录,详细介绍后续补充。可先查看厂商、分类与来源。
Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations. The model craters note in the Github Repository > We train unsupervised SimCSE on 106 randomly sampled sentences from English Wikipedia, and train supervised SimCSE on the combination of MNLI and SNLI datasets (314k).
基础档案已收录,详细介绍后续补充。可先查看厂商、分类与来源。
Basic profile indexed · More details to follow
基础档案已收录,详细介绍后续补充。可先查看厂商、分类与来源。
基础档案已收录,详细介绍后续补充。可先查看厂商、分类与来源。
Basic profile indexed · More details to follow
Adjust provider, category or input/output filters. New models are added continuously.
Adjust monthly token usage to compare estimated API costs.
Uses collected, comparable USD-per-million-token rates. Snapshots exclude cache discounts, tiers, extra reasoning output and taxes. Check current source prices.
Separate input, output and cache rates, then estimate the same workload. Image, audio and video rates may use items, duration or resolution rather than tokens.
Not necessarily. Check each version’s license, commercial restrictions and policies. Self-hosting also incurs compute and maintenance costs.
Rankings depend on tasks, versions and methods. Compare accuracy, latency, reliability and costs on your own workload.