**Language model:** deepset/roberta-base-squad2-distilled **Language:** English **Training data:** SQuAD 2.0 training set **Eval data:** SQuAD 2.0 dev set **Code:** See an example extractive QA pipeline built with Haystack **Infrastructure**: 4x V100 GPU **Published**: Dec 8th, 2021 Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial.
This is the roberta-large model, fine-tuned using the SQuAD2.0 dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Question Answering. **Language model:** roberta-large **Language:** English **Downstream-task:** Extractive QA **Training data:** SQuAD 2.0 **Eval data:** SQuAD 2.0 **Code:** See an example extractive QA pipeline built with Haystack **Infrastructure**: 4x Tesla v100 Please note that we have also released a distilled version of this model called deepset/roberta-base-squad2-distilled. The distilled model has a comparable prediction quality and runs at twice the speed of the large model.
This is the *distilled* version of the deepset/roberta-base-squad2 model. This model has a comparable prediction quality and runs at twice the speed of the base model. **Language model:** tinyroberta-squad2 **Language:** English **Downstream-task:** Extractive QA **Training data:** SQuAD 2.0 **Eval data:** SQuAD 2.0 **Code:** See an example extractive QA pipeline built with Haystack **Infrastructure**: 4x Tesla v100 This model was distilled using the TinyBERT approach described in this paper and implemented in haystack. Firstly, we have performed intermediate layer distillation with roberta-base as the teacher which resulted in deepset/tinyroberta-6l-768d. Secondly, we have performed task-specific distillation with deepset/roberta-base-squad2 as the teacher for further intermediate layer distillation on an augmented version of SQuADv2 and then with deepset/roberta-large-squad2 as the teacher for prediction layer distillation.
**Language model:** xlm-roberta-base **Language:** Multilingual **Downstream-task:** Extractive QA **Training data:** SQuAD 2.0 **Eval data:** SQuAD 2.0 dev set - German MLQA - German XQuAD **Code:** See an example extractive QA pipeline built with Haystack **Infrastructure**: 4x Tesla v100 Corresponding experiment logs in mlflow: link Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack:
**Language model:** deepset/xlm-roberta-base-squad2-distilled **Language:** Multilingual **Downstream-task:** Extractive QA **Training data:** SQuAD 2.0 **Eval data:** SQuAD 2.0 **Code:** See an example extractive QA pipeline built with Haystack **Infrastructure**: 1x Tesla v100 Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial.
**Language model:** xlm-roberta-large **Language:** Multilingual **Downstream-task:** Extractive QA **Training data:** SQuAD 2.0 **Eval data:** SQuAD dev set - German MLQA - German XQuAD **Training run:** MLFlow link **Code:** See an example extractive QA pipeline built with Haystack **Infrastructure**: 4x Tesla v100 Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial.
- Model Details - How To Get Started With the Model - Uses - Risks, Limitations and Biases - Training - Evaluation - Environmental Impact - Technical Specifications - Citation Information - Model Card Authors **Model Description:** The DistilBERT model was proposed in the blog post Smaller, faster, cheaper, lighter: Introducing DistilBERT, adistilled version of BERT, and the paper DistilBERT, adistilled version of BERT: smaller, faster, cheaper and lighter. DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than *bert-base-uncased*, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark. This model is a fine-tune checkpoint of DistilBERT-base-cased, fine-tuned using (a second step of) knowledge distillation on SQuAD v1.1.
- Model Details - How To Get Started With the Model - Uses - Risks, Limitations and Biases - Training - Evaluation - Environmental Impact - Technical Specifications - Citation Information - Model Card Authors **Model Description:** The DistilBERT model was proposed in the blog post Smaller, faster, cheaper, lighter: Introducing DistilBERT, adistilled version of BERT, and the paper DistilBERT, adistilled version of BERT: smaller, faster, cheaper and lighter. DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than *bert-base-uncased*, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark. This model is a fine-tune checkpoint of DistilBERT-base-uncased, fine-tuned using (a second step of) knowledge distillation on SQuAD v1.1.
Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is cased: it makes a difference between english and English. Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same. The training is identical -- each masked WordPiece token is predicted independently.
Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is uncased: it does not make a difference between english and English. Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same. The training is identical -- each masked WordPiece token is predicted independently.
Llama3.1-Aloe-Beta-8B is an **open healthcare LLM** achieving **state-of-the-art performance** on several medical tasks. Aloe Beta is made available in four model sizes: 7B, 8B, 70B, and 72B. All models are trained using the same recipe, on top of two different families of models: Llama3.1 and Qwen2.5. Aloe is trained on 20 medical tasks, resulting in a robust and versatile healthcare model. Evaluations show Aloe models to be among the best in their class. When combined with a RAG system (also released) the 7B and 8B version gets close to the performance of closed models like MedPalm-2, GPT4. With the same RAG system, Llama3.1-Aloe-Beta-70B and Qwen2.5-Aloe-Beta-72B outperforms those private alternatives, producing state-of-the-art results. **Aloe-8B-Beta** is the latest iteration in the **Aloe family**, building and improving on the success of its predecessor, Aloe-8B-Alpha. Beta more than triples the training data used by Alpha, for a total of **1.8B tokens**, including a wider variety of medical tasks and instructions (e.g., text summarization, explanation, diagnosis, text classification, treatment recommendation, ...).
UlizaLlama is a 7B Parameters language model that builds upon the foundation of Jacaranda/kiswallama-pretrained. Jacaranda/kiswallama-pretrained is a large language model continually-pretrained with 321,530,045 swahili tokens and a customized tokenizer with a swahili vocabulary of 20,000 tokens to extend the capabilities of Meta/Llama2. It offers significant improvements in both encoding and decoding for Swahili text, surpassing the Swahili performance of Meta/Llama2. Moreover, Jacaranda/kiswallama-pretrained excels in providing accurate next-word completions in Swahili, a capability which Meta/Llama2 falls short of. - Origin: Adaptation of the Jacaranda/kiswallama-pretrained model which is continually pretrained from Meta/Llama2. - Data: Instructional dataset in Swahili and English consisting of prompt-response pairs. - Training: Alignment to standard methodologies, incorporation of task-centric heads, neural network weight optimization via backpropagation, and task-specific adjustments. - Fine-tuning: Utilized the LoRA approach, refining two matrices that mirror the main matrix from Jacaranda/kiswallama-pretrained. This Low Rank Adapter (LoRa) was vital for instruction-focused fine-tuning. Post-training, the developed LoRa was extracted, and Hugging Face's merge and unload() function facilitated the amalgamation of adapter weights with the base model. This fusion enables standalone inference with the merged model Meanwhile, Jacaranda/kiswallama-pretrained) offers versatility in: - Text Summarization - Autoregressive Text Completion - Content Generation - Text Rewording - Grammar Refinement and Editing - Further Research-The current UlizaLlama is available as a 7 Billion parameters model, further research can also explore availing bigger variants of UlizaLlama. ## Sample Evaluation Prompts The use of the developed Large Language Model (LLM) capabilities is for research,social good and internal use purposes only. For commercial use and distribution, organisations/individuals are encouraged to contactJacaranda Health). To ensure the ethical and responsible use of UlizaLlama, we have outlined a set of guidelines. These guidelines categorize activities and practice