--- license: llama2 datasets: - liuhaotian/LLaVA-Instruct-150K --- # Ada-LLaVA Model Card __AdaLLaVA-H: This model is another verson of Ada-LLaVA-13B where we drop attention heads and neurons.__ Ada-LLaVA-13B-drop_heads (AdaLLaVA-H) is an open-source adaptive inference framework for multimodal Large Language Models (MLLMs) that dynamically adjusts its operations based on available computational resources and latency requirements. See the paper for more details: [Learning to Inference Adaptively for Multimodal Large Language Models]() **Model details**: https://zhuoyan-xu.github.io/ada-llava/ ## Model Details **Model Type:** Ada LLaVA 13B follows the [LLaVA-v1.5](https://arxiv.org/abs/2310.03744) stage-2 training pipeline, with [CLIP-ViT-L-336px](https://huggingface.co/openai/clip-vit-large-patch14-336) as visual encoder (336*336 image resolution), [Vicuna-v1.5-13B](https://huggingface.co/lmsys/vicuna-13b-v1.5) as base LLM and a two-layer MLP as vision-language connector, customized embedding model and MLP as latency scheduler. It was trained with stage-2 pipeline as LLaVA: Instruction tuning: Freeze vision encoder, train the remaining model with multimodal instruction following data of tabular and non-tabular tasks. **Code Base:** We use the official code of [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA) for model training and inference, and the saved model checkpoint is uploaded to this repository. **Model Date:** Ada-LLaVA 13B was trained in Oct 2024. ## License AdaLLaVA is based on LLaVA-1.5 and thus follows its license. Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved. ## Intended use **Primary intended uses:** The primary use of Ada LLaVA is research on multimodal large multimodal models and chatbots, especially for resource-constrain inference and deployment. **Primary intended users:** The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence. ## Training dataset - 665K image level instruction data from LLaVA-1.5 stage-2, see details in original [LLaVA repo](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#visual-instruction-tuning). ## Limitations While Ada-LLaVA is currently limited to processing one image at a time and only applies adaptive operations in its later half of layers, future work could explore multi-image input support and extend the adaptive mechanisms throughout the entire model architecture, including the vision encoder. These improvements would make the model more versatile and applicable to a broader range of real-world scenarios.