|
---
|
|
license: llama2
|
|
datasets:
|
|
- liuhaotian/LLaVA-Instruct-150K
|
|
---
|
|
|
|
# Ada-LLaVA Model Card
|
|
|
|
__AdaLLaVA-H: This model is another verson of Ada-LLaVA-13B where we drop attention heads and neurons.__
|
|
|
|
<!-- Provide a quick summary of what the model is/does. -->
|
|
|
|
Ada-LLaVA-13B-drop_heads (AdaLLaVA-H) is an open-source adaptive inference framework for multimodal Large Language Models (MLLMs) that dynamically adjusts its operations based on available computational resources and latency requirements.
|
|
|
|
|
|
|
|
See the paper for more details: [Learning to Inference Adaptively for Multimodal Large Language Models]()
|
|
|
|
**Model details**: https://zhuoyan-xu.github.io/ada-llava/
|
|
|
|
## Model Details
|
|
|
|
<!-- Provide a longer summary of what this model is. -->
|
|
|
|
**Model Type:** Ada LLaVA 13B follows the [LLaVA-v1.5](https://arxiv.org/abs/2310.03744) stage-2 training pipeline,
|
|
with [CLIP-ViT-L-336px](https://huggingface.co/openai/clip-vit-large-patch14-336) as visual encoder (336*336 image resolution),
|
|
[Vicuna-v1.5-13B](https://huggingface.co/lmsys/vicuna-13b-v1.5) as base LLM and a two-layer MLP as vision-language connector, customized embedding model and MLP as latency scheduler.
|
|
|
|
It was trained with stage-2 pipeline as LLaVA:
|
|
|
|
Instruction tuning: Freeze vision encoder, train the remaining model with multimodal instruction following data of tabular and non-tabular tasks.
|
|
|
|
**Code Base:** We use the official code of [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA) for model training and inference,
|
|
and the saved model checkpoint is uploaded to this repository.
|
|
|
|
**Model Date:** Ada-LLaVA 13B was trained in Oct 2024.
|
|
|
|
|
|
## License
|
|
|
|
AdaLLaVA is based on LLaVA-1.5 and thus follows its license. Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
|
|
|
|
## Intended use
|
|
|
|
**Primary intended uses:** The primary use of Ada LLaVA is research on multimodal large multimodal models and chatbots, especially for resource-constrain inference and deployment.
|
|
|
|
**Primary intended users:** The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
|
|
|
|
## Training dataset
|
|
- 665K image level instruction data from LLaVA-1.5 stage-2, see details in original [LLaVA repo](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#visual-instruction-tuning).
|
|
|
|
## Limitations
|
|
|
|
While Ada-LLaVA is currently limited to processing one image at a time and only applies adaptive operations in its later half of layers, future work could explore multi-image input support and extend the adaptive mechanisms throughout the entire model architecture, including the vision encoder. These improvements would make the model more versatile and applicable to a broader range of real-world scenarios.
|
|
|
