Model Details
This model is an int4 model with group_size 128 and symmetric quantization of allenai/Molmo-72B-0924 generated by intel/auto-round.
How To Use
INT4 Inference
from auto_round import AutoRoundConfig ## must import for auto-round format
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from PIL import Image
import requests
quantized_model_path = "OPEA/Molmo-72B-0924-int4-sym-inc"
# load the processor
processor = AutoProcessor.from_pretrained(
quantized_model_path,
trust_remote_code=True,
torch_dtype='auto',
device_map='auto'
)
# load the model
model = AutoModelForCausalLM.from_pretrained(
quantized_model_path,
trust_remote_code=True,
torch_dtype='auto',
device_map='auto',
)
image_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
text = "Describe this image."
# process the image and text
inputs = processor.process(
images=[Image.open(requests.get(image_url, stream=True).raw)],
text=text
)
# move inputs to the correct device and make a batch of size 1
inputs = {k: v.to(model.device).unsqueeze(0) for k, v in inputs.items()}
inputs["images"] = inputs["images"].to(model.dtype)
# generate output; maximum 200 new tokens; stop generation when <|endoftext|> is generated
output = model.generate_from_batch(
inputs,
GenerationConfig(max_new_tokens=200, stop_strings="<|endoftext|>"),
tokenizer=processor.tokenizer
)
# only get generated tokens; decode them to text
generated_tokens = output[0,inputs['input_ids'].size(1):]
generated_text = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)
# print the generated text
print(generated_text)
##INT4:
## In this heartwarming photograph, a woman and her dog are captured sharing a tender moment on a sandy beach. The woman, with her long brown hair cascading down, is seated on the sand, her legs extended in front of her. She is dressed in a cozy flannel shirt and black pants, and she is barefoot, allowing the sand to embrace her feet. Her dog, a light brown canine with a white underbelly, sits upright on its hind legs, its front paws gently resting on the woman's hands. The dog is adorned with a harness that features a playful pattern of bones and fire hydrants. Both the woman and her furry companion are smiling warmly, their eyes locked in a loving gaze. The backdrop of the scene is a serene beach with gentle waves rolling in, and the sky above is a soft, light gray, adding a calm and peaceful ambiance to the moment.
##FP32:
## In this heartwarming photograph, a woman and her golden retriever are captured enjoying a serene moment on a sandy beach. The woman, with her long brown hair cascading down, is dressed in a blue and white plaid shirt paired with black pants. She sits barefoot on the sand, her right leg extended outward and her left leg bent at the knee, creating a relaxed posture. Her right hand gently holds the paw of her golden retriever, who is seated beside her, facing her with a joyful expression. The dog's left paw rests on the woman's extended leg, and its tail is stretched out behind it. The golden retriever sports a blue harness adorned with red and green paw prints, adding a splash of color to the scene. The sandy beach is marked with numerous footprints, hinting at the activity of other beachgoers. In the background, the tranquil ocean stretches out, with gentle waves rolling in, completing this picturesque and touching moment between a woman and her
image_url = "http://images.cocodataset.org/train2017/000000411975.jpg"
text = "How many people are there on the baseball field in the picture?"
##INT4:
## Counting the <points x1="10.2" y1="95.2" x2="47.7" y2="40.3" x3="57.1" y3="87.3" x4="58.6" y4="50.2" x5="76.1" y5="38.8" alt="people are there on the baseball field">people are there on the baseball field</points> shows a total of 5.
##FP32:
## Counting the <points x1="10.2" y1="94.9" x2="47.7" y2="40.3" x3="56.8" y3="87.1" x4="59.1" y4="50.0" x5="75.8" y5="37.8" alt="people are there on the baseball field">people are there on the baseball field</points> shows a total of 5.
image_url = "https://intelcorp.scene7.com/is/image/intelcorp/processor-overview-framed-badge:1920-1080?wid=480&hei=270"
text = "Which company does this image represent?"
##INT4:
## The image represents Intel Corporation. The logo prominently displays the word "intel" in lowercase letters, followed by "INSIDE" in uppercase letters, which is Intel's well-known branding slogan. This iconic logo design is instantly recognizable and has been associated with Intel's processors and computing technology for decades.
##FP32:
## The image represents Intel Corporation. The iconic "Intel Inside" logo is prominently displayed, featuring the distinctive blue color scheme and typography that has become synonymous with Intel's brand identity. This logo has been a hallmark of Intel's marketing strategy for decades, appearing on computer components and systems that use Intel processors.
Generate the model
Here is the sample command to reproduce the model.
pip install auto-round
auto-round-mllm
--model allenai/Molmo-72B-0924 \
--device 0 \
--group_size 128 \
--bits 4 \
--iters 1000 \
--nsamples 512 \
--seqlen 2048 \
--format 'auto_round' \
--output_dir "./tmp_autoround"
Ethical Considerations and Limitations
The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
Therefore, before deploying any applications of the model, developers should perform safety testing.
Caveats and Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
Here are a couple of useful links to learn more about Intel's AI software:
- Intel Neural Compressor link
Disclaimer
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
Cite
@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }
- Downloads last month
- 0