metadata

library_name: transformers
tags:
  - 4bit
  - AWQ
  - AutoAWQ
  - llama
  - llama-2
  - facebook
  - meta
  - 7b
  - quantized
license: llama2
pipeline_tag: text-generation

Model Card for alokabhishek/Llama-2-7b-chat-hf-4bit-AWQ

This repo contains 4-bit quantized (using AutoAWQ) model of Meta's meta-llama/Llama-2-7b-chat-hf

Model Details

Model creator: Meta
Original model: Llama-2-7b-chat-hf

About 4 bit quantization using AutoAWQ

AutoAWS github repo: bitsandbytes github repo

How to Get Started with the Model

Use the code below to get started with the model.

How to run from Python code

First install the package

!pip install autoawq
!pip install accelerate

Import

import torch
import os
from torch import bfloat16
from huggingface_hub import login, HfApi, create_repo
from transformers import AutoTokenizer, pipeline
from awq import AutoAWQForCausalLM

Use a pipeline as a high-level helper

# define the model ID
model_id_llama = "alokabhishek/Llama-2-7b-chat-hf-4bit-AWQ"

# Load model
tokenizer_llama = AutoTokenizer.from_pretrained(model_id_llama, use_fast=True)
model_llama = AutoAWQForCausalLM.from_quantized(model_id_llama, fuse_layer=True, trust_remote_code = False, safetensors = True)

# Set up the prompt and prompt template. Change instruction as per requirements.
prompt_llama = "Tell me a funny joke about Large Language Models meeting a Blackhole in an intergalactic Bar."
fromatted_prompt = f'''[INST] <<SYS>> You are a helpful, and fun loving assistant. Always answer as jestfully as possible. <</SYS>> {prompt_llama} [/INST] '''

tokens = tokenizer_llama(fromatted_prompt, return_tensors="pt").input_ids.cuda()

# Generate output, adjust parameters as per requirements
generation_output = model_llama.generate(tokens, do_sample=True, temperature=1.7, top_p=0.95, top_k=40, max_new_tokens=512)

# Print the output
print(tokenizer_llama.decode(generation_output[0], skip_special_tokens=True))

Uses

Direct Use

[More Information Needed]

Downstream Use [optional]

[More Information Needed]

Out-of-Scope Use

[More Information Needed]

Bias, Risks, and Limitations

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]