Initial commit

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,133 @@
+---
+language:
+- de
+license: apache-2.0
+library_name: transformers
+tags:
+- deutsch
+- german
+- seedbox
+- mistral
+datasets:
+- seedboxai/multitask_german_examples_32k
+pipeline_tag: text-generation
+---
+
+![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/645ded34a45b4182d7f5c385/oh7yRzqtRlDtdu8sJoAdV.jpeg)
+
+
+# KafkaLM-7B-German-V0.1
+
+**KafkaLM 7b** is a Mistral 7b model - further pre-trained on a large German dataset from Björn Plüster and LAION. [leo-mistral-hessianai-7b](https://huggingface.co/LeoLM/leo-mistral-hessianai-7b) - which was finetuned on an ensemble of popular high-quality open-source instruction sets (translated from English to German). 
+
+KafkaLM 7b is a [Seedbox](https://huggingface.co/seedboxai) project trained by [Dennis Dickmann](https://huggingface.co/doubledsbv).
+
+**Why Kafka?** 
+The models are proficient, yet creative, and have some tendencies to linguistically push boundaries 😊
+
+
+## Model Details
+
+The purpose of releasing the **KafkaLM series** is to contribute to the German AI community with a set of fine-tuned LLMs that are easy to use in everyday applications across a variety of tasks.
+
+The main goal was to provide LLMs proficient in German, especially to be used in German-speaking business contexts where English alone is not sufficient.
+
+
+### Dataset
+
+I used a 8k filtered version of the following [seedboxai/multitask_german_examples_32k](https://huggingface.co/datasets/seedboxai/multitask_german_examples_32k)
+
+### Prompt Format
+
+
+This model follows the subsequent prompt format:
+
+```
+<|system|>
+Du bist ein freundlicher und hilfsbereiter KI-Assistent. Du beantwortest Fragen faktenorientiert und präzise, ohne dabei relevante Fakten auszulassen.</s>
+<|user|>
+Welche Möglichkeiten der energetischen Sanierung habe ich neben Solar und Energiespeicher?</s>
+<|assistant|>
+```
+
+### Inference
+
+Getting started with the model is straightforward
+
+```python
+import transformers
+
+model_id = "seedboxai/KafkaLM-7B-German-V0.1"
+
+model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, trust_remote_code=True)
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+tokenizer.padding_side = "right" 
+tokenizer.pad_token = tokenizer.unk_token
+tokenizer.add_eos_token = False
+
+def generate_prompt(input):
+    prompt = ''
+    sys_prompt = "Du bist ein freundlicher und hilfsbereiter KI-Assistent. Du beantwortest Fragen faktenorientiert und präzise, ohne dabei relevante Fakten auszulassen."
+    
+    prompt += f"<|system|>\n{sys_prompt.strip()}</s>\n"
+    prompt += f"<|user|>\n{input.strip()}</s>\n"
+    prompt += f"<|assistant|>\n"
+
+    return prompt.strip()
+
+
+def evaluate(
+    input,
+    temperature=0.7,
+    top_p=0.95,
+    top_k=50,
+    num_beams=3,
+    max_new_tokens=512,
+    #max_length=8192,
+    **kwargs,
+):
+    prompt = generate_prompt(input)
+
+    #print(prompt)
+    
+    inputs = tokenizer(prompt, return_tensors="pt")
+    input_ids = inputs["input_ids"].to(device)
+    attention_mask=inputs["attention_mask"].to(device)
+    generation_config = GenerationConfig(
+        temperature=temperature,
+        top_p=top_p,
+        top_k=top_k,
+        num_beams=num_beams,
+        no_repeat_ngram_size=3,
+        do_sample=True,
+        **kwargs,
+    )
+
+    with torch.no_grad():
+        generation_output = model.generate(
+            early_stopping=False,
+            #eos_token_id=tokenizer.eos_token_id,
+            #pad_token_id=tokenizer.pad_token_id,
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            generation_config=generation_config,
+            return_dict_in_generate=True,
+            output_scores=True,
+            max_new_tokens=max_new_tokens,
+            #max_length= max_length
+        )
+    s = generation_output.sequences[0]
+    output = tokenizer.decode(s)
+    return output #.split("<|assistant|>")[1].strip()
+
+
+print(evaluate("Wer ist eigentlich dieser Kafka?"))
+
+```
+
+## Disclaimer
+
+The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model.
+This model should only be used for research purposes. The original Llama2 license and all restrictions of datasets used to train this model apply.

kafkalm-7b-german-v0.1.Q2_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7c01c579d3b30dfc829f67f72944e2cd53eeba62e279d8d9f877bb3f827c5f0
+size 2719242432

kafkalm-7b-german-v0.1.Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c829e08ea749dec7fe4ceefff7d73bf3bbd8d64997a5d906846b409be73be5e1
+size 3518986432

kafkalm-7b-german-v0.1.Q3_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0002a92d48ef9c9f87327ed9716e1b5e1fed95850748a8676bf52fe6194b108
+size 3164567744

kafkalm-7b-german-v0.1.Q4_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c913cd7f7e31b3a0041bb44b6332ff88d245de636db36a9523ca102d3e62208
+size 4108916928

kafkalm-7b-german-v0.1.Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5323e0752fe093f1f472b7d70384c758ec0e57d4aa21301d01ccf1658e395fcc
+size 4368439488

kafkalm-7b-german-v0.1.Q4_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:589acda152b3dc7218b8be04e5d145df5c5dfea26c3b19015bfc28286abeca39
+size 4140374208

kafkalm-7b-german-v0.1.Q5_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f6ddc226ef8b11c45c7cfcbf2a1390c4007c22efd4131a3eddcbe1f81b6fa10
+size 4997716160

kafkalm-7b-german-v0.1.Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:703b7bf64fdc7b61d588926b7a10ceb59a3a69f1afafa9ff74384fdefb20305d
+size 5131409600

kafkalm-7b-german-v0.1.Q5_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32f04ff9c1e9afb38e8b478b83e6b89365c526d0ef90fc94e8a0a4147ba7b168
+size 4997716160

kafkalm-7b-german-v0.1.Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee882a080f749b45a87e8f1460ec50963ab28f1e55cdf06c977a4bbe92d4ed98
+size 5942065344

kafkalm-7b-german-v0.1.Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:422cea635582eafd29b249ffda15fefcf5f464d0ce25ae3f50967b742d80aecd
+size 7695857856