How can I create a 128K GGUF of a non-instruct model?
#1
by
nitinrathi
- opened
I am using Qwen2.5 Coder 7B for FIM and code completion in my editor. I am using repo-level completion, but the 32K context is not enough. How can I create a 128K GGUF of the non-instruct model?
how are you doing FIM right now. can you share tcode
I wrote an Emacs extension. I used the following template:
<|repo_name|>{repo_name}
<|file_sep|>{file_path1}
{file_content1}
<|file_sep|>{file_path2}
{file_content2}
<|file_sep|>{file_path3}
<|fim_prefix|>{code_pre}<|fim_suffix|>{code_suf}<|fim_middle|>{code_fim}<|endoftext|>
For inference I am using llama.cpp and the /completion endpoint.
I will open source the Emacs extension once it matures.