bartowski

/

Behemoth-123B-v1-GGUF

like 3

I am having trouble loading this Q6K quant in text-generation-webui. I'm at a loss, because your Mistral-Large-Instruct-2411-Q5_K_M loads perfectly fine, and is also in three split files. The error is not related to total RAM, and happens within seconds of trying to load the model. (Edit: also, SHA256 checks out)
Happens with both llama_cpp_cuda_tensorcores and llama_cpp_cuda.
Any ideas?
Thank you

02:56:50-075384 ERROR    Failed to load the model.
Traceback (most recent call last):
  File "P:\text-generation\text-generation-webui-1.8\modules\ui_model_menu.py", line 232, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "P:\text-generation\text-generation-webui-1.8\modules\models.py", line 93, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "P:\text-generation\text-generation-webui-1.8\modules\models.py", line 278, in llamacpp_loader
    model, tokenizer = LlamaCppModel.from_pretrained(model_file)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "P:\text-generation\text-generation-webui-1.8\modules\llamacpp_model.py", line 85, in from_pretrained
    result.model = Llama(**params)
                   ^^^^^^^^^^^^^^^
  File "P:\text-generation\text-generation-webui-1.8\installer_files\env\Lib\site-packages\llama_cpp_cuda_tensorcores\llama.py", line 369, in __init__
    internals.LlamaModel(
  File "P:\text-generation\text-generation-webui-1.8\installer_files\env\Lib\site-packages\llama_cpp_cuda_tensorcores\_internals.py", line 56, in __init__
    raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: F:\AI\Models\LLM\Behemoth-123B-v1-GGUF-Q6K-bartowski\Behemoth-123B-v1-Q6_K-00001-of-00003.gguf

Exception ignored in: <function LlamaCppModel.__del__ at 0x0000018B89531580>
Traceback (most recent call last):
  File "P:\text-generation\text-generation-webui-1.8\modules\llamacpp_model.py", line 33, in __del__
    del self.model
        ^^^^^^^^^^
AttributeError: 'LlamaCppModel' object has no attribute 'model'