| Current best model from our experiments to finetune CLIP on 5k archaeological record photos plus another 300 from the Met. | |
| See [blog post 1](https://carleton.ca/xlab/2023/archaeclip-or-building-a-visual-search-engine-for-archaeology/) and a companion post at [open context](https://alexandriaarchive.org/2023/10/08/artificial-intelligence-ai-and-open-context/) | |
| The goal is to use with `LLM`, Simon Willison's package for working with large language models, and in particular, `LLM-CLIP`, to make our own embeddings-powered search engine. | |
| https://github.com/simonw/llm-clip | |
| requires LLM: https://llm.datasette.io/en/stable/ | |
| So: | |
| ``` | |
| $ pip install llm | |
| $ llm install llm-clip | |
| ``` | |
| Then, assuming you are doing this in an environment (I create mine with conda), find the site packages directory, and the llm-clip.py file: | |
| `/Users/username/mambaforge/envs/clip/lib/python3.10/site-packages` is where mine hides. | |
| Change | |
| ``` | |
| if self._model is None: | |
| self._model = SentenceTransformer('clip-ViT-B-32') | |
| ``` | |
| to point to your new model, like so: | |
| ``` | |
| def embed_batch(self, items): | |
| # Embeds a mix of text strings and binary images | |
| if self._model is None: | |
| self._model = SentenceTransformer('/path/to/your/retrained-model') | |
| ``` | |
| The folder with your model should contain a pytorch_model.bin and config.json inside a subfolder called 0CLIP_Model. You will need the extra json files and so on from here [https://huggingface.co/sentence-transformers/clip-ViT-B-32/tree/main](https://huggingface.co/sentence-transformers/clip-ViT-B-32/tree/main) . You need all those .json files, arranged that way. And since you're not otherwise futzing with the basic CLIP-ness, it should be ok. | |
| Once you create your embeddings, these will be in your ~Library/Application Support/io.datasette.llm folder. |