PrimeDepth: Efficient Monocular Depth Estimation with a Stable Diffusion Preimage

[**Denis Zavadski**](https://scholar.google.com/citations?user=S7mDg00AAAAJ)^\* · [**Damjan Kalšan**](https://scholar.google.com/citations?user=6NAxnFUAAAAJ)^\* · [**Carsten Rother**](https://scholar.google.com/citations?user=N_YNMIMAAAAJ) Computer Vision and Learning Lab,
IWR, Heidelberg University ^*equal contribution ACCV 2024

PrimeDepth is a diffusion-based monocular depth estimator which leverages the rich representation of the visual world stored within Stable Diffusion. The representation, termed preimage, is extracted in a single diffusion step from frozen Stable Diffusion 2.1 and adjusted towards depth prediction. PrimeDepth yields detailed predictions while simulatenously being fast at inference time due to the single-step approach.

![teaser](images/teaser.png) ## Introduction These are the weights for the inference codebase for [PrimeDepth](https://arxiv.org/abs/2409.09144) based on Stable Diffusion 2.1. Further details and visual examples can be found on the [project page](https://vislearn.github.io/PrimeDepth/). ## Installation 1. Create and activate a virtual environment: ``` conda create -n PrimeDepth python=3.9 conda activate PrimeDepth ``` 2. Install dependencies: ``` pip3 install -r requirements.txt ``` 3. Download the [weights](https://huggingface.co/CVL-Heidelberg/PrimeDepth) 4. Adjust the attribute `ckpt_path` in `configs/inference.yaml` to point to the downloaded weights from the previous step ## Usage ``` from scripts.utils import InferenceEngine config_path = "./configs/inference.yaml" image_path = "./goodBoy.png" ie = InferenceEngine(pd_config_path=config_path, device="cuda") depth_ssi, depth_color = ie.predict(image_path) ``` PrimeDepth predicts in inverse space. The raw model predictions are stored in `depth_ssi`, while a colorized prediction `depth_color` is precomputed for visualization convenience: ``` depth_color.save("goodBoy_primedepth.png") ``` ## Citation ```bibtex @misc{zavadski2024primedepth, title={PrimeDepth: Efficient Monocular Depth Estimation with a Stable Diffusion Preimage}, author={Denis Zavadski and Damjan Kalšan and Carsten Rother}, year={2024}, eprint={2409.09144}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2409.09144}, } ```