add visual comparison

README.ipynb

@@ -21,7 +21,29 @@
     "\n",
     "- [Project page and documentation](https://danjacobellis.net/walloc)\n",
     "- [Paper: \"Learned Compression for Compressed Learning\"](https://danjacobellis.net/_static/walloc.pdf)\n",
-    "- [Additional code accompanying the paper](https://github.com/danjacobellis/lccl)"
+    "- [Additional code accompanying the paper](https://github.com/danjacobellis/lccl)\n",
+    "\n",
+    "WaLLoC (Wavelet-Domain Learned Lossy Compression) is an architecture for learned compression that simultaneously satisfies three key\n",
+    "requirements of compressed-domain learning:\n",
+    "\n",
+    "1. **Computationally efficient encoding** to reduce overhead in compressed-domain learning and support resource constrained mobile and remote sensors. WaLLoC uses a wavelet packet transform to expose signal redundancies prior to autoencoding. This allows us to replace the encoding DNN with a single linear layer (<100k parameters) without significant loss in quality. WaLLoC incurs <5% of the encoding cost compared to other neural codecs.\n",
+    "\n",
+    "2. **High compression ratio** for storage and transmission efficiency. Lossy codecs typically achieve high compression with a combination of quantization and entropy coding. However, naive quantization of autoencoder latents leads to unpredictable and unbounded distortion. Instead, we apply additive noise during training as an\n",
+    "entropy bottleneck, leading to quantization-resiliant latents. When combined with entropy coding, this provides nearly 12× higher compression ratio compared to the VAE used in Stable Diffusion 3, despite offering a higher degree of dimensionality reduction and similar quality.\n",
+    "\n",
+    "3. **Dimensionality reduction** to accelerate compressed-domain modeling. WaLLoC’s encoder projects high-dimensional signal patches to low-dimensional latent representations, providing a reduction of up to 20×. This allows WaLLoC to be used as a drop-in replacement for resolution reduction while providing superior detail preservation and downstream accuracy.\n",
+    "\n",
+    "WaLLoC does not require perceptual or adversarial losses to represent high-frequency detail, making it compatible with a wide variety of signal types. It currently supports 1D and 2D signals, including mono, stereo, and multi-channel audio and grayscale, RGB, and hyperspectral images."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "84b21fc4-875c-468e-b0ee-201e71bd5737",
+   "metadata": {},
+   "source": [
+    "![](https://danjacobellis.net/walloc/_images/walloc.svg)\n",
+    "WaLLoC’s encode-decode pipeline. The entropy bottleneck and entropy coding steps are only required to achieve high compression ratios for storage and transmission. For compressed-domain learning where dimensionality reduction is the primary goal, these steps can be skipped to reduce overhead and completely eliminate CPU-GPU transfers.\n",
+    "\n"
    ]
   },
   {
@@ -35,24 +57,39 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
-   "id": "241f408a-7a78-414a-b03c-a28c39fc6275",
+   "id": "6dd81040-1e6e-45e3-b37d-1a1aabce8962",
+   "metadata": {},
+   "source": [
+    "![](https://danjacobellis.net/walloc/_images/walloc_4x.svg)\n",
+    "\n",
+    "![](https://danjacobellis.net/walloc/_images/sd3.svg)\n",
+    "\n",
+    "![](https://danjacobellis.net/walloc/_images/walloc_16x.svg)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3651a78e-c5c9-44e3-9f6d-ef44928ef0a7",
    "metadata": {},
    "source": [
-    "![](https://danjacobellis.net/walloc/_images/wpt.svg)\n",
-    "Example of forward and inverse WPT with $J=2$ levels. Each level applies filters $\\text{L}_{\\text{A}}$ and $\\text{H}_{\\text{A}}$ independently to each of the signal channels, followed by downsampling by two $(\\downarrow 2)$. An inverse level consists of upsampling $(\\uparrow 2)$ followed by $\\text{L}_{\\text{S}}$\n",
-    "and $\\text{H}_{\\text{S}}$, then summing the two channels. The full WPT $\\tilde{\\textbf{X}}$ of consists of $J$ levels."
+    "![](https://danjacobellis.net/walloc/_images/audio_comparison.svg)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "84b21fc4-875c-468e-b0ee-201e71bd5737",
+   "id": "e756f6fb-6caa-4587-9767-9735043291f4",
    "metadata": {},
    "source": [
-    "![](https://danjacobellis.net/walloc/_images/walloc.svg)\n",
-    "WaLLoC’s encode-decode pipeline. The entropy bottleneck and entropy coding steps are only required to achieve high compression ratios for storage and transmission. For compressed-domain learning where dimensionality reduction is the primary goal, these steps can be skipped to reduce overhead and completely eliminate CPU-GPU transfers.\n",
-    "\n"
+    "```\n",
+    "@article{jacobellis2024learned,\n",
+    "  title={Learned Compression for Compressed Learning},\n",
+    "  author={Jacobellis, Dan and Yadwadkar, Neeraja J.},\n",
+    "  year={2024},\n",
+    "  note={Under review},\n",
+    "  url={http://danjacobellis.net/walloc}\n",
+    "}\n",
+    "```"
    ]
   },
   {
@@ -60,10 +97,6 @@
    "id": "612736c3-c47f-4447-9d1a-07a07cfe4c47",
    "metadata": {},
    "source": [
-    "# Wavelet Learned Lossy Compression (WaLLoC)\n",
-    "\n",
-    "WaLLoC sandwiches a convolutional autoencoder between time-frequency analysis and synthesis transforms using CDF 9/7 wavelet filters. The time-frequency transform increases the number of signal channels, but reduces the temporal or spatial resolution, resulting in lower GPU memory consumption and higher throughput. WaLLoC's training procedure is highly simplified compared to other $\\beta$-VAEs, VQ-VAEs, and neural codecs, but still offers significant dimensionality reduction and compression. This makes it suitable for dataset storage and compressed-domain learning. It currently supports 1D and 2D signals, including mono, stereo, or multi-channel audio, and grayscale, RGB, or hyperspectral images.\n",
-    "\n",
     "## Installation\n",
     "\n",
     "1. Follow the installation instructions for [torch](https://pytorch.org/get-started/locally/)\n",
@@ -775,7 +808,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 1,
    "id": "0ae9e5bb-d616-4d22-a9c9-6743e4216802",
    "metadata": {},
    "outputs": [
@@ -785,7 +818,7 @@
      "text": [
       "[NbConvertApp] Converting notebook README.ipynb to markdown\n",
       "[NbConvertApp] Support files will be in README_files/\n",
-      "[NbConvertApp] Writing 11629 bytes to README.md\n"
+      "[NbConvertApp] Writing 12900 bytes to README.md\n"
      ]
     }
    ],
@@ -795,7 +828,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 2,
    "id": "3d4794d5-a26a-4789-804e-7aa543db89d6",
    "metadata": {},
    "outputs": [],
@@ -805,7 +838,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 3,
    "id": "8a386d35-9cdc-4e0b-8d5a-b58c4e055c28",
    "metadata": {},
    "outputs": [],

README.md

@@ -9,23 +9,45 @@ datasets:
 - [Paper: "Learned Compression for Compressed Learning"](https://danjacobellis.net/_static/walloc.pdf)
 - [Additional code accompanying the paper](https://github.com/danjacobellis/lccl)
 
-![](https://danjacobellis.net/walloc/_images/radar.svg)
-Comparison of WaLLoC with other autoencoder designs for RGB Images and stereo audio.
+WaLLoC (Wavelet-Domain Learned Lossy Compression) is an architecture for learned compression that simultaneously satisfies three key
+requirements of compressed-domain learning:
+
+1. **Computationally efficient encoding** to reduce overhead in compressed-domain learning and support resource constrained mobile and remote sensors. WaLLoC uses a wavelet packet transform to expose signal redundancies prior to autoencoding. This allows us to replace the encoding DNN with a single linear layer (<100k parameters) without significant loss in quality. WaLLoC incurs <5% of the encoding cost compared to other neural codecs.
 
+2. **High compression ratio** for storage and transmission efficiency. Lossy codecs typically achieve high compression with a combination of quantization and entropy coding. However, naive quantization of autoencoder latents leads to unpredictable and unbounded distortion. Instead, we apply additive noise during training as an
+entropy bottleneck, leading to quantization-resiliant latents. When combined with entropy coding, this provides nearly 12× higher compression ratio compared to the VAE used in Stable Diffusion 3, despite offering a higher degree of dimensionality reduction and similar quality.
 
+3. **Dimensionality reduction** to accelerate compressed-domain modeling. WaLLoC’s encoder projects high-dimensional signal patches to low-dimensional latent representations, providing a reduction of up to 20×. This allows WaLLoC to be used as a drop-in replacement for resolution reduction while providing superior detail preservation and downstream accuracy.
 
-![](https://danjacobellis.net/walloc/_images/wpt.svg)
-Example of forward and inverse WPT with $J=2$ levels. Each level applies filters $\text{L}_{\text{A}}$ and $\text{H}_{\text{A}}$ independently to each of the signal channels, followed by downsampling by two $(\downarrow 2)$. An inverse level consists of upsampling $(\uparrow 2)$ followed by $\text{L}_{\text{S}}$
-and $\text{H}_{\text{S}}$, then summing the two channels. The full WPT $\tilde{\textbf{X}}$ of consists of $J$ levels.
+WaLLoC does not require perceptual or adversarial losses to represent high-frequency detail, making it compatible with a wide variety of signal types. It currently supports 1D and 2D signals, including mono, stereo, and multi-channel audio and grayscale, RGB, and hyperspectral images.
 
 ![](https://danjacobellis.net/walloc/_images/walloc.svg)
 WaLLoC’s encode-decode pipeline. The entropy bottleneck and entropy coding steps are only required to achieve high compression ratios for storage and transmission. For compressed-domain learning where dimensionality reduction is the primary goal, these steps can be skipped to reduce overhead and completely eliminate CPU-GPU transfers.
 
 
 
-# Wavelet Learned Lossy Compression (WaLLoC)
+![](https://danjacobellis.net/walloc/_images/radar.svg)
+Comparison of WaLLoC with other autoencoder designs for RGB Images and stereo audio.
+
 
-WaLLoC sandwiches a convolutional autoencoder between time-frequency analysis and synthesis transforms using CDF 9/7 wavelet filters. The time-frequency transform increases the number of signal channels, but reduces the temporal or spatial resolution, resulting in lower GPU memory consumption and higher throughput. WaLLoC's training procedure is highly simplified compared to other $\beta$-VAEs, VQ-VAEs, and neural codecs, but still offers significant dimensionality reduction and compression. This makes it suitable for dataset storage and compressed-domain learning. It currently supports 1D and 2D signals, including mono, stereo, or multi-channel audio, and grayscale, RGB, or hyperspectral images.
+
+![](https://danjacobellis.net/walloc/_images/walloc_4x.svg)
+
+![](https://danjacobellis.net/walloc/_images/sd3.svg)
+
+![](https://danjacobellis.net/walloc/_images/walloc_16x.svg)
+
+![](https://danjacobellis.net/walloc/_images/audio_comparison.svg)
+
+```
+@article{jacobellis2024learned,
+  title={Learned Compression for Compressed Learning},
+  author={Jacobellis, Dan and Yadwadkar, Neeraja J.},
+  year={2024},
+  note={Under review},
+  url={http://danjacobellis.net/walloc}
+}
+```
 
 ## Installation
 
@@ -90,7 +112,7 @@ img
 
 
     
-![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_11_0.png)
+![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_13_0.png)
     
 
 
@@ -115,7 +137,7 @@ ToPILImage()(x_hat[0]+0.5)
 
 
     
-![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_13_0.png)
+![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_15_0.png)
     
 
 
@@ -151,7 +173,7 @@ plt.xlim([-25,25]);
 
 
     
-![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_16_0.png)
+![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_18_0.png)
     
 
 
@@ -183,7 +205,7 @@ print("compression_ratio: ", x.numel()/os.path.getsize("latent.png"))
 
 
     
-![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_20_0.png)
+![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_22_0.png)
     
 
 
@@ -205,7 +227,7 @@ print("compression_ratio: ", x.numel()/os.path.getsize("latent.webp"))
 
 
     
-![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_22_0.png)
+![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_24_0.png)
     
 
 
@@ -228,7 +250,7 @@ print("compression_ratio: ", x.numel()/os.path.getsize("latent.tif"))
 
 
     
-![jpeg](README_files/README_24_0.jpg)
+![jpeg](README_files/README_26_0.jpg)
     
 
 
@@ -357,7 +379,7 @@ plt.xlim([-25,25]);
 
 
     
-![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_39_0.png)
+![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_41_0.png)
     
 
 
@@ -398,7 +420,7 @@ webp
 
 
     
-![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_42_1.png)
+![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_44_1.png)
     
 
 
@@ -438,7 +460,7 @@ plt.yticks([]);
 
 
     
-![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_45_0.png)
+![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/README_47_0.png)
     
 
 
@@ -449,10 +471,15 @@ plt.yticks([]);
 
     [NbConvertApp] Converting notebook README.ipynb to markdown
     [NbConvertApp] Support files will be in README_files/
-    [NbConvertApp] Writing 1409744 bytes to README.md
+    [NbConvertApp] Writing 11629 bytes to README.md
 
 
 
 ```python
 !sed -i 's|!\[png](README_files/\(README_[0-9]*_[0-9]*\.png\))|![png](https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/\1)|g' README.md
 ```
+
+
+```python
+!sed -i 's|src="README_files/\(README_[0-9]*\.wav\)"|src="https://huggingface.co/danjacobellis/walloc/resolve/main/README_files/\1"|g' README.md
+```