# Proper Pixel Art

## Summary

Converts noisy, high resolution pixel-art style images produced by generative models or sourced from low-quality web uploads to clean, true-resolution pixel-art assets.

### Challenges

Generative pixel-art style images are noisy and high resolution, often with a non-uniform grid and random artifacts. Standard downsampling techniques do not work. The current approach is to either use naive downscaling techniques or manually re-create the asset pixel by pixel.

This tool addressed both of these issues by automating the process of recovering true-resolution pixel-art assets.

## Installation

### From Source

Clone the Repository

```bash
git clone git@github.com:KennethJAllen/proper-pixel-art.git
cd proper-pixel-art
```

#### Create Virtual Environment

- Install [uv](https://docs.astral.sh/uv/getting-started/installation/) if not already installed.
- Sync environments:
  - `uv sync`

### From Pip

```bash
pip install proper-pixel-art
```

Or with uv:

```bash
uv add proper-pixel-art
```

## Usage

First, obtain a source pixel-art-style image (e.g. a pixel-art-style image generated by GPT-4o or a screenshot of pixel-art).

### CLI

```bash
uv run ppa <input_path> -o <output_path> -c <num_colors> -s <result_scale> [-t]
# or directly using uvx
uvx --from "proper-pixel-art" ppa <input_path> -o <output_path> -c <num_colors> -s <result_scale> [-t]
```

#### Flags

| Flag                             | Description                                                                                               |
| -------------------------------- | --------------------------------------------------------------------------------------------------------- |
| INPUT (positional)        | Source file in pixel-art-style                                                                      |
| `-o`, `--output` `<path>`        | Output directory or file path for result. (default: '.')                                                  |
| `-c`, `--colors` `<int>`         | Number of colors for output (1-256). Omit to skip quantization and preserve all colors. May need to try a few different values. (default None)                         |
| `-s`, `--scale-result` `<int>`     | Width/height of each "pixel" in the output. (default: 1)                                                          |
| `-t`, `--transparent` `<bool>`   | Output with transparent background. (default: off)                                                        |
| `-u`, `--initial-upscale` `<int>` | Initial image upscale factor. Increasing this may help detect pixel edges. (default 2)                    |
| `-w`, `--pixel-width` `<int>`    | Width of the pixels in the input image. If not set, it will be determined automatically. (default: None)  |

#### Example

```bash
uv run ppa assets/blob/blob.png -c 16 -s 20 -t
# or directly using uvx
uvx --from "proper-pixel-art" ppa path/to/input.png -o . -c 16 -s 20 -t
```

### Python

```python
from PIL import Image
from proper_pixel_art.pixelate import pixelate

image = Image.open('path/to/input.png')
result = pixelate(image, num_colors=16)
result.save('path/to/output.png')
```

#### Parameters

- `image` : `PIL.Image.Image`

  - A PIL image to pixelate.

- `num_colors` : `int | None`

  - The number of colors in result (1-256). Omit to skip quantization and preserve all colors.
  - May need to try a few values if the colors don't look right.
  - 8, 16, 32, or 64 typically works for quantized output.

- `initial_upscale` : `int`

  - Upscale initial image. This may help detect lines.

- `scale_result` : `int`

  - Upscale result after algorithm is complete if not None.

- `transparent_background` : `bool`
  - If True, flood fills each corner of the result with transparent alpha.

- `intermediate_dir` : `Path | None`
  - Directory to save images visualizing intermediate steps of algorithm. Useful for development.

- `pixel_width` : `int | None`
  - Width of the pixels in the input image. If not set, it will be determined automatically. It may be helpful to increase this parameter if not enough pixel edges are being detected.

#### Returns

A PIL image with true pixel resolution and quantized colors.

### Web Interface

Local:

```bash
uv sync --extra web
uv run ppa-web
# Opens http://127.0.0.1:7860
```

Without cloning:

```bash
uvx --from "proper-pixel-art[web]" ppa-web
```

## Examples

The algorithm is robust. It performs well for images that are already approximately alligned to a grid.

Here are a few examples. A mesh is computed, where each cell corresponds to one pixel.

### Bat

- Generated by GPT-4o.

<table align="center" width="100%">
  <tr>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/bat/bat.png" style="width:100%;" />
      <br><small>Noisy, High Resolution</small>
    </td>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/bat/mesh.png" style="width:100%;" />
      <br><small>Mesh</small>
    </td>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/bat/result.png" style="width:100%;" />
      <br><small>True Pixel Resolution</small>
    </td>
  </tr>
</table>

### Ash

- Screenshot from Google images of Pokemon asset.

<table align="center" width="100%">
  <tr>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/ash/ash.png" style="width:100%;" />
      <br><small>Noisy, High Resolution</small>
    </td>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/ash/mesh.png" style="width:100%;" />
      <br><small>Mesh</small>
    </td>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/ash/result.png" style="width:100%;" />
      <br><small>True Pixel Resolution</small>
    </td>
  </tr>
</table>

### Demon

- Original image generated by GPT-4o.

<table align="center" width="100%">
  <tr>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/demon/demon.png" style="width:100%;" />
      <br><small>Noisy, High Resolution</small>
    </td>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/demon/mesh.png" style="width:100%;" />
      <br><small>Mesh</small>
    </td>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/demon/result.png" style="width:100%;" />
      <br><small>True Pixel Resolution</small>
    </td>
  </tr>
</table>

### Pumpkin

- Screenshot from Google Images of Stardew Valley asset. This is an adversarial example as the source image is both low quality and the object is round.

<table align="center" width="100%">
  <tr>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/pumpkin/pumpkin.png" style="width:100%;" />
      <br><small>Noisy, High Resolution</small>
    </td>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/pumpkin/mesh.png" style="width:100%;" />
      <br><small>Mesh</small>
    </td>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/pumpkin/result.png" style="width:100%;" />
      <br><small>True Pixel Resolution</small>
    </td>
  </tr>
</table>

## Real Images To Pixel Art

- This tool can also be used to convert real images to pixel art by first requesting a pixelated version of the original image from GPT-4o, then using the tool to get the true pixel-resolution image.

- Consider this image of a mountain

<img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/mountain/real.jpg" width="50%" alt="Original mountain"/>

- Here are the results of first requesting a pixelated version of the mountain, then using the tool to get a true resolution pixel art version.

<table align="center" width="100%">
  <tr>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/mountain/mountain.png" style="width:100%;" />
      <br><small>Noisy, High Resolution</small>
    </td>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/mountain/mesh.png" style="width:100%;" />
      <br><small>Mesh</small>
    </td>
    <td width="33%">
      <img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/mountain/result.png" style="width:100%;" />
      <br><small>True Pixel Resolution</small>
    </td>
  </tr>
</table>

## Algorithm

- The main algorithm solves these challenges. Here is a high level overview. We will apply it step by step on this example image of blob pixel art that was generated from GPT-4o.

<img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/blob/blob.png" width="80%" alt="blob"/>

- Note that this image is high resolution and noisy.

<img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/blob/zoom.png" width="80%" alt="The blob is noisy."/>

1) Trim the edges of the image and zero out pixels with more than 50% alpha.
    - This is to work around some issues with models such as GPT-4o not giving a perfectly transparent background.

2) Upscale by a factor of 2 using nearest neighbor.
    - This can help identify the correct pixel mesh.

3) Find edges of the pixel art using [Canny edge detection](https://docs.opencv.org/4.x/da/d22/tutorial_py_canny.html).

<img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/blob/edges.png" width="80%" alt="blob edges"/>

4) Close small gaps in edges with a [morphological closing](https://docs.opencv.org/4.x/d9/d61/tutorial_py_morphological_ops.html).

<img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/blob/closed_edges.png" width="80%" alt="blob closed edges"/>

5) Take the [probabalistic Hough transform](https://docs.opencv.org/4.x/d3/de6/tutorial_js_houghlines.html) to get the coordinates of lines in the detected edges. Only keep lines that are close to vertical or horizontal giving some grid coordinates. Cluster lines that are closeby together.

<img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/blob/lines.png" width="80%" alt="blob lines"/>

6) Find the grid spacing by filtering outliers and taking the median of the spacings, then complete the mesh.

<img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/blob/mesh.png" width="80%" alt="blob mesh"/>

7) Quantize the original image to a small number of colors.
    - Note: The result is sensitive to the number of colors chosen.
    - The parameter is not difficult to tune, but the script may need to be re-run if the colors don't look right.
    - 8, 16, 32, or 64 typically works.

8) In each cell specified by the mesh, choose the most common color in the cell as the color for the pixel. Recreate the original image with one pixel per cell.

    - Result upscaled by a factor of $20 \times$ using nearest neighbor.

<img src="https://raw.githubusercontent.com/KennethJAllen/proper-pixel-art/main/assets/blob/result.png" width="80%" alt="blob pixelated"/>
