quantize() reduces to far fewer colors than requested in some cases

[boringhexi]

What did you do?

This image, named test-in.png, has 508 distinct colors according to getcolors().

I tried to quantize it to 256 colors with the following code.

from PIL import Image
MAXCOLORCOUNT = 1024
with Image.open("test-in.png") as im:
    colsbefore = im.getcolors(maxcolors=MAXCOLORCOUNT)
    colsbefore = len(colsbefore) if colsbefore is not None else f"{MAXCOLORCOUNT}<"
    im = im.quantize(colors=256)
    colsafter = len(im.getcolors(maxcolors=MAXCOLORCOUNT))
    print(f"colors {colsbefore} -> {colsafter}")
    im.save("test-out.png") 

What did you expect to happen?

Reduced to 256 colors, i.e. print colors 508 -> 256

What actually happened?

Reduced to only 25 colors, i.e. prints colors 508 -> 25

What are your OS, Python and Pillow versions?

• OS: Ubuntu 20.04 (Kubuntu)
• Python: 3.9.1 (via pyenv)
• Pillow: 8.1.0

Other remarks

I'm guessing it's something about the image having only a few distinct RGB values but with a wide range of alpha values. A test image with over 256 distinct RGB values had no such issue.

Why don't I use pngquant or imagemagick instead? I depend on Pillow because I need a quantization method that preserves the colors of fully transparent pixels. (I'm modding a game whose texture filtering causes the colors of fully-transparent pixels to be faintly but noticeably visible around the edges of the fully-transparent regions.)

Edit: Using im.convert("P") on an image that already has 256 or fewer colors (according to len(im.getcolors())) can cause it to lose more colors unnecessarily during the conversion.

[gofr]

The reason why an identical RGB works fine is because quantize() uses different methods by default for RGB and RGBA. The only method available for RGBA is Image.FASTOCTREE.

If you change your code to do im.quantize(colors=256, method=Image.FASTOCTREE) you'd also use this Octree method for an RGB image. The result isn't exactly the same, but I still get way fewer colors than 256 in that case too.

This looks like a limitation in the FastOctree implementation. As far as I can tell it has very little to do with the Octree algorithm described on Wikipedia.

Your example image really has only 4 RGB colors: red, green, blue and black, where the red, green and blue have multiple alpha values. The best the FastOctree implementation can do is subdivide those alpha values into 8 buckets. So the result is 8 reds + 8 greens + 8 blues + 1 transparent black = 25.

I don't think this is going to be easy to fix. It would be possible to increase the number of buckets, but this is gonna affect performance pretty badly before you get to near 256 colors. I don't think that would be acceptable for a general fix, so it would probably just have to mean implementing a new quantization method.

If you just want to fix your own problem you could build Pillow yourself. Fiddle with the first 4 numbers in the {} on this line and see what gives you the best results. Values can be 0-8. Higher is better, but higher also means it uses more CPU and memory. My computer stopped liking it if the sum of those 4 was higher than about 24:

Pillow/src/libImaging/QuantOctree.c

Line 377 in c377d8c

const unsigned int CUBE_LEVELS_ALPHA[8] = {3, 4, 3, 3, 2, 2, 2, 2};

You could also try building Pillow with libimagequant support. That's supposed to be better. See https://pillow.readthedocs.io/en/stable/installation.html#external-libraries. With that enabled you can use quantize(method=Image.LIBIMAGEQUANT). But that library is also what powers pngquant, so if that's not giving the results you want, this may not work for you either.