Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory issue when using composite function #2759

Closed
scottbradybuckii opened this issue Jun 17, 2021 · 5 comments
Closed

Memory issue when using composite function #2759

scottbradybuckii opened this issue Jun 17, 2021 · 5 comments
Labels

Comments

@scottbradybuckii
Copy link

Are you using the latest version? Is the version currently in use as reported by npm ls sharp the same as the latest version as reported by npm view sharp dist-tags.latest?
npm ls sharp -> 0.28.3
npm view sharp dist-tags.latest -> 0.28.3

What are the steps to reproduce?
When compositing multiple images, some of the memory does not seem to be fully released back when garbage collection is run. This isn't a problem when only doing a few images, but it is a problem when running a web server that can conceivably compose thousands of images before it is shut down.

I have read all of the other threads on memory usage with this library and have tried the fixes listed in them. I have tried changing the LD_PRELOAD env to libjemalloc, I have tried using Alpine instead of Debian-based systems, I have even tried running it on ARM processors, and it still happens across the board.

What is the expected behaviour?
When sharp is finished composing two images, the memory is released back to the system instead of building up.

Are you able to provide a minimal, standalone code sample, without other dependencies, that demonstrates this problem?

const fs = require('fs');
const path = require('path');
const sharp = require('sharp');

setInterval(() => {
  var memMB = process.memoryUsage().rss / 1048576;
  console.log(memMB);
}, 500);

async function test() {
  const watermark = fs.readFileSync(path.resolve('watermark.svg'));
  const background = fs.readFileSync(path.resolve('landscape.jpg'));
  sharp.cache(false);

  for (let i = 0; i <= 500; i++) {
    console.log(i);
    await sharp(background)
      .composite([{ input: watermark }])
      .toBuffer();
  }
}

test();

I realize that this is printing out RSS numbers, which might not be totally accurate when it comes to showing how much memory is freed vs how much is collected - but I also monitor the ram usage using htop and docker stats (since I'm running this in a container) and it is consistently climbing across the board. Also, when I do any other operation to the image besides compose it, the RSS (and htop and docker stats) memory using will immediately fall back down after finishing this loop.

Also, this problem is much more pronounced when larger (and different) files are loaded, like in the web server application that I am using it for. In that case, I usually can't get past 300-400 requests before my ram usage is between 500MB and 1G.

Are you able to provide a sample image that helps explain the problem?
Any two images composed together should cause this issue

What is the output of running npx envinfo --binaries --system?
Debian

  System:
    OS: Linux 4.4 Debian GNU/Linux 9 (stretch) 9 (stretch)
    CPU: (2) x64 AMD EPYC 7571
    Memory: 508.70 MB / 7.72 GB
    Container: Yes
    Shell: 4.4.12 - /bin/bash
  Binaries:
    Node: 12.22.1 - /usr/local/bin/node
    Yarn: 1.22.5 - /usr/local/bin/yarn
    npm: 6.14.12 - /usr/local/bin/npm

Alpine

  System:
    OS: Linux 5.4 Alpine Linux
    CPU: (2) arm64 unknown
    Memory: 178.69 MB / 952.52 MB
    Container: Yes
    Shell: 1.31.1 - /bin/ash
  Binaries:
    Node: 12.22.1 - /usr/bin/node
    Yarn: 1.22.10 - /usr/bin/yarn
    npm: 6.14.12 - /usr/bin/npm
@lovell
Copy link
Owner

lovell commented Jun 19, 2021

Hi, I've been unable to reproduce this locally. Please can you provide sample images that trigger this problem.

Are you able to reproduce this when using non-SVG images? It's possible you might have run into libvips/libvips#2293

@scottbradybuckii
Copy link
Author

That was a good thought, because up until now I have been using an SVG as one of the images, but I just changed the sample code to compose the same jpg on itself and it still has the issue:

const watermark = fs.readFileSync(path.resolve('landscape.jpg'));
const background = fs.readFileSync(path.resolve('landscape.jpg'));

Here is the image I am using:
landscape

And here is a look at what docker stats shows once it finishes compositing all the images after changing the for loop to loop 1000 times (instead of 500):
Screenshot from 2021-06-22 10-13-38

And here is what docker stats shows immediately when I kill the program by pressing ctrl + c:
Screenshot from 2021-06-22 10-14-45

And it does really seem to be an issue with it actually holding on to some memory, because when I changed it to loop 2,000 times, it eventually hit 500MB of ram and slowed down to a crawl - taking 30 seconds to a minute to do each iteration of the loop

@lovell
Copy link
Owner

lovell commented Jun 26, 2021

Thank you for the update, I'm still unable to reproduce this based on the above image and code.

Here's what I see locally using the above image as in.jpg with a slightly modified version of your code that shows RSS every 10 seconds.

const fs = require('fs');
const path = require('path');
const sharp = require('sharp');

setInterval(() => {
  console.log(process.memoryUsage().rss);
}, 10000);

async function test() {
  const background = fs.readFileSync(path.resolve('in.jpg'));
  sharp.cache(false);

  for (let i = 0; i <= 2000; i++) {
    await sharp(background)
      .composite([{ input: background }])
      .toBuffer();
  }
}

test();
94154752
133918720
120516608
111300608
107692032
134144000
91254784
114487296
97988608
130732032
110534656
105365504
93372416
133812224
128036864
97951744
97951744
97951744
...

Given you're using Docker, please can you create a separate repo with a Dockerfile that allows someone else to reproduce the problem inside a container.

@lovell lovell added question and removed triage labels Jun 26, 2021
@scottbradybuckii
Copy link
Author

Thanks for the reply, it helped me track down what I can do to get around this issue for my use case!

This seems to be an issue on Node version 12 (I only checked 12, 14, and 16 just now - 14 and 16 did not have this issue). So there does seem to be some sort of memory leak on Node 12, but for my purposes, I can easily switch to either 14 or 16.

Here is a Dockerfile that should reproduce the issue for you:

FROM node:12
COPY . /
RUN yarn
CMD node index.js

index.js is the same as your last comment:

const fs = require('fs');
const path = require('path');
const sharp = require('sharp');

setInterval(() => {
	console.log(process.memoryUsage().rss);
}, 10000);

async function test() {
	const background = fs.readFileSync(path.resolve('in.jpg'));
	sharp.cache(false);

	for (let i = 0; i <= 2000; i++) {
		await sharp(background)
			.composite([{input: background}])
			.toBuffer();
	}
}

test();

package.json is just the sharp package:

{
  "dependencies": {
    "sharp": "^0.28.3"
  }
}

And the in.jpg is the same we have been using.

Then I run it using docker run -m 512m --rm -it $(docker build -q .). This should show a climbing number as it continues to run, and it will probably even crash towards the end. But if you just change the node version in the Dockerfile to 14 or 16, everything works as it should.

Thanks for helping me track down how I can fix this in my project. Would you like me to keep the issue open for Node 12?

@lovell
Copy link
Owner

lovell commented Jun 27, 2021

Thanks, there was a bug in Node.js 12 and earlier that prevented garbage collection from running when inside a cgroup-constrained container, which was fixed via nodejs/node#27508

@lovell lovell closed this as completed Jun 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants