-
-
Notifications
You must be signed in to change notification settings - Fork 935
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DNS cache (cacheable-lookup@^4.2.3
) causes a memory leak
#1220
Comments
I need these information:
Alternatively you can try this:
Ummm... OS shouldn't kill the process I think. On my machine the Node.js process usually crashes with a C++ stack trace dump. If available, please provide one.
Sure. |
I too experience memory issues (even with `pagination.stackAllItems: false) For exemple when iterating a large set of data while doing zero processing, node crashes:
|
I just found the same thing. Simply FOUND THE LEAK https://github.com/sindresorhus/got/blob/master/source/index.ts#L57 Having this code multiple times (like Jest tests) will lead to running out of memory. const CL = require('cacheable-lookup');
new CL(); |
Great job fixing the leak @szmarczak! The fixes aren't published yet so I just used the source from master https://github.com/szmarczak/cacheable-lookup with |
My case is not exactly similar. I hardly do require once or twice in code.
I already tried with 12.16.3.
I used cache option in one module and it's disabled in another module. Both seem to crash. The one with cache option crashes faster though.
I don't do any I have mailed you the heap stacks. |
Are you using |
Do you have |
No I don't. |
And @adityapatadia ? |
I neither use http2 nor pagination. I create new agents with keepalive options and use them for http and https. |
@PopGoesTheWza What Got version were you using when you got the dump? Edit: nvm, it was 11.1.0 |
Also, my dump may be more relevant to #1223 |
@adityapatadia What's the ratio of requests - memory usage? |
My production system runs from 20 requests per second to 60 requests per second. Memory usage gradually rises and stabilises at around 2.93 GB. Sometimes it crashes anytime after that. Let me put GOT 11 in my code which has more frequent crash issue and send you heap stacks. |
@PopGoesTheWza Can you try again with @adityapatadia Can you try again with these:
Make sure to include // ...
const CacheableLookup = require('cacheable-lookup');
const dnsCache = new CacheableLookup();
got('...', {dnsCache}); |
@szmarczak do the two test cases I submitted pass with Got@11.1.2 ? |
no |
Are you sure that the memory leak came from the hook duplicates? |
Without the kludgy hooks deduping, it would fail around 4500-5000 records :/ |
My use cases relies heavily on the pagination feature in fact. Otherwise I would probably not experience as much issues. Note that it is not a complain. Edit: my memory leak issue was due to bad definition of |
cacheable-lookup@^4.2.3
) causes a memory leak
I'm able to reproduce the memory leak. To reproduce:
const CacheableLookup = require('cacheable-lookup');
const cacheable = new CacheableLookup();
cacheable.servers = ['127.0.0.1'];
setInterval(() => {
cacheable.lookup(`${Math.random().toString().substr(2)}.test`, () => {});
}, 1); The memory usage slowly increases every minute. To cause an instant crash, replace |
I ran it for 12 minutes and got a crash:
|
If you run const CacheableLookup = require('.');
const cacheable = new CacheableLookup();
cacheable.servers = ['127.0.0.1'];
while (true) {
cacheable.lookup(`${Math.random().toString().substr(2)}.test`, console.log);
console.log(cacheable._cache.size, Object.keys(cacheable._pending).length, cacheable._nextRemovalTime);
} you will see some logs. It crashes in less than a minute. The first numbers shows how many entries are in the cache, the next one shows how many requests are pending, the next one tells if there's any timeout set to remove outdated entries. You'll see
or
This is definitely a bug in the DNS cache... |
Describe the bug
Actual behavior
When I use GOT v10, my production systems work perfectly fine but when I switch to v11 without any other change, the memory keeps increasing in a period of 6-7 hours and eventually, OS kills the process.
...
Expected behavior
Memory should not grow when GOT version is upgraded
...
Code to reproduce
I am no expert in debugging memory leak but I can privately provide my heap snapshot to help debug this.
Checklist
The text was updated successfully, but these errors were encountered: