1.27rc slow down after half a day

[xanatos]

Purely anecdotal. I've been trying the 1.27rc on an old machine I have together with the new Lighthouse 5.2.0 . I've rebuilt both dbs from scratch using the 1.26 . Then I switched to the 1.27rc. There are no running validators or other programs (but I use the machine for websurfing).

Note that the machine is "old" (Intel i5-8500, 32gb ram, Crucial P3 2TB m.2), AND I'm using Windows 11 23H2.

The 1.27rc is FAST. The MGas/s are very high (compared to my main validator). But tonight at around 3:28 in the morning it slowed down and it remained slow. I had to relaunch it this morning to have it return to its "best" values. In the EventViewer there was nothing "strange".

I've exctracted from the log files the Mgas/s and plotted them. Yellow part is the slow down. Green part is after relaunching.

I'm adding some screenshots made with the ProcessExplorer (one yesterday evening, one this morning) to compare the use of resources.

Yesterday:

This morning:

And I'm including a zip file with the logs.

mainnet.logs.zip

As I've said, this could be totally anecdotal. If in the next 24h it will repeat then I'll add a comment.

[xanatos]

While not in a so evident way, the moving average of the MGas/s seems to be still going down.

[LukaszRozmej]

Hi, yes it is a known issue on Windows (not replicating on Linux). We tracked it down to how RocksDb manages memory. It is generally not great and especially bad on windows. We are trying to build a custom RocksDb package with a changed allocator, but are having a bit of trouble getting everything linked statically not to have dependency issues. Hopefully we can resolve that in 1.28.

Another thread about changing the allocator is here: #6107.

[benaadams]

Partially resolved by #7327

[benaadams]