[fix](memtracker) Refactor load channel mem tracker to improve accuracy #12791
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
morningman
reviewed
Sep 20, 2022
| @@ -78,11 +82,13 @@ class MemInfo { | |||
| static int64_t _s_mem_limit; | |||
| static int64_t _s_hard_mem_limit; | |||
| static size_t _s_allocator_physical_mem; | |||
| static size_t _s_pageheap_unmapped_bytes; | |||
There was a problem hiding this comment.
please update document
docs/zh-CN/docs/admin-manual/maint-monitor/monitor-metrics/metrics.md
There was a problem hiding this comment.
These detailed properties of tcmalloc are not added to metrics,
I will update memory related metrics later.
xinyiZzz
force-pushed
the
20220920_fix_load_flush
branch
from
609cff1
to
056b19f
Compare
xinyiZzz
force-pushed
the
20220920_fix_load_flush
branch
from
056b19f
to
cdfd84c
Compare
xinyiZzz
added a commit
to xinyiZzz/incubator-doris
that referenced
this pull request
Sep 24, 2022
…cy (apache#12791) The mem hook record tracker cannot guarantee that the final consumption is 0, nor can it guarantee that the memory alloc and free are recorded in a one-to-one correspondence. In the life cycle of a memtable from insert to flush, the memory free of hook is more than that of alloc, resulting in tracker consumption less than 0. In order to avoid the cumulative error of the upper load channel tracker, the memtable tracker consumption is reset to zero on destructor.
13 tasks
dataroaring
pushed a commit
that referenced
this pull request
Sep 25, 2022
…cy (#12791) (#12948) The mem hook record tracker cannot guarantee that the final consumption is 0, nor can it guarantee that the memory alloc and free are recorded in a one-to-one correspondence. In the life cycle of a memtable from insert to flush, the memory free of hook is more than that of alloc, resulting in tracker consumption less than 0. In order to avoid the cumulative error of the upper load channel tracker, the memtable tracker consumption is reset to zero on destructor.
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
…cy (apache#12791) The mem hook record tracker cannot guarantee that the final consumption is 0, nor can it guarantee that the memory alloc and free are recorded in a one-to-one correspondence. In the life cycle of a memtable from insert to flush, the memory free of hook is more than that of alloc, resulting in tracker consumption less than 0. In order to avoid the cumulative error of the upper load channel tracker, the memtable tracker consumption is reset to zero on destructor.
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
…cy (apache#12791) The mem hook record tracker cannot guarantee that the final consumption is 0, nor can it guarantee that the memory alloc and free are recorded in a one-to-one correspondence. In the life cycle of a memtable from insert to flush, the memory free of hook is more than that of alloc, resulting in tracker consumption less than 0. In order to avoid the cumulative error of the upper load channel tracker, the memtable tracker consumption is reset to zero on destructor.
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
…cy (apache#12791) The mem hook record tracker cannot guarantee that the final consumption is 0, nor can it guarantee that the memory alloc and free are recorded in a one-to-one correspondence. In the life cycle of a memtable from insert to flush, the memory free of hook is more than that of alloc, resulting in tracker consumption less than 0. In order to avoid the cumulative error of the upper load channel tracker, the memtable tracker consumption is reset to zero on destructor.
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
…cy (apache#12791) The mem hook record tracker cannot guarantee that the final consumption is 0, nor can it guarantee that the memory alloc and free are recorded in a one-to-one correspondence. In the life cycle of a memtable from insert to flush, the memory free of hook is more than that of alloc, resulting in tracker consumption less than 0. In order to avoid the cumulative error of the upper load channel tracker, the memtable tracker consumption is reset to zero on destructor.
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
…cy (apache#12791) The mem hook record tracker cannot guarantee that the final consumption is 0, nor can it guarantee that the memory alloc and free are recorded in a one-to-one correspondence. In the life cycle of a memtable from insert to flush, the memory free of hook is more than that of alloc, resulting in tracker consumption less than 0. In order to avoid the cumulative error of the upper load channel tracker, the memtable tracker consumption is reset to zero on destructor.
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
…cy (apache#12791) The mem hook record tracker cannot guarantee that the final consumption is 0, nor can it guarantee that the memory alloc and free are recorded in a one-to-one correspondence. In the life cycle of a memtable from insert to flush, the memory free of hook is more than that of alloc, resulting in tracker consumption less than 0. In order to avoid the cumulative error of the upper load channel tracker, the memtable tracker consumption is reset to zero on destructor.
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #xxx
Problem summary
The mem hook record tracker cannot guarantee that the final consumption is 0, nor can it guarantee that the memory alloc and free are recorded in a one-to-one correspondence.
In the life cycle of a memtable from insert to flush, the memory free of hook is more than that of alloc, resulting in tracker consumption less than 0.
In order to avoid the cumulative error of the upper load channel tracker, the memtable tracker consumption is reset to zero on destructor.
Checklist(Required)
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...