Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature-wip] (memory tracker) (step6, End) Fix some details #9301

Merged
merged 6 commits into from
May 10, 2022
Merged

[feature-wip] (memory tracker) (step6, End) Fix some details #9301

merged 6 commits into from
May 10, 2022

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Apr 28, 2022 •
edited
Loading

Proposed changes

Issue Number: close #7196

Problem Summary:

  1. Fix LoadTask, ChunkAllocator, TabletMeta, Brpc, the accuracy of memory track.
  2. Modified some MemTracker names, deleted some unnecessary trackers, and improved readability.
  3. More powerful MemTracker debugging capabilities.
  4. Avoid creating TabletColumn temporary objects and improve BE startup time by 8%.
  5. Fix some other details.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

For related tests of impact and accuracy, can refer to the previous pr: #9145

In future:

  1. More tests, more accurate statistics.
  2. Faster tracking, there is now a 2% performance penalty.
  3. Use MemTracker raw pointers to improve performance by 1%.
  4. Optimized to switch existing Tracker and Task Tracker in TLS.
  5. Supports memory leak monitoring.

@xinyiZzz xinyiZzz force-pushed the switch_tls_tracker6_fix_datils branch 2 times, most recently from a04f145 to f2a8708 Compare April 29, 2022 10:39
morningman
morningman reviewed Apr 30, 2022
View reviewed changes
be/src/exec/tablet_sink.cpp Outdated Show resolved Hide resolved
be/src/http/action/compaction_action.h Outdated Show resolved Hide resolved
be/src/olap/tablet_manager.cpp Outdated Show resolved Hide resolved
be/src/olap/tablet_manager.h Outdated Show resolved Hide resolved
be/src/olap/tablet_schema.cpp Outdated Show resolved Hide resolved
be/src/runtime/mem_pool.cpp Outdated Show resolved Hide resolved
be/src/runtime/mem_tracker.h Show resolved Hide resolved
be/src/runtime/mem_tracker.cpp Show resolved Hide resolved
be/src/runtime/tcmalloc_hook.h Outdated Show resolved Hide resolved
be/src/service/internal_service.cpp Outdated Show resolved Hide resolved
@xinyiZzz xinyiZzz force-pushed the switch_tls_tracker6_fix_datils branch 2 times, most recently from 22d4988 to 80f58df Compare May 6, 2022 03:13
@morningman morningman added kind/improvement area/memory-consumption dev/backlog waiting to be merged in future dev branch labels May 7, 2022
morningman
morningman previously approved these changes May 7, 2022
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 7, 2022
@github-actions
Copy link
Contributor

github-actions bot commented May 7, 2022

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 7, 2022

PR approved by anyone and no changes requested.

@xinyiZzz xinyiZzz force-pushed the switch_tls_tracker6_fix_datils branch from 28e4be0 to 5b56690 Compare May 9, 2022 02:04
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 9, 2022
@morningman morningman merged commit b34ed43 into apache:master May 10, 2022
zhengshiJ pushed a commit to zhengshiJ/incubator-doris that referenced this pull request May 13, 2022
@xinyiZzz
[feature-wip] (memory tracker) (step6, End) Fix some details (apache#…
3b3e65c 
…9301)

1. Fix LoadTask, ChunkAllocator, TabletMeta, Brpc, the accuracy of memory track.
2. Modified some MemTracker names, deleted some unnecessary trackers, and improved readability.
3. More powerful MemTracker debugging capabilities.
4. Avoid creating TabletColumn temporary objects and improve BE startup time by 8%.
5. Fix some other details.
starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request May 19, 2022
@xinyiZzz @starocean999
[feature-wip] (memory tracker) (step6, End) Fix some details (apache#…
0a5f18f 
…9301)

1. Fix LoadTask, ChunkAllocator, TabletMeta, Brpc, the accuracy of memory track.
2. Modified some MemTracker names, deleted some unnecessary trackers, and improved readability.
3. More powerful MemTracker debugging capabilities.
4. Avoid creating TabletColumn temporary objects and improve BE startup time by 8%.
5. Fix some other details.
englefly pushed a commit to englefly/incubator-doris that referenced this pull request May 23, 2022
@xinyiZzz
[feature-wip] (memory tracker) (step6, End) Fix some details (apache#…
d129d51 
…9301)

1. Fix LoadTask, ChunkAllocator, TabletMeta, Brpc, the accuracy of memory track.
2. Modified some MemTracker names, deleted some unnecessary trackers, and improved readability.
3. More powerful MemTracker debugging capabilities.
4. Avoid creating TabletColumn temporary objects and improve BE startup time by 8%.
5. Fix some other details.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
area/memory-consumption area/vectorization dev/backlog waiting to be merged in future dev branch kind/improvement reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature] Refactored memory statistics framework MemTracker
2 participants
@xinyiZzz @morningman