mprof each child process independently

[bbengfort]

Moved here as a feature request from the following SO question:

http://stackoverflow.com/questions/38358881/how-to-profile-multiple-subprocesses-using-python-multiprocessing-and-memory-pro

The mprof script allows you to track memory usage of a process over time, and includes a -C flag which will also sum up the memory usage of all child processes (forks) spawned by the primary process.

Instead of summation, I would like the mprof script to include a flag that will identify each process by pid in the generated .dat file, allowing the plot command to visualize each process' memory usage independently of each other, over time.

[bbengfort]

I've created a proof of concept in my fork thanks to @fabianp pointing out the _get_memory function. I've taken out the parts of this function that summed the child memory to the main process using psutil, and moved it to its own function, _get_child_memory which you can see on L76. This is called from _get_memory on L119.

The problem is that I don't know where I should adapt the memory_usage function to include children. I was thinking of doing logging similar to how the FUNC objects are logged. I created a proof of concept of how this would work in another executable script mpmprof. The while loop in this file is an adaptation of the loop on L297 in memory_usage specifically for a Popen object. It changes how it writes to mprofile_ dat files to include child processes as follows:

CMDLINE python examples/multiprocessing_example.py
MEM 0.207031 1468627462.8746
MEM 7.730469 1468627462.9888
CHLD0 17.343750 1468627462.9992
CHLD1 86.722656 1468627462.9992
CHLD2 17.359375 1468627462.9992
CHLD3 84.859375 1468627462.9992
MEM 7.730469 1468627463.0995
CHLD0 17.343750 1468627463.1084
CHLD1 154.644531 1468627463.1084

I then also created an adapted plot function to visualize this new mprofile file, which produces results as follows from profiling the examples/multiprocessing_example.py:

I'm not sure if there are any tests for this - but make test seems to work for the changes I made. I guess the question now is how should we integrate this work into the primary memory_profiler API, if at all.

[fabianp]

Hi, Thanks for looking into this. Could you make a pull request so I can see the diff and comment on your code? Thanks

[bbengfort]

@fabianp - the only diff is a small function in memory_profiler; otherwise I put the proof of concept in a new file called mpmprof -- are you sure you want me to pull request this?

[fabianp]

Ideally it would be a switch (like --include-children) for mprof that prints the different children instead of adding them. Do you think this is possible?

[bbengfort]

I can certainly look into doing this, but I wanted to avoid that as a start because I'd have to make changes at so many points in the original code. I'm pretty busy these past few weeks, but hopefully I can get a chance to spin back to this next week.

[cachedout]

👍 on this. I need this ability quite badly as well and would be willing to help out with development.

[davidgbe]

Did this ever get resolved?

[fabianp]

No. I'm happy to include it if someone sends me a pull request but I won't work on it for the moment.

[bbengfort]

Sorry - I just haven't had a chance to get into it. The proof of concept is still there, let me see if I can get a pull request together real quick.

[petroslamb]

Apologies for the direct approach, I wanted to point out a possible misreport on the the total memory usage when multiprocessing.

My thought was that I could save some time from everyone by using this open issue. I can go into more details or a new issue, if this does not suffice.

My report is on https://github.com/fabianp/memory_profiler/blob/master/memory_profiler.py#L146

I added a comment on the MR that seems to introduce it: https://github.com/fabianp/memory_profiler/pull/134#pullrequestreview-43443387

Thanks

[bbengfort]

My thought is that this is a new issue and potentially doesn't matter to being able to plot things independently; rather if there is an issue it needs to be fixed in _get_child_memory() and this change will ensure that the plotting functions work as expected.

@fabianp what do you think; can we close this and start a new issue?

[fabianp]

Indeed, I agree

…

On Jun 12, 2017 12:18 PM, "Benjamin Bengfort" ***@***.***> wrote: My thought is that this is a new issue and potentially doesn't matter to being able to plot things independently; rather if there is an issue it needs to be fixed in _get_child_memory() <https://github.com/fabianp/memory_profiler/blob/master/memory_profiler.py#L89> and this change will ensure that the plotting functions work as expected. @fabianp <https://github.com/fabianp> what do you think; can we close this and start a new issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/fabianp/memory_profiler/issues/118#issuecomment-307891979>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQ8h3cXcvrjoJN1GMKcvEKLoTOa-imdks5sDY7rgaJpZM4JMjxT> .

[bbengfort]

@petroslamb I'm going to go ahead and close this issue; do you want to create the new issue regarding the memory reporting?