WIP: Independent child process monitoring #118 #134

bbengfort · 2017-03-18T20:45:44Z

This pull request is a start at resolving #118 -- as mentioned in that issue, I've created a function _get_child_memory that iterates through all child processes yielding their memory. This is summed if the include_children flag is true.

I've also updated the reference from #71 to wrap the entire yield in try/except and yield 0.0 if the race condition occurs; but this needs to be checked.

Including this into the main mprof library is the next step. I've created a demo mpmprof which logs child and main process memory separately in th same .dat file, then plots them correctly. Here is an example:

The next steps are to merge mpmprof with mprof with some flag, e.g. independent_children or something like that (suggestions welcome) to make it behave like mpmprof and to have the plot command correctly handle both cases.

bbengfort · 2017-03-18T20:49:38Z

memory_profiler.py

-            raise NotImplementedError('The psutil module is required when to'
-                                      ' monitor memory usage of children'
-                                      ' processes')
+            raise NotImplementedError((


Sorry about this, I think this was just a carry over from the historical merge; I can reset back to the original if needed.

bbengfort · 2017-03-18T20:51:51Z

mpmprof

+            lines += 1
+
+            # Collect memory usage of spawned children and write to profile
+            for idx, mem in enumerate(mp._get_child_memory(proc.pid)):


So here is the line where I independently collect the memory of each child and add them to the .dat file with the "CHLD#" key. If include_children=True then "main" will be the total memory consumed, and if not, it will be only the main process, but all processes will be plotted with independent_children=True or whatever flag we come up with.

bbengfort · 2017-03-18T20:52:26Z

mpmprof

+        mpoint = (0, 0)
+
+        # Plot all of the series, the main process and the child.
+        for proc, data in series.items():


And then obviously, plot needs to be adapted to plot all the children.

bbengfort · 2017-03-18T20:54:28Z

mpmprof uses argparse e.g. #128

fabianp · 2017-03-18T21:10:30Z

Thanks @bbengfort! . Do you think it is possible to have only one executable file, i.e., merge mpmprof into mprof?

bbengfort · 2017-03-19T10:24:40Z

@fabianp yes, it's possible; just not very straightforward. I'll take a crack at it.

fabianp · 2017-03-20T12:35:59Z

Great!

…

On 19 Mar 2017, at 11:24, Benjamin Bengfort ***@***.***> wrote: @fabianp <https://github.com/fabianp> yes, it's possible; just not very straightforward. I'll take a crack at it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/fabianp/memory_profiler/pull/134#issuecomment-287606977>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQ8h8D_Fjoqwm0monZTeaGq2RMoQTR2ks5rnQJpgaJpZM4MhiXU>.

bbengfort · 2017-03-20T13:41:08Z

@fabianp ok, I've pushed a version that does it - would you mind taking a look and running your tests/examples?

fabianp · 2017-03-20T15:11:45Z

Great work @bbengfort ! The code looks great, let me try it on some of my own problems. In the meantime, you can add your name to the Authors part of the README :-)

bbengfort · 2017-03-20T15:37:55Z

@fabianp thanks! I've updated the README with the example and usage and also added max value markers to the plot for the largest child.

fabianp · 2017-03-20T16:38:35Z

If I run the example with mprof run -M multiprocessing_example.py then it fails with an _pickle.PicklingError exception. It works however with mprof run -M python multiprocessing_example.py (not the python after -M). However, both should give the same result as the python mode should be picked up by the .py extension.

fabianp · 2017-03-20T15:40:11Z

README.rst

+the total memory of the program as well as each child individually.
+
+.. warning:: currently the child tracking only works if a ``stream`` is provided to the ``profile`` (e.g. from the command line or in the decorator).
+


How hard do you think it would be to remove this limitation?, i.e., to track children even if the function is not @Profile'd

fabianp · 2017-03-20T16:39:50Z

README.rst

+the total memory of the program as well as each child individually.
+
+.. warning:: Currently the child tracking only works if a ``stream`` is provided to the ``profile`` (e.g. from the command line or in the decorator). If you are using the API to retrieve values then the flag will not do anything.
+


I am confused, as the example works fine even without decorated functions

I meant that the stream argument has to be passed in, if stream = None then I can't return multiple values. I could if you wanted me to return a list instead of a single float. I didn't necessarily mean it wouldn't work with the decorator.

Perhaps I could rephrase as follows:

"Currently tracking individual children requires a reporting output stream; if you'd like direct access to the memory usage of individual children, see the _get_child_memory function."

Though of course, that then points the user to an internal function. I'm open to suggestions.

I've put placeholder comments in the code (line 363 and 403) where this issue occurs.

Ah, thanks for the explanation. I think the API would be simpler if we allow memory_usage to return a nested list instead embedding this into the stream. What do you think?

Ok, I've updated memory_usage to return a nested list and updated the README accordingly.

bbengfort · 2017-03-20T16:58:13Z

Unfortunately, I think that the pickle error is a product of multiprocessing library: functions can only be pickled if they're defined at the top level and when you run it by creating a subprocess in mprof, the example function is no longer at the top level and therefore cannot be pickled.

I can see if I can change the example (though that may be difficult), but I also wanted to note that the problem exists even with the original version of the mprof, so it's not related to the code changes I proposed here. It's likely that any user that uses the multiprocessing library would have to use the python command to ensure their code is serialized/forked correctly.

http://stackoverflow.com/questions/8804830/python-multiprocessing-pickling-error

bbengfort · 2017-03-20T16:59:12Z

memory_profiler.py

                else:
                    ret.append(mem_usage)
+                    if multiprocess:


Here is the first place where I don't return multiple values and instead warn

bbengfort · 2017-03-20T16:59:22Z

memory_profiler.py

                else:
                    ret.append(mem_usage)
+
+                    if multiprocess:
+                        warnings.warn("use include_children not multiprocess without a stream")


Here is the second place where I don't return multiple values and instead warn

fabianp · 2017-03-20T21:29:20Z

For the pickling, what I find strange is that both commands (with and without the "python" word in mprof) should be identical, as per lines 217-223 of mprof. Not sure why multiprocessing fails only on one of these cases since they should be executed the same way.

bbengfort · 2017-03-21T12:13:51Z

So I see what you mean, however there is a subtle difference. The args to Popen if options.python = True are:

['python', '-m', 'memory_profiler', '--timestamp', '-o', 'mprofile_20170321080505.dat', 'examples/multiprocessing_example.py']

Whereas the args to Popen if passed as an executable are

`['python', 'examples/multiprocessing_example.py']`

In the first case, __main__ is memory_profiler.__main__ and examples/multiprocessing_example.py is run with execfile() on L1108 or exec() on L1119. This is correctly implemented (as discussed here) by embedding the exec into its own namespace (ns), thus making the exec'd file the equivalent of an import -- as a result, the functions inside examples/multiprocessing_example.py are not at the top level (but are nested under ns) and therefore cannot be pickled.

So that took me a while to figure out, but I think it makes sense. What do you think?

fabianp · 2017-03-21T20:43:58Z

Thanks for the explanation and your time. I've merged the PR and added a small patch (52c5788) so that the example can be run without the python keyword.

Thanks!

bbengfort · 2017-03-21T20:52:44Z

Sure thing, let me know if there is anything else I can help with; sorry that this feature took so long!

petroslamb · 2017-06-12T13:56:46Z

memory_profiler.py

-                except psutil.NoSuchProcess:
-                    # https://github.com/fabianp/memory_profiler/issues/71
-                    pass
+                mem +=  sum(_get_child_memory(process, meminfo_attr))


Hi, quick question, as processes in modern linux share common memory after forking from their parent and only copy it on writes, wouldn't this sum report the wrong number?

Unfortunately I'm not sure of the behavior of psutil with regards to linux forks; I simply used the same meminfo_attr as the summation function. If that's true then this is indeed a problem, but I'd guess that we'd have to dig into psutil to figure it out.

fabianp · 2017-06-12T20:30:59Z

It would indeed be great to know (but I'm clueless here) On Jun 12, 2017 12:09 PM, "Benjamin Bengfort" <notifications@github.com> wrote: *@bbengfort* commented on this pull request. ------------------------------ In memory_profiler.py <https://github.com/fabianp/memory_profiler/pull/134#discussion_r121500525>:

@@ -111,12 +143,7 @@ def ps_util_tool():

else 'get_memory_info' mem = getattr(process, meminfo_attr)()[0] / _TWO_20 if include_children: - try: - for p in process.children(recursive=True): - mem += getattr(p, meminfo_attr)()[0] / _TWO_20 - except psutil.NoSuchProcess: - # https://github.com/fabianp/memory_profiler/issues/71 - pass + mem += sum(_get_child_memory(process, meminfo_attr)) Unfortunately I'm not sure of the behavior of psutil with regards to linux forks; I simply used the same meminfo_attr as the summation function. If that's true then this is indeed a problem, but I'd guess that we'd have to dig into psutil to figure it out. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <https://github.com/fabianp/memory_profiler/pull/134#discussion_r121500525>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQ8h-Xtff8eOlmZVoMv3dePOSis4Wziks5sDYzxgaJpZM4MhiXU> .

bbengfort added 4 commits July 15, 2016 17:31

multiprocessing example

0d956a1

child process memory now recorded seperately in mpmprof

a846fa6

plot function for mpmproc

e32679b

merge with upstream

bdf9995

bbengfort commented Mar 18, 2017

View reviewed changes

merge mpmprof into mprof

aab15ed

bbengfort added 2 commits March 20, 2017 11:34

added max child usage marker and update readme

f1cee23

fixed image

099c5c8

fabianp requested changes Mar 20, 2017

View reviewed changes

bbengfort commented Mar 20, 2017

View reviewed changes

bbengfort added 2 commits March 21, 2017 07:59

return child as nested list from memory_usage

9c62f0a

minor update to readme

d333fda

fabianp merged commit d333fda into pythonprofilers:master Mar 21, 2017

rokroskar mentioned this pull request Apr 6, 2017

Improve child handling #140

Open

petroslamb reviewed Jun 12, 2017

View reviewed changes

petroslamb mentioned this pull request Jun 12, 2017

mprof each child process independently #118

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Independent child process monitoring #118 #134

WIP: Independent child process monitoring #118 #134

bbengfort commented Mar 18, 2017

bbengfort Mar 18, 2017

bbengfort Mar 18, 2017

bbengfort Mar 18, 2017

bbengfort commented Mar 18, 2017

fabianp commented Mar 18, 2017

bbengfort commented Mar 19, 2017

fabianp commented Mar 20, 2017 via email

bbengfort commented Mar 20, 2017

fabianp commented Mar 20, 2017

bbengfort commented Mar 20, 2017

fabianp commented Mar 20, 2017

fabianp Mar 20, 2017

fabianp Mar 20, 2017

bbengfort Mar 20, 2017 •

edited

Loading

bbengfort Mar 20, 2017

fabianp Mar 20, 2017

bbengfort Mar 21, 2017

bbengfort commented Mar 20, 2017

bbengfort Mar 20, 2017

bbengfort Mar 20, 2017

fabianp commented Mar 20, 2017

bbengfort commented Mar 21, 2017

fabianp commented Mar 21, 2017

bbengfort commented Mar 21, 2017

petroslamb Jun 12, 2017

bbengfort Jun 12, 2017

fabianp commented Jun 12, 2017 via email

		the total memory of the program as well as each child individually.

		.. warning:: currently the child tracking only works if a ``stream`` is provided to the ``profile`` (e.g. from the command line or in the decorator).

WIP: Independent child process monitoring #118 #134

WIP: Independent child process monitoring #118 #134

Conversation

bbengfort commented Mar 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bbengfort commented Mar 18, 2017

fabianp commented Mar 18, 2017

bbengfort commented Mar 19, 2017

fabianp commented Mar 20, 2017 via email

bbengfort commented Mar 20, 2017

fabianp commented Mar 20, 2017

bbengfort commented Mar 20, 2017

fabianp commented Mar 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bbengfort Mar 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bbengfort commented Mar 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabianp commented Mar 20, 2017

bbengfort commented Mar 21, 2017

fabianp commented Mar 21, 2017

bbengfort commented Mar 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabianp commented Jun 12, 2017 via email

bbengfort Mar 20, 2017 •

edited

Loading