Improve python wrapper

[shelhamer]

This work-in-progress seeks to polish the python wrapper, document, and include more detailed examples.

• standardize python mean to binaryproto mean 47ec9ac
• input preprocessing for Caffe (mean subtraction, scaling, channel order) 8da2a32, 96cd02d
• automagic input and output blobs (no matter the number) 56ca978
• batch inputs to net 1b23680
• friendly forward/backward to prepare blobs/diffs and unpack output 0e5a5cf, ac5e6fa
• handle ndarrays throughout, instead of lists 31907b5
• properly separate responsibilities between the pycaffe wrapper, the imagenet wrapper example (wrapper.py) and detection example (detector.py)
• rewrite examples for completeness to show how to format inputs, compute the forward pass, and read off outputs and extracted features
• raise exceptions instead of crashing the library. Done in part by 76c2554 thanks to @longjon–adopted for blob arg checking in 9d4324e
• document, with feeling: docstrings written. TODO: an introductory guide and discussion of blob wrapping / synced memory.
• fix handling of gpu memory seems like gpu blob access is alright for now (provided Forward or Backward are called, since they will trigger syncing) see Improve python wrapper #311 (comment)
• save example notebooks on durian for timing reference

[longjon]

Let's also make sure everything that should has a docstring once the interface is stable.

[shelhamer]

@longjon re: gpu memory, it seems it's not actually an issue. Although mutable_cpu_data() is always used, further calls to Caffe actions like Forward() will sync properly. I checked by zeroing out params in gpu mode–the change was seen in the forward pass.

I don't know what problem I had before. This seems fine, unless I'm tired and somehow missing the issue.

[longjon]

Yes, I can confirm that, e.g., this works in GPU mode:

net.params['fc8'][0].data[...] = 0
net.ForwardPrefilled()   # or your preferred forward call

and this fits my model of what SyncedMemory is doing. However, note that referential transparency is subtly violated:

p = net.params['fc8'][0].data
net.ForwardPrefilled()
p[...] = 0
net.ForwardPrefilled()   # uses unchanged parameters

which makes me a bit uncomfortable. However, getting blobs has the same issue. In fact, in this code data behaves like a value:

bl = net.blobs['fc8'].data
net.ForwardPrefilled()
# bl does not contain up-to-date fc8, and writing to it has no effect

but in this code like a reference:

bl = net.blobs['fc8'].data
net.ForwardPrefilled()
bl2 = net.blobs['fc8'].data
# bl now contains the same data as bl2, and writing to it works

I think it's fine and probably the best compromise to keep things like they are, but to maximize sanity one should consider data to be a reference that becomes invalid whenever forward or backward are called (and this is true even for params, which don't change on forward/backward).

[shelhamer]

@longjon Please review my proposed changes to the caffe.Net interface. @sergeyk Let me know what you think too.

Rewriting the rest of pycaffe (imagenet and detector wrappers + examples) according to the new interface will follow.

[longjon]

Really? See python.org, Python wiki page, etc. Just sayin'.

[shelhamer]

Ok, fair enough. Capital is fine and used throughout now.

[longjon]

Always the standard for Python is four-space indents (PEP8 or Google style guide).

[longjon]

Should the axis permutation also be part of this?

[shelhamer]

H x W x K is the scikit-image standard, so I'm comfortable leaving it out. It'd be easy to add on the model of the existing options if it turns out to be annoying.

[longjon]

Yep, H x W x K is pretty much the standard for images, I am thinking of non-image data. But I agree we don't need to be eager to add things that aren't used.

[longjon]

Some time ago I suggested (which I am now recalling and still partial to) moving the blob copying in forward from C++ to Python. This would mean that Net.forward could be simplified to (just sketching here)

for k, v in kwargs.iteritems():
    self.blobs[k].data[...] = np.asarray(v)
net._forward_prefilled()
# now deal with output

where CaffeNet.Forward goes away and CaffeNet.ForwardPrefilled gets renamed CaffeNet._forward_prefilled and need not be called by the user.

The advantages of doing it this way:

• replace a bunch of C++ shenanigans with a few lines of Python
• noncontiguous or non-float32 data are only copied once, whereas now they are copied twice
• the extra checks and C++ exceptions simply go away, the exception for wrongly-sized input is automatic

What do you think?

[longjon]

Final note for now, regarding the blobs= mechanism. I see four ways this could be accomplished:

• how it is now, specifying the blobs you want in addition to the output blobs as an argument
• return the output blobs if you don't ask for anything specifically, otherwise return exactly what you asked for
• always return output blobs only, the user can dig in net.blobs for anything else she wants
• provide a mechanism (to=?) to stop computation at some layer, and always return the last computed blobs

The fourth (and optionally the second?) saves time if the user wants some intermediate level features and not later ones. Of course, that option can be implemented in addition to one of the others.

And three related comments:

• the returned blobs have the caveats described above re: GPU mode (if we keep this let's document it)
• it might be nice to be scolded if passing in a blob that will be clobbered in computation
• I can't decide if the convenience justifies the grating-ness of blobs= + **kwargs. At least one can still run nets with blobs named blobs by filling in net.blobs oneself.

[shelhamer]

return the output blobs if you don't ask for anything specifically, otherwise return exactly what you asked for

This is equally reasonable, and one can ask for outputs + additional blobs by blobs=self.outputs + ['conv2', 'pool5'] or the like. For now I'm keeping it as it is, since the output blobs are always computed so they might as well be packaged in.

user can dig in net.blobs for anything else she wants

While true, it feels a bit awkward, and if we're going to have a convenience method let's make it convenient.

provide a mechanism (to=?) to stop computation at some layer, and always return the last computed blobs
[...]
saves time if the user wants some intermediate level features and not later ones

This is a good idea... for the future. For now the usual workaround of defining the decapitated net and running it is fine (ok, more like "barely tolerable").

the returned blobs have the caveats described above re: GPU mode (if we keep this let's document it)

I'm not sure how to fix it right now, so let's document it. We should make docs pages for the python wrapper and matlab wrapper in general with details like this.

it might be nice to be scolded if passing in a blob that will be clobbered in computation

Right now you're yelled at if you don't pass all the named input blobs. I'm happy with that (feel free to add further checks). Now it's going to yell whenever the args don't match the input blobs.

I can't decide if the convenience justifies the grating-ness of blobs= + **kwargs. At least one can still run nets with blobs named blobs by filling in net.blobs oneself.

I can relate–the collusion of named keywords + **kwargs feels like sin. But it's nice and I've made my peace.

[shelhamer]

moving the blob copying in forward from C++ to Python

This is an excellent idea. I'll do this, barring unforeseen issues.

[shelhamer]

Ready to brew with python.