DenseNet feature pyramid computation #307

moskewcz · 2014-04-08T19:33:56Z

DenseNet PR current state / TODOs (for discussion)

explanatory tech report:

DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
Forrest Iandola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer

additional integration notes (see main notes below inlined from the DENSENET_MERGE_TODO file)

last rebase was a while ago now; there were some delays getting our tech report polished up
in the mean time, some work on replacements for the GPL'd code is in progress; see:
- BLF: https://github.com/moskewcz/boda/blob/master/src/blf_pack.cc
- CNN padding/support/sizes calcs: https://github.com/moskewcz/boda/blob/master/src/conv_util.cc
- hosting repo / framework: https://github.com/moskewcz/boda/

inlined from: DENSENET_MERGE_TODO

list of the issues blocking the merge of the DenseNet feature branch:

critical
- replacement of GPL'd code (including removal from history)
- update build process / Makefile to match current practice
- tests
- trivial: remove DenseNet README.md header
- trivial: remove this todo file
unclear neccessity, semantics, and/or priority
- general cleanup of commit sequence (probably mostly squashing)
- input interface changes (i.e. jpeg filname as input -> ?)
- output interface changes (?, but probably something: image support size, multiple layer output, alignment, etc.)
- if still reading image files after any iface changes and removal of GPL code, use XX instead of YY

…and its python iface

…ake pycaffe'

…ors (as Sergey suggested)

…. works.

…ew data type is dict['feat'] = list of feature scales

…thon user

-- note: this commit almost certainly breaks compilation of matcaffe. mwm

…caffe output in matlab wrapper. mwm fix matlab vs octave compile stuff for matlab pyramid API

…-> caffe/src/stitch_pyramid

…to mean; corner regions still TODO

…rk; other minor changes. -- fill corners of padding with interpolation of edge padding (which is in turn already an interpolation from the image edge to the imagenet mean). -- add .PHONY stitch target to top level makefile for building just libStitchPyramid -- minor fix to testing makefile: add a -L. to src/stitch_pyramid/build/Makefile -- update matcaffe.cpp comment with current mkoctfile-based build command (can be used to test building matcaffe under octave) -- condense str() in featpyramid_common.hpp (str() is for debugging printfs) -- add a copy of str() in JPEDPyramid.cpp (FIXME: use some common copy?) mwm

…ject*) casts (to fix compile errors for some build envs). note also that there is a new SHARED_LDFLAGS makefile var that can be set to include -Wl,--no-undefined (for gcc) to avoid accidentally linking an .so missing some of its dependencies. but, since people use other compilers, it's not enabled by default, and only the stitch library build line uses the macro at this point. mwm

…port note that matcaffe was broken (would compile, but would have incorrect behavior) when feat_min{Width,Height} were added these since the function that should have been taking them has 5 int args, all with defaults, and new args were added in middle of the arg list.

… matlab. also, commenting out InitGoogleLogging() in matcaffe until we decide how to avoid multiple calls to InitGoogleLogging

…ong. at a minimum, maybe we should have a way to disable trying to init logging (i.e. as a param to caffe('init')) as a workaround for the multiple-library usage case?

…ed image.bits().

…the brave)

…d' for building pycaffe

…on/license note there too.

Added some DenseNet API documentation. This will probably percolate from this top-level README.md to the caffe.berkeleyvision.org gh-pages.

shelhamer · 2014-04-08T20:15:28Z

Hey Matt, I'm excited to see this code go public! However, you should close this PR and open one against the dev branch instead. See step 4 of https://help.github.com/articles/creating-a-pull-request. Thanks.

moskewcz · 2014-04-08T20:54:10Z

duh. will do. new PR is #308

On Tue, Apr 8, 2014 at 1:15 PM, Evan Shelhamer notifications@github.comwrote:

Hey Matt, I'm excited to see this code go public! However, you should
close this PR and open one against the dev branch instead. See step 4 of
https://help.github.com/articles/creating-a-pull-request. Thanks.

Reply to this email directly or view it on GitHubhttps://github.com//pull/307#issuecomment-39896265
.

amiltonwong · 2014-04-15T04:23:19Z

Hi, Matt,

Could you provide the link for downloading the DenseNet code? I was
overwhelmed and couldn't find the DenseNet code location.

Thanks~

Milton

2014-04-09 3:33 GMT+08:00 moskewcz notifications@github.com:

DenseNet PR current state / TODOs (for discussion) explanatory tech
report:

DenseNet: Implementing Efficient ConvNet Descriptor Pyramidshttp://arxiv.org/abs/1404.1869
Forrest Iandola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor
Darrell, and Kurt Keutzer
additional integration notes (see main notes below inlined from the
DENSENET_MERGE_TODO file)

last rebase was a while ago now; there were some delays getting our
tech report polished up

in the mean time, some work on replacements for the GPL'd code is in
progress; see:

BLF: https://github.com/moskewcz/boda/blob/master/src/blf_pack.cc

CNN padding/support/sizes calcs:
https://github.com/moskewcz/boda/blob/master/src/conv_util.cc

hosting repo / framework: https://github.com/moskewcz/boda/

inlined from: DENSENET_MERGE_TODO

list of the issues blocking the merge of the DenseNet feature branch:

critical
- replacement of GPL'd code (including removal from history)
- update build process / Makefile to match current practice
- tests
- trivial: remove DenseNet README.md header
- trivial: remove this todo file
-

unclear neccessity, semantics, and/or priority
- general cleanup of commit sequence (probably mostly squashing)
- input interface changes (i.e. jpeg filname as input -> ?)
- output interface changes (?, but probably something: image
support size, multiple layer output, alignment, etc.)
- if still reading image files after any iface changes and removal
of GPL code, use XX instead of YY

You can merge this Pull Request by running

git pull https://github.com/forresti/caffe featpyra_int_rc

Or view, comment on, or merge it at:

#307
Commit Summary

import of ffld patchwork code; initial skeleton of pyramid stitching
and its python iface

stitching 2000x2000 planes from within pycaffe.cpp

designing jpeg pyramid -> caffe copying stuff

setting up input and output blobs for featpyramid

getting stitched planes and visualizing them

add DenseNet prototxt files (from the future, without padding layers)

creating multiple scales / planes and returning them to python

add disabled code for rand padding scales inside planes

remove old featpyramid test; keeping the multiscale test

convinced the make system to also build PyramidStitcher when we do
'make pycaffe'

add STITCHPYRAMID_{SRC,HDRS} to Makefile (but no usage yet)

use single imagenet mean pixel for data centering in feature pyramid

streamlining upsampling/downsampling code. going to return scale
factors (as Sergey suggested)

new downsampling param passing looks good.

remove Eigen debris

testing boost::python functonality of returning dictionaries from
C++. works.

now returning a DICT instead of a LIST from extract_features(). the
new data type is dict['feat'] = list of feature scales

avoid having 'scales' go out of scope...

returning scales that make sense

cleanup. not officially supporting 'return unstitched features' to
python user

add disabled code for writing pyramid to image files

-- initial work on matlab exports for feature pyramid extraction.

-- use p_vect_float instead of mxArray for buffering feature pyramid
caffe output in matlab wrapper.

moved stitch_pyramid from caffe/python/caffe/imagenet/stitch_pyramid
-> caffe/src/stitch_pyramid

fixed demo

rename extract_featpyramid -> convnet_featpyramid in matlab API

partially implimented padding: background->mean; edges linear interp
to mean; corner regions still TODO

add interpolation-to-mean to corner padding for image pyramid
patchwork; other minor changes.

pycaffe/stitchpyramid related makefile tweaks; add missing
(PyArrayObject*) casts (to fix compile errors for some build envs).

indentation and comments

add visualization to demo

setting up code to rip a bbox out of a pyra

running hog pyra for regression test

testing scale calc

bbox slicing from pyra looks reasonable now. (in hog space... will
test in convnet space soon)

working on only using pyramids and no on-demand features in DPM
training. root_model() now updates spos{} to 'know' its root filter shape

first pass at precomputing features instead of warping

calling precompute_gt_bbox_features() from the voc5 training code

change default plane size in featpyramid matlab demo to 1100x1100.

looks like there's systematic shift of bboxes slightly left from
where they should be. (when rounding)

centering approx bbox on each exemplar in hog space. (previously,
was anchored to top left)

centering hog features, this time with ceil() instead of round().
also, looking at results without truncation features.

python API now returns imwidth, imheight of input image

setting up shared DenseNet_Params class for both the matlab and
python APIs to share

factoring shared matlab+python code up to featpyra_common.cpp

convnet_featpyramid matlab function takes parameters as second arg

changes to convnet_featpyramid matlab interface

remove test_io function from matcaffe

convnet_featpyramid matlab iface: add imwidth and imheight to output

convnet_featpyramid matlab iface: add placeholder feat_padx and
feat_pady fields to output (always set = 1)

convnet_featpyramid matlab/python iface changes

adding the ability to ignore big scales if they don't fit in planes

solving problem of 'some pyra scales are too big for planes.'
pruning scales that are too big. added some additional error checking too.

add a matcaffe debugging printout (only printed prior to assert
failure).

pruning scales that are smaller than template size

debugging voc5 + densenet

assert that individual images fit in the planes

getting 33% ap on inria with densenet + 1component.

add layer_strides() vector/accessor to net.{cpp,h}

weaken too-strong assertion in PyramidStitcher.cpp on {x,y}Max
versus plane size

rename convnet_subsampling_ratio -> sbin

use get_sbin(net_) to get sbin in pycaffe and matcaffe.

matcaffe: init google logging with "matcaffe" as name

makefile: remove unneeded -UNDEBUG, add commented
-DENABLE_ALLOC_TRACE example cflags

enabling user-configurability of 'minimum desired scale' in the form
of feat_minHeight,feat_minWidth. this is now usable in the the Python
wrapper. We should add it to Matlab wrapper soon.

add timers

swap matcaffe demo plane size 2000 -> 1100 (again)

matcaffe: almost-minimal commit to fix/add feat_min{Width,Height}
support

matcaffe demo: set interval=10 (the default) to agree with comment

remove unneeded and unused defaults from stitch_pyramid() decl

forgot to initialize img_minHeight_. fixed.

improve google logging init for matcaffe but note that it is still
wrong.

better scheme for convnet_featpyramid to unpack user's params

testing out the 'user-selectable min image scale' stuff in python
and matlab. also, commenting out InitGoogleLogging() in matcaffe until we
decide how to avoid multiple calls to InitGoogleLogging

simplified demo

handling 'pull bbox out of pyra' for boxes that are near the edge.

fix paths in demo

tweaking demos; looking at receptive fields

remove 'keyboard' call

automating pascal evaluation, and setting up feature caching

fleshing out cached feature functionality

sped stitch_pyramid up a bit (maybe 2x?) by avoiding calls to
uninlined image.bits().

tweaks to model and training script

various cpu mode tweaks

less restrictive check on filename

pyramid padding calculation

Makefile: add mkoctfile_mat rule to build .mex file with octave (for
the brave)

fix Makefile to use $(SHARED_LDFLAGS) instead of a hard-coded
'-shared' for building pycaffe

add DENSENET_MERGE_TODO and add DenseNet header to README.md

update README.md with DenseNet arXiv paper link; fix/polish
attribution/license note there too.

File Changes

A DENSENET_MERGE_TODOhttps://github.com/DenseNet feature pyramid computation #307/files#diff-0(15)

M Makefile https://github.com/BVLC/caffe/pull/307/files#diff-1(43)

M README.md https://github.com/BVLC/caffe/pull/307/files#diff-2(14)

A include/caffe/featpyra_common.hpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-3(79)

A include/caffe/imagenet_mean.hpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-4(18)

M include/caffe/net.hpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-5(3)

A matlab/caffe/convnet_featpyramid.mhttps://github.com/DenseNet feature pyramid computation #307/files#diff-6(44)

A matlab/caffe/convnet_featpyramid_cache.mhttps://github.com/DenseNet feature pyramid computation #307/files#diff-7(120)

A matlab/caffe/featpyramid_matcaffe_demo.mhttps://github.com/DenseNet feature pyramid computation #307/files#diff-8(62)

A matlab/caffe/get_featureSlice.mhttps://github.com/DenseNet feature pyramid computation #307/files#diff-9(97)

M matlab/caffe/matcaffe.cpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-10(326)

A matlab/caffe/pascal_000001.jpghttps://github.com/DenseNet feature pyramid computation #307/files#diff-11(0)

A matlab/caffe/test_get_featureSlice.mhttps://github.com/DenseNet feature pyramid computation #307/files#diff-12(50)

A python/caffe/featpyramid_demo.pyhttps://github.com/DenseNet feature pyramid computation #307/files#diff-13(73)

A python/caffe/imagenet/featpyramid_tests.pyhttps://github.com/DenseNet feature pyramid computation #307/files#diff-14(101)

A
python/caffe/imagenet/imagenet_rcnn_batch_1_input_1100x1100_output_conv5.prototxthttps://github.com/DenseNet feature pyramid computation #307/files#diff-15(208)

A
python/caffe/imagenet/imagenet_rcnn_batch_1_input_2000x2000_output_conv5.prototxthttps://github.com/DenseNet feature pyramid computation #307/files#diff-16(208)

A python/caffe/imagenet/pascal_009959.jpghttps://github.com/DenseNet feature pyramid computation #307/files#diff-17(0)

M python/caffe/pycaffe.cpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-18(330)

M src/caffe/net.cpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-19(1)

A src/stitch_pyramid/COPYING.txthttps://github.com/DenseNet feature pyramid computation #307/files#diff-20(674)

A src/stitch_pyramid/JPEGImage.cpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-21(390)

A src/stitch_pyramid/JPEGImage.hhttps://github.com/DenseNet feature pyramid computation #307/files#diff-22(113)

A src/stitch_pyramid/JPEGPyramid.cpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-23(232)

A src/stitch_pyramid/JPEGPyramid.hhttps://github.com/DenseNet feature pyramid computation #307/files#diff-24(106)

A src/stitch_pyramid/Patchwork.cpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-25(347)

A src/stitch_pyramid/Patchwork.hhttps://github.com/DenseNet feature pyramid computation #307/files#diff-26(120)

A src/stitch_pyramid/PyramidStitcher.cpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-27(95)

A src/stitch_pyramid/PyramidStitcher.hhttps://github.com/DenseNet feature pyramid computation #307/files#diff-28(45)

A src/stitch_pyramid/README.txthttps://github.com/DenseNet feature pyramid computation #307/files#diff-29(29)

A src/stitch_pyramid/Rectangle.cpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-30(151)

A src/stitch_pyramid/Rectangle.hhttps://github.com/DenseNet feature pyramid computation #307/files#diff-31(121)

A src/stitch_pyramid/SimpleOpt.hhttps://github.com/DenseNet feature pyramid computation #307/files#diff-32(1060)

A src/stitch_pyramid/build/Makefilehttps://github.com/DenseNet feature pyramid computation #307/files#diff-33(50)

A src/stitch_pyramid/build/demo.shhttps://github.com/DenseNet feature pyramid computation #307/files#diff-34(9)

A src/stitch_pyramid/build/stitch_pyramid_pascal.shhttps://github.com/DenseNet feature pyramid computation #307/files#diff-35(21)

A src/stitch_pyramid/imagenet_mean.hpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-36(1)

A src/stitch_pyramid/test_stitch_pyramid.cpphttps://github.com/DenseNet feature pyramid computation #307/files#diff-37(251)

Patch Links:

https://github.com/BVLC/caffe/pull/307.patch

https://github.com/BVLC/caffe/pull/307.diff

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/307
.

moskewcz · 2014-04-15T05:11:50Z

this PR is closed, you should be looking at DenseNet feature pyramid computation #308
there is no 'download' per-se, just this PR and the branch/fork is it from. the source branch for this (or any, AFAIK) PR is listed at the top of the PR (but is not hyperlinked). in this case it is:
forresti:featpyra_int_rc (here i've linked to to the tree view for that branch).

forresti and others added 30 commits March 18, 2014 14:01

import of ffld patchwork code; initial skeleton of pyramid stitching …

fc3503f

…and its python iface

add DenseNet prototxt files (from the future, without padding layers)

176b0cc

stitching 2000x2000 planes from within pycaffe.cpp

1821069

designing jpeg pyramid -> caffe copying stuff

f00eb3f

setting up input and output blobs for featpyramid

ecbd74e

getting stitched planes and visualizing them

f723879

creating multiple scales / planes and returning them to python

c005e6b

add disabled code for rand padding scales inside planes

96bfea8

remove old featpyramid test; keeping the multiscale test

4f25c7c

add STITCHPYRAMID_{SRC,HDRS} to Makefile (but no usage yet)

a99c574

convinced the make system to also build PyramidStitcher when we do 'm…

5944af5

…ake pycaffe'

use single imagenet mean pixel for data centering in feature pyramid

f76ca85

streamlining upsampling/downsampling code. going to return scale fact…

78fb085

…ors (as Sergey suggested)

new downsampling param passing looks good.

c05ae56

remove Eigen debris

d506d66

testing boost::python functonality of returning dictionaries from C++…

b1c06e0

…. works.

now returning a DICT instead of a LIST from extract_features(). the n…

7c1ea80

…ew data type is dict['feat'] = list of feature scales

avoid having 'scales' go out of scope...

0af3167

returning scales that make sense

45a327f

cleanup. not officially supporting 'return unstitched features' to py…

e904f25

…thon user

-- initial work on matlab exports for feature pyramid extraction.

178c693

-- note: this commit almost certainly breaks compilation of matcaffe. mwm

add disabled code for writing pyramid to image files

3decf2f

-- use p_vect_float instead of mxArray for buffering feature pyramid …

a2f3361

…caffe output in matlab wrapper. mwm fix matlab vs octave compile stuff for matlab pyramid API

moved stitch_pyramid from caffe/python/caffe/imagenet/stitch_pyramid …

30b8049

…-> caffe/src/stitch_pyramid

fixed demo

0cbbe39

rename extract_featpyramid -> convnet_featpyramid in matlab API

12e2fcd

partially implimented padding: background->mean; edges linear interp …

0e17940

…to mean; corner regions still TODO

add timers

b0af64c

moskewcz and others added 24 commits March 18, 2014 15:09

matcaffe demo: set interval=10 (the default) to agree with comment

0d1cfb2

remove unneeded and unused defaults from stitch_pyramid() decl

cbaea77

forgot to initialize img_minHeight_. fixed.

294054f

better scheme for convnet_featpyramid to unpack user's params

30d999a

testing out the 'user-selectable min image scale' stuff in python and…

42dde72

… matlab. also, commenting out InitGoogleLogging() in matcaffe until we decide how to avoid multiple calls to InitGoogleLogging

improve google logging init for matcaffe but note that it is still wr…

67ffe15

…ong. at a minimum, maybe we should have a way to disable trying to init logging (i.e. as a param to caffe('init')) as a workaround for the multiple-library usage case?

simplified demo

0c83487

handling 'pull bbox out of pyra' for boxes that are near the edge.

745c0f7

fix paths in demo

869ef4d

tweaking demos; looking at receptive fields

7c9966f

remove 'keyboard' call

41b3e9b

automating pascal evaluation, and setting up feature caching

1b634f8

fleshing out cached feature functionality

ea449b1

various cpu mode tweaks

4db11fa

sped stitch_pyramid up a bit (maybe 2x?) by avoiding calls to uninlin…

cd364b2

…ed image.bits().

tweaks to model and training script

819aaf4

less restrictive check on filename

8e8c3a0

pyramid padding calculation

25a399d

Makefile: add mkoctfile_mat rule to build .mex file with octave (for …

d208df7

…the brave)

fix Makefile to use $(SHARED_LDFLAGS) instead of a hard-coded '-share…

d342fca

…d' for building pycaffe

add DENSENET_MERGE_TODO and add DenseNet header to README.md

3c85b8f

update README.md with DenseNet arXiv paper link; fix/polish attributi…

bd3dfa8

…on/license note there too.

Update README.md

8717483

Added some DenseNet API documentation. This will probably percolate from this top-level README.md to the caffe.berkeleyvision.org gh-pages.

moskewcz mentioned this pull request Apr 8, 2014

Sliding Window, Varying input/output size and Dense, multiscale extraction #189

Closed

moskewcz closed this Apr 8, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DenseNet feature pyramid computation #307

DenseNet feature pyramid computation #307

moskewcz commented Apr 8, 2014

shelhamer commented Apr 8, 2014

moskewcz commented Apr 8, 2014

amiltonwong commented Apr 15, 2014

moskewcz commented Apr 15, 2014

DenseNet feature pyramid computation #307

DenseNet feature pyramid computation #307

Conversation

moskewcz commented Apr 8, 2014

DenseNet PR current state / TODOs (for discussion)

explanatory tech report:

additional integration notes (see main notes below inlined from the DENSENET_MERGE_TODO file)

inlined from: DENSENET_MERGE_TODO

shelhamer commented Apr 8, 2014

moskewcz commented Apr 8, 2014

amiltonwong commented Apr 15, 2014

moskewcz commented Apr 15, 2014