Build Caffe on CentOS 6 with dependencies included

[cypof]

First draft of how we could get Caffe to run on every Linux distrib. cppbuild.sh now compiles all dependencies from source, and keeps the libs in cppbuild/lib for inclusion in the jar file. I only worked on x86-64.

If we run this on CentOS 6, which has the oldest glibc (2.12), it should run anywhere. There is only one additional dependency, but it is only needed at compile time for OpenBLAS:
sudo yum install gcc-gfortran

An easy way to try on a brand new CentOS is with docker:
docker run -it centos:6 bash

A strange thing is that one of the libraries seems to #define FLOAT, which breaks the compilation of the OpenCV headers in Caffe, so it is necessary to modify caffe/examples/cpp_classification/classification.cpp to add #undef FLOAT before the opencv headers (line 2).

[saudet]

Looking good! Thanks a lot! Though, if you look under the ffmpeg module, I keep the archive download and unarchiving separate from the build phase, because we can basically reuse the same source for all platforms. I'll probably check myself linux-x86 and macosx eventually if I have to, but organizing the script file in advance would help. Anyway, let me know when you have this tested and I'll merge what you have! Thanks

And to fix any compile errors we can also include patch files, but it's better if we figure out a way, for example, to not compile the examples. :)

[bhack]

How do you think we will handle dependencies modularity after BVLC/caffe#2523 and BVLC/caffe#2619 (android)?

[cypof]

@bhack Not sure, but if it is easy to embed them all in the jar that should do it for now.

[bhack]

Minimizing dependencies would be a great thing for apk size and forward pass only use in android. So the problem it is about how to compile preset with the minimal dependencies we want.

[cypof]

Ah OK, we should import the Makefile.config options from cppbuild.sh, and only build and package the corresponding libraries.

[bhack]

Yes or pilot the relative cmake flags. One positive thing of Cmake it is that support external git dependencies. We could add that in caffe directly.

[cypof]

Interesting, I will look at this.

[cypof]

@saudet I refactored things to be closer to ffmpeg, and added Mac support. It's not finished but most of it should be there.

[bhack]

I really hope that we can remove boost dependencies with BVLC/caffe#2537

[saudet]

Great, thanks @cypof! A couple of questions though. I understand that Caffe's build doesn't have a proper "make install", but for the other libraries, if we configure the builds with a prefix to, say, our local cppbuild directory, and run make install, that might neatly place all header files under include and the shared libraries in lib. Have you tested that and found it did not work properly?

And, just for reference, what are the issues with building HDF5 from source on Linux?

[cypof]

I tried to avoid make install because for many libraries it also installs the static one (.a), and symbolic links to the library version (like .so.8.0) which might not be handled correctly when unzipping the jar. Also some file systems have problems with links, like when virtualbox mounts a host folder in a VM. That's useful e.g. for when running docker.

[saudet]

The Builder of JavaCPP already takes care of not bundling symlinks, and the shared librairies are used by default on Linux, so there shouldn't be any problems using standard 'make install'...

[saudet]

So, would you have any more recent changes to the build script? Or should I go, merge this, and polish it on my end?

[cypof]

I will try to go back to that this weekend, or maybe next week. I have something a bit better.

[saudet]

libcaffe.so doesn't link with the static libraries and the dynamic ones don't get included in the JAR file, but otherwise seems to be working fine on my machine so I'll merge and finish the work on my end :) Thanks!!

[cypof]

Thanks. After playing a bit with it I feel it would be even better to separate the dependencies and caffe itself in different jars. This way we could use maven to download the dependencies and work on our own version of caffe without to worry about which version of library gets loaded, between the maven one and the locally compiled. It would also be great with the include-files-in-a-jar stuff I am working on. Over time the dependencies library might turn into full fledged presets.

[saudet]

Yes, lots of stuff to be done :)