Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ONNX (v1.2.2) preset #547

Merged
merged 60 commits into from
Aug 4, 2018
Merged
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
79076dd
Initial commit for ONNX 1.0.1 JavaCPP Preset
EmergentOrder Feb 1, 2018
3e2e5d9
Remove default constructors, reinstate attributes declaration, fix En…
EmergentOrder Feb 16, 2018
37ec5fd
Fix get_all_schemas and EnforceConsumed and AllowConsumed
EmergentOrder Feb 16, 2018
60fc3ac
Fix FillUsing
EmergentOrder Feb 16, 2018
3ea1303
Fix access to TypeConstraintParam by restoring default constructor
EmergentOrder Mar 30, 2018
f071d63
Add LICENSE / Copyright and latest generated code
EmergentOrder Apr 3, 2018
ed8103f
Update pom.xml and generated code to JavaCPP 1.4.2-SNAPSHOT
EmergentOrder Apr 3, 2018
19df87c
Merge branch 'master' of https://github.com/bytedeco/javacpp-presets
EmergentOrder Apr 3, 2018
51da3a9
Add ONNX preset to .travis.yml, install miniconda in Travis, patch sc…
EmergentOrder Apr 7, 2018
8fb5ea9
Add onnx project to top-level pom.xml; patch the correct schema.h
EmergentOrder Apr 8, 2018
2a711c4
Add onnx module for linux-x86_64 to pom.xml
EmergentOrder Apr 9, 2018
9e99098
Remove use of sudo from cppbuild.sh, put onnx .so in the right place …
EmergentOrder Apr 10, 2018
90c7f17
Merge branch 'master' into master
EmergentOrder Apr 10, 2018
6eb8cb3
Explicitly use python3
EmergentOrder Apr 11, 2018
2c7829e
Patch ONNX build script to use python3 as well
EmergentOrder Apr 12, 2018
6657ac7
put miniconda on PATH in cppbuild.sh just to be sure
EmergentOrder Apr 12, 2018
d2e3ba6
Use python2 instead of python3, try installing miniconda from within …
EmergentOrder Apr 13, 2018
5e1e366
attempt to temporarily disable other Travis builds
EmergentOrder Apr 13, 2018
81fc397
temporarily disable Appveyor builds, install correct protobuf with pr…
EmergentOrder Apr 14, 2018
97277b0
Merge branch 'master' into master
EmergentOrder Apr 14, 2018
ea0a777
export CFLAGS so protobuf headers are found; remove onnx section from…
EmergentOrder Apr 15, 2018
96ff7af
Merge branch 'master' of https://github.com/EmergentOrder/javacpp-pre…
EmergentOrder Apr 15, 2018
f807074
Account for older GCC version in Travis; Switch to protobuf3 which ca…
EmergentOrder Apr 15, 2018
1b48fe3
fix path
EmergentOrder Apr 15, 2018
6c8b0de
bump protobuf version
EmergentOrder Apr 15, 2018
cdd3cf9
add setup.py patchfile
EmergentOrder Apr 15, 2018
76a91f4
another CFLAG
EmergentOrder Apr 15, 2018
5d90c05
try to update Travis's gcc
EmergentOrder Apr 15, 2018
858bb58
try to get a newer gcc
EmergentOrder Apr 15, 2018
be52a71
use centos 7
EmergentOrder Apr 15, 2018
f504ed5
fix
EmergentOrder Apr 15, 2018
f56889f
fix
EmergentOrder Apr 15, 2018
332c93b
revert to protobuf2
EmergentOrder Apr 15, 2018
b738c56
fix
EmergentOrder Apr 16, 2018
742745b
add autogen.sh.patch
EmergentOrder Apr 16, 2018
4fd1d76
link protobuf
EmergentOrder Apr 16, 2018
3dadeeb
link to protobuf from JavaCPP
EmergentOrder Apr 16, 2018
3f71ef9
fix
EmergentOrder Apr 16, 2018
7e7bc95
use LIBRARY_PATH instead of LD_LIBRARY_PATH
EmergentOrder Apr 16, 2018
b0ec9a1
clean up
EmergentOrder Apr 16, 2018
b2299fc
more cleanup
EmergentOrder Apr 16, 2018
9fad665
restore appveyor and travis builds; update generated code
EmergentOrder Apr 16, 2018
eb63f55
Install miniconda to instead of /home/laptx
EmergentOrder Apr 18, 2018
b1952cd
Merge branch 'master' into master
EmergentOrder Apr 18, 2018
c3c2a61
Remove unneeded installation of Miniconda
saudet Apr 30, 2018
765ad5e
Merge remote-tracking branch 'upstream/master'
saudet May 2, 2018
7c5d20a
Install Python 3.5 on Travis CI for CentOS 7
saudet May 2, 2018
7f76f73
Fix installation for Python 3.5 on CentOS 7
saudet May 3, 2018
3f58a76
Merge branch 'master' into master
saudet May 23, 2018
72b9f86
Merge branch 'master' into master
EmergentOrder Jun 18, 2018
f03580f
Use Protobuf 3.3.2
EmergentOrder Jun 18, 2018
b3d02fa
Merge branch 'master' of https://github.com/EmergentOrder/javacpp-pre…
EmergentOrder Jun 18, 2018
ef45c79
Bump ONNX version to v1.1.2
EmergentOrder Jun 19, 2018
5825cc8
Merge branch 'master' into master
EmergentOrder Jul 5, 2018
7689163
Bump ONNX version to 1.2.2
EmergentOrder Jul 6, 2018
6a05a63
Merge branch 'master' into master
saudet Aug 1, 2018
3198726
Fix, clean up, add README.md with example, and adjust license
saudet Aug 1, 2018
70caffc
Merge branch 'master' into master
saudet Aug 2, 2018
93402b0
Fix build script and presets
saudet Aug 3, 2018
862477e
Merge branch 'master' into master
EmergentOrder Aug 3, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,10 @@ matrix:
env: PROJ=tensorflow OS=android-x86_64
install: true
script: ./ci/install-travis.sh
- os: linux
env: PROJ=onnx OS=linux-x86_64
install: true
script: ./ci/install-travis.sh

- os: osx
osx_image: xcode7.3
Expand Down
2 changes: 1 addition & 1 deletion ci/install-travis.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ if [ "$TRAVIS_OS_NAME" == "osx" ]; then export JAVA_HOME=$(/usr/libexec/java_hom
if [[ "$OS" == "linux-x86" ]] || [[ "$OS" == "linux-x86_64" ]] || [[ "$OS" =~ android ]]; then
CENTOS_VERSION=6
SCL_ENABLE="rh-maven33 python27"
if [[ "librealsense chilitags llvm caffe mxnet tensorflow tensorrt ale skia " =~ "$PROJ " ]] || [[ "$OS" =~ android ]]; then
if [[ "librealsense chilitags llvm caffe mxnet tensorflow tensorrt ale skia onnx " =~ "$PROJ " ]] || [[ "$OS" =~ android ]]; then
CENTOS_VERSION=7
SCL_ENABLE="rh-maven33"
fi
Expand Down
12 changes: 12 additions & 0 deletions onnx/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Open Neural Network Exchange

Copyright (c) Facebook, Inc. and Microsoft Corporation.
All rights reserved.

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the ""Software""), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
14 changes: 14 additions & 0 deletions onnx/autogen.sh.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
--- protobuf-2.6.1/autogen.sh 2014-10-20 20:06:06.000000000 -0400
+++ autogen.sh 2018-04-15 19:38:24.032970600 -0400
@@ -19,8 +19,9 @@
# directory is set up as an SVN external.
if test ! -e gtest; then
echo "Google Test not present. Fetching gtest-1.5.0 from the web..."
- curl http://googletest.googlecode.com/files/gtest-1.5.0.tar.bz2 | tar jx
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably fixed with recent versions of protobuf. Let's try to get that running maybe? Please show me the kind of errors you get with 3.5.x.

- mv gtest-1.5.0 gtest
+ wget https://github.com/google/googletest/archive/release-1.5.0.zip
+ unzip release-1.5.0.zip
+ mv googletest-release-1.5.0 gtest
fi

set -ex
72 changes: 72 additions & 0 deletions onnx/cppbuild.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
#!/bin/bash
# This file is meant to be included by the parent cppbuild.sh script
if [[ -z "$PLATFORM" ]]; then
pushd ..
bash cppbuild.sh "$@" onnx
popd
exit
fi

if [[ $PLATFORM == windows* ]]; then
#No Windows support yet
:
else

export PROTO=2.6.1
export INSTALL_PATH=`pwd`/$PLATFORM
wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O ~/miniconda.sh
bash ~/miniconda.sh -f -b -p $HOME/miniconda
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Miniconda need to be installed in the $HOME directory? What happens if we place it in $INSTALL_PATH?

export PATH="$HOME/miniconda/bin:$PATH"
conda install -y -c conda-forge protobuf=2.6.1 numpy setuptools
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to create files in $HOME as hinted by the uninstallation procedure?
https://conda.io/docs/user-guide/install/linux.html#uninstalling-anaconda-or-miniconda
If we cannot get this to live somewhere else than $HOME, let's let users install it by themselves.
That is, place these commands in install-travis.sh for the CI servers as well as document them here:
https://github.com/bytedeco/javacpp-presets/wiki/Build-Environments#linux-x86-and-x86_64

export CFLAGS="-I$HOME/miniconda/include"
export CONDA_PREFIX="$HOME/miniconda"
wget https://github.com/onnx/onnx/archive/v1.0.1.tar.gz

mkdir -p $PLATFORM
cd $PLATFORM
wget https://github.com/google/protobuf/archive/v$PROTO.tar.gz
mv v$PROTO.tar.gz protobuf-$PROTO.tar.gz
tar -xvf protobuf-$PROTO.tar.gz

export LIBRARY_PATH=`pwd`/lib
export PATH=$PATH:`pwd`/bin
export CFLAGS="-I`pwd`/include"
export CXXFLAGS="-I`pwd`/include"
cd protobuf-$PROTO

patch ./autogen.sh ../../../autogen.sh.patch
./autogen.sh
./configure "--prefix=$INSTALL_PATH" CFLAGS=-fPIC CXXFLAGS=-fPIC
make -j `nproc`
make install

cd ..
tar -xzvf ../v1.0.1.tar.gz
cd onnx-1.0.1/third_party/pybind11
wget https://github.com/pybind/pybind11/archive/v2.2.1.tar.gz
tar -xzvf v2.2.1.tar.gz
mv pybind11-2.2.1/* .
rm -r pybind11-2.2.1/
cd ../..

#to build with "Traditional ML" support. Untested.
#export ONNX_ML=1
export BASEDIR=build/temp.linux-x86_64-2.7/`pwd`/onnx/

python setup.py install --single-version-externally-managed --record=record.txt
g++ -v -std=c++11 -shared -Wl,-soname,libonnx.so -fPIC -o libonnx.so $BASEDIR/onnx-operators.pb.o $BASEDIR/onnx.pb.o $BASEDIR/checker.o $BASEDIR/defs/schema.o $BASEDIR/defs/tensor/old.o $BASEDIR/defs/tensor/defs.o $BASEDIR/defs/generator/defs.o $BASEDIR/defs/math/defs.o $BASEDIR/defs/data_type_utils.o $BASEDIR/defs/traditionalml/defs.o $BASEDIR/defs/experiments/defs.o $BASEDIR/defs/nn/defs.o $BASEDIR/defs/nn/old.o $BASEDIR/defs/reduction/defs.o $BASEDIR/defs/logical/defs.o $BASEDIR/defs/rnn/defs.o -pthread
cd ..

cd include
mkdir defs
mkdir onnx
cd ..

cp onnx-1.0.1/onnx/*.h include/onnx/
cp onnx-1.0.1/onnx/defs/*.h include/defs/
#TODO: Fix so the workaround isn't needed here
#next line to workaround by commenting out parts of schema.h that cause failures
patch include/defs/schema.h ../../schema.h.patch
cp onnx-1.0.1/libonnx.so lib
fi
cd ..
64 changes: 64 additions & 0 deletions onnx/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp-presets</artifactId>
<version>1.4.2-SNAPSHOT</version>
</parent>

<groupId>org.bytedeco.javacpp-presets</groupId>
<artifactId>onnx</artifactId>
<version>1.0.1-1.4.2-SNAPSHOT</version>
<packaging>jar</packaging>
<name>JavaCPP Presets for onnx</name>

<dependencies>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp</artifactId>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
</configuration>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp</artifactId>
<configuration>
<compilerOptions>
<compilerOption>-std=c++11</compilerOption>
<compilerOption>-fpermissive</compilerOption>
</compilerOptions>
</configuration>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
</plugin>
<plugin>
<artifactId>maven-source-plugin</artifactId>
</plugin>
<plugin>
<artifactId>maven-javadoc-plugin</artifactId>
</plugin>
</plugins>
</build>

</project>
61 changes: 61 additions & 0 deletions onnx/schema.h.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
--- cppbuild/linux-x86_64/onnx-1.0.1/onnx/defs/schema.h 2018-01-25 21:25:37.000000000 -0500
+++ schema.h 2018-04-07 10:36:14.887478089 -0400
@@ -311,6 +311,7 @@

// Type constraint.
struct TypeConstraintParam {
+ TypeConstraintParam(): description("default description") {}
TypeConstraintParam(
const std::string& type_param_str_,
const std::vector<std::string>& allowed_type_strs_,
@@ -389,6 +390,7 @@
int since_version() const {
return since_version_;
}
+
const std::map<std::string, Attribute>& attributes() const {
return attributes_;
}
@@ -427,9 +429,9 @@
int max_output() const {
return max_output_;
}
- std::pair<UseType, int> consumed(int i) const {
- return consumed_(i);
- }
+// std::pair<UseType, int> consumed(int i) const {
+// return consumed_(i);
+// }

private:
friend class OpSchemaRegistry;
@@ -448,8 +450,10 @@
std::string doc_;
// Default domain value ("") means it's ONNX domain.
std::string domain_ = "";
+
std::map<std::string, Attribute> attributes_{};
bool allows_unchecked_attributes_ = false;
+
std::vector<FormalParameter> inputs_;
std::vector<FormalParameter> outputs_;
std::vector<TypeConstraintParam> type_constraint_params_;
@@ -577,9 +581,9 @@
}
}

- private:
+// private:
// OpSchemaRegistry should not need to be instantiated.
- OpSchemaRegistry() = delete;
+// OpSchemaRegistry() = delete;

/**
* @brief Returns the underlying string to OpSchema map.
@@ -623,5 +627,5 @@
op_schema_register_once##name) = OpSchema(#name, __FILE__, __LINE__)

// Helper function
-size_t ReplaceAll(std::string& s, const char* from, const char* to);
+//size_t ReplaceAll(std::string& s, const char* from, const char* to);
} // namespace onnx
Loading