Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose IOBinding features via C/C++/C# language bindings. #4646

Merged
merged 52 commits into from
Aug 10, 2020
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
8506fc9
Implement Create/Release Allocator
yuslepukhin Jun 27, 2020
10833d8
Add C++ API for CreateAllocator/ReleaseAllocator
yuslepukhin Jun 27, 2020
83b7742
Move Allocator API up the structure.
yuslepukhin Jun 29, 2020
e308a9d
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Jun 29, 2020
39d33f9
Add IoBnding.
yuslepukhin Jul 1, 2020
50fdda1
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Jul 1, 2020
e4e39c9
Add Seesion:Run() with binding and tests.
yuslepukhin Jul 1, 2020
c3bb59a
Expose new allocator and io_binding api to c# nativemethods.
yuslepukhin Jul 2, 2020
ef5d5be
Do not use outside pointers for meminfo names as they may be shortlived.
yuslepukhin Jul 2, 2020
8503904
Convert MemoryInfo to a pubvlic class exposing properties
yuslepukhin Jul 3, 2020
e27c884
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Jul 13, 2020
10db4f6
Fix unused return value error.
yuslepukhin Jul 14, 2020
189300d
Implement MemoryInfo, MemoryAllocator and MemoryAllocation
yuslepukhin Jul 14, 2020
3184e82
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Jul 14, 2020
a243331
Add tests and make them run.
yuslepukhin Jul 14, 2020
90e1571
Introduce IOBinding and OrtValue
yuslepukhin Jul 16, 2020
618486d
Implement IoBinding and some auxiallry functionality
yuslepukhin Jul 16, 2020
96b522c
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Jul 17, 2020
80a7361
Adjust for the recent changes in Api global instance.
yuslepukhin Jul 17, 2020
29c1b8e
Introduce BindOutputToDevice API
yuslepukhin Jul 17, 2020
ae8e408
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Jul 23, 2020
a568fa4
Some refactoring.
yuslepukhin Jul 24, 2020
eea3851
Imlpement GetBoudnOutputNames and test it.
yuslepukhin Jul 24, 2020
0b22c98
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Jul 27, 2020
202d9e9
Fix docs and merge issues.
yuslepukhin Jul 27, 2020
0ea6132
Implement GetBoundOutputValues and test.
yuslepukhin Jul 27, 2020
2910a0c
Finish implementation of OrtIoBinding
yuslepukhin Jul 28, 2020
61b48f8
Complete testing.
yuslepukhin Jul 28, 2020
c7e466e
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Jul 28, 2020
764d401
Fix build failures, fix some leaks.
yuslepukhin Jul 28, 2020
224e7e7
Adjust some protos.
yuslepukhin Jul 29, 2020
e011a01
Address some review comments. Fix CI build failures.
yuslepukhin Jul 29, 2020
dddd368
Advance ptr when creating output values.
yuslepukhin Jul 29, 2020
38c3686
Save work before Bitlocker restart.
yuslepukhin Jul 29, 2020
39b78f8
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Jul 31, 2020
cb536fd
Add necessary member functions to MemoryInfo.
yuslepukhin Jul 31, 2020
9a1c2cd
Address review comments.
yuslepukhin Aug 1, 2020
b438547
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Aug 1, 2020
48aecca
Fix centos compilation.
yuslepukhin Aug 1, 2020
b4c5d23
Fix up info comparision for CPU.
yuslepukhin Aug 3, 2020
53dd860
Move factory methods to OrtValue
yuslepukhin Aug 3, 2020
df825e4
Don't throw out of destructor
yuslepukhin Aug 3, 2020
084a85e
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Aug 3, 2020
b76f3b9
Fix CI issues.
yuslepukhin Aug 4, 2020
a755a08
Refactor inferencing Run() methods.
yuslepukhin Aug 4, 2020
dc0e730
Minor fixes.
yuslepukhin Aug 4, 2020
83b28ea
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Aug 4, 2020
db70346
Fix encoding conversion and OrtIoBinding.GetOutputNames()
yuslepukhin Aug 5, 2020
0d1fbc8
Fix Run() bugs.
yuslepukhin Aug 5, 2020
4710ee8
Move some unsafe test code to the main project so it can be tested th…
yuslepukhin Aug 6, 2020
bcf368a
Merge branch 'master' into yuslepukhin/io_binding
yuslepukhin Aug 10, 2020
facd52a
Fix up merge leftovers
yuslepukhin Aug 10, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 30 additions & 48 deletions csharp/src/Microsoft.ML.OnnxRuntime/DisposableNamedOnnxValue.cs
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

using Microsoft.ML.OnnxRuntime.Tensors;
using System;
using System.Buffers;
using System.Collections.Generic;
using Microsoft.ML.OnnxRuntime.Tensors;
using System.Runtime.InteropServices;


namespace Microsoft.ML.OnnxRuntime
Expand All @@ -30,27 +29,23 @@ protected virtual void Dispose(bool disposing)
{
if (disposing)
{
// TODO: dispose managed state (managed objects).
for (int i = 0; i < this.Count; i++)
// Dispose in the reverse order.
// Objects should typically be destroyed/disposed
// in the reverse order of its creation
// especially if the objects created later refer to the
// objects created earlier. For homogeneous collections of objects
// it would not matter.
for (int i = this.Count - 1; i >= 0; --i)
{
this[i]?.Dispose();
}
this.Clear();
}

// TODO: free unmanaged resources (unmanaged objects) and override a finalizer below.
// TODO: set large fields to null.

disposedValue = true;
}
}

~DisposableList()
{
// Do not change this code. Put cleanup code in Dispose(bool disposing) above.
Dispose(false);
}

// This code added to correctly implement the disposable pattern.
public void Dispose()
{
Expand Down Expand Up @@ -116,22 +111,17 @@ internal static DisposableNamedOnnxValue CreateTensorFromOnnxValue(string name,

/* Get Tensor element type */ //TODO: Assumed value is Tensor, need to support non-tensor types in future
IntPtr typeAndShape = IntPtr.Zero;
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetTensorTypeAndShape(nativeOnnxValue, out typeAndShape));
TensorElementType elemType = TensorElementType.DataTypeMax;
try
{
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetTensorTypeAndShape(nativeOnnxValue, out typeAndShape));
unsafe
{
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetTensorElementType(typeAndShape, new IntPtr(&elemType)));
}

IntPtr el_type;
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetTensorElementType(typeAndShape, out el_type));
elemType = (TensorElementType)el_type;
}
finally
{
if (typeAndShape != IntPtr.Zero)
{
NativeMethods.OrtReleaseTensorTypeAndShapeInfo(typeAndShape);
}
NativeMethods.OrtReleaseTensorTypeAndShapeInfo(typeAndShape);
}

switch (elemType)
Expand Down Expand Up @@ -182,19 +172,14 @@ internal static DisposableNamedOnnxValue CreateTensorFromOnnxValue(string name,

internal static DisposableNamedOnnxValue CreateFromOnnxValue(string name, IntPtr nativeOnnxValue)
{
IntPtr allocator = IntPtr.Zero;
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetAllocatorWithDefaultOptions(out allocator));
var ret = CreateFromOnnxValue(name, nativeOnnxValue, allocator);
return (DisposableNamedOnnxValue)ret;
return CreateFromOnnxValue(name, nativeOnnxValue, OrtAllocator.DefaultInstance);
}

internal static DisposableNamedOnnxValue CreateFromOnnxValue(string name, IntPtr nativeOnnxValue, IntPtr allocator)
internal static DisposableNamedOnnxValue CreateFromOnnxValue(string name, IntPtr nativeOnnxValue, OrtAllocator allocator)
{
OnnxValueType onnxValueType;
unsafe
{
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetValueType(nativeOnnxValue, new IntPtr(&onnxValueType)));
}
IntPtr valueType;
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetValueType(nativeOnnxValue, out valueType));
OnnxValueType onnxValueType = (OnnxValueType)valueType;
switch (onnxValueType)
{
case OnnxValueType.ONNX_TYPE_TENSOR:
Expand All @@ -207,28 +192,31 @@ internal static DisposableNamedOnnxValue CreateFromOnnxValue(string name, IntPtr
for (int i = 0; i < count.ToInt32(); i++)
{
IntPtr nativeOnnxValueSeq;
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetValue(nativeOnnxValue, i, allocator, out nativeOnnxValueSeq));
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetValue(nativeOnnxValue, i, allocator.Pointer, out nativeOnnxValueSeq));
sequence.Add(CreateFromOnnxValue(string.Empty, nativeOnnxValueSeq, allocator));
}
return new DisposableNamedOnnxValue(name, sequence, OnnxValueType.ONNX_TYPE_SEQUENCE, TensorElementType.DataTypeMax, null);

case OnnxValueType.ONNX_TYPE_MAP:
IntPtr typeAndShape = IntPtr.Zero;
IntPtr nativeOnnxValueMapKeys = IntPtr.Zero;
IntPtr nativeOnnxValueMapValues = IntPtr.Zero;
TensorElementType elemType = TensorElementType.DataTypeMax;
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetValue(nativeOnnxValue, 0, allocator, out nativeOnnxValueMapKeys));
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetValue(nativeOnnxValue, 1, allocator, out nativeOnnxValueMapValues));
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetTensorTypeAndShape(nativeOnnxValueMapKeys, out typeAndShape));
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetValue(nativeOnnxValue, 0, allocator.Pointer, out nativeOnnxValueMapKeys));
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetValue(nativeOnnxValue, 1, allocator.Pointer, out nativeOnnxValueMapValues));

unsafe
IntPtr typeAndShape = IntPtr.Zero;
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetTensorTypeAndShape(nativeOnnxValueMapKeys, out typeAndShape));
TensorElementType elemType = TensorElementType.DataTypeMax;
try
{
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetTensorElementType(typeAndShape, new IntPtr(&elemType)));
IntPtr el_type;
NativeApiStatus.VerifySuccess(NativeMethods.OrtGetTensorElementType(typeAndShape, out el_type));
elemType = (TensorElementType)el_type;
}
if (typeAndShape != IntPtr.Zero)
finally
{
NativeMethods.OrtReleaseTensorTypeAndShapeInfo(typeAndShape);
}

switch (elemType)
{
case TensorElementType.Int64:
Expand Down Expand Up @@ -317,12 +305,6 @@ protected virtual void Dispose(bool disposing)
}
}

~DisposableNamedOnnxValue()
yuslepukhin marked this conversation as resolved.
Show resolved Hide resolved
{
// Do not change this code. Put cleanup code in Dispose(bool disposing) above.
Dispose(false);
}

// This code added to correctly implement the disposable pattern.
public void Dispose()
{
Expand Down
5 changes: 0 additions & 5 deletions csharp/src/Microsoft.ML.OnnxRuntime/FixedBufferOnnxValue.cs
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,6 @@ protected virtual void Dispose(bool disposing)
}
}

~FixedBufferOnnxValue()
{
Dispose(false);
}

public void Dispose()
{
Dispose(true);
Expand Down
Loading