RFC: Tensorflow Core as a product #331

sun51 · 2020-11-20T23:39:01Z

Tensorflow Core as a product RFC

Status	Proposed
Authors	Yanhua Sun (yanhuasun@google.com)
Sponsor	Rohan Jain (rohanj@google.com)
Updated	2020-11-13

Objective

This RFC proposes Tensorflow core as a standalone product and describes the high level components and the principles behind it.

rfcs/20201113-tf-core.md

ebrevdo · 2020-11-30T22:45:36Z

rfcs/20201113-tf-core.md

+ExtensionType (as known as CompositeTensor, RFC): provides a pair of protocols for defining new types that can be directly supported by TensorFlow APIs.  (This API is also known as the "CompositeTensor" API). Example ExtensionType in core includes IndexSlices.
+TypeSpec:(go/tf-type-spec) provides a standard way to define and reason about TensorFlow-specific types that are supported by TensorFlow APIs. 
+
+Note: FullType (go/tf-fulltype) definitions to be added once the design is ready.


need a public link for fulltype

ebrevdo · 2020-11-30T22:47:59Z

rfcs/20201113-tf-core.md

+
+
+## Objective
+


This needs some more nuance.

When I first read "TF core" I thought it was about separating the C++ library from the python; but below it seems like you want to include some python API as part of core.

So for example when it comes to XLA/MLIR, either JITted or AOT (e.g., the functionality in saved_model_cli), do you plan to keep those in core, or do you expect that all XLA functionality be split off in separate symbolic libraries? If so, how do you expect to support jit_compile=True extension of tf.function?

+1 for separating C++ library from the Python one. The Python API might serve for prototyping purpose if we'd like to collect feedbacks from a broader audience though, e.g. https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/distribute/v1/all_reduce.py

I'll try and clarify the proposal a bit here. There are couple of directions this RFC is pushing towards

Python API modularity - Currently TF core includes Keras, tf.data, distribution strategies etc. without any clear layering and dependency structure. We want to define a small core on which all these libraries can be layered on top of. We don't plan to remove any APIs from TF or add too many new APIs etc.

C++ first API - We want to think of the "Core API" as primarily a C++ one with a thin (probably 1-1 mapping) layer of python on top. Users can choose to interact with either API. Arguably, one can think of this as an implementation detail but its a real important implementation detail that we wanted to bring folks attention to.

Concretely, the first step we're taking now is defining the boundaries of what this small core API surface is going to look like and ensuring that the other TF APIs (tf.data, distribute, keras etc.) are dependent on it (and not the other way). This involves removing circular dependencies, BUILD file refactoring, moving some files around, creating right extension points / interfaces in TF core with no API surface changes expected whatsoever.

@rohan100jain +1
If there is design meeting kindly let us know. I would like to join the call.

Yes, design review meeting is scheduled at 10AM PT Dec 15, 2020

Sorry, we need more time to figure out things better, the design review is cancelled.

bhack · 2020-12-01T00:14:40Z

rfcs/20201113-tf-core.md

+
+### Keys to get there
+
+Understand users’ needs


I think this will require a SIG for the core.

Or "users" here refer to the high-level API owners, tf-keras/data/distribute etc. then one could argue internal communication might be more effective/efficient at the initial stage.

Please involve SIGs leads at least.

bhack · 2020-12-01T00:23:38Z

rfcs/20201113-tf-core.md

+
+Performance
+
+TF Core will be extremely lightweight and performant. We’ll try to do as much computation as possible in C++ with a thin layer on top. For example, Python APIs will be one thin layer on top. This also involves building a clean C API layer where state of the art runtimes like TFRT and Graph Compilers like MLIR can plug in and improve performance. 


What Is the python layer here? A 1-to-1 wrapper? Or an higher level layer with a reduced surface?

We would like see layered Python APIs, with Keras, tf-data, distributed, etc built on core APIs and extension/composite mechanics. This core APIs are a reduced and complete surface. One high level API could be composed by a few core APIs.

bhack · 2020-12-01T00:25:15Z

rfcs/20201113-tf-core.md

+
+* Is this API serving any use cases?
+
+When we add/remove APIs, the first thing we should ask is ‘what is the use case for this APIs’. This API should serve users’ purpose, not developers’ purpose. 


I think that we cannot do too much claims for users without a regular SIG.

Also what about custom c++ ops in SIGS?
Can we really analyze composabilty in python space before evaluate a core API change/extension?

There are some essential and basic C++ Ops. Also a lot of custom c++ ops can be replaced with python code. currently one of main reason for these custom C++ ops is performance. We do have some plans to bring the python code performance closer to custom c++ op performance, so that custom c++ is not required, instead, same functions are implemented in the high level languages, this provides easier programability and flexibility to modify.

Yes this was my point and this seems to me to give a quite high responsibility to the core API set perimeter definition + compiler stack performance to not create a bottleneck.

quick question, when I add a new hardware device(TensorFlow modular plugin), for ABI stable, I need register Ops/Kernels with C API, How could register them into TF-Core? also , I see (https://github.com/tensorflow/community/pull/77)， if user need a new Python API(eg. for plugin device new Ops/Kernels), they need to interact TF-Core(C++ libraries) with C API, What do you think？

@sun51 In the meantime can you expose a little but more if and how this is going to interact with https://github.com/tensorflow/tensorflow/tree/master/tensorflow/compiler/mlir/tfr.
Cause I expect that reducing the core API surface is going to create more pressure over ecosystem repos that have the infrastructure for custom ops.

bhack · 2020-12-01T00:26:26Z

rfcs/20201113-tf-core.md

+  Are the error messages for the APIs easy to understand? Are the error messages clear enough to give the feedback on what is going wrong, and how to fix it? 
+
+* Is this API performant? 
+


On what reference runtime? TFRT?

A general suggestion is to avoid over-emphasizing "performant" without going into detailed analysis to the trade-offs, e.g. why this API design enables high performance implementation and its alternatives do not.

bhack · 2020-12-01T00:36:12Z

rfcs/20201113-tf-core.md

+
+* Is this API similar or redundant with existing ones? 
+
+Corresponding to the principle ‘Primitive/Essential/Orthogonality’: If one API can be written by using existing a few APIs together with mechanics core provides, then the API is not essential. Instead of including that particular APIs, the primitive APIs that API is composed of should be included in the core.


I think the "few" term Is different in the c++ domain instead of python domain

bhack · 2020-12-01T00:40:14Z

rfcs/20201113-tf-core.md

+
+### Functional components of TF core
+* Control flow: tf.cond, tf.while
+* tf.function: Trace compile tensorflow programs to accelerate them 


Can the quality of the compiler performance in a function have an impact on the evaluation about extending the core? Or we will be forwarded on compiler improvements?

bhack · 2020-12-01T13:17:31Z

Generally I think that this RFC will have some positive impacts if the SIGs leads will be involved in the process and not only in the RFC revision.

What I don't understand is if the scope of this RFC is to have a TF-core only repository in the near future maintained by a specific TF-core team.
This Is related to the C++/C/Python API relationship. Just to make some examples:

Keras Is a python only SiG but when It needs c++ ops are contributed in the TF monolitich repo as needed (e.g. image processing)
TF.data, if It will be a separare SIG, It requires c++
TF Addons and TFIO have custom ops when composabilty or tf.function is not enought to reach performance. But we know the portability ISSUES related to custom ops (Custom Op TPU custom-op#53)

Are we going in a TF world where many SIGs will Need to maintain its own c++/c API to offer High level python API for all the cases not covered by python composabilty on TF-core or when the compiler stack (tf.function) is not efficient enought?

Also, are we sure that we can still minimize duplications across the SIGs ecosystem if the "centralized" core is going to have a minimal surface?
I strongly belive that we will need a robust ecosystem review process with a smaller core proposal.

mihaimaruseac · 2020-12-01T23:35:03Z

~~There will be a design meeting for this on 15th of December at 10 am Pacific.~~

~~If you want to join, here is the Meet link~~

Edit: meeting has been postponed, see #331 (comment)

wangpengmit · 2020-12-08T21:32:55Z

rfcs/20201113-tf-core.md

+Corresponding to the principle ‘Primitive/Essential/Orthogonality’: If one API can be written by using existing a few APIs together with mechanics core provides, then the API is not essential. Instead of including that particular APIs, the primitive APIs that API is composed of should be included in the core.
+
+* Is this API easy to use?
+  Usability - representation


Some formatting (especially line breaks) are broken in this doc when rendered. For example, you need a blank line here to break the line.

wangpengmit · 2020-12-08T21:36:34Z

rfcs/20201113-tf-core.md

+* Testing: APIs and functions for users to test Tensorflow programs.
+* Basic Utilities: Essential and primitive utilities functions, APIs. 
+
+### Math/NN APIs


What about RNG ops (both stateless RNGs and tf.random.Generator)?

sun51 · 2020-12-14T16:07:23Z

There will be a design meeting for this on 15th of December at 10 am Pacific.

If you want to join, here is the Meet link

Sorry we need more time to figure out things better, the design review is cancelled for now, we will let you know if there is one later.

sun51 · 2020-12-14T16:09:05Z

Thanks everyone for the discussion and comments, we will need more time to figure out the design, so we will hold this design RFC and have cancelled the design review for now. Will update later.

bhack · 2020-12-14T16:23:18Z

Thanks everyone for the discussion and comments, we will need more time to figure out the design, so we will hold this design RFC and have cancelled the design review for now. Will update later.

Do you mean #331 (comment) ?

Craigacp · 2020-12-19T23:15:12Z

Would this TF-core have a C API as well as a C++ one, or is that planned to live elsewhere?

SidneyLann · 2021-04-18T11:51:04Z

Hope most things do in c/c++

ematejska · 2022-01-31T18:57:38Z

@rohan100jain Are we still pursuing this RFC?

bhack · 2022-05-03T14:56:38Z

It would be really useful if we could revamp and approve this RFC.

bhack · 2022-05-05T12:29:35Z

/cc @learning-to-play

bhack · 2022-07-22T11:29:17Z

Do we have an opportunity to re-evalute/refresh this and all the useful feedback we had collected?

learning-to-play · 2022-07-31T03:03:49Z

@joker-eph Do you think there is an opportunity to incorporate the collected feedback into the current refactoring projects? See @bhack's comment above.

Create 20201113-tf-core.md

b1bab03

sun51 requested review from ematejska, ewilderj, martinwicke and theadactyl as code owners November 20, 2020 23:39

google-cla bot added the cla: yes label Nov 20, 2020

ematejska changed the title ~~Create 20201113-tf-core.md~~ RFC: Tensorflow Core as a product Nov 30, 2020

ebrevdo reviewed Nov 30, 2020

View reviewed changes

rfcs/20201113-tf-core.md Outdated Show resolved Hide resolved

ebrevdo reviewed Nov 30, 2020

View reviewed changes

bhack reviewed Dec 1, 2020

View reviewed changes

angerson mentioned this pull request Dec 3, 2020

Guide for installing Tensorflow-GPU in WSL2 tensorflow/docs#1757

Closed

sun51 added 2 commits December 4, 2020 14:22

Update 20201113-tf-core.md

a27f17f

Update 20201113-tf-core.md

59d193c

wangpengmit reviewed Dec 8, 2020

View reviewed changes


		Performance

		TF Core will be extremely lightweight and performant. We’ll try to do as much computation as possible in C++ with a thin layer on top. For example, Python APIs will be one thin layer on top. This also involves building a clean C API layer where state of the art runtimes like TFRT and Graph Compilers like MLIR can plug in and improve performance.


		* Is this API serving any use cases?

		When we add/remove APIs, the first thing we should ask is ‘what is the use case for this APIs’. This API should serve users’ purpose, not developers’ purpose.

		Are the error messages for the APIs easy to understand? Are the error messages clear enough to give the feedback on what is going wrong, and how to fix it?

		* Is this API performant?


		* Is this API similar or redundant with existing ones?

		Corresponding to the principle ‘Primitive/Essential/Orthogonality’: If one API can be written by using existing a few APIs together with mechanics core provides, then the API is not essential. Instead of including that particular APIs, the primitive APIs that API is composed of should be included in the core.

RFC: Tensorflow Core as a product #331

Are you sure you want to change the base?

RFC: Tensorflow Core as a product #331

Conversation

sun51 commented Nov 20, 2020 • edited by ematejska Loading

Tensorflow Core as a product RFC

Objective

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bhack Dec 1, 2020 • edited Loading

Choose a reason for hiding this comment

bhack Dec 1, 2020 • edited Loading

Choose a reason for hiding this comment

bhack commented Dec 1, 2020

mihaimaruseac commented Dec 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sun51 commented Dec 14, 2020

sun51 commented Dec 14, 2020

bhack commented Dec 14, 2020

Craigacp commented Dec 19, 2020

SidneyLann commented Apr 18, 2021

ematejska commented Jan 31, 2022

bhack commented May 3, 2022

bhack commented May 5, 2022

bhack commented Jul 22, 2022

learning-to-play commented Jul 31, 2022

sun51 commented Nov 20, 2020 •

edited by ematejska

Loading

bhack Dec 1, 2020 •

edited

Loading

bhack Dec 1, 2020 •

edited

Loading

mihaimaruseac commented Dec 1, 2020 •

edited

Loading