TensorFlow 2 support for win-64 nonexistent

[QuaternionsRock]

Unlike the package provided by Anaconda, this feedstock (1) is out-of-date (latest version is v2.5.0, not v2.4.1), and (2) does not support anything above TensorFlow v1.14 for win-64.

[maresb]

Hello @QuaternionsRock, we are well aware. Please see #111. We need some help from someone with Windows. Are you able to help?

[h-vetinari]

The main bottleneck here is that conda-forge is limited to what the free version of azure provides (and times out, meaning everything needs to be done by a core-member manually, rather than being able to distribute the work to the community with the normal pull request workflow), whereas Anaconda doesn't have such limits on their CI.

Any help (both on the recipe, and even more so on figuring a way around the CI limitations - see tensorflow/tensorflow#49555 & conda-forge/conda-forge.github.io#1272) is appreciated.

[gschramm]

I have access to a win64 machine (at work). So if someone provides me with detailed instructions on what to do, I could build a few of newer tensorflow packages manually.

[h-vetinari]

Very cool, thanks for the offer @gschramm!
For now, we could use some help with someone bashing on #111 until things run far enough that the only failure is a timeout. Then we could hopefully start building packages.

[mooch443]

I also have access to a win-64 machine (at home), and could profit from a working tensorflow2 win64 binary on conda-forge. What can I do?

[izahn]

@mooch443 I think the basic idea is just "try to get the recipe to build on windows". Are you familiar with building conda-forge packages in general? This one is like all the rest, except much bigger and more fragile :-(

[mooch443]

@izahn I am familiar with building my own program as a conda package, but I have not tried anything quite as big as tensorflow yet. Where best to start? I guess I'd clone the repo and see what happens when I try conda build.

[izahn]

I guess I'd clone the repo and see what happens when I try conda build.

Yeah, you might want to start with #111

Keep in mind that if it were easy someone would have done it already :-)

[bjarthur]

Keep in mind that if it were easy someone would have done it already :-)

what i don't understand, is that anaconda already did it, so can't conda-forge just copy from them?

[h-vetinari]

what i don't understand, is that anaconda already did it, so can't conda-forge just copy from them?

Because our artefacts are not necessarily compatible with each other (e.g. if we have different versions anywhere for ABI-relevant host dependencies; which we do, because we migrate faster than anaconda).

One of the strengths of conda-forge (and anaconda main) is that they ensure a consistent ecosystem where stuff doesn't break, but that means having to rebuild things, and tensorflow is too big to fit into public CI.

[bjarthur]

sorry if this is a dumb question, but i'm new around here.

what i meant, is copy their build process / recipe, not to rely on their binary dependencies.

[h-vetinari]

copy their build process / recipe

It's not a problem of the recipes, that we'd be able to solve. The problem is that conda-forge operates on azure's "free" CI, which does not have very powerful agents, and has a hard-wired 6h timeout, which is not enough to build tensorflow. Anaconda has more powerful machines, so they don't have this problem.

[ethanbb]

Am I understanding correctly that the bottleneck for having tf 2 on Windows is still basically access to a Windows machine? Maybe I can help - I work in a lab with 4 Windows workstations and we don't always need all that compute. Are we talking like a day-long process? A week?

(but also, seems like this can't be the whole story - nobody's been able to do this for years??)

[hmaarrfk]

(but also, seems like this can't be the whole story - nobody's been able to do this for years??)

1. The build is generally difficult with shared protobuf (the preferred way to do things at conda-forge).
2. It gets more difficult with cuda. (for which support was pulled for upstream, so i guess thats not a concern anymore)

Feel free to make PR. Honestly, many of us just don't have windows machines handy.

The builds take about 6 hours even on a good machine, so talking about 4 different python versions would mean a full day for each rebuild.

[ethanbb]

OK thanks - so I may be willing to, but I don't necessarily want to be spending a lot of mental effort on this. So I want to clarify - when you say the build is difficult, do you mainly mean it will take a long time for the compiler to figure out, or is there a lot of potential finagling that I would have to do to find some configuration that works?

[hmaarrfk]

is there a lot of potential finagling that I would have to do to find some configuration that works

this, and you have to wait 1-2 hours per try at times.

[ethanbb]

OK so, I'm looking at #111, do I basically just try to run azure-pipelines-win.yml using something like this?

I can just run the script and report what happens, and you (or whoever) can tell me what to change, but if that's not an arrangement you're interested in, I probably can't help.

[hmaarrfk]

i would create your own PR from main, and see how far you get, we can comment in the PR itself.

[ethanbb]

Alright. And I found CFEP-03, so never mind to what I suggested in my previous comment, I'll follow that.

[hmaarrfk]

part of CFEP03 is just to let Azure fail after 6 hours of runtime. So after debugging locally, do make the PR and let the CI fails due to timeout