Hosting OpenJDK upstream reference builds for JDK 8u and JDK 11u

[jerboaa]

After Red Hat took over upstream OpenJDK leadership of the jdk8u project and the jdk11u stream of the jdk-updates project there is a desire to provide reference binary builds for the community. This is similar to what Oracle does via jdk.java.net. AdoptOpenJDK seems to have the required infrastructure to host those binaries. We, Red Hat, would like to host them on AdoptOpenJDK and link to them from the upstream JDK 8u wiki and JDK 11u wiki pages for the community to use.

At this point, Red Hat has JDK 11 early access builds available for Linux x86_64, version 11.0.3+2. JDK 8 will follow shortly as soon as the first upstream tag for 8u212 arrives. The builds have been done on Red Hat infrastructure and scripts as to how those builds have been produced could be shared with the community if there is a desire.

Initially, there will be Linux x86_64 builds. Other reference binary builds may come for future updates.

[karianna]

Added a TSC Agenda label for this, some early thoughts:

1. Do we want to host these binaries through the API and the Website?
2. What technical work is required to achieve hosting and to differentiate between the RI provider, Adopt, Corretto etc (there will be refactoring :-))
3. Does Red Hat want the binary to go through our test pipeline?

I'm personally in favor of supporting this.

[jerboaa]

Thanks.

1. Do we want to host these binaries through the API and the Website?

Not sure about the technicalities, but if it's easy enough to achieve, I'd rather not have it listed on the website or API. If that's technically problematic, not actively advertising them should be sufficient. They will show up at the upstream Wiki page. Consider that those builds are not plain AdoptOpenJDK binaries (built in a different way, tested in a different way) so this could be confusing for users. Yet, the source they were produced from would be the same.

3. Does Red Hat want the binary to go through our test pipeline?

Final builds will have been tested by Red Hat. So not necessarily. However, it would be good to run EA builds as soon as they are available through the pipeline so as to increase test coverage.

Depending on the tree/tag JDK 11 nightlies from Adopt build from I don't really expect differences, at this point. Provided nightlies build from jdk-updates/jdk11u and jdk8u/jdk8u testing of nightlies should have the same effect.

[theRealAph]

This is getting urgent. We need to get this done soon, in order to give the pre-CPU binaries a decent amount of public Beta testing before the CPU release. I'd love to put these on Adopt.

[theRealAph]

2. What technical work is required to achieve hosting and to differentiate between the RI provider, Adopt, Corretto etc (there will be refactoring :-))

I guess these are all questions for Adopt's internal processes, not questions for us.

[karianna]

I've asked the TSC to meet ASAP to work through this.

[theRealAph]

I've asked the TSC to meet ASAP to work through this.

Thanks. Our Windows binaries aren't quite ready yet, so it'll be just x86 linux for now.

[karianna]

We did not have Qurom to vote on this so I'm writing this up further and if clear enough we 'll attempt a virtual asynchronous vote.

A broad question for the TSC is whether AdoptOpenJDK through it's API and website distribute binaries built elsewhere? And as a subtext, is doing this for the Reference Implementation a special case?

Option 1 - do this all at AdoptOpenJDK:

• Upload 8u and 11u binaries built and signed by Red Hat (on behalf of the updates project) to an AdoptOpenJDK GitHub repo (maybe the existing openjdk8-binaries and openjdk11-binaries repos, maybe new ones).
• Run the binaries through our test pipeline (new pipelines may need to be built and/or existing pipelines amended).
• Make those binaries available through our API (handily hosted on OpenShift). This is required because of GitHub rate limiting which the API works around. (code work to be done here)
• Optionally Make those available through our website (code work to be done here)
• Update the Dashboard to split out download stats

Option 2 - Use AdoptOpenJDKs infrastructure as code

If the elapsed time of option 1 takes too long (due to technical and/or policy work) then Red Hat could still easily use our infrastructure as code. That is:

• Upload 8u and 11u binaries built and signed by Red Hat (on behalf of the updates project) to a canonical OpenJDK GitHub repo.
• Optionally run the binaries through our test pipeline (new pipelines may need to be built and/or existing pipelines amended).
• Run a copy of our API code on OpenShift or wherever. This is required because of GitHub rate limiting which the API works around.
• Utilise parts of our website code to host a site for the binaries

[tellison]

@jerboaa Please help clarify the ask:

there is a desire to provide reference binary builds for the
community. This is similar to what Oracle does via jdk.java.net.

The term "Reference Implementation" (RI) relates to the JCP JSR triform delivery of Spec, RI, and TCK. OpenJDK produces the RI for each platform JSR at the point it goes GA, as you can see on the page you linked to above. The RI binaries for Java 8 and Java 11 are published once and don't change.

So it seems we are talking about hosting builds of the JDK8 updates and JDK11 stream JDK updates projects' releases - but we do that already. So what is the ask? That we host binaries of those releases that are built by Red Hat rather than AdoptOpenJDK?

The mission for AdoptOpenJDK is to have an "open and auditable" build, test, and distribution community. I don't yet see the proposal for how we achieve that by taking binaries built elsewhere. Is the plan that we link to Red Hat's open build system/build definitions somehow?

[jerboaa]

@jerboaa Please help clarify the ask:

there is a desire to provide reference binary builds for the
community. This is similar to what Oracle does via jdk.java.net.

The term "Reference Implementation" (RI) relates to the JCP JSR triform delivery of Spec, RI, and TCK. OpenJDK produces the RI for each platform JSR at the point it goes GA, as you can see on the page you linked to above. The RI binaries for Java 8 and Java 11 are published once and don't change.

Sorry for the confusion. I probably shouldn't have used the term "reference binary builds" in this context. However, those would be builds done elsewhere and would give you a baseline in terms of bugs/regressions in AdoptOpenJDK binaries too. I could see this being helpful for triage purposes.

So it seems we are talking about hosting builds of the JDK8 updates and JDK11 stream JDK updates projects' releases - but we do that already. So what is the ask?

Not quite. AdoptOpenJDK builds use "AdoptOpenJDK" in the version string (vendor). Plain upstream binaries would not have that.

AdoptOpenJDK has a concept of "nightly" builds. But it's not clear what those nightly builds are built from (specific tag? head of some repository? which repository?). We hope to provide early access (EA) builds too. See below.

That we host binaries of those releases that are built by Red Hat rather than AdoptOpenJDK?

Yes. As a replacement for Oracle provided JDK 11 GPL builds. They didn't have JDK 8 builds AFAIK. We want to provide them for 8 too.

Another item we'd like to address is for people to get EA builds of the upcoming CPU releases (modulo embargoed patches of course). Historically speaking there have been patches in CPU releases which weren't security fixes, but didn't get pushed into the open until the CPU release. Some of those fixes may cause regressions. We intend to change that. EA builds would be based on upstream build tags like jdk8u212-b01 or jdk-11.0.3+3. The hope is to publish those builds ahead of the actual CPU release and give consumers a way to test the builds with their apps minimizing potential regressions.

The mission for AdoptOpenJDK is to have an "open and auditable" build, test, and distribution community. I don't yet see the proposal for how we achieve that by taking binaries built elsewhere. Is the plan that we link to Red Hat's open build system/build definitions somehow?

We'd be providing instructions on how to reproduce our builds in their own setup. Scripts can be shared. There is absolutely no secret sauce. The binary builds would have the same "open and auditable" promises AFAICS. Once the CPU releases are GA, AdoptOpenJDK would pick those changes up as well and builds would converge to the same basically (modulo bugs, vendor string, etc.) Rinse, repeat.

Another hope is to be able to run Red Hat builds through the AdoptOpenJDK test pipeline. As I understand it, though, that can be done without actually hosting the binaries at AdoptOpenJDK.

[theRealAph]

On 3/19/19 11:09 PM, Tim Ellison wrote: So it seems we are talking about hosting builds of the JDK8 updates and JDK11 stream JDK updates projects' releases - but we do that already. So what is the ask? That we host binaries of those releases that are built by Red Hat rather than AdoptOpenJDK?

This is not about Red Hat. When it was maintaining OpenJDK 8, Oracle provided two sets of binaries: their own customized ones which make up the proprietary JDK and "community" OpenJDK builds with no changes at all from the upstream open sources. The latter came with no support. There is now a gap in that the vanilla builds aren't available, and as 8u and 11u lead I need to fill that gap. Red Hat, like many other suppliers of JDKs based on OpenJDK, customizes its builds for a variety of reasons, in particular integration with the OS. The purpose of these binaries, on the other hand, is to replace the community builds that Oracle provided. They are not a Red Hat product, although I am grateful that we can use the Red Hat infrastructure to test and build them. Of course there are other places that the jdk updates projects could use to host these binaries, but appearances matter. I believe that Adopt would be a good place to come together as a community and show unity of purpose.

…

-- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. <https://www.redhat.com> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

[sxa]

While I'm a touch nervous about a precedent of hosting binaries built outside our setup at adoptopenjdk, I think this is a special case and I'm tentatively in favour of it I think. As per previous comments we'd need to thrash out how to store and link to it. Perhaps it'd be another variant optionally available via the API but not the web site for those who want it, but we'd need to be clear on how it'd be labelled - @theRealAph do you have a fixed naming scheme for the tarball downloads or are you flexible on it? We host multiple variants in the openjdkxx-binaries repo so we could either:

1. Have a pre-release in the existing -binaries repo with some identifiable name (such as an openjdk11u prefix) to distinguish them from our regular nightlies
2. Use a separate repo for these to keep them totally separate to avoid any potential confusion
3. Host them some other way rather than our github release repos

[jerboaa]

do you have a fixed naming scheme for the tarball downloads or are you flexible on it?

Right now they are called: openjdk-8u212-b01-ea-linux-x86_64.tar.gz and openjdk-11u-11.0.3+3-ea-linux-x86_64.tar.gz. It should be reasonably flexible to adjust to your needs. The ea name will be dropped once GA'ed.

We are going off on a tangent as this would be concerned about the implementation not the generic TSC yes/no kind of decision. Anyhow. What would be the expectation of a tarball file name?

[sxa]

We are going off on a tangent as this would be concerned about the implementation not the generic TSC yes/no kind of decision.

Understood - I was thinking out loud on how practical it will be before making my decision :-)

As long as it has an identifier in the filename (^openjdk- as a regex may by adequate) it shouldn't be a major concern but either way there shouldn't be an issue keeping all your identifiers in there too.

[karianna]

While I'm a touch nervous about a precedent of hosting binaries built outside our setup at adoptopenjdk, I think this is a special case and I'm tentatively in favour of it I think. As per previous comments we'd need to thrash out how to store and link to it. Perhaps it'd be another variant optionally available via the API but not the web site for those who want it, but we'd need to be clear on how it'd be labelled - @theRealAph do you have a fixed naming scheme for the tarball downloads or are you flexible on it? We host multiple variants in the openjdkxx-binaries repo so we could either:

On the variant issue this is a bug in our code / API etc. We need to split variant (which really means VM, i.e. Hotspot, OpenJ9 and possibly DCEVM one day) from provider. This is an opportunity to do that but may require an API v3

[mstoodle]

Slightly massaged statement from @theRealAph (hopefully I didn't alter his meaning) that sparked a question for me :

When it was maintaining OpenJDK 8, Oracle provided ... "community" OpenJDK builds with no changes at all from the upstream open sources ... There is now a gap in that the vanilla builds aren't available.

I had thought the Hotspot based AdoptOpenJDK builds use "pure" OpenJDK source code with no additional patches other than maybe some "cosmetic" ones to change vendor/provider properties. So, question for the AdoptOpenJDK folks: are there any other patches being applied by AdoptOpenJDK in those builds?

In terms of differences between AdoptOpenJDK "Hotspot" builds and the vanilla builds @theRealAph describes, there would also be the small number of third party libraries (e.g. bundling or not of Freetype, based on platform), which aren't actually part of OpenJDK.

Call me naive (wouldn't be the first time!), but if the only significant differences come down to cosmetics and some small set of bundled libraries, would it not make sense to align how these two OpenJDK binary builds are done and then only do one of them? I would think that would benefit everyone operating in the open Java community...

[karianna]

Slightly massaged statement from @theRealAph (hopefully I didn't alter his meaning) that sparked a question for me :

When it was maintaining OpenJDK 8, Oracle provided ... "community" OpenJDK builds with no changes at all from the upstream open sources ... There is now a gap in that the vanilla builds aren't available.

I had thought the Hotspot based AdoptOpenJDK builds use "pure" OpenJDK source code with no additional patches other than maybe some "cosmetic" ones to change vendor/provider properties. So, question for the AdoptOpenJDK folks: are there any other patches being applied by AdoptOpenJDK in those builds?

In terms of differences between AdoptOpenJDK "Hotspot" builds and the vanilla builds @theRealAph describes, there would also be the small number of third-party libraries (e.g. bundling or not of Freetype, based on platform), which aren't actually part of OpenJDK.

Call me naive (wouldn't be the first time!), but if the only significant differences come down to cosmetics and some small set of bundled libraries, would it not make sense to align how these two OpenJDK binary builds are done and then only do one of them? I would think that would benefit everyone operating in the open Java community...

We initially thought that but it's possible that AdoptOpenJDK will add patches going forwards with regards to cacerts and making sure that things like IcedTea-Web and OpenJFX are first class citizens (I'm musing out loud here - this is not any official statement)

[theRealAph]

There are a couple of other issues. Firstly, there's the build dependencies: we build on a RHEL 6 system (i.e. pretty old) because glibc is backwards compatible, thus allowing the OpenJDK system to run on anything more recent than that. I don't think you would really want to restrict yourself always to use vanilla sources, but you could choose to do that. It is possible, though, that on some systems you'd need to patch in order to pass compatibility tests.

[mstoodle]

So, AdoptOpenJDK would need to be able to build their master branch (presumably these other things are brought in via separate branch(es) ?) and would need RHEL 6 systems in their farm to do the builds (I guess the RHEL6 dependency is from Oracle's original OpenJDK builds? =). Aren't the other things just variations (along the lines of "do you want Hotspot or OpenJ9?" and "What platform do you want it for?") ? Oversimplifying but: "Do you want IcedTea-Web?" "Do you want OpenJFX?" "Whose cacerts file do you want?" "Do you want pure source OpenJDK build?" (which would clearly decide many of the other questions).

Sorry if I seem pushy, I just want to see where the concept really breaks down. I realize variations can become difficult to describe and complicates lots of things from UI to reporting and triaging problems to verifying fixes, but having the build, test, and distribution in one open community seems like a really nice "feature" for the community (and really puts the "OpenJDK" in "AdoptOpenJDK" :) ).

[karianna]

TSC debated this at length and the vote was yes to approve this. I'll raise tickets in the various repos to track this work.

[karianna]

So, AdoptOpenJDK would need to be able to build their master branch (presumably these other things are brought in via separate branch(es) ?) and would need RHEL 6 systems in their farm to do the builds (I guess the RHEL6 dependency is from Oracle's original OpenJDK builds? =). Aren't the other things just variations (along the lines of "do you want Hotspot or OpenJ9?" and "What platform do you want it for?") ? Oversimplifying but: "Do you want IcedTea-Web?" "Do you want OpenJFX?" "Whose cacerts file do you want?" "Do you want pure source OpenJDK build?" (which would clearly decide many of the other questions).

Sorry if I seem pushy, I just want to see where the concept really breaks down. I realize variations can become difficult to describe and complicates lots of things from UI to reporting and triaging problems to verifying fixes, but having the build, test, and distribution in one open community seems like a really nice "feature" for the community (and really puts the "OpenJDK" in "AdoptOpenJDK" :) ).

It was decided that AdoptOpenJDK may well add new features and patches that were significantly different (at the core)

[karianna]

This has been done (first incarnation) - we'll hook the upstream builds properly into APIv3 going forwards.