JAR release to Maven Central

[tlf30]

Hello,
I saw some prior discussion on the matter, but would it be possible to get the jni binding jars released to maven central?
Maven central is the key location for libraries to be accessible via build tools.

Thanks,
Trevor

[MarkCallow]

I'm not against it. I don't know how to accomplish it. If someone can provide a PR, if changes are necessary to source or build system, and clear instructions for what I need to do then this can be done.

[tlf30]

Does Khronos have a maven central account that is approved? The process is actually very easy it just looks daunting at first.

[MarkCallow]

I'll check.

[MarkCallow]

Khronos does not currently have an account.

What is involved in syncing with the Maven repository? My colleague took a quick look, and it seems we can either host with Sonotype OSS repository hosting, or we can self host the repository and sync with Smart Proxy.

https://central.sonatype.org/publish/large-orgs/sync/
OSSRH
Smart Proxy

What is the right approach here?

[tlf30]

You will probably want to setup the OSSRH to let them host it. The link you provided are the instructions. It is a bit clunky making a jira account to request a 'project'.

Depending on how your organization is setup, you will want to request your highest level domain (no subdomain) for your packages. You can add the appropriate subdomain classpaths to each package you release.

[MarkCallow]

Turns out Khronos does have a Maven Central account and an established group id: org.khronos. We have now set up an account I can use for publishing the KTX jar files to this group id.

Please look at pom.xml starting at the linked line. First there is stuff to do with signing and then, commented out, with publishing to Maven Central.

Regarding signing, I couldn't spot where I'm supposed to provide the GPG signature. I also wonder if we can use our SSL.com-issued software signing certificate instead of a GPG signature.

Regarding deployment, as well as fixing this up for our group id and figuring out how to provide the account name and pw to use, I need to know how to call this part of the POM from a CI deployment section.

Can you make a PR with the necessary changes and clear indications of where I should provide signing key and account credentials?

We do CI builds for Linux, macOS and Windows. Does it matter which one we use to deploy to Maven Central? The JAR files will all be the same right?

[tlf30]

@MarkCallow I'll need to look at how the build is setup. Normally for jars that are for jni bindings, the natives are packages in their own jar that makes the native available for java to bind to, as such, usually a separate jar would be produced for each platform. Here is an example from LWJGL: https://repo1.maven.org/maven2/org/lwjgl/lwjgl-vma/3.3.1/
In the example there is a jar that contains the jni binding, then there are multiple jars that are wrappers around the natives, and finally the source and javadoc jars. So I imagine that there will need to be changes to each CI build platform, but I will need to look into how the CI is setup to know for sure.

As for the SSL.com, I'm not sure and will have to look into that. I have always used my own GPG signature when publishing to maven central.

[tlf30]

@MarkCallow It looks like you will need to generate and publish a GPG key. I do not see where SSL.com has services for GPG, but according to https://central.sonatype.org/publish/requirements/gpg/#distributing-your-public-key these are the only supported gpg hosts:

keyserver.ubuntu.com
keys.openpgp.org
pgp.mit.edu

This (https://central.sonatype.org/publish/requirements/gpg/) has instructions on getting a GPG key setup and published. It will need to be available to the CI environment for the publishing, but the key itself should not be made public.

[javagl]

Normally for jars that are for jni bindings, the natives are packages in their own jar that makes the native available for java to bind to, as such, usually a separate jar would be produced for each platform.

For the case of KTX, one could/should consider to add the actual ktx.dll (and respective OS-specific counterpats) in the JAR that contains the ktx-jni.dll. This way, users may not even have to install KTX-Software as a whole, but can just add the dependency to their POM, and it will automatically pull in what they need.

The DLLs cannot be loaded from the JAR files directly, though. They have to be written to a "temp"-directory to be loaded. And they have to be loaded in the "reverse order" of their dependency direction - meaning that it has to load ktx.dll first, and then ktx-jni.dll. (I did set up some utilities for that a while ago).

Creating the JARs ... can be a bit tricky, too. In order to create the JARs, the native libraries have to be present. So building the natives and the JARs can hardly happen on the same machine, and even less in a lean-and-clean mvn package step. (It might be possible nowadays, but certainly not easy...)

Signing the JARs will require a key, but this does not necessarily have to happen in the Maven build or CI.

So... setting all this up properly can be a pain in the back. Depending on whether someone is willing to set up the signing as part of the official CI, an intermediate solution could be to create a build process that takes the precompiled natives, packs them into JARs, sign the JARs (manually), and upload them as 'Staging Bundles' in Sonatype (roughly as described in https://central.sonatype.org/publish/publish-manual/ )

[MarkCallow]

I have no expertise in this. I would welcome a PR that sets up this publishing. To create a JAR files that contains the libs (dll/dylib/so) for each platform a manually triggered action similar to the one we use for publishing pyktx could be set up. We trigger this after all builds are completed and it downloads the assets to be included from GitHub releases.

The DLLs cannot be loaded from the JAR files directly, though. They have to be written to a "temp"-directory to be loaded. And they have to be loaded in the "reverse order" of their dependency direction - meaning that it has to load ktx.dll first, and then ktx-jni.dll. (I did set up some utilities for that a while ago).

What has to load the dlls, the application? What do you mean here by a "temp" directory?

Note that on macOS dylibs have to be installed in the location matching their install-name attributes and libktx.dylib's location has to be listed in the RPATH attribute of libktx-jni.dylib. The install names for the dylibs included in the install package are /usr/local/lib/libktx{,-jni}.dylib. sudo is required to install there. It may be necessary to make a different build of libktx.dylib to meet this requirements using a location that does not require sudo.

[javagl]

Some context: My first goal would be to have a library that is not the KTX JAR itself, but a simpler, more Java-idiomatic convenience layer that is wrapped around that. But if that was supposed to be deployed to Maven, it would either

1. have to include the native libs or
2. rely on the KTX JARs (with the native libs) already being available in Maven.

Sooo... paradoxically, it would safe me a ton of work if this issue was resolved, but ... I guess that I'll at least have to spend some time to look closer into this, and see whether I can contribute something to resolve it.

---

What has to load the dlls, the application?

The DLLs have to be loaded "into the running JVM". On this level, it does not matter whether it is triggered by the application or by the (Java) library itself. But it does matter (a lot!), in terms of the user expecience.

The current 'Usage' section at https://github.com/KhronosGroup/KTX-Software/tree/main/interface/java_binding#usage shows the example that includes the block

    static {
        // Load libktx-jni, which provides the JNI stubs for natively
        // implemented Java methods. This should also load libktx
        // automatically! If it doesn't, you may need to load libktx manually.
        System.loadLibrary("ktx-jni");
    }

So there, it is loaded by the application.

But this System.loadLibrary call makes a ton of assumptions. It appends the file extension (like .dll or .so, depending on the operating system), and then tries to load this, as a file, from any path that is visible for the JVM (for example, the PATH environment variable and such). The ktx.dll (dependency) would also have to also be visible for the JVM.

So when someone wanted to deploy such an application, every user would have to manually twiddle with the (OS-specific) native libraries and put them into directories where they are "visible" for the JVM.

---

What do you mean here by a "temp" directory?

That's the common solution for this problem. It's unfortunate that this workaround is necessary, but ... it is.

The point is: There is no function for loading a library from a byte[] array or so. It has to be a file. And there are two options:

• The System.loadLibrary function takes a vague name, like "ktx-jni", and tries its magic to find the right file (e.g. the "C:/path/ktx-jni.dll").
• The System.load function directly takes a full path, like System.load("C:/path/ktx-jni.dll")

So when the DLL is deployed in a JAR via Maven, then there is only only feasible, user-transparent way to load it:

(You can skip the links - these are only given as examples)

• Create a file in the actual "temp" directory of the user (i.e. the real /var/tmp or C:\Users\User\AppData\Local\Temp\ directory) - Example
• Fill this file with the data from the DLL that was read from the JAR file - Example
• Load this 'temp' file with System.load and the full path of that file - Example
• Call all this from a static initializer block of one of the Java classes of the library (so that it is done when that class is loaded, and the application does not have to do this manually)

All this is surrounded by a load of boilerplate code for detecting the operating system and architecture. You'll have to differentiate between that ktx-jni-linux-x86.so and that ktx-jni-windows-x86_64.dll somehow. But it works reasonably well, and makes it really easy for others: Developers just add the dependency to their Maven POM. The end-user does not have to download and fiddle with obscure DLLs. This is as close as it gets to the idea(l) of Write once, run anywhere.

---

You specifically pointed out an issue on MacOS (and I think that's your main development platform):

Note that on macOS dylibs have to be installed in the location matching their install-name attributes and libktx.dylib's location has to be listed in the RPATH attribute of libktx-jni.dylib. The install names for the dylibs included in the install package are /usr/local/lib/libktx{,-jni}.dylib. sudo is required to install there. It may be necessary to make a different build of libktx.dylib to meet this requirements using a location that does not require sudo.

I don't know the technical backround here. What I do remember, though, is that I once stumbled over an issue in one of my projects (or a depencency of my projects) where that RPATH setting was relevant, and eventually, this was "solved" by something as trivial as
set(CMAKE_MACOSX_RPATH 1)
or
set(CMAKE_SKIP_RPATH FALSE)
(either of them, or both?).

Related questions would be:

• Is this still relevant when loading the libktx.dylib and libktx-jni.dylib manually, with their full path (even if that path is inside a temp directory)?
• Can there be multiple paths in the "install-name" attribute or in the RPATH?

(There's always the option to say: "The Java bindings don't work on MacOS", depending on which hoops we can or cannot jump through here...)

[MarkCallow]

So when someone wanted to deploy such an application, every user would have to manually twiddle with the (OS-specific) native libraries and put them into directories where they are "visible" for the JVM.

The KTX installers install both libraries into a location found on the PATH environment variable. (On Windows the install location is added to PATH.) So one user can install the libraries making them available to everyone else.

---

All this is surrounded by a load of boilerplate code for detecting the operating system and architecture. You'll have to differentiate between that ktx-jni-linux-x86.so and that ktx-jni-windows-x86_64.dll somehow. But it works reasonably well, and makes it really easy for others: Developers just add the dependency to their Maven POM. The end-user does not have to download and fiddle with obscure DLLs. This is as close as it gets to the idea(l) of Write once, run anywhere.

I think I understand. I'd be happy to receive a PR to add this stuff.

---

On macOS binaries contain the paths to their dependent libraries which are set at link time. Those paths can be full paths or the path can use one of 3 prefixes that allow binaries to be installable anywhere:

1. @executable_path
2. @loader_path
3. @rpath

The latter is constructed from the LC_RPATH commands in the binary.

A library's install name can be a relative path or can use the @rpath prefix which permits dyld to match and load it for any application that has its location on its RPATH.

Is this still relevant when loading the libktx.dylib and libktx-jni.dylib manually, with their full path (even if that path is inside a temp directory)?

I'm not 100% sure. I think so, hence the @rpath stuff. Another issue is that on Apple Silicon all binaries must be signed and any attempts to modify an install name or rpath after signing will render the signature invalid.

Can there be multiple paths in the "install-name" attribute or in the RPATH?

Only one install name. Multiple LC_RPATH entries are allowed. I miswrote earlier. The install names for libktx.dylib and libktx-jni.dylib in our packages are @rpath/libktx-jni.4.dylib and @rpath/libktx.4.dylib. The LC_LOAD_DYLIB libktx-jni uses to load libktx is @rpath/libktx.4.dylib.

What you want to do is possible.

[javagl]

Poor @rpath 🙂 I can imagine your notifications.
One of my other GitHub accounts is @gpu , a lot of stuff is happening "at the GPU"...

---

The point about the users having to handle the libraries may be a bit too focussed on the actual end-user experience. I'd like to correct the statement

So one user can install the libraries making them available to everyone else.

to say "So one user has to install the libraries making them available to everyone else."

(Imagine you downloaded an arbitrary application that could display, for example, glTF files. And during the installation it said: "You also have to install KTX-Software". You don't know what this is. You don't know what it does. You don't know why you need it. For an end-user, that's annoying...)

---

How easy it is to make things really transparent and easy to use... is hard to say. But that @rpath thing looks like it could make things much harder for developers. I'll have to read more about that, because I didn't fully understand all aspects or your response. But it sounds like it might not even really "use" that @rpath information when the libraries are loaded brutally and manually with the full file path in System.load, and that this was only used for "discovering dependencies" when they are not loaded manually. (At least, I hope so 🤞 )

[MarkCallow]

I just read the dlopen man page. If the path passed to dlopen contains a slash then dlopen opens that file. If just a leaf name. @rpath comes into play.