Can't attach to jre/jdk 9 using ByteBuddy

[GotoFinal]

ByteBuddy agent fails for jdk/jre 9.

java version "9.0.1"
Java(TM) SE Runtime Environment (build 9.0.1+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)

For jdk i got:

Exception in thread "main" java.lang.IllegalStateException: Could not self-attach to current VM using external process
	at net.bytebuddy.agent.ByteBuddyAgent.installExternal(ByteBuddyAgent.java:433)
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:371)
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:352)
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:320)
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:306)

I tried to debug that code, and all I can see is that process return 1, but I don't see anything in output/error stream of process. But if throw is skipped (via debugger) it still does fail (returns null instead of instrumentation)
For sure it does see my java_home, as I changed command to -version and it does work fine.

For JRE I got:

Exception in thread "main" java.lang.IllegalStateException: No compatible attachment provider is not available
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:367)
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:352)
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:320)
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:306)

I used 1.7.9 version of ByteBuddy agent

[raphw]

It is expected to not work for a JRE as the required libraries are missing. It is supposed to work for a JDK. What operating system are you using?

[GotoFinal]

@raphw Windows 10. (Version 10.0.16299.19)

About JRE, it should work fine on JRE I think, in JRE 8 you could run byte buddy agent with manually provided tools.jar and added attach.dll to command line or via reflections directly to class loader - ofc you need to choose valid lib/tools for each system etc, but it works fine. So something like this should be possible in JRE 9 too. By somehow loading jdk.attach.jmod (I think this is the only one missing from jre?) and that attach library, but not sure how to do that yet. (like I said, it works perfectly fine in JRE 8, you just need to add that libraries to your .jar or download them on startup)

[raphw]

If it was possible to include the attachment module in a JRE, this would be possible but I have never tried that.

Could you help me debug this by checking what arguments are provided here: https://github.com/raphw/byte-buddy/blob/master/byte-buddy-agent/src/main/java/net/bytebuddy/agent/ByteBuddyAgent.java#L423

Could you then try to run https://github.com/raphw/byte-buddy/blob/master/byte-buddy-agent/src/main/java/net/bytebuddy/agent/Attacher.java with these arguments? I assume an exception is thrown, maybe some folder with spaces that is not escaped correctly.

[GotoFinal]

@raphw

"C:\Program Files\Java\jdk-9\bin\java.exe" -cp C:\Users\Admin\AppData\Local\Temp\byteBuddyAttacher13896466634965179790.jar net.bytebuddy.agent.Attacher com.sun.tools.attach.VirtualMachine 5488 C:\Users\Admin\AppData\Local\Temp\byteBuddyAgent11878628151824075180.jar 

And like I said, output is empty:
https://i.imgur.com/eZ1wpeX.png

https://github.com/raphw/byte-buddy/blob/master/byte-buddy-agent/src/main/java/net/bytebuddy/agent/Attacher.java#L56 ugh, that would explain empty output

[raphw]

The result is communicated using a status code what is the easiest. Status code 1 means: the self attachment failed.

[GotoFinal]

@raphw I think that it should still print that stractrace and bytebuddy should read it if process failed, as now it is just hard to debug.

oh...
https://github.com/raphw/byte-buddy/blob/master/byte-buddy-agent/src/main/java/net/bytebuddy/agent/Attacher.java#L45
here
There are 3 arguments, vm attach class, process id and path, but it is trying to check 4th one.

java.lang.ArrayIndexOutOfBoundsException: 3
	at net.bytebuddy.agent.Attacher.main(Attacher.java:45)

So just bad args.length check, probably typo?

[raphw]

This is unfortunately difficult to implement as Byte Buddy should not print to standard output, this can cause quite some trouble in managed environments. But I agree that it is difficult to debug.

The argument check is guarded against a missing argument as Windows translates the missing argument to "null" whereas on Linux, it is translated to an empty string. The check is however overly careful (off by one error). This should however not cause an error.

[GotoFinal]

@raphw it does, as when 3 args are provided args.length < 3 -> 3 < 3 -> false, so it does second side of OR, and then it fails as it want to fetch 4th element: args[3]

(args.length < 3 || args[3].isEmpty())

[raphw]

Of course, too late at night. I have fixed the off by one index check what should fix that by the next release. Could you build Byte Buddy and verify the fix, just to make sure it now works?

[GotoFinal]

@raphw seems to works fine, returned instrumentation instance on jdk 9

[GotoFinal]

@raphw also, maybe some argument can be used to provide information if stacktrace should be printed/ignored/saved to some .log file, it would make debugging simple while default behavior will be still this same.

[raphw]

Unfortunately, the argument is already used by custom instrumentation. It could be done by an additional argument. In general, this does however add some overhead, I wil think about it.

[GotoFinal]

@raphw
Also about

If it was possible to include the attachment module in a JRE, this would be possible but I have never tried that.

// this reflection hack can be replaced with -Djava.library.path starup argument
System.setProperty("java.library.path", System.getProperty("java.library.path") + ";B:\\Java\\tools\\j8"); // path to where attach.dll is
Field fieldSysPath = ClassLoader.class.getDeclaredField("sys_paths");
fieldSysPath.setAccessible(true);
fieldSysPath.set(null, null);
File toolsJar = new File("B:\\Java\\tools\\j8\\tools-min.jar");
Instrumentation instrumentation = ByteBuddyAgent.install(
    () -> {
        try {
            return Accessor.Simple.of(new URLClassLoader(new URL[]{toolsJar.toURI().toURL()}, ClassLoader.getSystemClassLoader()), toolsJar);
        }
        catch (MalformedURLException e) { throw new InternalError("Can't load tools"); }
    });
System.out.println(instrumentation);

And it works fine on JRE
Where tools.jar are by default in jdk8/bin/tools.jar and attach.dll (on windows) is in jdk8/jre/bin/attach.dll
Also tools.jar are pretty big 17MB, and different for each system, but they can be much smaller - as only sun\tools\attach\ and com\sun\tools\attach\ packages are needed, and only one service: com.sun.tools.attach.spi.AttachProvider
Then whole tools.jar is just 35KB, and some classes can be shared between systems too.

So something like this can be used to provide instrumentation on JRE - but for library it would require multiple versions for multiple JRE versions are systems. But it should be possible to use this same tools/attach lib for java 6/7/8, I didn't test that tho.

But I can't do this same for JRE 9 yet, as now there are that modules... and even if I try to run custom Attacher with manually provided modules for attach (only jdk.attach module should be needed, but it also depend on some jdk.internal.jvmstat module for unknown for me reason) then it fails with hash mismatch - so it should be possible if I would find a way to define module manually, but this is much more work to do, and probably some strong reflections/unsafe (should be possible to prepare that module once tho).
The problem is with AttacherProvider as no providers are added after loading all needed methods, probably because it is looking for them in different class loader, I was able to get implementation of it manually, but module rules are block usage of it.

So maybe you know something more about java 9 and how it can be done.
Also you can try to use that to provide additional attach metod in byte-buddy-agent that will be able to work even on JRE ;)