Include hdf.hdf5lib.* java packages #1327

mkitti · 2023-02-23T22:04:02Z

This includes the hdf5_java.{dll, so, dylib} as well as the following Java packages:

hdf.hdf5lib
hdf.hdf5lib.callbacks
hdf.hdf5lib.exceptions
hdf.hdf5lib.structs

mkitti · 2023-02-23T22:07:53Z

We currently end up compiling the hdf.hdf5lib.* java classes twice. Once when building HDF5 and then again when assembling the classes. Could we avoid this? I'm tempted just to expand the jarhdf5 that is produced.

$ jar tvf hdf5/cppbuild/linux-x86_64/lib/jarhdf5-1.14.0.jar 
     0 Thu Feb 23 16:56:30 EST 2023 META-INF/
    77 Thu Feb 23 16:56:30 EST 2023 META-INF/MANIFEST.MF
     0 Thu Feb 23 11:45:16 EST 2023 hdf/hdf5lib/
  3510 Thu Feb 23 11:45:16 EST 2023 hdf/hdf5lib/HDFNativeData.class
  4180 Thu Feb 23 11:45:16 EST 2023 hdf/hdf5lib/ArrayDescriptor.class
 12551 Thu Feb 23 11:45:16 EST 2023 hdf/hdf5lib/HDFArray.class
 46612 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/HDF5Constants.class
     0 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/
  1749 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5FD_ros3_fapl_t.class
   804 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5L_info_t.class
   714 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5O_token_t.class
  1779 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5AC_cache_config_t.class
   720 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5O_info_t.class
   592 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5_ih_info_t.class
   794 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5F_info2_t.class
   921 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5O_native_info_t.class
  1105 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5O_hdr_info_t.class
   664 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5E_error2_t.class
   481 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5A_info_t.class
   393 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5G_info_t.class
  1879 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/structs/H5FD_hdfs_fapl_t.class
 57116 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/H5.class
     0 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/
   252 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_cls_copy_func_cb.class
   121 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/Callbacks.class
   127 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5D_append_t.class
   227 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_prp_set_func_cb.class
   129 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_iterate_t.class
   123 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5E_walk_t.class
   143 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_cls_close_func_t.class
   129 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5D_iterate_t.class
   232 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_prp_create_func_cb.class
   257 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_cls_create_func_cb.class
   129 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5A_iterate_t.class
   288 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5O_iterate_t.class
   230 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_prp_close_func_cb.class
   258 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5E_walk_cb.class
   288 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5L_iterate_t.class
   228 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_prp_copy_func_cb.class
   227 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_prp_get_func_cb.class
   143 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5O_iterate_opdata_t.class
   232 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5D_append_cb.class
   141 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_cls_copy_func_t.class
   251 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_iterate_cb.class
   254 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_cls_close_func_cb.class
   143 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5L_iterate_opdata_t.class
   218 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_prp_compare_func_cb.class
   238 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5D_iterate_cb.class
   233 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_prp_delete_func_cb.class
   283 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5A_iterate_cb.class
   145 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/callbacks/H5P_cls_create_func_t.class
     0 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/
   350 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5DataFiltersException.class
   370 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5PropertyListInterfaceException.class
   354 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5InternalErrorException.class
   346 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5ReferenceException.class
   362 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5DatatypeInterfaceException.class
   364 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5DataspaceInterfaceException.class
   338 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5BtreeException.class
   362 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5FunctionEntryExitException.class
  7490 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5LibraryException.class
   346 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5AttributeException.class
   332 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5IdException.class
   348 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5LowLevelIOException.class
   350 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5SymbolTableException.class
   329 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5JavaException.class
   366 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5ResourceUnavailableException.class
   360 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5DatasetInterfaceException.class
   354 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5FileInterfaceException.class
   360 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5FunctionArgumentException.class
   352 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5ObjectHeaderException.class
   350 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5DataStorageException.class
   447 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5Exception.class
   336 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5HeapException.class
   354 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5MetaDataCacheException.class
   360 Thu Feb 23 11:45:14 EST 2023 hdf/hdf5lib/exceptions/HDF5ExternalFileListException.class

mkitti · 2023-02-23T22:19:57Z

We may want to have h5.hdf5lib.loadH5Lib load org.bytedeco.hdf5.hdf5_java.

saudet · 2023-02-24T00:33:43Z

We currently end up compiling the hdf.hdf5lib.* java classes twice. Once when building HDF5 and then again when assembling the classes. Could we avoid this? I'm tempted just to expand the jarhdf5 that is produced.

I think it's fine having our project (re)compile the source. It makes sure we have all the dependencies in there. And we can make those dependencies "optional" so they are not forced upon the user.

We may want to have h5.hdf5lib.loadH5Lib load org.bytedeco.hdf5.hdf5_java.

Sure, I guess that should be fine.

mkitti · 2023-02-24T03:24:59Z

Great. Is there anything else that needs to be done?

I am mainly pausing to test this out, but that could also be done from the snapshots.

I'm also evaluating if there are any elements from the SIS JHDF5 bindings that may be useful:

https://sissource.ethz.ch/sispub/jhdf5/-/tree/master/source/c

saudet · 2023-02-24T08:35:33Z

It looks fine, we can merge this, yes, but let's do it after it's been tested a bit locally, yes.

mkitti · 2023-02-24T18:38:30Z

Why do half of the platforms use cmake and the other half use autotools?

HDF Group seems to moving towards retiring autotools:
https://forum.hdfgroup.org/t/can-we-retire-the-autotools/10362

saudet · 2023-02-24T22:08:02Z

CMake wasn't available previously. We can switch to CMake, of course.

saudet · 2023-03-01T08:59:45Z

hdf5/cppbuild.sh

+# Copyright 2007 - 2018 ETH Zuerich, CISD and SIS
+# Apache License, Version 2.0
+# https://sissource.ethz.ch/sispub/jhdf5
+cp ../../../src/main/c/jhdf5/*.c java/src/jni/


Please don't add them to this repository. Download them just like HDF5.

The files from JHDF5 need significant modification since they were last used with HDF 1.10.

I could start a branch on my fork and pull from there. Would that be better?

https://github.com/JaneliaSciComp/jhdf5

In the meantime, could we run the CI so I can see if there are any cross platform issues?

Why not just rewrite it in Java then?

I think that might be the easiest route. I'll revert recent changes

mkitti · 2023-03-02T23:40:13Z

I'm willing to move ahead with this. I'll try the full Java approach to addressing the JHDF5 legacy.

mkitti · 2023-03-06T04:07:45Z

To be clear, the future Java work lies outside of the scope of this PR. I'm done here unless there is something else to fix.

Let me know if you there are any other issues to address.

mkitti · 2023-03-07T21:39:14Z

It may be useful to enable SZIP compression via libaec https://gitlab.dkrz.de/k202009/libaec .

The conda-forge build of HDF5 recently added support for SZIP compression via libaec in December 2022:
conda-forge/hdf5-feedstock#179

I plan to work on this in a separate branch based on this branch combined with changing all the builds to use cmake.

mkitti · 2023-03-08T00:35:01Z

hdf5/src/main/java/org/bytedeco/hdf5/hdf5_java.java

@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2018-2022 Samuel Audet
+ * Copyright (C) 2023 Mark Kittisopikul


Ah. Thank you.

Initial build with hdf.hdf5lib.*

62da0ee

Fix javadoc, only copy src directory

0775e1e

saudet reviewed Mar 1, 2023

View reviewed changes

mkitti force-pushed the mkitti/hdf_hdfgroup branch from fcdec66 to 0775e1e Compare March 2, 2023 23:38

mkitti marked this pull request as ready for review March 2, 2023 23:39

mkitti added 2 commits March 3, 2023 14:46

Make copying of java/src to ../java more robust

3f43cfa

Fix reference to opencv

6895ca2

mkitti mentioned this pull request Mar 6, 2023

Java Binding Enhancements from Janelia HDFGroup/hdf5#2402

Open

4 tasks

Update CHANGELOG.md and fix nits

f5305f9

mkitti commented Mar 8, 2023

View reviewed changes

saudet merged commit 138c9b0 into bytedeco:master Mar 8, 2023

mkitti mentioned this pull request May 26, 2023

H5DataSet read and write H5std_string functions are likely to get called accidentally HDFGroup/hdf5#2501

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include hdf.hdf5lib.* java packages #1327

Include hdf.hdf5lib.* java packages #1327

mkitti commented Feb 23, 2023 •

edited

Loading

mkitti commented Feb 23, 2023

mkitti commented Feb 23, 2023

saudet commented Feb 24, 2023

mkitti commented Feb 24, 2023

saudet commented Feb 24, 2023

mkitti commented Feb 24, 2023

saudet commented Feb 24, 2023 via email

saudet Mar 1, 2023

mkitti Mar 1, 2023

saudet Mar 1, 2023

mkitti Mar 2, 2023

mkitti commented Mar 2, 2023

mkitti commented Mar 6, 2023

mkitti commented Mar 7, 2023

mkitti Mar 8, 2023

Include hdf.hdf5lib.* java packages #1327

Include hdf.hdf5lib.* java packages #1327

Conversation

mkitti commented Feb 23, 2023 • edited Loading

mkitti commented Feb 23, 2023

mkitti commented Feb 23, 2023

saudet commented Feb 24, 2023

mkitti commented Feb 24, 2023

saudet commented Feb 24, 2023

mkitti commented Feb 24, 2023

saudet commented Feb 24, 2023 via email

saudet Mar 1, 2023

Choose a reason for hiding this comment

mkitti Mar 1, 2023

Choose a reason for hiding this comment

saudet Mar 1, 2023

Choose a reason for hiding this comment

mkitti Mar 2, 2023

Choose a reason for hiding this comment

mkitti commented Mar 2, 2023

mkitti commented Mar 6, 2023

mkitti commented Mar 7, 2023

mkitti Mar 8, 2023

Choose a reason for hiding this comment

mkitti commented Feb 23, 2023 •

edited

Loading