Type hash in interface codegen (rep2011)

[emersonknapp]

Related to ros-infrastructure/rep#358
Part of ros2/ros2#1159
Related to ros2/rcl#1027
No dependencies.

Feature addition: New generator package rosidl_generator_type_hash that creates JSON representations and their corresponding SHA256 hashes for ROS 2 interfaces during code generation. A new type rosidl_type_hash_t that contains these versioned hash values and can be used to store them in generated code and be used in RMW discovery.

Change brief by repository:

• rosidl_cmake - Add new argument to generator
• rosidl_generator_c - depend on new typehash generator, embed hashes in generated C code using the new rosidl_runtime_c struct
• rosidl_generator_cpp - depend on typehash generator, embed hashes in generated C++ code using the new rosidl_runtime_c struct
• rosidl_generator_tests - new tests for the typehash generator
• rosidl_generator_type_hash - new generator, reads ROS IDL representations, serializes to hashable JSON format, and produces SHA256 hashes for all interface types, according to REP-2011. See README.md for more details
• rosidl_pycommon - utility for loading and embedding hashes in code generators
• rosidl_runtime_c - struct rosidl_type_hash_t for representing type hashes, with string utilities for encoding
• rosidl_typesupport_introspection_c - add the type hash to introspection alongside type name for use by RMW implementation
• rosidl_typesupport_introspection_cpp - add the type hash to introspection alongside type name for use by RMW implementation

The hashes are provided to the install tree, and so are available to language code generators (C, C++, Python) to be embedded in generated code for calculation-free use at runtime.

For Messages, a simple hash is output. For Services and Actions, all generated subinterfaces are also represented and hashed, so that every Service and Message in the resulting installation has its own unique hash. See the rosidl_generator_type_hash/resource/HashedTypeDescription.schema.json file, as mentioned in the README for a strict description of the structure.

Attached are archives containing the output tree of build/example_interfaces/rosidl_generator_type_hash/rosidl_generator_c for representative output.

rosidl_generator_c.tar.gz

[emersonknapp]

@clalancette @wjwwood This still needs testing but I feel good about it functionally - feedback, especially on high level approach and architecture would be very valuable now

[methylDragon]

Some small comments

[methylDragon]

Also additional note: The hashes aren't getting populated/generated for rosidl_generator_py, but I don't think we need it to at the moment

[emersonknapp]

@methylDragon see ros2/rosidl_python#191 for rosidl_generator_py - but that one is fairly simple and is just a quick copy of the patterns established here

[emersonknapp]

Converting back to draft while I do the CLI and figure out a couple other bits I'm discovering that I need downstream for RMW implementations to distribute the type hash during discovery.

Didn't hear any major architectural concerns, so I am thinking the main thing needed here is documentation and tests, not overhaul.

[emersonknapp]

The Rpr runner is showing an interesting side effect right now - since the PR runner only builds packages in this repository. The rosidl_typesupport_introspection_tests package defines a service and therefore refers to ServiceEventInfo.msg (this is just the first example that comes up in the build). But, since the tree underneath that test has not been built with this PR, those messages don't have the generated hashes that are expected by our generated code locally. Do we reasonably expect that a new generator will require that all interfaces packages must be rebuilt, or should it be able to handle this case?

[emersonknapp]

Sanity check time!

• Linux
• Linux-aarch64
• Windows

[emersonknapp]

@clalancette @wjwwood @methylDragon @james-rms
This is now ready for review. It's unfortunately gotten large, but some percentage of that is copyright headers... I've updated the description up top with a brief on the changes.

I have prototypes working donwstream of this passing type hashes in FastDDS and CycloneDDS via user_data, which validates to me that this PR is now providing all required interfaces to enable Type Hash Distribution.

[emersonknapp]

@methylDragon yes that is true - referenced type descriptions are sorted alphabetically (rather than, for example, some topological ordering), but fields are added in the order they are specified in each individual type definition

[clalancette]

I've done a first review pass here. I generally understand what you are doing in here, and I don't see any major flaws. I've left a few things to fix up, and I also need to go back and look at the empy things more carefully, but we can at least start with this stuff.

[emersonknapp]

Next phase sanity checking (windows not ready for it just yet)

• Linux

[emersonknapp]

• Windows

[clalancette]

OK, I did a more thorough review and also looked at the generated output.

Despite the fact that I left quite a few comments, they are almost all minor or questions. Overall, this looks really fantastic. I think once we get these things solved and also get Windows happy, we can proceed with this.

[emersonknapp]

Gist: https://gist.githubusercontent.com/emersonknapp/190b20823a334c72086b26bd139e29d6/raw/adb81acaee0b780fc260e3461ca5f8efba166a46/ros2.repos
BUILD args: --packages-above-and-dependencies rosidl_cmake rosidl_generator_c rosidl_generator_cpp rosidl_generator_tests rosidl_generator_type_description rosidl_pycommon rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_cpp
TEST args: --packages-above rosidl_cmake rosidl_generator_c rosidl_generator_cpp rosidl_generator_tests rosidl_generator_type_description rosidl_pycommon rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_cpp
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/11586

• Linux
• Linux-aarch64
• Windows

[emersonknapp]

OK - I think I've pushed an update to all the actionable comments and answered all the questions. Green CI!

[clalancette]

OK, I went back through, and there is one minor renaming thing left. After that, this is good to merge.

[clalancette]

Thanks for all of the back-and-forth here!

This now looks good to me. I'll suggest we run CI with just this PR in place, since we'll only be merging this one right away.

Once that CI is green, I'm going to wait until after our weekly meeting to merge this in.

[emersonknapp]

Gist: https://gist.githubusercontent.com/emersonknapp/54b2023bfdc0fc52f06b54c83afd1d59/raw/adb81acaee0b780fc260e3461ca5f8efba166a46/ros2.repos
BUILD args:
TEST args:
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/11595

• Linux
• Linux-aarch64
• Windows

[wjwwood]

I skimmed the C code part and it lgtm, with some nitpick/comments, but nothing blocking.

[wjwwood]

nitpick:

Suggested change

	for (size_t i = 0; i < ROSIDL_TYPE_HASH_SIZE; i++) {
	for (size_t i = 0; i < ROSIDL_TYPE_HASH_SIZE; ++i) {

https://google.github.io/styleguide/cppguide.html#Preincrement_and_Predecrement