Discussion: new iceoryx organisation structure.

[elfenpiff]

To give new comers, fans and other contributors an overview of all iceoryx extension we created an eclipse-iceoryx organization under which all iceoryx related repositories should be stored. This is inspired by the eclipse-zenoh project.

At the moment there are some projects out there and it would be nice if they would all have the same home. This list provides you with a glimpse of the extensions where people are currently working on or where there are ideas to realize them. This could then be structured under eclipse-iceoryx like that:

repository name	description
iceoryx-meta	integration repository
iceoryx	the core repository which is developed according to our safety standards
iceoryx-utils	STL constructs, Posix wrapper, concurrent structures developed according to our safety standards
iceoryx-experimental	the newest and coolest stuff
iceoryx-platforms	required to build iceoryx in windows, mac os and on a C64
iceoryx-binding-c	language binding for C
iceoryx-binding-python	language binding for python
iceoryx-binding-rust	language binding for rust
iceoryx-binding-lua	language binding for lua
iceoryx-binding-bash	language binding for bash for command line usage
iceoryx-gateway-dds	iceoryx cyclone dds gateway
iceoryx-gateway-mqtt	iceoryx mqtt gateway
iceoryx-demonstrator	demonstrator
iceoryx-introspection	introspection client

Now some questions come into my mind:

• Continuous integration: When changing on repository how do we make sure that all depending repositories are still building?
• Should we move the original eclipse/iceoryx repository into eclipse-iceoryx/iceoryx?
• Are there some open questions or issues I am missing?
• Should all repositories which are intended for high safety usage have a special name like iceoryx-safety?

The goal of this issue is to have a good and long lasting strategy on howto organize a project which has extensions, tools etc. which are developed by "third parties".

@budrus , @elBoberido, @FerdinandSpitzschnueffler , @dkroenke , @MatthiasKillat , @mossmaurice , @marthtz, @ithier what are your opinions?

Todo

• Untangle RouDi / TOML dependency and create certification strategy for TOML

[elBoberido]

My suggestion would be to move eclipse/iceoryx to eclipse-iceoryx/iceoryx else it would be confusing to have repositories in two separate organizations. It should be possible to transfer the repo to another organization and keep all the issues, PRs, forks, stars, etc.
https://docs.github.com/en/github/administering-a-repository/transferring-a-repository

Not sure about the -binding- in the language bindings. I do not know that from other projects and e.g. eclipse-zenoh also doesn't has it.
I wouldn't use the safety suffix either.

Could we add iceoryx-website?
Should we move RouDi to an own repo?

The continuous integration is an interesting topic. These guys (https://invent.kde.org/frameworks) manage around 100 repos for a release, so it's not impossible. Maybe we need some scripting for this.

[elfenpiff]

Not sure about the -binding- in the language bindings. I do not know that from other projects and e.g. eclipse-zenoh also doesn't has it.
I wouldn't use the safety suffix either.

I on the other hand like expressive names. I know the binding suffix from package names from some linux distributions and I liked it.

Could we add iceoryx-website?

This list is not complete so definitely yes!

Should we move RouDi to an own repo?

I would leave RouDi in iceoryx as long as we do not have a specific need to separate him from iceoryx.

The continuous integration is an interesting topic. These guys (https://invent.kde.org/frameworks) manage around 100 repos for a release, so it's not impossible. Maybe we need some scripting for this.

[elBoberido]

The idea was to have just one binary/library per repo

[orecham]

I think the "one binary per repo" idea is a bit excessive, too many repos can get annoying...
I like the current breakdown.

Also I was initially a little unsure about the -binding category but it kind of makes sense to indicate that they are built on top of our C++ core.

[dkroenke]

I like the idea too that we get a better structuring of our repositories, especially when more components like gateways, bindings etc. will come up in future. Other big OSS projects like ROS 2 do it the similar way.

Regarding the naming:
Yes we should have a eclipse-iceoryx organisation in github, i vote for it. But i would recommend to remove the prefix "iceoryx_" from the underlying repo names because it seems a little bit redundant for me. Of course there should be a core-repository with the name iceoryx in it to have a starting point for newbies.

Regarding the structure:
The main point is that more repositories lead to more complexity in the integration of them, because we need consistency for releases and we want always have a running iceoryx + dependencies. In ROS 2 there are also a lot of subrepos and they found a good way how to deal with it.

1. Repository structure:
   They have a ros2 core repository where only a list of repositories is in it which is used by the developer + jenkins to checkout different versions of the complete iceoryx. See https://github.com/ros2/ros2/blob/master/ros2.repos
   You can see that they are defining the repos and the according versions to it. with that we can define releases which point to a certain version in the sub repositories.
   The main aspect here is that they are defining the master as main develop branch for all subrepos and therefore also the PRs in the subrepos go into master.

This is a bit similar to the iceoryx-meta package which aims also to manage everything from iceroyx.

2. Build structure:
   Using Cmake for putting all repositories together is not really feasible because it's not designed for that. We have then to think of alternatives like Colcon or Conan to integrate the packages. It must be good to manage and easy to use for devs.
   Possible solution: Colcon https://colcon.readthedocs.io/en/released/#
   It is used by the devs and jenkins to build multiple packages at your own choice. In ROS2 the colcon build system is reading the .repos file from the core repository to build the packages. You can of course also build only a subset of packes like only the gateways, Colcon will resolve the dependcies automatically, i like.
   My advise is that we also create a core repo with all subrepos in it and then create colcon packages in all you submodules. Its kinda easy and the existing CMake files will still persist. This should be done before we split up the repos.
   My guess is that having colcon packages in every subrepo is not that much work.
   iceoryx_posh and utils are already colcon packages and used in rmw_iceroyx.

3. Jenkins
   My guess is that the github workflow will not meet our requirements anymore when we want to build multiple packages. We need a jenkins instance where we can put scripts into it to have a scalable build.
   For consistency we need also define some rules for the jenkins:
   In every PR of a subrepo the code and the dependencies should be build and tests needs to be run. This needs to be good defined because if you have a change in iceroyx_utils (which is used everywhere, i guess) it should be verified that the subrepos are still green.

You see that this topic covers more aspects than naming which we should think of to have a clean solution. As far as i can see ROS 2 has no problem when putting all repos together, so we should maybe stick to it.

[elBoberido]

I think the "one binary per repo" idea is a bit excessive, too many repos can get annoying...
I like the current breakdown.

How about one library/binary per CMakeLists.txt?

[elBoberido]

Regarding the naming:
Yes we should have a eclipse-iceoryx organisation in github, i vote for it. But i would recommend to remove the prefix "iceoryx_" from the underlying repo names because it seems a little bit redundant for me. Of course there should be a core-repository with the name iceoryx in it to have a starting point for newbies.

Even if it's redundant, I would keep the prefix. Our cmake projects would have the same name like the folder with the root CMakeLists.txt and when we create deb/rpm packages we would just use the project name and still have distinct package names.

I don't have strong feelings about the naming, though.

[elBoberido]

1. Repository structure:
   They have a ros2 core repository where only a list of repositories is in it which is used by the developer + jenkins to checkout different versions of the complete iceoryx. See https://github.com/ros2/ros2/blob/master/ros2.repos
   You can see that they are defining the repos and the according versions to it. with that we can define releases which point to a certain version in the sub repositories.
   The main aspect here is that they are defining the master as main develop branch for all subrepos and therefore also the PRs in the subrepos go into master.

This is a bit similar to the iceoryx-meta package which aims also to manage everything from iceroyx.

While this is quite a convenient solution, I think this makes bisecting quite hard. We would need a consistent snapshot of all these repositories with each merged PR.
I don't know if something like this is possible, but what do you think of this:

eclipse-iceoryx
    |- iceoryx (repo)
    |    |- iceoryx-posh (submodule)
    |    |- iceoryx-utils (submodule)
    |    |- iceoryx-foo (submodule)
    |    |- git-tools
    |        |- checkout.py
    |        |- branch.py
    |        |- pull.py
    |        |- push.py
    |- iceoryx-posh (repo)
    |- iceoryx-utils (repo)
    |- iceoryx-gateway-dds (repo)

Instead of using git directly, we would use the scripts from the git-tools folder, which would e.g. step into the submodules and pull each of them or commit all of them.
The PR would be created on iceoryx and a git hook would (magically) also merge the branches in iceoryx-posh, etc.
This would be a lot of work but maybe there is already something like this available.

[elBoberido]

One problem would also be the issue numbers we need for tracing. With multiple repos, each repo has it's own issue numbers.
We would either have

• use a separate prefix for each repo, e.g. iox-#13, iox-dds-#13, iox-lua-#13, etc
• create a repo just for tracing and use those issue numbers in the particular repos; we would then also need to close the issues for all repos except the issue tracking repo and would also loose such nice features like automatically closing issues when a branch is merged
  Rust does something similar like the latter suggestion, but only for larger stuff and not bugfixes or documentation changes
  https://github.com/rust-lang/rfcs/
  I think it would be quite cumbersome to do this for all issues and also complicated the relation of issues to the respective repos. On the other hand, we would have consistent tracking numbers across repos.

[elBoberido]

accidently closed the issue

[bishibashiB]

Stumbled over the following github option: declaring code owners e.g. for a few of your planed subfolders.
Thought it might help in organizing ( ... if you didn't consider it already.)
.../creating-cloning-and-archiving-repositories/about-code-owners

[dkroenke]

For all Committer: You should have a pending invitation for the organization "eclipse-iceoryx", if so please join, if not let me know. We have the plan to move the iceoryx repo into eclipse-iceoryx for having a better management of repositories.
Iceoryx itself and the access to it will not change and all existing links (including weblinks) will be redirected by Github. There is no immediate action on your local fork needed and all your issues will still be on "eclipse-iceoryx/iceoryx". We decided to move iceoryx into an own organization because we want to create repositories beside it like the iceoryx-website

Ticket on eclipse: https://bugs.eclipse.org/bugs/show_bug.cgi?id=565706