Return to Tralla La or: RPM in C++

[pmatilai]

Ever since the RPM upstream reboot in 2006, we've been striving to replace the old-school, hand-written pointer gymnastics with general purpose APIs in the codebase. We've come a long way, but all this time we've been dreaming about richer data structures than C has to offer. Of course in C, you roll your own, but then it's time wasted on crafting basic tools instead of solving the actual problem you had. We believe RPM is being held back by the current implementation language, and we don't want to play Robinson Crusoe anymore.

Few people know this, but I actually ported RPM to C++ in 2010. Ported in the sense that it builds with a C++ compiler, and then ported some internal data structures to C++ native ones to get a feel for things. There were advantages certainly, but I came out of the experience with a resounding "ugh no" conclusion. The world just wasn't ready for that, for several reasons. It didn't stop the dreaming though, and the C++ topic has raised it's head a few times in the intervening years.

It happened again just recently, and this time we woke up to a language that seems almost like a distant cousin to the C++ I cursed at in 2010 (and before). Ever since C++11, the language usability has improved in leaps and bounds, whereas C has completely stagnated at C99. Which is at least part of the reason I haven't paid much attention to C++ either.

With the first major RPM package format update in two decades on the horizon, what better opportunity there could there possibly be?

This is NOT a rewrite from scratch operation, not even a rewrite, just a mere implementation detail. For RPM 6.0 in 2025, the plan is to enable use of C++ in the codebase. Fully taking advantage of it is for the following years. Here's the initial sketch for the transition, which will happen after 4.20 is branched (in the coming weeks):

1. RPM (or most of) is minimally ported to the common subset of C99 and C++17, with a build option to switch between them
2. The build option default is swapped to C++
3. The relevant sources are renamed to a C++ extension and the option removed.

As in: open the door, step through it and then close it. There will be no looking back.

The work on step 1 is already underway. Much of the required cleanup work from the "2010 expedition" was merged into the RPM codebase despite remaining in C, so this initial transition step is mostly about adding explicit casts in the places that C++ requires and C doesn't. Some new incompatibilities have appeared since then of course.

The public C API will remain of course, and will be the only available API in 6.0. There will eventually be a C++ native API alongside it though. For 6.0, the only C++ related goal is to enable use of C++ in the major data structures of rpm internally. In the process some amount of code will of course be moved to native C++, but there are no coverage goals.

Of course, moving to C++ is no panacea. It is a ridiculously large and complicated language, there will be quite the learning curve for us. But, we do believe that this will eventually pay back big time in terms of developer productivity for both old and new, and co-operation with other teams and projects in the RPM stack.

We have no intentions of going haywire with abstract design patterns here. The main interest initially is to gain access to more advanced data structures and RAII, but certainly there are places in rpm that will be nicer in proper OOP.

This transition also concludes our build infra upgrade: first we made the leap from autotools to cmake, then fakechroot to containers and as the final piece, the codebase itself gets dragged to this millenium. The initial plan is to target C++17 as this seems mature and widely available.

There are of course a thousand details not answered by this announcement, so ask away.

[cgwalters]

I have some experience with this; we did a similar thing in rpm-ostree starting around coreos/rpm-ostree#2336 (comment)
The rationale there was actually to aid porting to Rust, because we could use https://cxx.rs/

There were a lot of things there, see various PRs like https://github.com/coreos/rpm-ostree/issues?q=label%3Aproject%2Fc%2B%2B+is%3Aclosed around that time (I didn't label all of them, there were a lot).

Definitely dealing with NUL terminated strings versus C++ strings and Rust strings was and is a notable pain.

Probably the biggest ergonomic problem was exceptions...Rust and C align there, and "standard" C++ doesn't. Of course it's possible to use C++ without exceptions...but, it's an ecosystem bifurcation.

It is a ridiculously large and complicated language

Yes. Not to be That Person but...I dunno, I have a pretty strong opinion that Rust is the right choice for systems software, and for those bits you are rewriting, it can pay itself back. Rust is also very complex and was a huge learning curve for me (and others). The trap with C++ in a nutshell IMO is that it can feel very high level, yet low-level things like string_view use-after-frees will just come and bite you.

[pmatilai]

Yes, people keep bringing up Rust. It's just not a viable option for rpm, for a multitude of reasons. For starters, it's a show-stopper as a bootstrapping dependency for something as early in that chain as rpm. I won't go further because it'd be pointless because of that first reason.

[pmatilai]

As for exceptions: rpm's allocator has the terminate-on-failure behavior, which in practise means any call to the rpm API could abruptly terminate your process. Which is not a whole lot different from -fno-exceptions. That said, what exactly to do with exceptions remains to be seen. It's one of the thousand details to work out.

[jengelh]

Uncaught C++ exceptions also terminate the program, which is perhaps slightly better than a -fno-exceptions program not knowing what to do: While new T(); would return nullptr under libstdc++ with -fno-exceptions, no such check could be made for e.g. std::string s; s.resize(verylargevalue); because the return type is void.

[pmatilai]

Yeah I''ve been slowly approaching he conclusion that uncaught exceptions are probably the lesser evil afterall.

I mumble about rpm's allocator in the above and your comment made me realize that rpm's allocator does not get used currently for new, so all these places that assume allocation cannot return NULL are now in danger. Ooooops. Thanks for the comment!

[pmatilai]

Submitted #3325 while this is still fresh in memory...

[cgwalters]

For starters, it's a show-stopper as a bootstrapping dependency for something as early in that chain as rpm.

(Threading this) do you have a link to these discussions? There's a ton of work on bootstrapping Rust (and systems in general) on self-hosting OSes/distributions. The GUIX one is pretty cool. And for sure Rust gets handled as part of that, it's well documented and understood. I get that people who want to do something similar for self-hosting rpm-based systems would want to build rpm pretty early and switch over a bootstrap process to using it, but I'd imagine there's quite a bit of stuff that needs to be done using not-RPM before that that maintaining such a system would be quite close to needing to maintain two different build systems anyways, and having Rust in that set in addition to other compilers and toolchains that are already needed seems like not a large addition.

[pmatilai]

There was plenty enough surrounding rpm-sequoia.

Rust is not an option, that is not up to discussion. Like I said, the bootstrap issue is just one of multitude of reasons.

[nwalfield]

I understand that rpm is not ready for Rust today. But I wonder if rpm will be ready for Rust in 3 or 4 years.

In particular, the Rust ecosystem is quickly becoming a fundamental dependency. For instance, Rust is already used in the Linux kernel. So far, it is only used by non-essential code, but it is only a question of time before it is also used there. Given this, my expectation is that Rust's bootstrapping story will improve even more in the near future.

The flip side is it's easier to use Rust with C than it is with C++, as @cgwalters pointed out regarding error handling. As such, moving to C++ now will probably make it harder to move to Rust later.

So, my suggestion is: since you've already waited 15 years for C++, what do you think about waiting another 2 or 3 years to see what happens with Rust?

[cgwalters]

As such, moving to C++ now will probably make it harder to move to Rust later.

Well, maybe. My original comment here remember was about how we very intentionally moved rpm-ostree to "C compiling as C++" explicitly to bridge with cxx.rs. This...kind of worked in some ways, and definitely didn't in others - although I'd say at least 50% of that is just not executing well enough probably.

Using some C++ definitely doesn't exclude Rust in the future either.

[pmatilai]

If somebody wants to rewrite rpm from scratch in Rust in another 15 years when I'm retired, be my guest.

[dralley]

For what it's worth, it kind of already exists in a very limited capacity.

https://github.com/rpm-rs/rpm

It only handles parsing, signing, verifying and building (simple) packages via code, no CLI, no specfile parsing, no rpmdb handling etc.. basic functionality works well though.

[pmatilai]

Let me put it this way: if we had a will to move to Rust, maybe sitting out a few more years waiting for technical matters to sort themselves would be a meaningful option. That's just not the case.

[nwalfield]

Got it. My comment was meant to be constructive. I'm sorry if it didn't come across that way.

[Conan-Kudo]

For what it's worth, I'm excited about the transition to C++, because as a C++ programmer, I feel much more comfortable working my way through and cleaning things up leveraging the things I know well. So I'm looking forward to this!

[pmatilai]

Nice to see somebody besides ourselves being excited about this 😄

And yeah that is really a big part of the point: rpm's data structures aren't really that exotic, but to someone new it's all lost in the details of this specific implementation, and then we have like three different string buffer things and a whole ecosystem of C-string helpers that nobody but us knows are there, or how to use.
Once you know a handful of std:: stuff you know what they do and how to use them, everywhere.

[Conan-Kudo]

Yeah, I've always been afraid of broaching the idea seriously. I had joked about this with @ffesti a few times at the openSUSE Conference, but I'm really glad to see us doing this.

[jerome-diver]

@pmatilai because of poor documentation. Please share to make things alive and lovely the constructive way.

[pmatilai]

And we're live now with #3028 🥳 🙈

For the morbidly curious, there was certainly more to clean up before that, the major preparatory work being in
#2977, #2978, #3019, #3022 and #3027

That was quite the push up the hill, it'll be nice to get to the real thing now.

[jerome-diver]

1. What ever politic choice and speach has been done (for good reason, for sure), do you have an intention to document the API ? (i know it is something boring to do, but it is so helpfull for users)
2. And / Or take some 5 minutes of your time to provide some help about how to use it when user post the maximum information including there bugged code about what they failed to do because of wrong or not complete documentation and confused functions at any places when they try to use them and failed miserably ?