support alternative TLS backends

[decathorpe]

It looks like the in-progress HTTP/3 support for hyper (as implemented in the h3 crate) will use quinn as its backend. Currently, quinn only supports rustls for its TLS backend. This will cause problems for less-used compilation targets in the future:

rustls is built on ring, a BoringSSL fork with limited architecture support. So while rustls can claim that it's technically a "pure-Rust TLS implementation", it depends on the ring crate, which is mostly written in C / Assembly (due to it being a BoringSSL fork, duh).

This is a problem, especially for linux distributions, when HTTP/3 support in hyper will become mainstream. It will severely limit the architecture support of all applications that use it, given that ring has limited architecture support. I'm also concerned that the last stable release of ring is now over a year old, which strikes me as non-optimal, given that it's a crypto library, and there have surely been security problems in BoringSSL that have been fixed in BoringSSL upstream since the last release of ring.

I see that the rustls dependency in quinn is already "optional", but there is currently no support code for other TLS backends. It would be great if native-tls support were added, as this should be supported on all architectures that Rust itself is supported on (for example, native-tls uses OpenSSL as its backend on Linux, and cargo itself also depends on OpenSSL).

If you think that this task would be doable and not too hard, I can try to work on a pull request for this myself.

[Ralith]

This is difficult because QUIC requires special low-level cryptographic APIs from the TLS implementation, which are not normally exposed for TLS over TCP. I'm unsure what the state of these APIs are in OpenSSL (last I heard, they were scrapping them entirely in favor of shipping their own QUIC implementation...) and I'm not sure if other platform libraries expose the necessary primitives at all. native-tls itself certainly does not bind the necessary interfaces, and I would be surprised if it was possible to do so while abstracting over the underlying TLS implementation as iirc native-tls otherwise attempts to.

We'd be happy to have this, and I can offer limited guidance, but implementing it will require significant research and work on upstream projects, and may require platform-specific code for each target of interest.

[djc]

Apart from Ralith's comments (which I definitely agree with), some other thoughts/questions:

It looks like the in-progress HTTP/3 support for hyper (as implemented in the h3 crate) will use quinn as its backend. Currently, quinn only supports rustls for its TLS backend. This will cause problems for less-used compilation targets in the future:

Do you have particular targets in mind that you worry about? I guess this is mainly in your role on the Fedora Packaging Committee, or do you have a more concrete/specific use case?

rustls is built on ring, a BoringSSL fork with limited architecture support. So while rustls can claim that it's technically a "pure-Rust TLS implementation", it depends on the ring crate, which is mostly written in C / Assembly (due to it being a BoringSSL fork, duh).

I don't think it's still correct as this point to say that it's "mostly" written in C / Assembly, I think a decent amount of progress has been made converting C/Assembly to Rust over time (in part to enable support for more targets).

I'm also concerned that the last stable release of ring is now over a year old, which strikes me as non-optimal, given that it's a crypto library, and there have surely been security problems in BoringSSL that have been fixed in BoringSSL upstream since the last release of ring.

I am certain that, If any of the BoringSSL vulnerabilities would have had corresponding vulnerabilities in ring, there would've been a ring release. In fact, most BoringSSL security problems have not had corresponding vulnerabilities in ring to date, so I wouldn't worry about this particular part too much. (It is true that ring maintenance has not been very active in recent times, causing other problems. There are some ongoing discussions in the background about how to deal with that.)

I see that the rustls dependency in quinn is already "optional", but there is currently no support code for other TLS backends. It would be great if native-tls support were added, as this should be supported on all architectures that Rust itself is supported on (for example, native-tls uses OpenSSL as its backend on Linux, and cargo itself also depends on OpenSSL).

Given the amount of potential complexity in having a QUIC/TLS implementation abstracting over OpenSSL, SChannel and Secure Transport, I'm not super convinced that the proper place for such an implementation would be in this repo, but we'd certainly happy to provide guidance and/or modify the interface to support efforts in that direction.

[decathorpe]

Do you have particular targets in mind that you worry about? I guess this is mainly in your role on the Fedora Packaging Committee, or do you have a more concrete/specific use case?

Well, this is not a purely hypothetical problem. We are already dealing with this in Fedora Linux. And I'm not really speaking as a member of the Packaging Committee here, but as the person who is primary maintainer for 2000+ packages for Rust crates.

In Fedora Linux, our supported architectures are x86_64, i686, aarch64, powerpc64le, s390x, and, up to Fedora 36, armv7 (will no longer supported starting with Fedora 37). By default, all packages are built on all architectures, and architecture support is opt-out instead of opt-in. This means we have very comprehensive support for some architectures that could be considered "esoteric".

On the other hand, this also makes it rather painful to deal with Rust crates which only have limited architecture support: Builds of packages for the affected crates and every other package of a Rust crate that depends on them need to opt-out of building on, in this case, powerpc64le and s390x architectures. This is manageable for the 2-3 packages that we have which depend on ring, but right now, I'm in the process of actually removing optional features that need rustls where I can, because that support is unused and hard to support.

However, the problem will get much worse once widely-used crates, like hyper (via h3 and quinn) start adding a (non-optional) dependency on rustls / ring. At that point, it would probably be easier to stop building Rust crates on the two unsupported architectures completely - but we cannot do that, because some new distribution-critical components have been introduced, which were either written from scratch in Rust, or were ported from C or Python to Rust, and many of them are network stack related, with many of them using hyper.

I don't think it's still correct as this point to say that it's "mostly" written in C / Assembly, I think a decent amount of progress has been made converting C/Assembly to Rust over time (in part to enable support for more targets).

I apologize if this was a mischaracterization on my part. I based this on the statistics shown by GitHub on the ring project's repo, which currently lists 46.1% Assembly, 34.7% Rust, and 14% C, and the rest are scripts which won't end up in compiled binaries. Only counting the three compiled languages, that amounts to ~2/3 of the crate being written in Assembly and C.

I know that some code was being (re)written in Rust to support more architectures, but this effort seems to have stalled:

• powerpc64le support issue: Support Power/PowerPC briansmith/ring#389
• powerpc64le support PR: Initial ppc64le Support briansmith/ring#1057
• s390x support issue: "cargo build" failed on s390x platform briansmith/ring#986
• s390x support PR: Add basic support for s390x briansmith/ring#1297

All of these issues are still open after years, without comments from the project's maintainer for the past 4-5 months. (Side note: The commit activity in ring's git repo also stopped 4 months ago, and the maintainer's GitHub profile lists zero activity since then.)

I am certain that, If any of the BoringSSL vulnerabilities would have had corresponding vulnerabilities in ring, there would've been a ring release. In fact, most BoringSSL security problems have not had corresponding vulnerabilities in ring to date, so I wouldn't worry about this particular part too much. (It is true that ring maintenance has not been very active in recent times, causing other problems. There are some ongoing discussions in the background about how to deal with that.)

OK, that's good to know.

Given the amount of potential complexity in having a QUIC/TLS implementation abstracting over OpenSSL, SChannel and Secure Transport, I'm not super convinced that the proper place for such an implementation would be in this repo, but we'd certainly happy to provide guidance and/or modify the interface to support efforts in that direction.

I understand that I underestimated the complexity of this task, I did not realize that native-tls wouldn't expose enough of the underlying mechanisms as APIs for this purpose.

This is difficult because QUIC requires special low-level cryptographic APIs from the TLS implementation, which are not normally exposed for TLS over TCP. I'm unsure what the state of these APIs are in OpenSSL (last I heard, they were scrapping them entirely in favor of shipping their own QUIC implementation...)

At this point, I'm unsure what could be a way forward here. Using OpenSSL doesn't sound like a good idea if they're potentially not keeping around the necessary APIs, Rustls doesn't support all the architectures I would need, and native-tls doesn't expose the necessary low-level APIs. I haven't found any existing Pure-rust implementation of TLS, so ... do I need to write one myself to make this happen?

[djc]

On the other hand, this also makes it rather painful to deal with Rust crates which only have limited architecture support: Builds of packages for the affected crates and every other package of a Rust crate that depends on them need to opt-out of building on, in this case, powerpc64le and s390x architectures. This is manageable for the 2-3 packages that we have which depend on ring, but right now, I'm in the process of actually removing optional features that need rustls where I can, because that support is unused and hard to support.

That seems a pretty unfortunate trade-off given the fact that rustls (with webpki and ring) has had a much better security track record than OpenSSL (and even BoringSSL), so you're trading off substantially improved security on the most common architectures against building on (relatively) "esoteric" architectures. Probably not the way I would go.

However, the problem will get much worse once widely-used crates, like hyper (via h3 and quinn) start adding a (non-optional) dependency on rustls / ring. At that point, it would probably be easier to stop building Rust crates on the two unsupported architectures completely - but we cannot do that, because some new distribution-critical components have been introduced, which were either written from scratch in Rust, or were ported from C or Python to Rust, and many of them are network stack related, with many of them using hyper.

I'm pretty sure the H3 support in hyper will be optional (like the H2 support).

I don't think it's still correct as this point to say that it's "mostly" written in C / Assembly, I think a decent amount of progress has been made converting C/Assembly to Rust over time (in part to enable support for more targets).

I apologize if this was a mischaracterization on my part. I based this on the statistics shown by GitHub on the ring project's repo, which currently lists 46.1% Assembly, 34.7% Rust, and 14% C, and the rest are scripts which won't end up in compiled binaries. Only counting the three compiled languages, that amounts to ~2/3 of the crate being written in Assembly and C.

I know that some code was being (re)written in Rust to support more architectures, but this effort seems to have stalled:

* powerpc64le support issue: [Support Power/PowerPC briansmith/ring#389](https://github.com/briansmith/ring/issues/389)

* powerpc64le support PR: [Initial ppc64le Support briansmith/ring#1057](https://github.com/briansmith/ring/pull/1057)

* s390x support issue: ["cargo build" failed on s390x platform briansmith/ring#986](https://github.com/briansmith/ring/issues/986)

* s390x support PR: [Add basic support for s390x briansmith/ring#1297](https://github.com/briansmith/ring/pull/1297)

Yeah, the situation maybe isn't as good as I made it out to be.

At this point, I'm unsure what could be a way forward here. Using OpenSSL doesn't sound like a good idea if they're potentially not keeping around the necessary APIs, Rustls doesn't support all the architectures I would need, and native-tls doesn't expose the necessary low-level APIs. I haven't found any existing Pure-rust implementation of TLS, so ... do I need to write one myself to make this happen?

rustls is a pure-Rust implementation of TLS, it just depends on ring for its cryptography needs. So it would probably still be valuable to reuse rustls, while replacing/forking ring. Some discussion in rustls/rustls#521 and related concerns about the long-term viability of ring in rustls/rustls#1034 (comment). If you think Fedora/RH would be in a position to contribute to these efforts, maybe send me a private email?

[decathorpe]

That seems a pretty unfortunate trade-off given the fact that rustls (with webpki and ring) has had a much better security track record than OpenSSL (and even BoringSSL), so you're trading off substantially improved security on the most common architectures against building on (relatively) "esoteric" architectures. Probably not the way I would go.

I don't disagree, but I cannot wave a magic wand here. These are constraints that I cannot change.

I'm pretty sure the H3 support in hyper will be optional (like the H2 support).

Right - but at some point, some applications we need to support will start to use it, and then it's not really optional any more.

rustls is a pure-Rust implementation of TLS, it just depends on ring for its cryptography needs. So it would probably still be valuable to reuse rustls, while replacing/forking ring. Some discussion in rustls/rustls#521 and related concerns about the long-term viability of ring in rustls/rustls#1034 (comment). If you think Fedora/RH would be in a position to contribute to these efforts, maybe send me a private email?

Speaking for Fedora, there's only 1-2 people who keep everything Rust-related afloat. The only support we get from Red Hat is the maintenance of rustc and cargo themselves. Other than that, Red Hat employees who work on Rust code have been frustrating to work with - they do not contribute to Rust crates or Rust packages on Fedora at all, except for the handful of crates that they develop, publish, and maintain themselves, and often resort to "shurtcuts" or do things in broken or unorthodox ways, and basically, just throw things at the wall, but then not even stay around to look if things actually stuck.

However, a RustCrypto-based backend for rustls would be great, and would basically solve all problems with Rustls. RPM packages for the crates in the RustCrypto stack are relatively easy to maintain, and we already have many / most of them in Fedora. I cannot promise anything here, neither from the Fedora side (Rust packaging is an entirely volunteer- / community-led effort by 1-2 people, mostly just me), nor the Red Hat side (people are either difficult to work with, difficult to get a hold of, or are generally uninterested in contributing to anything that's not strictly required by - or related to - their work). However, I personally would be interested in helping make this happen, if it looks like a doable and worthwhile effort, just because it would make my life easier. 😄

[Ralith]

Using OpenSSL doesn't sound like a good idea if they're potentially not keeping around the necessary APIs

This is worth investigating in earnest to confirm, as we haven't been closely following their planning (rustls is far more convenient to work with, after all). You could also build something relying one of the (presumably portable) TLS implementations used by other QUIC impls, like https://github.com/h2o/picotls, though of course these less-supported implementations are a risk themselves.

[Ralith]

For comparison, ngtcp2 also supports a forked OpenSSL, GnuTLS, and BoringSSL. At least those latter two might provide more options for portability, at least until rustls can mitigate the issues with ring.

[decathorpe]

Small follow-up: I would consider this problem moot if support for different crypto backends were added in rustls (i.e. rustls/rustls#521 / rustls/rustls#1184), since the problem is not caused by rustls itself, but by the ring crate. So big thank you to everyone who's working on this. 🙇🏼‍♂️

[djc]

For now, the rustls backend in Quinn has a direct dependency on ring. So some extra work would need to be done on Quinn even if rustls exposed an interface to switch crypto backends.

[Ralith]

I think the scope of that work is pretty modest, though.

[Ralith]

A BoringSSL backend has been contributed at https://github.com/quinn-rs/quinn-boring.

[djc]

And pluggable crypto backends are coming to rustls soonish.