Be able to host & use private crate registries

[repi]

This used to be important for us as we have a lot of private crates and dependencies between them.

But we have since moved to a monorepo for all of our Rust crates where it is easy to set up dependencies within the repository and workspace.

But believe for other companies this will be critical to adopt Rust.

One workaround that we've used in the past when having multiple private repositories of crates with dependencies between them is to set up the dependencies as git dependencies. The draw back of that is that it is both slower to download and more to set up on local machines and esp. CI.

• Alternative Registries Stabilization Proposal rust-lang/cargo#6589
• Tracking issue for RFC 2141: Add support to Cargo for alternative registries rust-lang/rust#44931
• Using alternative registries names in text output rust-lang/cargo#6691

[repi]

Closing this as Cargo has the support now and there is at least a first private cloud hosted crate registry with Cloudsmith: https://blog.cloudsmith.io/2019/05/01/worlds-first-private-cargo-registry/

[repi]

Reopened this because I tested out the alternate registry support together with cloudsmith and unfortunately Cargo doesn't support the main use case we have.

What we would like to do is to have a mechanism to privately publish individual crates that we have, that are in development, to a private registry and reference them in other private crates and repos.
This one can do by setting up a cloudsmith registry, adding that to .cargo/config and when referencing that crate using the registry keyword to use the private registry, something like this:

[dependencies]
embark-test = { version = "0.2.0", registry = "embark" }

And this itself does work, but only if the crate doesn't have any dependencies itself or if it only have dependencies that are in the same private registry and they are not also used from crates.io.

The first issue here for us is that all transitive dependencies of the crate from the private registry also have to exist in the private registry, so if say embark-test here also depends on the log crate that has to be duplicated to the private registry. And the second issue (which I've read but haven't confirmed myself) is that even if the log crate is identical in the private registry compared to the crates.io version, it will be duplicated and treated as a separate crate so you can't have some code using the crate and the types from the crates.io version together with code that use it from the private registry.

What we are actually after in our use case is that we just want to publish our private crates in our private registry, and use everything that doesn't specify registry = "embark" get it from crates.io, including transitive dependencies to our private crate.

There doesn't seem to be a way to do this today and I'm unsure how we should proceed

[lskillen]

@repi Lee from @cloudsmith-io here!

This popped up on my radar, so thought I'd offer some assistance. We recently made a change that would help alleviate at least the first problem. Enabling "Use crates.io as default Cargo upstream?" in your repository settings would allow for transitive dependencies to exist on crates.io without having to change configuration on the client-side. This is now enabled by default for new repositories (created after the functionality enabled). See screenshot:

Don't forget, we're always happy to help out if you're having issues. Sometimes it's possible that we need to make some changes on our side to improve your quality of life, but that's why we're there. :-)

[repi]

@lskillen oh that is super interesting and relevant, will try it out and see if it solves our use case.

big thanks for reaching out!

[lskillen]

Not a problem! If it doesn't, let me know and we'll sort it out for you. 😊

[repi]

Tested it now and just checking that "Use crates.io as default Cargo upstream" checkbox worked perfectly for us and resolves our issues!

It picks up the private crate that we have uploaded and specified that it should use the private registry, and all its dependencies (including transitive) gets picked up by crates.io and there is no duplication of crates or issues. Sweet!

Did run into a minor potential issue with Cargo in that its log output writes out the full URL for the registry instead of the name of the registry. And in the cloudsmith case this can include the secret entitlement token for it which unnecessarily leaks it into log files and such.

  Downloaded embark-test v0.2.0 (registry `https://dl.cloudsmith.io/<SECRET_TOKEN>/org/test/cargo/index.git`)
    Checking nalgebra-glm v0.4.0
    Checking embark-test v0.2.0 (registry `https://dl.cloudsmith.io/<SECRET_TOKEN>/org/test/cargo/index.git`)

Think it would be cleaner, nicer, and safer it Cargo just writes out the name of the registry here instead of the URL for it. Should be an easy fix in Cargo and found an issue for it here:
rust-lang/cargo#6691

[lskillen]

@repi You're absolutely right; we've put our thumbs up on that as well. 👍

In the meantime, it's possible to get around this by configuring basic authentication instead of using entitlement tokens; where the former is authentication via a header, and the latter bakes the token into the URI itself. Screenshot for configuration (accessed via "Set Me Up" from the top-right):

As Cargo is built upon the git protocol, it requires the credentials to be registered with git, using the credentials store. Quick example (from our own examples):

git config --global credential.helper store
echo "https://token:${CLOUDSMITH_API_KEY}@dl.cloudsmith.io" > ~/.git-credentials

In this case we're using an API key, and token is literally the string "token" to indicate that we're using an API key (or entitlement token, that works too) as the password.

Configuring this means that your credentials won't be baked into any output or logs.

[repi]

Cool thanks, may try out that approach also

[snawaz]

@repi

And this itself does work, but only if the crate doesn't have any dependencies itself or if it only have dependencies that are in the same private registry and they are not also used from crates.io.

The first issue here for us is that all transitive dependencies of the crate from the private registry also have to exist in the private registry, so if say embark-test here also depends on the log crate that has to be duplicated to the private registry.

We're implementing our own private registry at TenX. I encountered this case as well, but then I realized I was doing it wrong. With each dependency, the index is supposed to store the registry as well, so that Cargo could download the dependencies from their own registry. Interestingly, crates.io-index does not have the field registry in the json objects which — to some extent — makes sense because crates.io cannot depend on any other (private) registry. So I looked at the doc which mentions this field and explains it as:

 // The URL of the index of the registry where this dependency is
 // from as a string. If not specified or null, it is assumed the
 // dependency is in the current registry.
 "registry": null,

That explains really well. So I added this to my index and it worked.

And the second issue (which I've read but haven't confirmed myself) is that even if the log crate is identical in the private registry compared to the crates.io version, it will be duplicated and treated as a separate crate so you can't have some code using the crate and the types from the crates.io version together with code that use it from the private registry.

Since the first issue does not exist, the second issue will not arise in the first place,

What we are actually after in our use case is that we just want to publish our private crates in our private registry, and use everything that doesn't specify registry = "embark" get it from crates.io, including transitive dependencies to our private crate.

We're doing exactly this. So far we're able to do that, though we've not deployed our registry to production, but I'm hoping it's going to work.

[gilescope]

Is https://github.com/cswindle/caesium the only open source private registry?

[redmar]

@gilescope (and for other coming here) you also have https://github.com/Hirevo/alexandrie

[rafaelcaricio]

@gilescope There is also this one https://github.com/mcorbin/meuse (I have not tried using it yet myself).

[gilescope]

I’ve been making good progress with using cargo-local-registry and using a non-rust package manager to provision the crates there.

_{Sent with GitHawk}

[repi]

We're interested in being able to do a local instance of docs.rs and point it to or private crate registry as well, filed a separate issue on that in #47

[repi]

Closing this as the last Rust issue around CLI output was fixed a while back rust-lang/cargo#6691, we've had no problems at all with Cloudsmith as a private crate registry either, it is great