Handling of multiple npm servers

[Daniel15]

Something to consider as a future enhancement, post-launch

It's possible to use custom servers with npm, it's just pretty annoying to handle it. The server is either set globally (via npm set registry) or every single time you run a command (via --registry flag, eg npm install --registry http://www.example.com/ test). Setting it globally only allows a single server to be selected We should be able to do better than that in Yarn.

I like the approach that NuGet takes. It has a default configuration file (%AppData%\NuGet\NuGet.Config on Windows) that only has their package source configured by default:

<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  </packageSources>
</configuration>

However you can easily add your own package sources to it, either by editing that file (which changes it globally) or adding a config to your project (which just changes it for that one project):

  <packageSources>
    <add key="ReactJS.NET Dev" value="http://reactjs.net/packages/" />
    <add key="ASP.NET Core Nightly Builds" value="https://dotnet.myget.org/F/aspnetcore-dev/api/v3/index.json" protocolVersion="3" />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  </packageSources>

How this works is that when you install a package, it looks in the first source to see if it's available there. If not, it continues down the list, and only fails if none of the sources contain the requested package.

Use cases:

• In other communities, it's common to have a separate package server for nightly builds, and save the "main" package feed just for stable releases.
• Corporate users may want to have an internal server for their private packages, while continuing to use public servers for public packages

---

Note that this issue is specifically for allowing the usage of multiple different servers (for example, the public npm server for some packages, and a private server for others). Allowing a single different server to be set (for example, if you have a caching proxy, or something like Sinopia that's both a package server and a proxy) is covered by #606. Thanks!

[sebmck]

I don't think this would work very well since the only way you can host your own registry is via npm enterprise which already handles this. Scopes and private packages are the solution to hosting your own packages.

[Daniel15]

the only way you can host your own registry is via npm enterprise

Says who? 😛

npm server just returns JSON and tarballs, there's no reason you couldn't host your own. You could fairly easily build your own like I did for NuGet with Simple NuGet Server. There's already software packages like sinopia (or its more well-maintained fork verdaccio) to handle it, too. sinopia/verdaccio handle caching of npm packages too, but a simple npm server that could be used with Yarn could avoid doing that and instead just serve the custom packages.

[bestander]

I have a mildly unsatisfying experience with sinopia.
Just saying :)

[sebmck]

In order to replicate the npm registry you need to implement a bunch of CouchDB endpoints. How would this interact with commands like publish, tag etc?

[bestander]

Afaik Sinopia implements publish command and other public api.
Ok for testing

On Friday, 7 October 2016, Sebastian McKenzie notifications@github.com
wrote:

In order to replicate the npm registry you need to implement a bunch of
CouchDB endpoints. How would this interact with commands like publish, tag
etc?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#547 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACBdWGEO5OWod9QicupOsPNdclsUjhz6ks5qxn68gaJpZM4KRQBU
.

[Daniel15]

I have a mildly unsatisfying experience with sinopia.

I just used it as an example, I'm sure there's better solutions. Nexus Repository Manager supports it too, for example (at Facebook we have a Nexus server for Java stuff + an internal Maven mirror)

In order to replicate the npm registry you need to implement a bunch of CouchDB endpoints

It doesn't have to be CouchDB specifically though, you just need something that returns data in the same format. NuGet uses OData for its old API, but I didn't use a proper OData server for my custom server, I just hacked something together in PHP.

How would this interact with commands like publish, tag etc?

NuGet's config has a default package source that it publishes to, and you can override it when publishing the package (eg. nuget push -Source http://www.example.com/). I imagine something similar would work here.

[0rvar]

We use a private NPM mirror/registry at my company. Currently we have a .npmrc file in the project directory, which Yarn seems to use (the registry url is correct in "info npm config"). However, yarn install cannot find the packages that only exist on our private registry. Yarn should respect the "registry" setting from the npm config.

[developit]

Just dropping a note here: Sinopia acts as a caching proxy to the official NPM registry, and layers its own local registry overtop. It implements the same CouchDB API but does not do any replication of CouchDB data. I wouldn't go so far as to call Sinopia stable, but we've been relying on it for almost 3 years now and have hundreds of modules published there.

[reflog]

same here, Sinopia user. would love to see yarn support!

[jdanyow]

Our private registry is proget with the npm feed type. We use scoped packages for our private stuff. I wasn't able to get the yarn command to work, even after adding a .yarnrc file as suggested here: #606 (comment)

[Daniel15]

Note that this issue is specifically for allowing the usage of multiple different servers (for example, the public npm server for some packages, and a private server for others). Allowing a single different server to be set (for example, if you have a caching proxy, or something like Sinopia that's both a package server and a proxy) is covered by #606. Thanks!

[nikolasleblanc]

Same thing, currently, it seems I can access either the private repo or npm, but not both. Granted, I've been using this for about 20 minutes, so maybe I've done something wrong. Current setup, I have a .yarnrc file with:

registry "[url]"

[benmonro]

I think I have the same issue as @nikolasleblanc

[jamietre]

We use Sinopia like @developit as both a caching server, and a private npm registry, with about 40 devs. It's been rock solid for almost 2 years now so (just as another data point).

Setting registry in .yarnrc seems to work, but it would be nice if yarn simply respected npm configuration for the default registry.

[schmod]

Artifactory can also host a private registry.

[ziir]

Same with https://github.com/verdaccio/verdaccio

[bcoe]

@kittens @bestander would be happy to lend a helping hand on this feature, in my copious amounts of OSS time -- one issue that immediately comes to mind is that we have special Varnish rules for scoped modules at npm, such that private content is never cached in our CDN ... before implementing private module support, there might need to be some additional logic added to the CloudFlare CDN fronting registry.yarnpkg.com.

What's the best way forward with work on this feature, should an RFC be drafted?

Update

1. looks like the CDN is setup to not cache if an authorization header is set, lovely :)
2. was playing with a interface that looks something like this; as a simple first pass at private module support that would work in a CI environment:

NPM_REGISTRY=https://registry.yarnpkg.com NPM_TOKEN=secret-token-4333-982-544444fa4966 yarn add @bcoe/hardwork26 --verbose

[bestander]

@wycats had great ideas how to go about community driven changes.
@bcoe IMHO use your best judgement depending on the size of change

[givankin]

It's an excellent discussion and it would be awesome if yarn could solve current problems with using multiple npm repos for in-company development.

Here's the typical scenario for a JS project within our organization:

• We have 2 instances of Nexus (which, as mentioned above, can serve a private npm registry), development and production
• Each instance has 2 repos - one for downloading packages (which in case of development also proxies public registry), another one for uploading. Weird maybe, but that's how it works.
• There is also separate Artifactory registry, so in some cases devs may need to use both of them (or to set up Nexus to proxy Artifactory)

Daily builds publish into development Nexus, production builds install from and publish to production registries. Remember, publishing is always to a different repo than installing from. Currently teams solving this by having:

• local (project-level) .npmrc with settings for development publishing registry. Note that publishConfig field in package.json looks perhaps a more logical place for this configuration, but it can't be used because it can't be overridden (--registry argument should override package.json publishConfig.registry npm/npm#5522), and it needs to be overridden for production build case.
• global (developer-level) .npmrc with developer's email and auth token for development Nexus.
• sh/bat npm_install files with hardcoded --registry argument - for installing from development registry (other registry than one in .npmrc).
• production servers with preconfigured global .npmrc, so that when npm install runs with no arguments, production registry is used to install dependencies.

This is a lot of boilerplate to set up, and even experienced developers still bump into installation failures when they forget that they must run npm_install.sh instead of npm install.

We tried to build some automation around this by switching registries on the fly but only bumped into other npm client peculiarities, such as npm/npm#7771

Now, I don't have any well-thought proposition for solving all these issues. I am adding it here to outline some real-world scenarios and pain that is currently associated with supporting them with npm, so that perhaps this discussion can contribute to yarn improvements.

I would personally be happy to contribute if/when there is an agreement on how yarn could work around such issues.

[schmod]

I think that a lot of us have built workarounds for several of NPM's idiosyncrasies, so I'd ask folks here to be open-minded towards solutions that might not align 100% with the (relatively contrived) ways that we're currently using NPM, and also mindful that we likely have a very wide range of distinct use-cases.

I'd also ask the participants in this thread to contribute their experiences with other package managers that have addressed some of NPM's shortcomings (without turning this into a referendum about all of the use-cases that NPM doesn't cover well).

So far, I've identified a few themes from others in this thread:

• We want the ability to resolve packages from multiple repositories.
  • It's unclear if scoped packages will accommodate all envisioned use-cases.
  • It's unclear how/where this should be configured. At the package level? Globally? Both?
• We want the ability to have more granular control over where packages are published.
  • There should be safeguards against accidentally publishing a private module to a public registry.
  • Apart from publishConfig, are there other things that Yarn should do differently during a production build?
• The purpose/structure of yarnrc has not been documented.

Also, a few thoughts of my own:

• The "out of the box" experience should not become any more complicated for users who are exclusively using the public registry.
• Should we support features of 3rd-party registries that are not present in NPM? Thinking again about production/internal workflows, NPM doesn't support snapshot dependencies, and has a fairly clunky process for prerelease builds.
• It would be nice if Yarn could produce a machine-readable report of any artifacts that were resolved or published during a build. Tools like Artifactory already have the ability to consume this kind or report from other package managers.
  • Do we want to provide hooks for CI servers to override resolution/publish settings for a particular build? This may be one way to have different "production build" configurations.

[jamietre]

The way we use this (with Sinopia) the only real concern is that configuration work the same way it does now with npm. Right now we can configure global default registry (npm set registry) and per-package with publishConfig. Sinopia is easily configurable to delegate packages reconciliation to an upstream registry based on the package name, so this solves all these problems already for us: prevents accidentally publishing to real npm registry, obtains/publishes scoped packages from correct repo, etc.

I would argue for compatibility first, rather than codifying more complex use cases that aren't currently supported by npm. While that might be something to think about as a future enhancement, presumably people are migrating from an existing, functioning environment given npm's current limitations. If the registry targets are resolved the same way they are using the real npm client, then I would think most people's environments should just work without any changes.

[devongovett]

See #839, which supports private registries the same way npm does.

[givankin]

@schmod , I second your point about "being open-minded towards solutions that might not align 100% with the (relatively contrived) ways that we're currently using NPM". npm is great for basic use cases, and it would be great to keep 100% compatibility for them so that it could be a drop-in replacement e.g. for nearly any open source project.

If you want to win the hearts of those who need a JS package manager for corporate development, then you could definitely do much better than npm. Commenting on your points, here are the immediate big (huge! awesome!) improvements I could see:

• allow publishConfig to be overridden by CLI argument (maybe you could think of better ways of publishing to different repos, this is obviously a low-hanging fruit)
• snapshot versions. This is huge advantage of e.g. Maven over npm. Very convenient for development.
• "machine-readable report of any artifacts that were resolved or published during a build" - great idea, especially if you add the ability to separate direct dependency tree from the tree of devDependencies and optional dependencies at the foundation. This is a common scenario - e.g. to hand over the list of direct dependencies to OSS evaluators or production registry gatekeepers.
• one last thing I can think of is that it may be convenient perhaps to have mutliple, easiliy-switchable "profiles" for a project. Such profile could include
  • (list of) registry(ies) to download artifacts from
  • registry to publish artifacts to
  • ability to inherit any value from global config (similarly to CSS inherit)

Can you elaborate a bit on the idea of CI hooks? Not sure I get how it would work.

[raghav135]

+1 yarn is not working with private npm registries. We are using sinopia.

[dsebastien]

At work we're also using the Nexus repository manager. We use it for Maven repositories but for npm as well. For npm our setup is as follows:

• we have a proxy registry that allows us to get packages from the official npm registry and cache them
• we have an internal registry where we publish our internal artifacts
• we have a virtual registry which exposes the contents of both the proxy and the internal registries

Developers all use the virtual registry in their global npm configuration. As pointed out above in this post, there are indeed ugly workarounds that we need to implement to be able to publish/retrieve artifacts to/from different registries using npm.

[Daniel15]

. As pointed out above in this post, there are indeed ugly workarounds that we need to implement to be able to publish/retrieve artifacts to/from different registries using npm.

Yeah, if we supported multiple servers natively in Yarn, you wouldn't need the "virtual registry" 😄

[schmod]

Virtual repositories are still commonly used with build tools (ie. Maven) that natively support multiple repositories.

[dramazani]

npm doesn't support any of the internal registry tools like proget... the problem for us is npm allows an extra hop after fetching something from the registry. This causes npm install to fail on projects of significant complexity like angular-cli

[KurtPreston]

Anyone know if there's currently any plans to add support for multiple servers in yarn? Looks like this issue's been stagnant for a few months now.

[bestander]

Not a priority of the core team. If it is easy to do then send a pr

…

On Fri, 24 Mar 2017 at 21:33, Kurt Preston ***@***.***> wrote: Anyone know if there's currently any plans to add support for multiple servers in yarn? Looks like this issue's been stagnant for a few months now. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#547 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBdWGSN-5rLijGbmnbLIRC1id36SxDzks5rpDaogaJpZM4KRQBU> .

[gforceg]

This would help with my use-case:

I am using TFS 2017 behind a firewall and publishing internal packages (specific to my organization) to the npm feed. However, the TFS npm feed is unable to cache upstream packages because of security restrictions.

It would be nice if I could specify a default registry and then a secondary named registry via dot notation.
e.g., I would say something like this:

$ yarn config set registry https://registry.npmjs.org
$ yarn config set registry.acme-dot-net https://local.registry.url

my .yarnrc would look something like this:

registry "https://registry.npmjs.org"
registry.acme-dot-net "https://local.registry.url"

(the above commands already product the above .yarnrc we just need to make yarn install consume the dot separeated named registry)

@Daniel15 would you consider working on a PR with me?

[gforceg]

addressing this might require 'sections' the way git configure handles sections
for each registry specified, you'll want to also be able to specify a proxy, https-proxy, ssl-verify, etc...

[ThePlenkov]

this is probably the reason why we cannot use yarn in our team.

we work at least with 3 different repositories:

1. npm
2. npm.sap.com
3. own artifactory instance

it's so easy to solve this issue in npm ( not proxies, scopes only ) with .npmrc file..

Why just not take same format?

Thanks!