Future direction support field

[Eomm]

After the publishing of the draft for support field in package.json and the feedback process form the community, the ideas to extend moreover that field are:

• reference a file via an URL field
• a separate package-support.json file

reference:
#220 (comment)
https://github.com/nodejs/package-maintenance/pull/220/files#r310909069

[ljharb]

I’d want a separate file - so it can be immutably stored in package registries - but only if the file can be updated without a full publish, and also read without downloading the full tarball.

[lholmquist]

I’d want a separate file - so it can be immutably stored in package registries - but only if the file can be updated without a full publish, and also read without downloading the full tarball.

If you are able to change a file without a full publish, wouldn't that make it mutable? since you are changing something inside a particular version(it seems sort of analogous to a force push to master, maybe not as bad, but the same idea).

[ljharb]

@lholmquist it's always "mutable" with the current proposal in the sense that you can just publish another "latest"; what is immutable is the history of edits (which could easily be made programmatically available), and the availability of the file.

[lholmquist]

right, "latest" is mutable since it is really just a pointer to a particular version. but the files that that tag is pointing should be immutable. And if you are able to update a file in a particular version without a full publish/version bump, then i don't think we can say that that version is immutable. I think that is what i was trying to say in my other comment.

If you want to change that file, then i think you need to bump the version, and publish it.

I get there could be a programttic way of reading the history of edits, but that is an extra step that someone could miss, or not know about.

[wesleytodd]

I was working on a json schema for the spec and made this example trying to exercise many of the features of the spec. Imagine this in our package.json 📦 🤦‍♂

I think it is clear that we need an external file. That said, I think there is a strong case for making it relative to the package (so the file must be shipped with the package), not any URL (as @ljharb poitns out leads to issues).

[ljharb]

I'm not personally bothered by tons of data being in package.json, fwiw.

[wesleytodd]

I think many people will be bothered by it. It seems to me we can start by supporting "support": "support.json" style because it is local to the package install. This would allow those who care to separate it out, and not hinder any of our core goals around history and immutable availability.

[Eomm]

The problem is that npm let you download only the package.json (in two sizes, one minimal with less information, and one with all the metadata).
If you want something else you need to download the tarball

[mhdawson]

Adding to @Eomm comment, if a tarball needs to be downloaded then that greatly increases the download required for tools that consume the new information.

[ljharb]

Right - that’s the reason why, until npm adds a feature or two, package.json is the only place it should go.

[kemitchell]

@ljharb, thanks for pointing me here from npm/cli#187. I hope folks don't mind a few stray thoughts and up-to-speed questions.

I think it's safe to say that a great many package consumers rely heavily on URL-based pointers from package.json files already. Specifically, once we know that repository has a GitHub or BitBucket URL, we know that's our best bet for up-to-date information on bugs, changelog, patch submission, and consumer-to-producer comms more generally. When we open the code host page, we can look to README.md---the latest on master, not the one that came from the registry---for CI badges, documentation site links, chat channel buttons, and so on. There's just no guarantee that yesterday's hot new channel will make it to the next version. We've all seen patch releases replacing Slack channels with Gitter, or Gitter with Slack, or Slack with IRC or just issues again. A lot of us have tried a few different sustainability-as-a-service platforms, and forgotten to update some of our packages.

If you're trying to do automated dependency assessment, perhaps by modeling risk on the availability or throughput of support channels, I don't think the pain point is lack of information. Rather, it's that some dev has to sift through all the repo webpages, click all the right links, and tally up all the actually relevant URLs. Much the same story if you're trying to take some kind of action---send thank-you e-mails, send donations, ask for support contracts---across many packages. Some of what PACKAGE-SUPPORT seems to be gunning for is structured data for facilities that haven't been well structured before, like support channels. But big chunks of it, specifically backing, basically map packages to other URLs.

How do we anticipate people using the data, wherever developers end up storing it? Is it safe to assume that requests for this data (npm support?) will be far less frequent than requests to install the packages (npm install)? Is it safe to assume that people will get to specific package versions through their node_modules directories, or will they come to packages as packages, with an assumption they'll keep up-to-date with the latest release?

I suspect the back-and-forth on where to put this data, no matter how it's encoded, reflects unexplored differences in how you imagine the data being consumed. But that's a guess. I'm a recent interloper here. ;-D

[wesleytodd]

I hope folks don't mind a few stray thoughts and up-to-speed questions.

The more input and questions the better!

I think it's safe to say that a great many package consumers rely heavily on URL-based pointers from package.json files already

You make a good point, but I would also like to add the counter argument that many package links are broken. I often click github links which have are a 404, I am sure the other cases are similar. I don't say this as a blocker to the idea, just that it is not a perfect solution.

Some of what PACKAGE-SUPPORT seems to be gunning for is structured data for facilities that haven't been well structured before

Yes this is IMO one of the main goals of this proposal. If we have structure we can write reliable tooling, educate the community on it in a reasonable way, and help a broad range of people instead of each group going in their own direction.

But big chunks of it, specifically backing, basically map packages to other URLs.

"Basically" I agree, but the devil is in the details. I think the structure we have defined gives more than just a list of links. The difference between "I want to donate to projects I use" and "I want to use projects which offer paid support plans" is quite different. Just a list of links does not help answer this question.

For example, I think a company might ask the second question, and IMO getting companies to "pony up" is one of the best ways to raise meaningful financial support for projects. Having this explicitly in the spec is important for this goal IMO.

Is it safe to assume that requests for this data (npm support?) will be far less frequent than requests to install the packages (npm install)?

Well, if we do it well I hope that we can do something like funding was doing where it only showed once and is a relatively clear and clean UX. So if we do that, then we can run npm support on all installs, and it would just show the new or important information to users. Very similar to npm audit's behavior.

Is it safe to assume that people will get to specific package versions through their node_modules directories, or will they come to packages as packages, with an assumption they'll keep up-to-date with the latest release?

I can see running npm support package-name and have it look locally then remotely. I can also see this interaction also being something that always does a network request to get the up-to-date file. Maybe doing npm support --local so that you can run w/o network.

I just think we can evolve the use case over time as we see clear reason. I think the important thing is making sure we don't make any of these impossible in the future. So to me, allowing url's which are not resolved into node_modules excludes the use case of --local, which to me means we should not allow it in the spec for now until we can understand more of the use cases.

[Eomm]

For tracking: the discussion in npm/cli#246 is continuing with a wide feedback from Isaacs

[wesleytodd]

Also, if anyone here wants to chime in there and provide different viewpoints than the one I am expressing, it would be helpful :)

[mhdawson]

The --local option is interesting, although my first thought is that if I was using the tool to verify support levels for the packages I'm going to deploy I would want to ensure I have the latest info. Having said that the info can change immediately after I check it so not having fully up to date info must be ok in some situations as well.

[wesleytodd]

not having fully up to date info must be ok in some situations as well.

I am not sure we can get around this, we are just in a world of imperfect information. But anything is better than what we have today. With tooling at least we can monitor and check for changes over time.

[mhdawson]

I am not sure we can get around this, we are just in a world of imperfect information. But anything is better than what we have today. With tooling at least we can monitor and check for changes over time.

+1

[Eomm]

Adding this to the agenda since the feedback from npm has a great impact on how to continue in this issue, in order to don't create confusion in the community of course

Ref: #241 (comment)

[Emuentes]

In our meeting today, we discussed the possibility of leaving BOTH options open, allow people to choose whether they want to drop a link or a valid JSON property within the package.json

Is there a reason we absolutely MUST pick one or the other before sharing our recommendation for the general structure of the support property?

[ljharb]

Yes, I remain opposed to allowing anything that's not inline, until such time as npm allows updating of metadata without a version bump.

[lholmquist]

until such time as npm allows updating of metadata without a version bump.

This is probably a discussion for another thread, but i feel like if that happens, then that is breaking the contract of that particular version, even if it is metadata and doesn't affect the functionality of what the code does. Sorry for the noise, but just wanted to get this thought out there

[ljharb]

This would have to be metadata separate from the version tarball.

[ljharb]

Surely anyone could write the PR, but the key question is, would they accept it :-)

[mhdawson]

@darcyclarke any comments on the suggestion in: #241 (comment)

[ghinks]

@Eomm when you said that there are 2 types of package.json that can be downloaded on August 28th one with meta data how are the mechanisms different to fetch them, I ask as I thought plan old curl https://registry.npmjs.org/expressjs was the only option. ( just part of my background on the blog for support :) )

[Eomm]

I was referring to this doc.

For what I saw, the full version is the double time the lighter's version in the worst case.

[ghinks]

I made a few updates to the google draft document as we suggested.

After some time for reflection, which is always good, I am hoping we can reach a compromise. As I do not believe either method could be fully satisfied currently without getting the registry involved.

[darcyclarke]

@Eomm #issue-481708057

• reference a file via an URL field
• a separate package-support.json file

I'm personally in favor of the URL and then having the exhaustive support information live outside of the package.

@mhdawson #issue-542424019
@darcyclarke any comments on the suggestion in: #241 (comment)

I'll try...

@mhdawson #issue-540748913

• Ask for both inline and url (ie URL which is link to latest version of the data as well as json for the current version) in the package.json (this is different suggestions which were OR, it is AND)

I'm not sure I understand the benefit of doubling up on where this information lives, even in the interim.

• At some point in the future we'd hope that the npm client could update the package.json on publish with the latest from the link automatically.

I'm a bit concerned with having npm agree to, at some point in the future, pull in any data automatically for a publisher; I feel like that strategy would be prone to conflicts and cause us a bit of pain from the tooling/support side.

• This ensures that the data is available both online and offline without adding to the load on npm in terms of fetching data.

Again, I'm not sure if package maintainers would want to keep two copies of the same information up-to-date even in the interim (not sure if that's what's being suggested here).

On a separate yet related note, I've filed an npm RFC for a much smaller chunk of work to support a funding field in package.json with the hope to land it in the next minor release (ie. v6.13). As I've noted in a few of the various calls we've had, I believe that the current scope of the support draft is trying to achieve a bit too much at one time.

I believe that focusing on the monetary opportunities, first, for maintainers helps move the needle forward and aligns with npm's commitment, we made to the community, at the end of August. I've taken some of the concepts outlined for the backing field, specifically, and scoped what the implementation would look like on our end.

I'd love to discuss this more in tomorrow's meeting with the hope that other active/vocal community members can join or provide their own feedback (ie. @ljharb @feross @arcanis @zkochan). Please let me know if there's anything I need to do to specifically add this as an agenda item. <3

[ljharb]

We had a great discussion in our meeting today.

Following that, @mhdawson and I had a productive chat, and here's what we'd like to propose: a package.json field called support which is one of:

• true: This will imply your package.json's repository URL (including directory), with a default path value of ./package-support.json
• a string: a relative path, starting with ./, that points to a file within the repository specified in the existing repository field
• object, with repository key matching the schema for the existing repository field, and optional path field that if provided, must be a relative path, starting with ./, within the specified repository.

The content of the file will match the schema we’ve already discussed.

Here’s the use cases we envision:

1. common case: "support": true with an associated package-support.json file in your repo at the top level.
2. much less common case: an alternate file path within the package’s repo, such as "support": "./path/to/support.json"
3. much much rarer case: sharing the same support data across multiple repos. Examples might be

• ”support”: { “repository”: { “type”: “git”, “url”: “node/.github” }, “path”: “support.json” }
• ”support”: { “repository”: { “type”: “git”, “url”: “node/.github” } }
• ”support”: { “repository”: { “type”: “git”, “url”: “node/.github”, “directory”: “packages” } }

These will resolve to the location of a JSON file (called package-support.json by default) that matches the schema discussed above.

Our recommendation will be to use our tooling to manage package.json, and to be sure to include package-support.json (or your alternative filename) in your files array, or, be sure that .npmignore does not exclude it.

The effect of this is that, by default, each publish will contain a snapshot of the support data, but the true source of truth is always a file within the package’s repository (or within an external repository). This supports offline and “no external internet” use cases as best as can be achieved.

By using a repository, we are using a known set of protocols (git, svn, etc), and creating a pit of success for folks to both store their support data in version control, and also include a snapshot of it in their published packages, with no additional effort.

Our tooling will not allow accessing any arbitrary repository URL by default - to avoid privacy concerns, we will survey npm packages for the most common repository hosts (presumably, github, bitbucket, gitlab, etc), and allow-list these by default. Users of our command line tool will be able to override the allowlist, or bypass it, as desired.

In the future, if npm sees value in this field, they could, for example, add validation into npm publish, surface this information on npmjs.com, and/or inline the support data into the packument so that npm view $package support or similar produces the data.

[arcanis]

I'd love to discuss this more in tomorrow's meeting with the hope that other active/vocal community members can join or provide their own feedback (ie. @ljharb @feross @arcanis @zkochan).

Thanks for the ping! I'm currently in vacations so attending is difficult, but I appreciate being at least aware of the discussion 👐

The proposal described by @ljharb would solve the main issue I had with this proposal. 👍

My only remark is that the repository field already has a way to specify a path within a repository. It currently doesn't have any effect, but we've been open to leverage it to connect a package to its real location (for example to print the readme / changelog). In this context, I think it would be a good idea to specify where the path is relative from when a directory field is present (and I'd suggest it to be the package directory rather than the root).

[ljharb]

@arcanis the intention is to respect the directory field (that the npm website already respects afaik); my comment includes an example of doing so.

[lholmquist]

@ljharb that outline looks really great.

I thought i had while reading is would it be beneficial to try and get the "common case" creation supported in npm init? probably another issue to discuss

[ljharb]

Absolutely it would :-) but that’s something we can do iteratively, after we’ve demonstrated some community adoption of the support field via our own tooling.

[mhdawson]

@nodejs/package-maintenance can you chime in on the latest proposal in: #241 (comment). If we can get consensus on this we can move on to the next phase of getting broader input.

[kemitchell]

Folks: Just FYI, I have to unsubscribe here. I'm slammed now, but finally getting some desperately needed time off next week.

Please feel free to @-mention me if there's some specific place where I can help. But please understand that I won't be responding for the next week or so.

[Eomm]

This supports offline and “no external internet” use cases as best as can be achieved.

Is this assuming the "no external internet" has access to the most common repository hosts right?
I just want to be sure to have understood all the cases.

[ljharb]

No, in those scenarios, they would be relying on the contents of the tarball.

[Eomm]

So should the 3rd use case ("sharing the same support data across multiple repos") download the file on installation?

I don't understand when the fetch happen in that scenario

[ljharb]

Yes, ideally - but i don’t think that will be a common use case, and our tooling could still download it prepublish for those making that extra effort.

[mhdawson]

Next steps:

• PR the latest suggestion from @ljharb and @mhdawson into the doc. (Michael)
• Update blog post
• Start brainstorming/working on tooling to support.
• Promote

[wesleytodd]

Start brainstorming/working on tooling to support.

This was the compliant JSON schema for the previous version. Maybe update that with the changes?

[Eomm]

This was the compliant JSON schema for the previous version. Maybe update that with the changes?

Yes, I think the direction more stable now and we could improve the pkgjs/support package: do you agree to open some issue there to split the job?