[code-infra] Add canary release scripts #41949

michaldudak · 2024-04-17T21:01:04Z

Created scripts in infra packages that can be used to publish them after each PR to the main (master or next) branch.

The workflow looks like this:

Check if there are any uncommitted changes. Fail if there are.
Set up .npmrc to look for an access token in an enviroment variable.
Figure out which packages changed between this and the previous commit.
Update package.json of the changed packages, adding the -dev.[yyMMddHHmmss] suffix.
Publish the changed packages.
Restore package.json files to the original state.

Runnning pnpm canary:release executes all these steps.
It requires an npm access token present in the NPM_TOKEN environment variable.

I'll create a GitHub action running the script in a separate PR, once this one is merged.

mui-bot · 2024-04-17T21:54:21Z

Netlify deploy preview

https://deploy-preview-41949--material-ui.netlify.app/

Bundle size report

No bundle size changes (Toolpad)
No bundle size changes

Generated by 🚫 dangerJS against c7d7190

Janpot · 2024-04-18T07:45:09Z

I'm not 100% sure, but I think there is a problem with only publishing changed packages in this scenario

suppose A has a dependency on B
start situation: A@v1=>B@v1 + B@v1
commit 1: only A changes => publish only A, new situation: A@v2=>B@v1 + B@v1, ✅
commit 2: only B changes => publish only B, new situation: A@v1=>B@v1 + B@v2, ❌
=> don't have a version of A that points to B@v2

And even if you also publish everything with a dependency on B, then you get

commit 2: only B changes => publish B and dependants, new situation: A@v2=>B@v2 + B@v2, ✅

but now:

commit 3: only A changes => publish only A, new situation: A@v3=>B@v1 + B@v1, ❌
=> doesn't know about B@v2 because repo always resets versions after publishing

Only correct way is to always publish everything, changes or not

commit 3: only A changes => publish everything, new situation: A@v3=>B@v3 + B@v3, ✅
=> now B@v2 is incorporated in B@v3

Or am I misunderstanding how this works? If we can't rely on semver resolution then shouldn't the whole dependency tree reflect the desired situation, regardless of changes?

I'm also touching on this in https://www.notion.so/mui-org/code-infra-Release-process-299a2fb7fe93441b8e345af0b2fc4fb4?pvs=4#7ba482d1725147bc8f312b6836f64b5e

michaldudak · 2024-04-18T09:55:18Z

I see. The script publishes changed packages and all the packages that depend on it, but "=> doesn't know about B@v2 because repo always resets versions after publishing" is indeed a problem.

Only correct way is to always publish everything, changes or not

This will create a massive number of packages that are essentially the same (especially considering that we don't change the infra code very often).
How about we check for changes between the last commit and the last stable release instead?

Janpot · 2024-04-18T10:04:52Z

How about we check for changes between the last commit and the last stable release instead?

That could also work, if that's what we already do for stable releases anyway, I don't see an immediate problem with it. I'm changing https://www.notion.so/mui-org/code-infra-Release-process-299a2fb7fe93441b8e345af0b2fc4fb4?pvs=4#7ba482d1725147bc8f312b6836f64b5e

michaldudak · 2024-04-24T13:54:48Z

I ended up actually implementing both: first, the script checks if there are any changes in public packages between this and the previous commit (there is no point in releasing packages if nothing changed). Then we look for differences between this commit and the latest tag (or any other commit - this is configurable).

michaldudak · 2024-04-24T16:30:09Z

I did a test run - the resulting packages were published to npm with the -dev.240424162023-9968b4889d suffix:

@mui/codemod
@mui/core-downloads-tracker
@mui/joy
@mui/material
@mui/docs
@mui/icons-material
@mui/lab
@mui/material-nextjs
@mui/styles
@mui/system

Janpot · 2024-05-03T14:16:58Z

scripts/canaryRelease.mts

+
+async function setVersion(packages: PackageInfo[]) {
+  const { stdout: currentRevisionSha } = await $`git rev-parse --short HEAD`;
+  const timestamp = formatDate(new Date());


🤔 I was just thinking, we should be able to read the date from the commit. That way we make it impossible to publish the same commit twice.

What's the issue with publishing the same commit twice?

To clarify, "Twice occasinally" is not really my main concern. It's about making this script idempotent. When multiple runs of the script result in the same outcome. This makes it have an inherent resiliency which comes in handy when we're going to automate it. It becomes safe to do automatic retries, and an accidental misconfiguration can't make you end up with 100 times publishing the same package.

Good point!

oliviertassinari · 2024-05-04T22:47:22Z

scripts/canaryRelease.mts

+        newVersion = version.slice(0, dashIndex);
+      }
+
+      newVersion = `${newVersion}-dev.${timestamp}-${currentRevisionSha}`;


In the format, is it possible to drop the time from the date? https://www.npmjs.com/package/typescript?activeTab=versions seems clearer and good enough since we have the commit too in our case.

Before:
6.0.0-dev.240424162023-9968b4889d

After:
6.0.0-dev.20240424-9968b4889d

Actually, maybe even?

6.0.0-dev-2024-04-24-9968b4889d

would be clearer.

This was done to ensure the version suffix is always increasing. In the TypeScript case, it's OK, as they publish one version a day. If we're targeting to publish each commit, having time is required to ensure increasing versions.

oliviertassinari · 2024-05-04T22:49:27Z

scripts/canaryRelease.mts

+  return date
+    .toISOString()
+    .replace(/[-:TZ]/g, '')
+    .slice(2, 14);


Why slice the year? It feels like it breaks the pattern matching of my brain (and others?) to recognize what the string is about.

Suggested change

.slice(2, 14);

.slice(0, 14);

Does it matter what this is about, though? It's just to ensure that version numbers increase.

michaldudak · 2024-05-07T08:16:31Z

Updated the version number to reflect the commit date and changed the format to -dev.yyyyMMdd-hhmmss-sha (for example @mui/material@6.0.0-dev.20240507-080616-4ea4f9e382)

Janpot

Good for me 👍

michaldudak · 2024-05-08T09:47:07Z

One more addition I made - the ability to ignore certain packages. I thought it could be useful to prevent uploading the huge @mui/icons-material after each commit.

michaldudak added 10 commits April 17, 2024 20:11

Version update script

363adfd

Set up npm scripts

8008925

Refactor

be2ce14

Use default package path

2a3577a

Add scripts to more packages

0f6720e

Clean up after publishing

7d7fd8c

Add trailing newline to package.json

a4648cc

HEAD instead of next

7300d3e

Date format

fcd0ec6

Set up npmrc

a3770ec

michaldudak added the scope: code-infra Specific to the core-infra product label Apr 17, 2024

michaldudak requested a review from a team April 17, 2024 21:01

Update lockfile

aa1371c

Detect changes between this commit and the last tag

c1c8399

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label Apr 23, 2024

michaldudak added 4 commits April 24, 2024 11:41

Merge remote-tracking branch 'upstream/next' into publish-script

232d6de

Add SHA to the version number

36272b7

Move all logic to a script

a65768e

Compare against the latest commit

463465f

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label Apr 24, 2024

michaldudak added 3 commits April 24, 2024 16:14

Fix passing access token with CLI

1765bd7

Revert unnecessary changes

b24c70b

Improve DX

bfbf7d9

michaldudak force-pushed the publish-script branch from 418630e to bfbf7d9 Compare April 24, 2024 15:26

michaldudak marked this pull request as ready for review April 24, 2024 15:28

michaldudak added 3 commits April 24, 2024 17:52

Fix argument passing

f60faf3

Build packages before publishing

9968b48

Add and use the build:public:ci script

c343af0

Janpot mentioned this pull request Apr 25, 2024

Bump @mui/monorepo digest to 2d02b94 mui/mui-toolpad#3427

Merged

1 task

Janpot reviewed May 3, 2024

View reviewed changes

oliviertassinari reviewed May 4, 2024

View reviewed changes

michaldudak added 4 commits May 7, 2024 09:47

Use commit date

fde6942

Merge remote-tracking branch 'upstream/next' into publish-script

d3bf046

Update the error message

761af2c

Update date formatting

233166a

Janpot approved these changes May 7, 2024

View reviewed changes

Ignore packages

c7d7190

michaldudak merged commit 8f88c8b into mui:next May 8, 2024
19 checks passed

michaldudak deleted the publish-script branch May 8, 2024 10:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[code-infra] Add canary release scripts #41949

[code-infra] Add canary release scripts #41949

michaldudak commented Apr 17, 2024 •

edited

Loading

mui-bot commented Apr 17, 2024 •

edited

Loading

Janpot commented Apr 18, 2024 •

edited

Loading

michaldudak commented Apr 18, 2024

Janpot commented Apr 18, 2024 •

edited

Loading

michaldudak commented Apr 24, 2024

michaldudak commented Apr 24, 2024

Janpot May 3, 2024

oliviertassinari May 4, 2024

Janpot May 5, 2024 •

edited

Loading

michaldudak May 7, 2024

oliviertassinari May 4, 2024 •

edited

Loading

oliviertassinari May 4, 2024

michaldudak May 7, 2024

oliviertassinari May 4, 2024 •

edited

Loading

michaldudak May 7, 2024

michaldudak commented May 7, 2024

Janpot left a comment

michaldudak commented May 8, 2024

[code-infra] Add canary release scripts #41949

[code-infra] Add canary release scripts #41949

Conversation

michaldudak commented Apr 17, 2024 • edited Loading

mui-bot commented Apr 17, 2024 • edited Loading

Netlify deploy preview

Bundle size report

Janpot commented Apr 18, 2024 • edited Loading

michaldudak commented Apr 18, 2024

Janpot commented Apr 18, 2024 • edited Loading

michaldudak commented Apr 24, 2024

michaldudak commented Apr 24, 2024

Janpot May 3, 2024

Choose a reason for hiding this comment

oliviertassinari May 4, 2024

Choose a reason for hiding this comment

Janpot May 5, 2024 • edited Loading

Choose a reason for hiding this comment

michaldudak May 7, 2024

Choose a reason for hiding this comment

oliviertassinari May 4, 2024 • edited Loading

Choose a reason for hiding this comment

oliviertassinari May 4, 2024

Choose a reason for hiding this comment

michaldudak May 7, 2024

Choose a reason for hiding this comment

oliviertassinari May 4, 2024 • edited Loading

Choose a reason for hiding this comment

michaldudak May 7, 2024

Choose a reason for hiding this comment

michaldudak commented May 7, 2024

Janpot left a comment

Choose a reason for hiding this comment

michaldudak commented May 8, 2024

michaldudak commented Apr 17, 2024 •

edited

Loading

mui-bot commented Apr 17, 2024 •

edited

Loading

Janpot commented Apr 18, 2024 •

edited

Loading

Janpot commented Apr 18, 2024 •

edited

Loading

Janpot May 5, 2024 •

edited

Loading

oliviertassinari May 4, 2024 •

edited

Loading

oliviertassinari May 4, 2024 •

edited

Loading