Experiment with Node.js Website Traffic on Vercel

[ovflowd]

Hey you all 👋 as our ongoing efforts to improve the reliability and performance of the Website. The nodejs.org website codebase reached maturity for us to start the cutoff tests for serving the Website related routes through Vercel.

Useful Information

• The Website is served on https://node-js-org.vercel.app
• The Website is automatically deployed to Vercel by Vercel Integration on https://vercel.com
• An ideal subdomain for this temporary experiment is https://vercel.nodejs.org

Important Notes

• Cloudflare cache should be disabled for all routes served by Vercel. Vercel has its own caching rules, and those might conflict with Vercel-ones
• After analysing our NGINX file, the following subdirectories should not be served to Vercel:
  • /dist
  • /download
  • /docs
  • /api
  • /documentation
  • /metrics
  • /github-webhook.log
  • /traffic-manager
• Vercel should serve all other routes

Miscellaneous Information

• In the future, api/ should be served by the website, as the latest API docs will be served and built directly from the website repository. But that's a topic for another time.
• Once we adopt Vercel, over time, we can update the NGINX config to remove some of thew website-specific configurations we had.

[targos]

As a first step, I created the DNS entry:

Next: figure out how to make the SSL config happy

[ovflowd]

I guess here are some docs: https://vercel.com/guides/using-cloudflare-with-vercel

[ovflowd]

Pretty much:

Set the target on your CNAME to cname.vercel-dns.com instead of node-js-org.vercel.app and ensure SSL mode is set to full!

[ovflowd]

Oh, it looks like it solved by itself, @targos when I added it on https://vercel.com > settings > domains, but I guess you can still update the CNAME so it becomes happy (and agnostic to the default .vercel.app domain)

[targos]

Updated the cname to cname.vercel-dns.com.
Now for the subdirectories that shouldn't be forwarded to Vercel, I don't really know what to do.
I found this documentation, but the feature is only available to Enterprise users (which we are not).

[ovflowd]

@jasnell if you can help us out here 🤔

[ovflowd]

cc @richardlau let me know once you have updates :)

[richardlau]

I've added a (now corrected) Redirect Rule in Cloudflare for the URLs that should not be served to Vercel:

(http.host eq "vercel.nodejs.org" and http.request.uri.path matches "^/(dist|download|docs|api|documentation|metrics|github-webhook.log$|traffic-manager)(\\/.*)*")

[ovflowd]

Hmm @richardlau I see that the redirects would work as long as the "vercel" website is not on the main domain (nodejs.org)

As I can see that it is redirecting https://vercel.nodejs.org to -> nodejs.org

Afaik the intent is that for anything from X routes it goes to one server and the others would go to another server.

[ovflowd]

Pretty much the desired final behaviour is that:

• Caching for nodejs.org is disabled except for the dist|docs|etc routes
• All routes that do not match that regex go to Vercel DNS (or subdomain)
• All routes that match the regex go to DgitialOcean DNS (or subdomain)

But the hand-off from one origin to another would need to be transparent, without actual 301 redirects I think. (Like a Reverse Proxy)...

cc @jasnell if you know how we can do this

[richardlau]

Hmm @richardlau I see that the redirects would work as long as the "vercel" website is not on the main domain (nodejs.org)

Yes, I don't know how to get the desired final behaviour without redirects and/or subdomains. At least via the various rules (Page, Redirect, Bulk Redirect) available to us in Cloudflare.

[targos]

As I wrote above, there seems to be a Page Rules feature: https://developers.cloudflare.com/support/page-rules/using-resolve-override-in-page-rules/
But we cannot use it as it's only available to Enterprise accounts (we have a Business account)

[ovflowd]

FYI: Our Cloudflare instance got upgraded to Enterprise.

[ovflowd]

After chatting with @targos here's the outlined plan:

• We're making a new subdomain download.nodejs.org which goes to our DigitalOcean origin
• For testing environment (vercel.nodejs.org) we're making all requests under these matches below to go through download.nodejs.org (it's not a redirect, but an "Origin Override"
  • /dist
  • /download
  • /docs
  • /api
  • /documentation
  • /metrics
  • /github-webhook.log
  • /traffic-manager
• Everything else goes to the default origin of "vercel.nodejs.org"

After this test goes successful, we can present the outcome to the TSC.

[richardlau]

After chatting with @targos here's the outlined plan:

• We're making a new subdomain download.nodejs.org which goes to our DigitalOcean origin
• For testing environment (vercel.nodejs.org) we're making all requests under these matches below to go through download.nodejs.org (it's not a redirect, but an "Origin Override"

I'm kind of confused here. Downloads should remain with the https://nodejs.org/ URLs.

[ovflowd]

I'm kind of confused here. Downloads should remain with the nodejs.org URLs.

They will remain. I guess the confusion here is that vercel.nodejs.org and download.nodejs.org are just references to the origins. For all intends and purposes you will see nodejs.org on your browser :)

[targos]

The URLs wouldn't change. This would just be a hidden level of indirection:

• nodejs.org/dist -> download.nodejs.org/dist -> DO
• nodejs.org/en -> vercel.nodejs.org/en -> Vercel

[targos]

I think this will make the mental model easier to understand. And in the future, download.nodejs.org may point to Cloud storage instead of our server.

[richardlau]

• nodejs.org/dist -> download.nodejs.org/dist -> DO

Does this break the load balancer we have set up for nodejs.org (Traffic -> Load balancing)?

[richardlau]

FWIW Cloudflare has been mostly serving requests over the weekend from our Equinix Metal server and not the DO one.

[ovflowd]

Yeah, I noticed that now, but, afaik, I don't think so. The whole download and vercel aliases are abstractions. I think current rules still apply. But we also will need to double-check all our page rules. As we might need to remove some and add some simplification here.

Edit: Cloudflare team updated us and Load Balancers are respected.

[targos]

We will probably need to update (or copy) the load balancer config, as it currently applies to the host name "nodejs.org".
I think creating a new host name will also help us to setup everything without touching at the production config, and make it easier to swap when we are ready.

[ovflowd]

All changes got applied by @richardlau.

[ovflowd]

Current Configuration on Cloudflare

[ovflowd]

This can be marked as done. We concluded the migration steps.

Below is a summary of the steps taken today with @UlisesGascon:

• All existing Page rules besides the github-webhook.log one were removed
  • These rules were not needed anymore as our Middlewares on Next.js already covered what they were doing
• The AAAA record of nodejs.org was changed to CNAME pointing to Vercel's Domain Servers
• The DNS record new.nodejs.org was removed due to being obsolete
• The DNS record blog.nodejs.org was updated to use Proxied Traffic by Cloudflare
  • A Redirect Rule for blog.nodejs.org was made to -> nodejs.org/blog/
• The Origin Rule for Vercel Traffic was updated:
  • Pointing to origin.nodejs.org instead of nodejs.org
  • The vercel. prefix from vercel.nodejs.org was removed
  • foundation.nodejs.org and benchmarking.nodejs.org were added as part of the Origin Rule (so their traffic goes through origin.nodejs.org
• The Load Balancer configuration was updated by using origin.nodejs.org as the Hostname for the Load Balancer
  • A new Load Balancer was added; the original one is disabled and can be removed.
  • The new load balancer uses the same Pools and the same rules as the original one
• A Cache Rule was created to cache all non-Vercel paths
• Cache Reserve was enabled on the account to store all cached paths into Cloudflare's R2 Storage
• The outdated Vercel Redirect Rule was removed (as it was never used)
• The vercel.nodejs.org DNS record was removed as it is not needed anymore

We've monitored traffic and tested common and edge scenario paths and everything seems to be working fine.

[ovflowd]

Also, a note that the Webhook on nodejs.org website for rebuilding the Website on DigitalOcean's server on every push got disabled. (I've disabled the repository settings).

So this should also solve some of our current traffic pains.

[ovflowd]

Today we had a little incident, since Cloudflare was caching Vercel-paths, because we forgot Cloudflare's default behaviour is to cache everything.

This led to one of the Website rebuilds to Cloudflare to cache legacyMain.js from the Website build exactly during the rebuild, causing a corrupted empty file to be cached.

This led to: nodejs/nodejs.org#5568

We've fixed this by forcefully telling Cloudflare to not cache Vercel paths as Vercel requests should be served directly by Vercel at all times. The Cache Rule we configured is as shown below: