Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: add a guide to running Lighthouse at scale #10511

Merged
merged 5 commits into from
Sep 6, 2023
Merged

docs: add a guide to running Lighthouse at scale #10511

Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c66f8e8
docs: add a guide to running Lighthouse at scale
paulirish Mar 26, 2020
b7d001e
add another link to variability
paulirish Mar 26, 2020
d7dbdd4
wording
paulirish Mar 26, 2020
025c778
wording
paulirish Mar 26, 2020
1fab957
Update running-at-scale.md
paulirish Aug 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions docs/running-at-scale.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@

# Running Lighthouse at Scale

Many Lighthouse users want to collect Lighthouse data for hundreds or thousands of URLs daily. First, anyone interested should understand [how variability plays into web performance measurement](./variability.md) in the lab.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

love that this is a prereq :)


There are three primary options for gathering Lighthouse data at scale.

## Option 1: Using the PSI API

The default quota of the [PageSpeed Insights API](https://developers.google.com/speed/docs/insights/v5/get-started) is 25,000 requests per day. Of course, you can't test localhost or firewalled URLs using the PSI API, unless you use a security-concerning solution like [ngrok](https://ngrok.com/) to web-expose them.

A huge benefit of using the PSI API is that you don't need to create and maintain [a stable testing environment](./variability.md#run-on-adequate-hardware) for Lighthouse to run. The PSI API has Lighthouse running on Google infrastructure which offers good reproducibility.

* PRO: You don't need to maintain testing hardware.
* PRO: A simple network request returns complete Lighthouse results
* CON: The URLs must be web-accessible.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the other big con here is limited control over configuration, even if the URL is web accessible, testing behind auth, different form factors or throttling isn't really possible


Approx eng effort: ~5 minutes for the first result. ~30 minutes for a script that evaluates and saves the results for hundreds of URLs.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as someone who has written multiple systems that save lighthouse results to some sort of database now I think ~30 minutes is a massive underestimation of trying to save and query LH results in any way that isn't "dump these reports to a local filesystem"

I think a big con of 1&2 and pro of 3 is that you don't need to worry about doing a bunch of work to consume the data. There's a great emphasis here already on creating and maintaining a test environment which is definitely a big stumbling block, but the storage of historical data is also pretty complex and annoying to build. Being upfront about that might help folks choose the right solution for them.

e.g. if you've got some big bespoke storage plan for how you consume the data on your platform then LHCI isn't really bringing much to the table for you, but if it's something you haven't thought about at all, then you're gonna be in for a lot of frustration pretty quick with option 1 or 2 when you realize you can't do anything useful with this hunk of reports on your filesystem

WDYT about breaking the eng effort into 3 components instead of 2? "first result", "setup of larger system to collect", "setup of larger system to query/consume"


## Option 2: Using the Lighthouse CLI on cloud hardware

The [Lighthouse CLI](https://github.com/GoogleChrome/lighthouse#using-the-node-cli) is the foundation of most advanced uses of Lighthouse and provides considerable configuration possibilities. For example, you could launch a fresh Chrome in a debuggable state (`chrome-debug --port=9222`) and then have Lighthouse repeatedly reuse the same Chrome. (`lighthouse <url> --port=9222`). That said, we wouldn't recommend this above a hundred loads, as state can accrue in a Chrome profile. Using a fresh profile for each Lighthouse run is the best approach for reproducible results.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we use an example that we would endorse? :)

maybe custom headers/throttling options/puppeteer/etc?


Many teams have wrapped around the Lighthouse CLI with bash, python, or node scripts. The npm modules [multihouse](https://github.com/samdutton/multihouse) and [lighthouse-batch](https://www.npmjs.com/package/lighthouse-batch) both leverage this pattern.

You'll be running Lighthouse CLI on your own machines, and we have guidance on the [specs of machines suitable](./variability.md#run-on-adequate-hardware) for running Lighthouse without skewing performance results. The environment must also be able to run either headful Chrome or headless Chrome.

* PRO: Ultimate configurability
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PRO: supports on-premise and private URLs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to have some recommendations if you want to run lighthouse inside a Docker container.
We have been running lighthouse + puppeteer in this way and we have seen drastic changes from v5 to v6.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our general recommendation on docker is: don't if you can help it

Probably worth adding to this doc @paulirish the various issues we've seen with docker and why it should be avoided (shared memory issue and flags to workaround, etc) :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this on and old issue: #6162 (comment)

I've been looking for documentation about "Device Class" or if is still being used.
Lately I've started to get with V6 very different results, but if I contrast this with the benchmarkIndex property, makes a lot of sense.
I don't know if is worth to add to the documentation in case anyone want to use it.

In my case, I will create a bucket for devices (similar to the device class) and group the results by ranges of benchmarkIndex

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shelved the device class targeting because the absolute value of BenchmarkIndex has different characteristics across OS and CPU architecture, so we didn't feel confident enough to automatically adjust thresholds with it. The new plan is to simply put warnings in the report if we think it was running on an underpowered device (see #9085).

* CON: Must create and maintain testing environment

Approx eng effort: 1 day for the first result, after provisioning and setup. Another 2-5 days for calibrating, troubleshooting, handling interaction with cloud machines.

## Option 3: Using Lighthouse CI in the cloud

[Lighthouse CI](https://github.com/GoogleChrome/lighthouse-ci#readme) leverages the CLI at its core and it provides a complete experience for those who want to understand how each commit in development affects their Lighthouse results. So while the product is designed for running Lighthouse for every pushed git commit, it's possible to use it for some production monitoring usecases. See [this recipe](https://github.com/GoogleChrome/lighthouse-ci/issues/5#issuecomment-591578507) to fake git commit data while testing production URLs.

* PRO: Easy multiple-run configuration, selection of median run
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PRO: supports on-premise and private URLs

* PRO: Server UI offers timeseries graphs [(example)](https://lhci-canary.herokuapp.com/app/projects/d1e4b15c-e644-4552-b136-e975f486a2ce/dashboard) supported by [straightforward APIs](https://github.com/GoogleChrome/lighthouse-ci/blob/master/packages/server/src/api/routes/projects.js).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

discussed previously but I think this is like the PRO

* CON: Must create and maintain testing environment
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in this context, PSI + LHCI suddenly makes a lot more sense. I was pretty dismissive of PSI as a core CI use case, but production monitoring it'd be kind of a game changer. could run a simple cron on the LHCI server to request PSI results even, so it's a single docker deploy to get setup.

with that lens we might then even say "here are your choices for collection: PSI, self-maintained CLI" and "here are your choices for storage/querying: LHCI, bespoke storage thing™ "

* CON: Must setup and maintain Lighthouse CI server

Approx eng effort: ½ day for the first result, after provisioning and setup. Another 1-3 days for calibrating, troubleshooting, working with the API/UI.