Benchmarking Test App

Overview

Benchmarking test app provides a simple harness for running React Native benchmarks on a real mobile device. The repository currently supports iOS benchmarks via Appium, and can be easily extended to support Android or any other platform supported by Appium and React Native. In this repository you'll find a few packages:

• apps/benchmarking-test-app: A React Native app with built-in benchmarks for bitECS, three.js, and ir-engine.
• packages/benchmarking: An Appium harness for running benchmarks within AWS device farm.
• packages/benchmark-parser: A utility library for parsing benchmark files generated for the app from AWS device farm.

There are currently two classes of benchmarks generated by the benchmarking test app: Xcode profiles and wall clock time benchmarks. The wall clock time benchmarks serve as a regression test for the benchmarks, tracking the performance of the profiled libraries over time, with results posted to: https://rebeckerspecialties.github.io/benchmarking-test-app/dev/bench/. When regressions happen, the Xcode instruments profiles can be used to dig into the root cause of the performance issues.

Using this repository

The benchmarks in this repository measure the performance of various Entity Component System (ECS) libraries and JavaScript graphics engines. However, the benchmarking test app can easily be used to profile any JavaScript code, and can even be extended to profile native modules. Anyone can use this repository as a template to create their own benchmark suite for their device test cloud setup.

We do not currently have a contribution guide or staffing to accept contributions, but it would make sense to set up extensions for running Android, Windows, and Mac benchmarks on a variety of test clouds. The repository is currently set up to run benchmarks on AWS, but it could be used on Firebase test lab with some wrangling.

Running benchmarks on a local device

In order to build and run the benchmarks on a local test device with Appium, there are a few prerequisistes:

General Development

1. Install node and npm

We recommend installing node with Node Version Manager (nvm): https://github.com/nvm-sh/nvm. This repository supports Node 20+

2. Install react native app dependencies

npm run install:hermes -w benchmarking-test-app

iOS Development (Mac Required)

1. Install/update Cocoapods

CocoaPods is installed by default on all Macs, but it may be out of date. Follow the instructions here: https://guides.cocoapods.org/using/getting-started.html to update Cocoapods. Cocoapods is used to manage native libraries for Xcode builds.

2. Install/update Xcode

Xcode is required for building iOS apps from native code. Download Xcode and Xcode command line tools from https://developer.apple.com/xcode/resources/

3. Install React Native pods for Hermes

cd apps/benchmarking-test-app
npm run install:ios:hermes

4. Connect an iOS device (or use an Xcode simulator)

5. Launch the benchmarking test app on the device

cd apps/benchmarking-test-app
npm run ios

6. (Optional) Build the app for release

Debug builds of the app may contain additional code that can affect performance. When collecting data, bundles should be generated for release.

cd apps/benchmarking-test-app
npm run build:ios:release
npm run ios -- --mode=Release

If you run into issues with "no development team provided", open the generated xcworkspace, click on ReactTestApp -> Signing, and add your development team.

Android Development

_ UNDER CONSTRUCTION _

Appium setup

1. Install Appium

npm i --location=global appium

2. Install the xcuitest driver for Appium

This allows Appium to run tests on an iOS device

appium driver install xcuitest

3. Run appium driver doctor to ensure that all dependencies are installed

appium driver doctor xcuitest

4. (Optional) Install the uiautomator2 driver for Appium

This allows Appium to run tests on an Android device. Android is not fully supported, but there is only iOS specific appium code in the xcode instruments profiling benchmarks.

appium driver install uiautomator2
appium driver doctor uiautomator2

5. Run appium

Run appium with the desired capabilities. the udid option can be omitted if using a simulator.

appium --relaxed-security --base-path=/wd/hub --default-capabilities \
  "{\"appium:deviceName\": \"<Your device name here>", \
  \"platformName\": \"iOS", \
  \"appium:app\": \"com.rbckr.TestApp", \
  \"appium:udid\":\"<Your device's udid>", \
  \"appium:automationName\": \"XCUITest\"}"

Running automated benchmarks

1. Install scripted benchmark dependencies

npm install -w benchmarking-scripts

2. Bundle the benchmark scripts

npm run bundle -w benchmarking-scripts

3. Run the benchmarks

node packages/benchmarking/dist/benchmark.js

Running automated benchmarks in AWS Device Farm

Creating an AWS Device Farm project

1. Navigate to https://us-west-2.console.aws.amazon.com/devicefarm
2. Click Create mobile project and follow instructions
3. Once the project is created, navigate to the Project settings tab and click Create device pool to select the devices you want your benchmarks to run on

Updating the GitHub Actions workflow

1. Update the devicePoolArn to match the device pool you created in the previous step. This will be configured via an environment variable in the future.
2. Add secrets for iOS provisioning. Follow the guide here for more information: https://www.andrewhoog.com/post/how-to-build-an-ios-app-with-github-actions-2023/
3. If you do not have a paid iOS developer account, you can find your DEVELOPMENT_TEAM by inspecting the Info.plist of any xcarchive you've generated. The ID will be a 10 digit alphanumeric value.
4. If you do have a paid iOS developer account, the development team ID can be found in the Mac Apple Developer application. You should also be able to remove the allowProvisioningUpdates option from the xcodebuild command in benchmark.yaml

Generating an IAM role for the DEVICE_FARM_IAM_ARN secret

1. Navigate to https://console.aws.amazon.com/iam
2. Click Roles in the side bar
3. Create a new role by clicking Create Role
4. Select Web Identity
5. Choose tokens.actions.githubusercontent.com as the Identity provider and fill in all of the fields below. You may need to create a new OpenID Connect Identity provider with this URL if it does not exist yet.
6. Once the role is created, paste it into the DEVICE_FARM_IAM_ARN secret in GitHub.

Running the benchmark GitHub action

Benchmarks will automatically run when a commit is pushed to the main branch on GitHub. Benchmarks can also be manually run from a branch by navigating to Actions -> Run Benchmarks -> Run Workflow.

Adding a benchmark

Benchmarks need to be registered in two places: within the React Native app and in the Appium test.

1. Write the JavaScript benchmark

// apps/benchmarking-test-app/src/benchmarks/fibonacci.ts
const ITERATIONS = 10000;

export const fibonacciBenchmark = async () => {
  let prev = 0,
    curr = 1,
    next = -1;

  for (let i = 1; i <= ITERATIONS; i++) {
    next = prev + curr;
    prev = curr;
    curr = next;
  }
};

2. Register the benchmark in App.tsx

// apps/benchmarking-test-app/App.tsx
import { fibonacciBenchmark } from "./src/benchmarks/fibonacciBenchmark";

const App = () => {
  return (
    <>
      ...
      <SafeAreaView ...>
        ...
        <Benchmark name="fibonnaciBenchmark" run={fibonacciBenchmark} /> // New benchmark
      </SafeAreaView>
    </>
  );
};

3. Register the benchmark in benchmark.ts

// packages/benchmarking/benchmark.ts
const runBenchmarkSuite = async () => {
  console.log("Running benchmarks with profiler");
  ...
  await benchmarkWithProfiler("fibonnaciBenchmark");

  console.log("Running benchmarks with wall clock time");
  ...
  await benchmarkWithWallClockTime("fibonnaciBenchmark");
};

4. Follow the directions above to rebuild the app and rebundle the appium benchmark

Profiling with Xcode Instuments

Currently, the Time Profiler is being used to profile the app with Appium. This can be swapped out for other profilers by chaning out the profileName argument in the startPerfRecord command in runBenchmarkWithProfiler.

A custom template can be used when profiling locally. For instance, if you want to profile CPU cache misses, you could create a CPU Counter template in XCode instruments and specify L1 cache misses in Instruments -> File -> Recording Options. For more details, see: https://www.advancedswift.com/counters-in-instruments/.

Profiling with Hermes flame graphs

To profile JavaScript and Hermes code, we can generate flame graphs with React Native Release Profiler: https://github.com/margelo/react-native-release-profiler. These profiles can be loaded into any performance trace tool to view JavaScript bottlenecks in the hot paths.

1. Launch the app.
2. Click the "Enable Flamegraph" button in the app.
3. Run the benchmark(s) of your choice.
4. Pull the cpu profiles from the device by running npm run download-profiles from the benchmarking-test-app workspace.
5. Generate source maps by running npm run build:android. Source maps are written to ./dist/sourceMap.js
6. Convert the CPU profile to a trace by running the following command for each file:

npx react-native-release-profiler --local Library/Caches/<filename>.cpuprofile --sourcemap-path dist/sourceMap.js

7. Import the trace into the performance trace of your choice, e.g. chrome://tracing.