[Performance] SearchForReports took 110602ms (~2min) to complete, seemingly blocking the whole app

[hannojg]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

What performance issue do we need to solve?

The customer was opening NewDot and doing a search. The account is quite big. When running search for the first time in the app the SearchForReports took 110 seconds. During that time the user already clicked on a search result, but the user was only brought to the chat once the search query finished.

This issue can be split into two aspects:

• A backend issue that should optimise the SearchForReports query
• A frontend issue thats about the app hanging while the query got executed

See this recording:

search_taking_forever.mov

What is the impact of this on end-users?

• Very high TTI to open search page
• Very long time for search results to come in

List any benchmarks that show the severity of the issue

The customer shared a profile trace with us:

Firefox 2024-07-25 10.42 profile.json.gz

(note: the trace also contains other test cases as well)

Additionally we have a HAR recording of the network requests that took that long. I hope that helps in identifying the query in the backend log system:

www.expensify.com_07-31-2024-16-53-27.har.zip

Proposed solution (if any)

None yet, I will go through the profile and see what can be optimised, what exactly caused those lags.

List any benchmarks after implementing the changes to show impacts of the proposed solution (if any)

not available yet

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android: Native
• Android: mWeb Chrome
• iOS: Native
• iOS: mWeb Safari
• MacOS: Chrome / Safari
• MacOS: Desktop

Version Number: v9.0.11-5
Reproducible in staging?: not tested
Reproducible in production?: yes
Email or phone of affected tester (no customers): customer
Logs: See performance file
Notes/Photos/Videos: See attached video
Expensify/Expensify Issue URL: n/a
Issue reported by: @hannojg
Slack conversation: https://expensify.slack.com/archives/C05LX9D6E07/p1721919928992729

View all open jobs on Upwork

[melvin-bot]

Auto-assigning issues to engineers is no longer supported. If you think this issue should receive engineering attention, please raise it in #whatsnext.

[hannojg]

cc @sakluger (feel free to assign me as I (or someone from my team) will work on this ticket!)

[hannojg]

Related to:

• [Performance] Opening search for the first time on a big account is causing the app to hang massively #46590
• [Performance] Typing in search is very laggy for big accounts #46591

[marcaaron]

Just for context, on the backend side, it is a known issue that SearchForReports can sometimes be extremely slow.

[sakluger]

No update

[sakluger]

No update

[sakluger]

No update

[hannojg]

I just opened a PR hat shows a PoC for using web workers for running the search, which will unblock the main thread:

• [WIP] WebWorker Demo (to improve search performance) #47432

I am now seeking approval on this new technical design here.

(Note: I am OOO for the rest of the week and will continue next week Monday 19.08)

[sakluger]

Moving to weekly for now since Hanno is out until Monday.

[hannojg]

@kirillzyusko do you have time to look into that in parallel? Should be able to just slow down the SearchForReports requests using a tool like Proxyman to try to reproduce this!

[kirillzyusko]

@hannojg sure, I'll look into this 👀

[sakluger]

@kirillzyusko thanks for helping! FYI I'll be OOO for the next week - hopefully, there will be some interesting new learnings when I return on 8/30.

[kirillzyusko]

@marcaaron may I kindly ask you to verify a duration of this request on BE:

POST https://www.expensify.com/api/SearchForReports? HTTP/1.1
Host: www.expensify.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Referer: https://new.expensify.com/
Content-Type: multipart/form-data; boundary=---------------------------224365322038742656772464040988
Content-Length: 2128
Origin: https://new.expensify.com
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
Priority: u=4
Pragma: no-cache
Cache-Control: no-cache
TE: trailers

-----------------------------224365322038742656772464040988
Content-Disposition: form-data; name="searchInput"

sasha@expensify.com
-----------------------------224365322038742656772464040988
Content-Disposition: form-data; name="canCancel"

true
-----------------------------224365322038742656772464040988
Content-Disposition: form-data; name="apiRequestType"

read
-----------------------------224365322038742656772464040988
Content-Disposition: form-data; name="authToken"

REDACTED
-----------------------------224365322038742656772464040988
Content-Disposition: form-data; name="referer"

ecash
-----------------------------224365322038742656772464040988
Content-Disposition: form-data; name="platform"

web
-----------------------------224365322038742656772464040988
Content-Disposition: form-data; name="api_setCookie"

false
-----------------------------224365322038742656772464040988
Content-Disposition: form-data; name="email"

expensify@nutanix.com
-----------------------------224365322038742656772464040988
Content-Disposition: form-data; name="isFromDevEnv"

false
-----------------------------224365322038742656772464040988
Content-Disposition: form-data; name="appversion"

9.0.11-5
-----------------------------224365322038742656772464040988
Content-Disposition: form-data; name="clientUpdateID"

730608865
-----------------------------224365322038742656772464040988--

Timings of request are:

As I can see the BE couldn't complete the request and returned:

{
  "jsonCode": 555,
  "message": "555 Timeout peeking command",
  "onyxData": [],
  "requestID": "8a8df5b19956171a-SJC"
}

But if you can double check that in your logs that would be a dope ❤️

[sakluger]

Adding @kirillzyusko as an assignee on the issue.

[marcaaron]

ah sorry, it's just outside the window of when we stop storing the logs.

[sakluger]

We're working on a PR that improves search speed (#48652). Hopefully that helps here too!

[sakluger]

No major updates, PR is still in progress.

[marcaaron]

What's the latest on this one?

[kirillzyusko]

@marcaaron I think @hannojg is working hard and pushing the PR forward. This PR is in review stage now 👀

[hannojg]

Hey, i have a little update on this one. It seems we identified one of the performance issues that were occurring for the customer.

Here is a profile trace that captures the lag/hang:

Trace-20241001T081231.json.zip

It specifically happens in OptionListContextProvider around these lines:

App/src/components/OptionListContextProvider.tsx

Lines 93 to 109 in a6d812c

	Object.keys(personalDetails).forEach((accountID) => {
	const prevPersonalDetail = prevPersonalDetails?.[accountID];
	const personalDetail = personalDetails?.[accountID];
	if (isEqualPersonalDetail(prevPersonalDetail, personalDetail)) {
	return;
	}
	Object.values(reports ?? {})
	.filter((report) => !!Object.keys(report?.participants ?? {}).includes(accountID) || (ReportUtils.isSelfDM(report) && report?.ownerAccountID === Number(accountID)))
	.forEach((report) => {
	if (!report) {
	return;
	}
	const newReportOption = OptionsListUtils.createOptionFromReport(report, personalDetails);
	const replaceIndex = options.reports.findIndex((option) => option.reportID === report.reportID);
	newReportOptions.push({

We will work on solutions to improve this! Its either:
a) we improve how this operates and make it more efficient
b) with our other PR we are implementing a suffix tree structure for very fast and efficient search. We might replace parts of the Context with that suffix tree, so this code will be restructured anyway

So the current PR won't fix that, but we are on the way to get this one here fixed as well; will keep you updated!

[marcaaron]

Sorry have been OOO - but that all sounds really promising, thanks!