Rate limiting during OneNote import

[altano]

Describe the bug
During an import of my OneNote notebooks the network requests started to fail with HTTP error code 429 (rate limiting).

To Reproduce
Steps to reproduce the behavior:

1. Go to https://importer.notesnook.com/
2. Select OneNote importer
3. Login and consent
4. import begins
5. I checked network dev tools and it looks like all requests to graph.microsoft.com started returning 429 http errors:

{
	"error": {
		"code": "20166",
		"message": "The app has issued too many requests on behalf of this user in a short time period.",
		"innerError": {
			"date": "2022-03-27T21:54:33",
			"request-id": "<redacted>",
			"client-request-id": "<redacted>"
		}
	}
}

NOTE: Import is actually still ongoing and at the "Transforming pages" phase, but will definitely not complete successfully given all the network errors.

Expected behavior
Import completes successfully

Desktop (please complete the following information):

• Firefox v98.0.2
• Notesnook Importer v1.2.1

Additional context

• I analyzed the network requests and it looks like I got a 429 http error response to requests # 292, 310, 646, and then EVERY request starting at # 938. I think I'm hitting the documented OneNote service limits. I would guess that at first I'm occasionally hitting the per-minute limit but at some point I hit the per-hour rate limit of 800.
• The Retry-After http response header mentioned in the documentation is NOT present in my http responses.

Ideas to fix it... I'm just brainstorming

• Can more requests be batched up to reduce the amount sent?
• Handle a 429 error response by backing off before either retrying or sending the next request. The Importer just keeps making requests after a 429 error right now. This can use exponential backoff. Unfortunately it might need to wait >1hr given the OneNote limits, but I'd be okay with that?
• Allow me to import one notebook at a time. That way I could manually eventually import all my notebooks as long as no single notebook was over the rate limit.

[altano]

FWIW the import eventually failed with a ton of errors, as expected:

[thecodrr]

@altano what's the state of the notes that did get converted? I didn't think about service limits so I might have to get Notebooks + section groups + sections in a single request — this might help make it a bit better. But some kind of throttling is still required in case of getting pages.

[altano]

UPDATE: Oops, I didn't originally test all these with the new import zip file (I got tripped up by the fact that deleting all my notebooks doesn't delete all my notes). Sorry for the noise. I went ahead and updated everything below to reflect the new import.

It's looking improved over the last time I tried this a while ago. I think this is almost usable for me!

These are fixed:

• [OneNote Importer] Heading 1 not imported as Heading 1 #25
• [OneNote Importer] Checkmarks/TODOs not imported #26
• [OneNote Importer] Multiple page parts not imported #27
• [OneNote Importer] Images not imported #30

I'm still seeing these but that might be caused by the 429 errors:

• [OneNote Importer] Most notes not imported #29

These I can't test:

• [OneNote Importer] Password protected sections are silently skipped #28 (I moved all my secure content out of OneNote into 1Password)

[altano]

Oh nice, were you able to fix this @thecodrr ?

I actually have an open PR with the graph sdk because they should be handling this better on their end: microsoftgraph/msgraph-sdk-javascript#1514

[thecodrr]

@altano yes! give it a try in v2.1.2 of the Importer. It should work much better.