Huge memory consumsation of isWithinTokenLimit

[aminsol]

I am experiencing 200MB increase after implementing gpt-tokenizer the only function that I am using from this library is isWithinTokenLimit. Here is an image of my memory consumtion before and after deployment.
Here is how I am using is

function getRequestTokenCount(req: ChatCompletionRequestMessage[]) {
  const extraTokensDueToPromptForEachMessage = 7
  return req.reduce((acc, curr) => {
    const tokensInText = isWithinTokenLimit(curr.content, Infinity) || 99999
    return acc + tokensInText + extraTokensDueToPromptForEachMessage
  }, 0)
}

[olboghgc]

same here, what seems even more weird, is that it made me run out of memory by just importing it, not even using it

[niieani]

hi @aminsol and @olboghgc, can you provide a little more context about the platform?
is this under node, bun, the browser?
are you using a bundler of any sort?
which GPT encoding are you trying use?

[olboghgc]

@niieani Node 18/typescript/babel 7.0.0 in a firebase cloud function, with gpt 3.5 and 4o(but that shouldn't really matter since a simple import already cause a spike in memory usage)

[olboghgc]

btw I was able to +/- ignore this issue by doing a conditional require so I don't always waste memory when I don't need to call it

[github-actions]

🎉 This issue has been resolved in version 2.3.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[niieani]

New version (2.3.0) should have more than 4x smaller memory footprint (data for GPT-3.5-turbo tokenizer):

Package	Version	Load time (ms)	Memory (Heap) consumption (MB)	Total Memory (RSS) (MB)	Encode Avg (ms)	Decode Avg (ms)
gpt-tokenizer	2.2.3	253.04	45.83	150.42	0.0208	0.0033
gpt-tokenizer	2.3.0	44.58	9.65	35.76	0.0102	0.0025