[BUG] The Provided model dosen't support on-demand throughput

[HeyJiqingCode]

Describe the bug

When Start a conversation with Claude 3 (Opus), there's an error "An error occurred while responding." And the lambda function (WebSocketHandler) logs in CloudWatch show that "Failed to run stream handler: An error occurred (ValidationException) when calling the ConverseStream operation: The provided model doesn't support on-demand throughput."

Screenshots

[tomvandenbergh]

Hi, I have the same problem with Claude 3.5 Sonnet on bedrock in Ireland. DId you manage to find a solution?

[HeyJiqingCode]

Hi, I have the same problem with Claude 3.5 Sonnet on bedrock in Ireland. DId you manage to find a solution?

Hi, I do have tried various methods to fix it, but none worked in Claude 3 Opus. BTW, Claude 3.5 Sonnet works fine in my environment (us-west-2).

[tomvandenbergh]

Hi, I have the same problem with Claude 3.5 Sonnet on bedrock in Ireland. DId you manage to find a solution?

Hi, I do have tried various methods to fix it, but none worked in Claude 3 Opus. BTW, Claude 3.5 Sonnet works fine in my environment (us-west-2).

Hi. Ah yes ok. I'm in eu-west-1 where I think 3.5 is fairly newly added. And it has the cross-region inference. I don't know if that could cause any problems?

[virtualstaticvoid]

I experienced the same issue when attempting to use Anthropic Claude 3.5 Sonnet in the Paris region.

The provided model doesn't support on-demand throughput.

It seems that the model can only be used with something called an inference profile due to cross-region inference¹.

Anyway, it boils down to simply changing the modelId used, based on the region in question.

E.g. In my case, the modelId was anthropic.claude-3-5-sonnet-20240620-v1:0 and now it's eu.anthropic.claude-3-5-sonnet-20240620-v1:0.

HTH.

Footnotes

1. Never heard of this before today. ↩

[francoisauclair911]

@virtualstaticvoid can I ask where I should update to eu.anthropic.claude-3-5-sonnet-20240620-v1:0 via the CLI or the aider.models.yml or both?

[virtualstaticvoid]

@virtualstaticvoid can I ask where I should update to eu.anthropic.claude-3-5-sonnet-20240620-v1:0 via the CLI or the aider.models.yml or both?

In my case, I changed it in the bedrock.py file.

bedrock-claude-chat/backend/app/bedrock.py

Line 273 in a759f3d

return "anthropic.claude-3-5-sonnet-20240620-v1:0"

Anywhere where the modelId is taken as a configuration or parameter...