DFK Subnet Latency on Trace Transactions

[gbartolome-avax]

Describe the bug
We have multiple workers using new header subscriptions for the DFK subnet via websockets in our data-development environment. We are seeing errors quite frequently when new blocks at the tip are calling the debug_traceTransaction method in geth for internal transactions.

To Reproduce
Test locally:

1. clone producer
2. This will need a valid DFK endpoint to reproduce
3. go into the producer directory and run

• make up this builds all the Docker services locally to run the application (Kafka and Redis)
• make run-binary this builds the go binary and runs the producer app with the correct arguments for the DFK subnet

Expected behavior
Based on the Screenshots you can see that the call is not able to get the debug trace data for that transaction.

The error is pretty intermittent now that the infra team has bumped up our resources for the nodes that power that endpoint though it still happens once in a while

Screenshots

Logs
block_error_dfk (1).csv
trace_error_dfk (1).csv

Metrics
N/A

Operating System
Mac and AWS EKS

Additional context
Add any other context about the problem here.

Avalanche Bug Bounty program can be found here.

[aaronbuchwald]

Could you please describe the API calls that are made on the node, the result, and a brief description of why you think this is unexpected behavior?

The repo that you linked to reproduce is private, so if that repo is the only way you have to reproduce the issue, a clear description of what it's doing would make a better starting place to debug.