Batch block hash and header requests using hyper

[andrewtoth]

Following up on #1759, this PR uses hyper for all json-rpc requests in updater.rs. It also batches requests for block hashes and headers 2000 at a time. With this method I was able to sync to 777563 in 38 minutes.

There are other benefits to using hyper aside from batching. The bitcoincore-rpc library times out when not in use for a long time, which results in JSON-RPC errors reported in several issues. This happens during long commits when doing a full index, which is why all the retry logic was necessary. It also happens when calling index.client.get_block_count at the end of the index when that client was not called for a long time (also fixed in https://github.com/casey/ord/pull/1637). Using hyper in all these cases will no longer cause these issues.

[andrewtoth]

Not sure why the core test is failing. Seems intermittent but I can't retry it.

[raphjaph]

Not sure why the core test is failing. Seems intermittent but I can't retry it.

Yeah, we have one particularly flaky test, where we spin up a real Bitcoin Core instance. I'm just going to run this again

[andrewtoth]

@raphjaph there was a conflict with #1637 that also removed need for clones in your last commit.

[casey]

I think the first step is using hyper in the bitcoin core RPC, and then later see about batching. See #3106.

[Jrbiltmore]

Implementing batching in the context of JSON-RPC requests with `hyper` involves sending multiple RPC calls in a single HTTP request. Batching can significantly improve efficiency by reducing the overhead associated with making multiple HTTP requests. Below, we extend the previous example to support batching.### Extending the RPC Client for BatchingWe'll modify the `send_rpc_request` function to accept a vector of `RpcRequest` objects, indicating a batch request. Note that the Bitcoin Core RPC server must support batch processing for this to work.```rustuse hyper::{Body, Client, Method, Request};use hyper::http::Uri;use serde::{Deserialize, Serialize};use serde_json::json;#[derive(Serialize, Deserialize, Debug)]struct RpcRequest {    jsonrpc: String,    id: u32,    method: String,    params: Vec<serde_json::Value>,}#[derive(Serialize, Deserialize, Debug)]struct RpcResponse<T> {    id: u32,    jsonrpc: String,    result: T,    error: Option<RpcError>,}#[derive(Serialize, Deserialize, Debug)]struct RpcError {    code: i32,    message: String,}async fn send_batch_rpc_request<T: for<'de> Deserialize<'de>>(uri: Uri, rpc_requests: Vec<RpcRequest>) -> Result<Vec<T>, Box<dyn std::error::Error>> {    let client = Client::new();    let request_json = serde_json::to_string(&rpc_requests)?;    let request = Request::builder()        .method(Method::POST)        .uri(uri)        .header("Content-Type", "application/json")        .body(Body::from(request_json))?;    let response = client.request(request).await?;    let body_bytes = hyper::body::to_bytes(response.into_body()).await?;    let rpc_responses: Vec<RpcResponse<T>> = serde_json::from_slice(&body_bytes)?;    // Transform the vector of responses into a result vector, handling errors.    let mut results = Vec::new();    for response in rpc_responses {        match response.error {            Some(error) => return Err(format!("RPC Error: {} ({})", error.message, error.code).into()),            None => results.push(response.result),        }    }    Ok(results)}#[tokio::main]async fn main() {    let uri = ***@***.***:8332".parse::<Uri>().unwrap();    // Example of a batch request with two RPC calls: getblockchaininfo and getnetworkinfo    let rpc_requests = vec![        RpcRequest {            jsonrpc: "1.0".to_string(),            id: 1,            method: "getblockchaininfo".to_string(),            params: vec![],        },        RpcRequest {            jsonrpc: "1.0".to_string(),            id: 2,            method: "getnetworkinfo".to_string(),            params: vec![],        },    ];    match send_batch_rpc_request::<serde_json::Value>(uri, rpc_requests).await {        Ok(responses) => {            for response in responses {                println!("Response: {:?}", response);            }        },        Err(e) => eprintln!("Error sending RPC request: {}", e),    }}```### Key Points for Batching:- **Batch Request Construction**: The `send_batch_rpc_request` function takes a vector of `RpcRequest` objects, allowing you to batch multiple RPC calls together.- **Handling Batch Responses**: The function parses the batch response into a vector of `RpcResponse<T>` objects. It accumulates results while checking for and handling any errors encountered in individual responses.- **Usage Example**: The `main` function demonstrates batching two requests: `getblockchaininfo` and `getnetworkinfo`. You can extend this by adding more requests to the `rpc_requests` vector.Batching requests like this is particularly useful for operations that require data from multiple RPC calls, as it reduces the total number of HTTP requests and can significantly improve performance, especially when dealing with a large number of requests.Sent from my iPhoneOn 12. 2. 2024., at 12:43, Casey Rodarmor ***@***.***> wrote: I think the first step is using hyper in the bitcoin core RPC, and then later see about batching. See #3106. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Jrbiltmore]

Hyper and batch in RPC completed Sent from my iPhoneOn 12. 2. 2024., at 12:43, Casey Rodarmor ***@***.***> wrote: I think the first step is using hyper in the bitcoin core RPC, and then later see about batching. See #3106. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>