Partitions not pushed down

[rtyler]

Environment

Delta-rs version: 0.16.5

Binding: Rust

Environment:

• Cloud provider: AWS/S3
• OS: Linux
• Other:

---

Bug

The table has a ds partition and whenever the DF query has more criteria it seems to fail to push down partition filters and a full table scan happens

use deltalake::arrow::util::pretty::print_batches;
use deltalake::datafusion::execution::context::SessionContext;
use deltalake::datafusion::prelude::*;
use deltalake::datafusion::common::*;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), anyhow::Error> {
    let ctx = SessionContext::new();
    let table = deltalake::open_table("s3://REDACTED")
        .await
        .expect("Failed to open the delta table");

    let df = ctx.read_table(Arc::new(table)).expect("Failed to load table");
    let batches = df.filter(col("ds").eq(Expr::Literal(ScalarValue::from("2024-01-18"))))?
                    // adding this filter results in an apparent full table scan.
                    //
                    // With this commented out the job runs in 13s
                    //.filter(col("event_name").eq(Expr::Literal(ScalarValue::from("REDACTED"))))?
                    .limit(0, Some(10))?
                    .collect()
                    .await
                    .expect("Failed to build dataframe");
    /*
     * This SQL query also appears to trigger a scan
    ctx.register_table("source", Arc::new(table)).expect("Failed to register table with datafusion");
    let batches = ctx
        .sql("SELECT * FROM source WHERE ds = '2024-01-18' AND event_name = 'REDACTED' LIMIT 10")
        .await
        .expect("Failed to execute query")
        .collect()
        .await
        .expect("Failed to collect batches");
    */

    print_batches(&batches).expect("Failed to print batches");
    Ok(())
}

What happened:

What you expected to happen:

How to reproduce it:

More details:

[rtyler]

I'm going to close this. I will re-open it if I can reproduce it, but I had a repro case last week and cannot reproduce it again 🤦