[help] Using format = "url" with alternative modified metadata?

[andrewheiss]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

In a pipeline I'm building, I'm accessing JSON data from a URL that doesn't include Last-Modified in the HTTP header, but does include it as an entry in the JSON itself, like this:

{
  "name": "thing",
  "last-modified": "2024-09-11 21:44:53",
  "value": 12345
}

(I just whipped up this little example server that spits out JSON data formatted in this way, with a last-modified timestamp that updates every 5 minutes: https://election-testing.andhs.co/county?county=example)

For a regular URL, I'd do something like this:

tar_target(remote_data, "https://election-testing.andhs.co/county?county=example", format = "url")

This doesn't work in this case, though, because of the missing metadata:

Last error message:
_paths_ no ETag or Last-Modified for url: https://election-testing.andhs.co/county?county=example Please choose a storage format other than format = "url" and pursue different workarounds to rerun the target when the remote data changes.

That all makes sense. It's not a normal website with normal HTTP headers or anything.

I'm curious what a recommended "different workaround" in this case might look like. The closest I've come is a process like this:

1. Similar to tarchetypes::tar_download(), make two targets: remote_url and remote_data. In the function for the remote_url target, save the timestamp in a temporary file somewhere
2. Use tarchetypes::tar_force() to make the remote_url target run every single time.
3. Add logic to the remote_data target that only grabs data from the URL if the timestamp in the temporary file is new

But that process feels wrong and un-targets-like (especially with the brute force of tar_force()). Is there a better way to use a last-modified field in JSON data to invalidate a target?

[andrewheiss]

Ahhh there's a similar question/use case from back in 2020 here: #131 (comment) - that led to the creation of tar_change() (ropensci/tarchetypes#2), where the recommended approach is something like this:

tar_change(remote_data, download_data(), change = remote_hash(dep1, dep2))

So in this case, I'd do something like

download_data <- function(url) {
  httr2::request(url) |>
    httr2::req_perform() |>
    httr2::resp_body_json()
}

tar_change(
  dataset, 
  download_data("https://election-testing.andhs.co/county?county=example"), 
  change = SOMETHING_HERE
)

I'm not sure how to structure the change function, though (or the remote_hash(dep1, dep2) part from the archetype in ropensci/tarchetypes#2)

[andrewheiss]

WAIT I FIGURED IT OUT! This works and seems to be the most {targets}-esque approach:

library(targets)
library(tarchetypes)

get_latest_timestamp <- function(url) {
  response <- httr2::request(url) |>
    httr2::req_perform() |>
    httr2::resp_body_json()

  response |>
    purrr::pluck("last-modified")
}

download_data <- function(url) {
  httr2::request(url) |>
    httr2::req_perform() |>
    httr2::resp_body_json()
}

url <- "https://election-testing.andhs.co/county?county=example"

list(
  tar_change(
    dataset,
    download_data(url),
    change = get_latest_timestamp(url)
  )
)

That change function just returns the timestamp from the API, and it stores it as text in the {targets} data store. When the pipeline runs, it will run get_latest_timestamp() and grab the current last-modified value from the API. If that value is the same as what {targets} has in its local data store, nothing happens—everything is up to date and fine. If the remote value is different, download_data() will run and grab the latest data and do stuff with it.