-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Some files missing from value returned by osf_ls_files() #142
Comments
Thanks for reporting and the excellent reprex! However, I'm not able to reproduce the issue with the current CRAN release of osfr (v0.2.8). When I run code_pkg_files <- osfr::osf_retrieve_node("https://osf.io/zyaqn") %>%
osfr::osf_ls_files(path = "Data for R package", n_max = Inf) The resulting I'm guessing this was caused by an intermittent server error. Although I would have expected the function to return an error rather than a partial result. If it occurs again you can enable verbose logging by defining the environment variable
This should give us more insight into what's happening at the request level. |
I've tried to investigate this in further detail, but not got much further. As far as I can tell is is either a problem with library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
library(osfr)
#> <Logging enabled: osfr.log>
# Trying to download a file directly using `osf_retrieve_file()` works as
# expected:
osf_retrieve_file("https://osf.io/avj2k")
#> # A tibble: 1 × 3
#> name id meta
#> <chr> <chr> <list>
#> 1 crime_open_database_core_mesa_2022.Rds avj2k <named list [3]>
# But the same file (`crime_open_database_core_mesa_2022.Rds`) is missing when
# we try to retrieve a list of files with `osf_ls_files()`:
osf_retrieve_node("https://osf.io/zyaqn") |>
osf_ls_files(path = "Data for R package", n_max = Inf) |>
filter(stringr::str_detect(name, "core_mesa")) |>
arrange(name)
#> # A tibble: 6 × 3
#> name id meta
#> <chr> <chr> <list>
#> 1 crime_open_database_core_mesa_2016.Rds 5c926f5d4712b400173cc144 <named list>
#> 2 crime_open_database_core_mesa_2017.Rds 5c926f5da743a900176287aa <named list>
#> 3 crime_open_database_core_mesa_2018.Rds 5c926f5d2286e80018c62e36 <named list>
#> 4 crime_open_database_core_mesa_2019.Rds 5f009012af1156016e3b61ca <named list>
#> 5 crime_open_database_core_mesa_2020.Rds 614da1c7ec7885002840fb0d <named list>
#> 6 crime_open_database_core_mesa_2021.Rds 6347e653ec7f3f2df2f5f524 <named list>
# Just to check, if we list the same files using the API directly, the 2022 file
# is there as expected:
jsonlite::fromJSON("https://api.osf.io/v2/nodes/zyaqn/files/osfstorage/5bbde32b7cb18100193c778a/?filter%5Bname%5D=core_mesa") |>
purrr::pluck("data", "attributes", "name")
#> [1] "crime_open_database_core_mesa_2016.Rds"
#> [2] "crime_open_database_core_mesa_2017.Rds"
#> [3] "crime_open_database_core_mesa_2018.Rds"
#> [4] "crime_open_database_core_mesa_2019.Rds"
#> [5] "crime_open_database_core_mesa_2020.Rds"
#> [6] "crime_open_database_core_mesa_2021.Rds"
#> [7] "crime_open_database_core_mesa_2022.Rds"
# And an extra twist: in some cases the same file is reported by
# `osf_ls_files()` multiple times:
osfr::osf_retrieve_node("https://osf.io/zyaqn") |>
osfr::osf_ls_files(path = "Data for R package", n_max = Inf) |>
dplyr::filter(stringr::str_detect(name, "core_st_louis_2009")) |>
dplyr::arrange(name)
#> # A tibble: 38 × 3
#> name id meta
#> <chr> <chr> <list>
#> 1 crime_open_database_core_st_louis_2009.Rds 5f009039af1156016a3b… <named list>
#> 2 crime_open_database_core_st_louis_2009.Rds 5f009039af1156016a3b… <named list>
#> 3 crime_open_database_core_st_louis_2009.Rds 5f009039af1156016a3b… <named list>
#> 4 crime_open_database_core_st_louis_2009.Rds 5f009039af1156016a3b… <named list>
#> 5 crime_open_database_core_st_louis_2009.Rds 5f009039af1156016a3b… <named list>
#> 6 crime_open_database_core_st_louis_2009.Rds 5f009039af1156016a3b… <named list>
#> 7 crime_open_database_core_st_louis_2009.Rds 5f009039af1156016a3b… <named list>
#> 8 crime_open_database_core_st_louis_2009.Rds 5f009039af1156016a3b… <named list>
#> 9 crime_open_database_core_st_louis_2009.Rds 5f009039af1156016a3b… <named list>
#> 10 crime_open_database_core_st_louis_2009.Rds 5f009039af1156016a3b… <named list>
#> # ℹ 28 more rows Created on 2023-11-06 with reprex v2.0.2 Session infosessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.3.1 (2023-06-16)
#> os macOS Sonoma 14.0
#> system aarch64, darwin20
#> ui X11
#> language (EN)
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz Europe/London
#> date 2023-11-06
#> pandoc 3.1.1 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/ (via rmarkdown)
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date (UTC) lib source
#> cachem 1.0.8 2023-05-01 [1] CRAN (R 4.3.0)
#> cli 3.6.1 2023-03-23 [1] CRAN (R 4.3.0)
#> crul 1.4.0 2023-05-17 [1] CRAN (R 4.3.0)
#> curl 5.0.2 2023-08-14 [1] CRAN (R 4.3.0)
#> digest 0.6.33 2023-07-07 [1] CRAN (R 4.3.0)
#> dplyr * 1.1.2 2023-04-20 [1] CRAN (R 4.3.0)
#> evaluate 0.21 2023-05-05 [1] CRAN (R 4.3.0)
#> fansi 1.0.4 2023-01-22 [1] CRAN (R 4.3.0)
#> fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.3.0)
#> fs 1.6.3 2023-07-20 [1] CRAN (R 4.3.0)
#> generics 0.1.3 2022-07-05 [1] CRAN (R 4.3.0)
#> glue 1.6.2 2022-02-24 [1] CRAN (R 4.3.0)
#> htmltools 0.5.6 2023-08-10 [1] CRAN (R 4.3.0)
#> httpcode 0.3.0 2020-04-10 [1] CRAN (R 4.3.0)
#> httr 1.4.7 2023-08-15 [1] CRAN (R 4.3.0)
#> jsonlite 1.8.7 2023-06-29 [1] CRAN (R 4.3.0)
#> knitr 1.43 2023-05-25 [1] CRAN (R 4.3.0)
#> lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.3.0)
#> logger 0.2.2 2021-10-19 [1] CRAN (R 4.3.0)
#> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.3.0)
#> memoise 2.0.1 2021-11-26 [1] CRAN (R 4.3.0)
#> osfr * 0.2.9 2022-09-25 [1] CRAN (R 4.3.0)
#> pillar 1.9.0 2023-03-22 [1] CRAN (R 4.3.0)
#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.3.0)
#> purrr 1.0.2 2023-08-10 [1] CRAN (R 4.3.0)
#> R.cache 0.16.0 2022-07-21 [1] CRAN (R 4.3.0)
#> R.methodsS3 1.8.2 2022-06-13 [1] CRAN (R 4.3.0)
#> R.oo 1.25.0 2022-06-12 [1] CRAN (R 4.3.0)
#> R.utils 2.12.2 2022-11-11 [1] CRAN (R 4.3.0)
#> R6 2.5.1 2021-08-19 [1] CRAN (R 4.3.0)
#> Rcpp 1.0.11 2023-07-06 [1] CRAN (R 4.3.0)
#> reprex 2.0.2 2022-08-17 [1] CRAN (R 4.3.0)
#> rlang 1.1.1 2023-04-28 [1] CRAN (R 4.3.0)
#> rmarkdown 2.24 2023-08-14 [1] CRAN (R 4.3.0)
#> rstudioapi 0.15.0 2023-07-07 [1] CRAN (R 4.3.0)
#> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.3.0)
#> stringi 1.7.12 2023-01-11 [1] CRAN (R 4.3.0)
#> stringr 1.5.0 2022-12-02 [1] CRAN (R 4.3.0)
#> styler 1.10.2 2023-08-29 [1] CRAN (R 4.3.0)
#> tibble 3.2.1 2023-03-20 [1] CRAN (R 4.3.0)
#> tidyselect 1.2.0 2022-10-10 [1] CRAN (R 4.3.0)
#> triebeard 0.4.1 2023-03-04 [1] CRAN (R 4.3.0)
#> urltools 1.7.3 2019-04-14 [1] CRAN (R 4.3.0)
#> utf8 1.2.3 2023-01-31 [1] CRAN (R 4.3.0)
#> vctrs 0.6.4 2023-10-12 [1] CRAN (R 4.3.1)
#> withr 2.5.0 2022-03-03 [1] CRAN (R 4.3.0)
#> xfun 0.40 2023-08-09 [1] CRAN (R 4.3.0)
#> yaml 2.3.7 2023-01-23 [1] CRAN (R 4.3.0)
#>
#> [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
#>
#> ────────────────────────────────────────────────────────────────────────────── Nothing gets written to the file specified in Log output (collapsed because it's quite long)
Two final things to note:
|
Hi, we've recently been running into this same issue (missing files and folders in osf_ls_files) in some projects where we host a large amount of raw data on OSF. Are there any updates on what might be causing the issue? Happy to provide details if this is actively being investigated. Thanks! |
osf_ls_files()
appears to be not listing some files in osfr version 0.2.8. The files that are missing are consistent (i.e. if I call the code again the same files are missing), but otherwise I can't see any pattern why some files are returned and not others. From the osfr documentation, settingn_max = Inf
should ensure all files are returned, but this does not appear to work.Created on 2022-07-23 by the reprex package (v2.0.1)
Session info:
The text was updated successfully, but these errors were encountered: