-
-
Notifications
You must be signed in to change notification settings - Fork 872
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
customizable cache (closes #2176) #2340
base: master
Are you sure you want to change the base?
Conversation
maybe preprocess and postprocess are not good names... 🤔 |
This comment was marked as resolved.
This comment was marked as resolved.
259fdd5
to
ccc14d7
Compare
R/cache.R
Outdated
cache_purge = function(hash) { | ||
for (h in hash) unlink(paste(cache_path(h), c('rdb', 'rdx', 'RData'), sep = '.')) | ||
for (h in hash) unlink(paste(cache_path(h), c('rds', 'rdb', 'rdx', 'RData'), sep = '.')) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cache_purge()
and clean_cache()
take into account of limited file types/names.
There may be some cases where knitr should remove more files.
The example in the description saved an extra file as a part of cache, which will not be removed by knitr.
registerS3method("knit_cache_preprocess", "data.frame", function(x) {
write.csv(x, "cache.csv") # NOTE: this file is not removed by `cache$purge()` or `clean_cache()`
structure("cache.csv", class = "knit_cache_csv")
}, envir = asNamespace("knitr"))
R/cache.R
Outdated
knit_cache_postprocess = function(x, ...) UseMethod('knit_cache_postprocess') | ||
knit_cache_postprocess.default = function(x, ...) x |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Postprocess is skipped if a package is not loaded.
To solve the above problems, I implemented the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel this implementation is too complicated, and I'd like to propose a different way: in the latest version of xfun, I added two functions lazy_save()
and lazy_load()
as a different and simple implementation of base R's lazyLoad()
and tools:::makeLazyLoadDB()
. By default, xfun::lazy_save()/lazy_load()
use the rds format, which should solve the problems #2176 and #2339 (I've only briefly tested #2176).
For backward compatibility, we can first test if *.rdb
/*.rdx
exist. If they do, we use the old approach (base R), otherwise we switch to xfun's lazy loading.
What do you think?
Correction: I don't remember how I tested #2176 now. I was expecting that this would just work:
```{r}
library(terra)
```
```{r, cache=TRUE}
r = rast(matrix(1:12,3,4))
```
```{r}
r
```
but it doesn't (even if we save r
to *.rds
). The object r
will fail to load in a new R session, and we still have to do wrap()
/unwrap()
:
```{r}
library(terra)
```
```{r, cache=TRUE}
r = rast(matrix(1:12,3,4))
p = wrap(r)
```
```{r}
unwrap(p)
```
That said, I had this issue in mind when designing the new cache system for litedown. With litedown, it's possible to customize the read/write methods for cache via the chunk option cache.rw
, e.g.,
---
title: "Caching terra objects with litedown"
knit: litedown:::knit
---
```{r}
library(terra)
rw_terra = list(
name = 'terra',
save = function(x, file) {
if (inherits(x, 'SpatRaster')) x = wrap(x)
saveRDS(x, file)
},
load = function(...) {
x = readRDS(...)
if (inherits(x, 'PackedSpatRaster')) x = unwrap(x)
x
}
)
```
```{r, cache=TRUE, cache.rw=rw_terra}
r = rast(matrix(1:12,3,4))
```
```{r}
r
```
This PR allows implementing
knit_cache_hook
methods which may preprocess objects (e.g., save to an external file) and define custom loaders.I will add a NEWS item after we agree with the design.
cache_save
replaces rdb/rdx files with rds filecache_load()
attemptslazyload()
if rdb/rdx files are availablefeat(cache): allow pre/postprocessing cache objects)knit_cache_preprocess
preprocesses objects being savedknit_cache_postprocess
postprocesses objects being loadedknit_cache_hook
methods on saving cache${cache_path(h)}__extra
directory${cache_path(h).rds}
With this PR, we can add some hooks on objects to be cached.
For example, we can use writeLines to save character objects.