Whitebox cannot create "settings.json" on shiny server after upgrade to 2.1.1

[hewag1975]

Hi,

we are using whitebox-tools through the R-client from within a shiny-application.
Installation on our shiny-server was performed as documented here by

• running install.packages("whitebox") as root (package directory is "/usr/local/lib/R/site-library")
• running whitebox::install_whitebox() likewise as root which installs the WBT executable into the folder of the R-package

So far it was possible to use the app as a standard shiny (=non-root) user.
Today when calling wbt_slope (or any other function) the app crashed with the following error message in the log:
thread 'main' panicked at 'Error creating output settings.json file.: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }', whitebox-common/src/configs/mod.rs:57:46 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

Error running WhiteboxTools (slope) whitebox.exe_path: '/usr/local/lib/R/site-library/whitebox/WBT/whitebox_tools'; File exists? TRUE Arguments: --run=slope --dem='' --output='/home/shiny/slope.tif' --units=degrees --compress_rasters=FALSE -v --max_procs=-1

To make sure that this is not related to the application itself I executed the code below on the same server in an R session as root user and in another session as non-root user (typically shiny users do not have root privileges) outside the application.

library(whitebox)
dem <- system.file("ex", "elev.tif", package="terra")
wbt_slope(dem, "/home/shiny/slope.tif")

In root mode the code calculates slope as expected and produces the output file.
In user mode I get the same error as when running the application.

The NEWS of the R-client states for version 2.1.1:
Updated how --compress_rasters parameter is passed via command line. Now the flag is added to all commands regardless of whether the value is TRUE or FALSE. This allows update of settings.json accordingly when --compress_rasters=FALSE. RE: https://github.com/jblindsay/whitebox-tools/issues/233#issuecomment-1065955783

Downgrading the R-package to version 2.1.0 resolves the issue.
Can this be changed to also allow non-root users to call the functions?
Best,
Hendrik

[brownag]

Hello,

It seems that I broke this by passing parameters to the WhiteboxTools binary even when they are not specified by the user. The whitebox_tools program will write updates to the options even if they aren't different from the current setting/defaults. It is normal for a non-root user to not be able to write to a folder owned by root. You were relying on default options before, which means you could run tools without whitebox creating a settings.json file. Now essentially any tool run will trigger an update to settings. Sorry about that

I believe in the short term you'll need to install WhiteboxTools to a directory that all users have ability to write to and set the executable path option.

Consider whitebox::install_whitebox(pkg_dir = "/path/to/common/directory") with an alternate pkg_dir argument so the settings.json file will be written there. And then when you call wbt_init() you can specify the path to the executable with exe_path argument. I realize that this is less simple than the prior installation sequence.

Something like:

whitebox::install_whitebox(pkg_dir = "/opt/whitebox")

Then when you go to use tools as a non-root user call this at beginning of session:

whitebox::wbt_init(exe_path = "/opt/whitebox/WBT/whitebox_tools")

Let me know if this resolves your immediate problem, and I will work on how to prevent passing of the default arguments that trigger this.

[brownag]

You may also need to chmod the folder and files so that any user can write.

[hewag1975]

Thanks for your reply. It helped me better understand the problem, though I was not able to solve it so far.

First I tested it on my local machine:

whitebox::install_whitebox(pkg_dir = "/home/hendrik")
if (!whitebox::wbt_init()) {
    usr_name = system2("whoami", stdout = TRUE)
    whitebox::wbt_init(
        exe_path = paste0("/home/", usr_name, "/WBT/whitebox_tools")
    )
}

This makes the executables available to the R-package. If I execute any whitebox command from my R-session, the settings.json file is created inside the specified WBT-folder (here /home/hendrik/WBT). However, if I launch my shiny app locally and create a call to a whitebox-function, the settings.json is created inside the shiny application folder. Took me a while to figure this out. On local machine this is no problem, because I have read/write access to the app folder. But it turns out that this is still a problem on the shiny server.

I applied the same code shown above there (as root) and installed the whitebox executable into the shiny user home. In order to make it writeable to the shiny user, I also changed permissions by comparing it to my local installation:

sudo chown -R shiny:shiny WBT/
sudo chmod g+w *
sudo chmod g-w whitebox_tools
sudo chmod g+w .DS_Store
sudo chmod g+w ._settings.json

Executing R code as shiny user in a R terminal creates the settings.json file within /home/shiny/WBT. Launching the App on the server obviously again tries to write settings.json to the app-folder (at least I still get the same error message in the logs as shown earlier and given the observations from my local machine that is the best explanation I have). We store all apps under /srv/shiny-server/... which is a root directory where the shiny user has no write access to. Of course we could change the way we store apps on the shiny-server, but this means quite some work wrt server config, so if possible I would like to avoid it. For the time being downgrading the R-package is the simplest solution.
Is there a way to get around this by e.g. enabling the user to decide where settings.json is stored?
Or do you have any further recommendations?

Thanks,
Hendrik

[brownag]

Thanks for your detailed response Hendrick, and thank you for raising this issue. You have alerted me to several things that need to be fixed and I got started on it last night. I am very interested in making sure whitebox package works well for your use case.

I believe if you set the working directory option e.g. wbt_init(wd = ...) subsequent settings.json may be written there, however this may rely on the initial location of settings.json being writable. I need to test this further and look at the whitebox_tools source to see how it works. Perhaps setting the working directory as root after install would be sufficient to have future non-root calls forwarded to a suitable directory?

There is not currently an option to have settings.json stored somewhere that is not the WBT install location or user-set working directory via whitebox_tools command line interface (that I am aware of).

I have committed a few things to master branch on my fork where I corrected the issues that I could see from your initial post:

remotes::install_github("brownag/whiteboxR")

If you have a chance to try this one out it would be great to know if it resolves your issues. As long as you don't change the default options in theory the above should work.

I will need to set up a test environment to try and better understand this shiny use case. Thanks for bringing it to my attention.

[hewag1975]

I installed the github version of the R package and also specified the working directory.

usr_name = system2("whoami", stdout = TRUE)
whitebox::wbt_init(
    exe_path = paste0("/home/", usr_name, "/WBT/whitebox_tools")
    , wd = paste0("/home/", usr_name, "/WBT")
)

Still when I launch the app and create a WBT function call, the settings.json is created inside the app folder, not within the working directory. As you mentioned that setting wd applies to subsequent writes of the settings.json, I also moved the settings.json from the app folder to the WBT folder manually (I thought maybe it needs to be found there so that it is not re-created). But a new one is just created again inside the app folder.
Regarding your comment on the initial location if settings.json being writable, that is the case on local machine (not on the shiny server), still whitebox writes to the app folder.
Let me know if I should test this differently or have misunderstood something.

[brownag]

This should fix the issue ensuring that additional flags related to settings.json are not added to the command string if the user has not explicitly set them.

I also did some digging into the settings.json. If we look here we see that std::env::current_dir() is used to decide where the settings.json will be written.

• If settings.json does not exist in that directory it will be created.
• If that directory or existing settings file is not writable the Error creating output settings.json file.: Os { code: 13, kind: PermissionDenied, message: "Permission denied" } error occurs.
• When we call whitebox_tools with R current_dir() will return the directory that system() is called from--which is the R working directory, not the WhiteboxTools working directory (unless you have set it up that way).

For your apps current_dir() will return/srv/shiny-server/... folders by default, I think. I should have realized this was the case--for example settings.json files get created by the unit tests and vignettes in tests/testthat/settings.json and vignettes/settings.json which is a result of the default working directory used when those run.

Here is a minimal example. I was wrong when I said an update to settings.json is always triggered; with a change in one or more options (verbosity, working directory, raster compression or maximum number of processes) an update to settings.json is triggered.

# run with reprex::reprex(std_out_err = TRUE)

library(whitebox)

wbt_verbose(TRUE)

if (file.exists("settings.json"))
  file.remove("settings.json")

wbt_slope(sample_dem_data(), output = "test.tif")
#> slope - Elapsed Time (excluding I/O): 0.5s

file.exists("settings.json")
#> [1] FALSE

wbt_slope(sample_dem_data(), output = "test.tif", compress_rasters = FALSE)
#> slope - Elapsed Time (excluding I/O): 0.4s

cat(readLines("settings.json", warn = FALSE))
#> {   "verbose_mode": true,   "working_directory": "",   "compress_rasters": false,   "max_procs": -1 }

wbt_slope(sample_dem_data(), output = "test.tif", compress_rasters = TRUE)
#> slope - Elapsed Time (excluding I/O): 0.6s

cat(readLines("settings.json", warn = FALSE))
#> {   "verbose_mode": true,   "working_directory": "",   "compress_rasters": true,   "max_procs": -1 }

file.exists("settings.json")
#> [1] TRUE

Sys.chmod("settings.json", "444")

# this doesn't work, it requires writing to settings.json
wbt_slope(sample_dem_data(), output = "test.tif", compress_rasters = FALSE)
#> thread 'main' panicked at 'Error creating output settings.json file.: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }', whitebox-common/src/configs/mod.rs:57:46
#> note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
#> 
#> Error running WhiteboxTools (slope)
#>   whitebox.exe_path: '/home/andrew/workspace/whitebox-tools/target/release/whitebox_tools'; File exists? TRUE
#>   Arguments: --run=slope  --dem='/home/andrew/R/x86_64-pc-linux-gnu-library/4.1/whitebox/extdata/DEM.tif' --output='test.tif' --units=degrees --compress_rasters=FALSE -v
#> System command had status 101
#> slope - Elapsed Time: NA [did not run]

# this works, compress_rasters=TRUE is already set
wbt_slope(sample_dem_data(), output = "test.tif", compress_rasters = TRUE)
#> slope - Elapsed Time (excluding I/O): 0.6s

# with https://github.com/brownag/whiteboxR/commit/4f72b173bb59bb8368279a05936f71167ec719a3 this works, 
# requires no change to existing (unwritable) settings.json
wbt_slope(sample_dem_data(), output = "test.tif")
#> slope - Elapsed Time (excluding I/O): 0.4s

# change working directory
dir.create('test')
setwd('test')

# this now works; creates a new settings.json in test
wbt_slope(sample_dem_data(), output = "test.tif", compress_rasters = FALSE)
#> slope - Elapsed Time (excluding I/O): 0.4s

If I understand correctly even on 2.1.0 you were not able to set the settings.json related flags on shiny server if your app directory/settings.json is not writable--unless you change your R working directory to somewhere you can write. I think it does not matter where whitebox_tools is installed to, what matters is the directory it is being called from. If you can use an R working directory other than the app directory which is writable by shiny user you would have ability to control the settings stored in the file as needed.

[brownag]

Here is an app I set up to test #68 on shiny server. With those fixes basic running of tools should work with the default R working directory (/srv/shiny-server/ app folder) as long as the settings.json options are not set.

If the user home folder is used (uncomment setwd("~") etc.), then changing settings.json options should work. This uses the normal install of whitebox package and WBT executable into the root user R library with install.packages("whitebox") (or equivalent for github until CRAN update) and whitebox::install_whitebox()

@hewag1975: I think this issue has been resolved but feel free to comment further / re-open the issue if needed.

library(shiny)
library(whitebox)
library(terra)
ui <- fluidPage(
  
  titlePanel(paste("WhiteboxTools-Shiny Test", getwd())),
  
  sidebarLayout(
      sidebarPanel(
          sliderInput("threshold", "Threshold:", min = 1, max = 40, value = 8),
          checkboxInput("compress", "Compress Rasters?")
      ),
      mainPanel(
         plotOutput("slopePlot")
      )
  )
)
server <- function(input, output) {
    # setwd("~")
    # wbt_wd("")
    output$slopePlot <- renderPlot({
        # generate threshold slope map based on input$threshold from ui.R
        x <- wbt_slope(sample_dem_data(destfile = "~/input.tif"), "~/output.tif")
                       # , compress_rasters = input$compress)
        terra::plot(terra::rast("~/output.tif") > input$threshold)
        unlink("~/output.tif")
    })
}
shinyApp(ui = ui, server = server)

[hewag1975]

Thanks for fixing this.
I launched the app locally and called wbt_slope from within the application. This generated settings.json in the application working directory. I re-deployed the app (incl. settings.json) to the shiny-server and installed the latest github version of whitebox. App can be launched by the shiny user and used as intended.