ifelse fails with multi-column xts object

[Eluvias]

ifelse() was working fine with multi-column xts object in 0.9-7 version.

Using v0.10-0 I get the following:

library(xts)

x <- xts(matrix(rnorm(30) > 0, 5, 3), seq.Date(as.Date('2017-07-01'),
 by ='day',  length.out = 5))

ifelse(x, -1, 1)
#> Error in `[.xts`(test, ok): 'i' or 'j' out of range

Similar (#36).

Session info

devtools::session_info()
#> Session info -------------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.4.1 (2017-06-30)
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                                      
#>  date     2017-07-11
#> Packages -----------------------------------------------------------------
#>  package   * version date       source        
#>  backports   1.1.0   2017-05-22 CRAN (R 3.4.0)
#>  base      * 3.4.1   2017-06-30 local         
#>  compiler    3.4.1   2017-06-30 local         
#>  datasets  * 3.4.1   2017-06-30 local         
#>  devtools    1.13.2  2017-06-02 CRAN (R 3.4.0)
#>  digest      0.6.12  2017-01-27 CRAN (R 3.4.0)
#>  evaluate    0.10.1  2017-06-24 CRAN (R 3.4.0)
#>  graphics  * 3.4.1   2017-06-30 local         
#>  grDevices * 3.4.1   2017-06-30 local         
#>  grid        3.4.1   2017-06-30 local         
#>  htmltools   0.3.6   2017-04-28 CRAN (R 3.4.0)
#>  knitr       1.16    2017-05-18 CRAN (R 3.4.0)
#>  lattice     0.20-35 2017-03-25 CRAN (R 3.4.1)
#>  magrittr    1.5     2014-11-22 CRAN (R 3.4.0)
#>  memoise     1.1.0   2017-04-21 CRAN (R 3.4.0)
#>  methods   * 3.4.1   2017-06-30 local         
#>  Rcpp        0.12.11 2017-05-22 CRAN (R 3.4.0)
#>  rmarkdown   1.6     2017-06-15 CRAN (R 3.4.0)
#>  rprojroot   1.2     2017-01-16 CRAN (R 3.4.0)
#>  stats     * 3.4.1   2017-06-30 local         
#>  stringi     1.1.5   2017-04-07 CRAN (R 3.4.0)
#>  stringr     1.2.0   2017-02-18 CRAN (R 3.4.0)
#>  tools       3.4.1   2017-06-30 local         
#>  utils     * 3.4.1   2017-06-30 local         
#>  withr       1.0.2   2016-06-20 CRAN (R 3.4.0)
#>  xts       * 0.10-0  2017-07-07 CRAN (R 3.4.1)
#>  yaml        2.1.14  2016-11-12 CRAN (R 3.4.0)
#>  zoo       * 1.8-0   2017-04-12 CRAN (R 3.4.0)

[joshuaulrich]

Thanks for the report! Interestingly, this also failed in 0.9-7 if the index was POSIXct, as created by .xts, but not as created by xts(). That might point to a possible cause/solution.

R> set.seed(21)
R> xxts <- xts(matrix(rnorm(30) > 0, 5, 3), .POSIXct(1:5))
R> x.xts <- .xts(matrix(rnorm(30) > 0, 5, 3), 1:5)
R> 
R> ifelse(xxts, -1, 1)  
                    [,1] [,2] [,3]
1969-12-31 18:00:01   -1   -1    1
1969-12-31 18:00:02   -1    1   -1
1969-12-31 18:00:03   -1    1    1
1969-12-31 18:00:04    1   -1   -1
1969-12-31 18:00:05   -1    1   -1
R> ifelse(x.xts, -1, 1)  
Error in `[.xts`(test, ok) : 'i' or 'j' out of range
R> 
R> y <- xxts
R> index(y) <- seq(as.Date('2017-07-01'), by='day', length.out=5)
R> ifelse(y, -1, 1)
           [,1] [,2] [,3]
2017-07-01   -1   -1    1
2017-07-02   -1    1   -1
2017-07-03   -1    1    1
2017-07-04    1   -1   -1
2017-07-05   -1    1   -1
R> packageVersion("xts")
[1] '0.9.7'

But it's consistently broken in version 0.10-0.

R> set.seed(21)
R> xxts <- xts(matrix(rnorm(30) > 0, 5, 3), .POSIXct(1:5))
R> x.xts <- .xts(matrix(rnorm(30) > 0, 5, 3), 1:5)
R> 
R> ifelse(xxts, -1, 1)  
Error in `[.xts`(test, ok) : 'i' or 'j' out of range
R> ifelse(x.xts, -1, 1)  
Error in `[.xts`(test, ok) : 'i' or 'j' out of range
R> 
R> y <- xxts
R> index(y) <- seq(as.Date('2017-07-01'), by='day', length.out=5)
R> ifelse(y, -1, 1)
Error in `[.xts`(test, ok) : 'i' or 'j' out of range
R> packageVersion("xts")
[1] '0.10.0'

I'll continue to investigate.

[joshuaulrich]

This is related to #163. The fix to that bug seems to be what's triggering the error now. This is because:

R> storage.mode(.index(xxts))
[1] "double"
R> storage.mode(.index(x.xts))
[1] "integer"

And that's what explains why ifelse() worked in 0.9-7 when the xts index was stored as a double. Looks like we need to special-case the [.xts call when dim(i)[2] > 1L.

[dnousias]

Hi Joshua,

Today, when I downloaded "quantsrat", it automatically updated xts to v0.10-0 and ever since I'm having the problem with ifelse() on xts objects.
I tried to use xts_0.9-7 with different R versions, but I'm getting the following:
Warning in install.packages :
package ‘~/Downloads/xts_0.09-7.tar’ is not available (for R version x.x.x.)

I'd be really thankful if you could point me towards a solution as this piece of code is paramount to my work...

Many Thanks!

[braverock]

xts v0.10-0 is the current version of xts.

The error you show has nothing to do with ifelse, and in general ifelse should work as it always has on newer xts objects (with the exceptions noted in this thread about use of the xts constructor).

Perhaps you could create a minimal reproducible example?

[dnousias]

The original error I'm getting, is:
Error in [.xts(test, ok) : 'i' or 'j' out of range]

The code reads large xlsx files, converts them to xts objects, performs rollapply() to extract rolling regression coefficients and then uses ifelse() to generate rules.
e.g.

Cl_Prices <- xts(workbook, order.by = dates, na.rm = TRUE)
ln_prices <- log(Cl_Prices)
coef_1 <- rollapplyr(ln_prices,
                            width = regr_width,
                            function(x){
                              coef(lm(x~index(x)))[1]
                             }
                            )
coef_2 <- rollapplyr(ln_prices,
                            width = regr_width,
                            function(x){
                              coef(lm(x~index(x)))[2]
                             }
                            )
Rule <-ifelse (coef_1 > 0 &  coef_2 <0, "Do_this", "Do_that")

This code is being applied to various sheets within a workbook via an lapply() function
It worked fine until the automatic update to v0.10-0 today.

thx!

[Eluvias]

Revert back to xts 0.9-7 on your latest R version.

devtools::install_version("xts", version = "0.9-7")

[dnousias]

Hi, thanks for confirming.
Please have a look at the error message I'm getting.
thx!

devtools::install_version("xts", version = "0.9-7")
Downloading package from url: https://cran.rstudio.com//src/contrib/Archive/xts/xts_0.9-7.tar.gz
Installing xts
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file --no-environ
--no-save --no-restore --quiet CMD INSTALL
'/private/var/folders/gs/_4d3qyf14h192_fnkzzpc77w0000gn/T/RtmpDylhAs/devtools80eb8e966f/xts'
--library='/Library/Frameworks/R.framework/Versions/3.4/Resources/library'
--install-tests

• installing source package ‘xts’ ...
  ** package ‘xts’ successfully unpacked and MD5 sums checked
  ** libs
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c add_class.c -o add_class.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c any.c -o any.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c attr.c -o attr.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c binsearch.c -o binsearch.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c coredata.c -o coredata.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c diff.c -o diff.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c dimnames.c -o dimnames.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c endpoints.c -o endpoints.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c extract_col.c -o extract_col.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c init.c -o init.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c isOrdered.c -o isOrdered.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c isXts.c -o isXts.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c leadingNA.c -o leadingNA.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -fPIC -Wall -g -O2 -c merge.c -o merge.o
  gfortran -fPIC -g -O2 -c period.max.f -o period.max.o
  make: gfortran: No such file or directory
  make: *** [period.max.o] Error 1
  ERROR: compilation failed for package ‘xts’
• removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/xts’
  Installation failed: Command failed (1)

[dnousias]

Case solved ;
gfortran had been (for an unknown reason) uninstalled.
I followed the instructions here:
http://thecoatlessprofessor.com/programming/rcpp-rcpparmadillo-and-os-x-mavericks-lgfortran-and-lquadmath-error/
Once re-installed, v0.9-7 also installed correctly.
Also, TTR had to be reverted to v0.23-2 to be compatible.

[joshuaulrich]

Notes from an email discussion with the zoo team. My initial email contained the example code below:

x <- xts(matrix(!c(0,0,0,1,0,0,1,1,0), 3, 3), .POSIXct(1:3, tz = "UTC"))
colnames(x) <- letters[1:ncol(x)]
z <- as.zoo(x)
s <- as.ts(z)
ifelse(s, 1, 0)  # works
ifelse(z, 1, 0)  # error
ifelse(x, 1, 0)  # error

---

Achim's initial thoughts:

Subset selection for the full zoo/xts object is always done "along" the time index, so subsetting with a logical vector or matrix does not work. Therefore, it cannot work in ifelse(). See:

## create a plain logical matrix
m <- coredata(z)
m
##         a     b     c
## [1,] TRUE FALSE FALSE
## [2,] TRUE  TRUE FALSE
## [3,] TRUE  TRUE  TRUE

## create a numeric zoo series
n <- 2 * z - 1
n
##                     a  b  c
## 1970-01-01 00:00:01 1 -1 -1
## 1970-01-01 00:00:02 1  1 -1
## 1970-01-01 00:00:03 1  1  1

## the subset selection along the time index keeps
## all time stamps with at least one TRUE (here: all three)
n[m]
##                     a  b  c
## 1970-01-01 00:00:01 1 -1 -1
## 1970-01-01 00:00:02 1  1 -1
## 1970-01-01 00:00:03 1  1  1

In "ts" this is different because the whole subset selection treats the object like a matrix. Only window() preserves the time index. Hence:

as.ts(n)[m]
## [1] 1 1 1 1 1 1

---

Josh's proposal:

There are 3 cases we might want to handle when i is a matrix. The
similar to the matrix cases documented in ?Extract:

1. logical, ncol(i) == ncol(x)
   returns vector of values for TRUE elements in i
2. numeric, ncol(i) == length(dim(x))
   returns vector of elements in the matrix/array specified by each row in i
3. character, ncol(i) == length(dim(x))
   returns vector of elements matching rownames/colnames specified by
   each row in i

I've a few requests for support for (1) and (2) in xts. I'm not sure (3) makes sense. I think ifelse() only relies on (1).

---

Gabor's thoughts:

Column at a time operations work but zoo subscripting applies to the time index, not the coredata (as Achim mentioned), so I am not sure that having ifelse work in the way ts works is feasible or desirable.

---

Achim's reply to Gabor:

I agree. Note that the matrix also does not work for data.frame objects:

d <- data.frame(a = 1:3, b = 4:6)
d[c(TRUE, FALSE, FALSE, FALSE, TRUE, TRUE)]
## Error in `[.data.frame`(d, c(TRUE, FALSE, FALSE, FALSE, TRUE, TRUE)) :
##   undefined columns selected

Of course, 'zoo' set out to be as consistent as possible with 'ts'. However, the subsetting was deliberately changed and is not always quite consistent.
Hence, to me personally the solution proposed by Josh sounds more confusing than helpful:

• If the extract index is a matrix...
• ..and of matching dimension,
• then we should drop the time attribute and use only the coredata
• and then drop the dimension and return a vector.

It also wouldn't be easy to document, I think.

[joshuaulrich]

Removing the milestone, since I'm not sure when (or if) this will be implemented.