Parallelize `mice::mice()` Using {future}

future_mice() parallelizes chains in Multivariate Imputation using Chained Equations (MICE) using the {furrr} package to create futures for chains. Chains are also assessed for convergence using the R-hat (potential scale reduction factor) statistic; if the largest R-hat is less than rhat_max for minit iterations, the function returns early (without completing maxit iterations). This can save a significant amount of computation and manual convergence checking, and it often works well in practice. However, a "good" R-hat is neither a necessary nor sufficient condition for MCMC convergence, nor is it a substitute for checking imputation quality once convergence is achieved.

Usage

future_mice(
  data,
  m = 5L,
  method = NULL,
  predictorMatrix = NULL,
  ignore = NULL,
  where = NULL,
  blocks = NULL,
  visitSequence = NULL,
  formulas = NULL,
  blots = NULL,
  post = NULL,
  defaultMethod = c("pmm", "logreg", "polyreg", "polr"),
  maxit = 100L,
  minit = min(5L, maxit),
  quiet = FALSE,
  seed = NA,
  data.init = NULL,
  chunk_size = 1L,
  rhat_max = 1.05,
  progressor = NULL,
  ...
)

Arguments

data

A data frame or a matrix containing the incomplete data. Missing values are coded as NA.

m

Number of multiple imputations. The default is m=5.

method

Can be either a single string, or a vector of strings with length length(blocks), specifying the imputation method to be used for each column in data. If specified as a single string, the same method will be used for all blocks. The default imputation method (when no argument is specified) depends on the measurement level of the target column, as regulated by the defaultMethod argument. Columns that need not be imputed have the empty method "". See details.

predictorMatrix

A numeric matrix of length(blocks) rows and ncol(data) columns, containing 0/1 data specifying the set of predictors to be used for each target column. Each row corresponds to a variable block, i.e., a set of variables to be imputed. A value of 1 means that the column variable is used as a predictor for the target block (in the rows). By default, the predictorMatrix is a square matrix of ncol(data) rows and columns with all 1's, except for the diagonal. Note: For two-level imputation models (which have "2l" in their names) other codes (e.g, 2 or -2) are also allowed.

ignore

A logical vector of nrow(data) elements indicating which rows are ignored when creating the imputation model. The default NULL includes all rows that have an observed value of the variable to imputed. Rows with ignore set to TRUE do not influence the parameters of the imputation model, but are still imputed. We may use the ignore argument to split data into a training set (on which the imputation model is built) and a test set (that does not influence the imputation model estimates). Note: Multivariate imputation methods, like mice.impute.jomoImpute() or mice.impute.panImpute(), do not honour the ignore argument.

where

A data frame or matrix with logicals of the same dimensions as data indicating where in the data the imputations should be created. The default, where = is.na(data), specifies that the missing data should be imputed. The where argument may be used to overimpute observed data, or to skip imputations for selected missing values.

blocks

List of vectors with variable names per block. List elements may be named to identify blocks. Variables within a block are imputed by a multivariate imputation method (see method argument). By default each variable is placed into its own block, which is effectively fully conditional specification (FCS) by univariate models (variable-by-variable imputation). Only variables whose names appear in blocks are imputed. The relevant columns in the where matrix are set to FALSE of variables that are not block members. A variable may appear in multiple blocks. In that case, it is effectively re-imputed each time that it is visited.

visitSequence

A vector of block names of arbitrary length, specifying the sequence of blocks that are imputed during one iteration of the Gibbs sampler. A block is a collection of variables. All variables that are members of the same block are imputed when the block is visited. A variable that is a member of multiple blocks is re-imputed within the same iteration. The default visitSequence = "roman" visits the blocks (left to right) in the order in which they appear in blocks. One may also use one of the following keywords: "arabic" (right to left), "monotone" (ordered low to high proportion of missing data) and "revmonotone" (reverse of monotone). Special case: If you specify both visitSequence = "monotone" and maxit = 1, then the procedure will edit the predictorMatrix to conform to the monotone pattern. Realize that convergence in one iteration is only guaranteed if the missing data pattern is actually monotone. The procedure does not check this.

formulas

A named list of formula's, or expressions that can be converted into formula's by as.formula. List elements correspond to blocks. The block to which the list element applies is identified by its name, so list names must correspond to block names. The formulas argument is an alternative to the predictorMatrix argument that allows for more flexibility in specifying imputation models, e.g., for specifying interaction terms.

blots

A named list of alist's that can be used to pass down arguments to lower level imputation function. The entries of element blots[[blockname]] are passed down to the function called for block blockname.

post

A vector of strings with length ncol(data) specifying expressions as strings. Each string is parsed and executed within the sampler() function to post-process imputed values during the iterations. The default is a vector of empty strings, indicating no post-processing. Multivariate (block) imputation methods ignore the post parameter.

defaultMethod

A vector of length 4 containing the default imputation methods for 1) numeric data, 2) factor data with 2 levels, 3) factor data with > 2 unordered levels, and 4) factor data with > 2 ordered levels. By default, the method uses pmm, predictive mean matching (numeric data) logreg, logistic regression imputation (binary data, factor with 2 levels) polyreg, polytomous regression imputation for unordered categorical data (factor > 2 levels) polr, proportional odds model for (ordered, > 2 levels).

maxit

A scalar giving the maximum number of iterations. future_mice() will use less than maxit iterations if convergence criteria are met; because of this, the default is maxit = 50, which is much larger than the {mice} default of maxit = 5 but is large enough to "just work" in many situations without potentially running for days on end if convergence is not achieved.

minit

The minimum number of iterations to run. This is also the number of iterations used to assess convergence. Convergence is defined as all(tail(rhat, minit) < rhat_max).

quiet

Should convergence messages and warning be suppressed?

seed

Seed for random number generation; either a scalar integer, NA, or NULL. This seed is not used directly in mice::mice(); instead, it is used to generate separate RNG streams for each future using the parallel-safe L'Ecuyer-CMRG algorithm.

data.init

A data frame of the same size and type as data, without missing data, used to initialize imputations before the start of the iterative process. The default NULL implies that starting imputation are created by a simple random draw from the data. Note that specification of data.init will start all m Gibbs sampling streams from the same imputation.

chunk_size

The average number of chains per future. Differs from the usual {future} parameter in that multiple chains ("chunks") will be evaluated in a single call to mice::mice() if there is an integer i such that 1 < i <= chunk_size and m %% i == 0.

rhat_max

The R-hat threshold used to assess convergence. Convergence is defined as all(tail(rhat, minit) < rhat_max).

progressor

An optional progressor function to signal progress updates. If supplied, you are responsible for ensuring that the number of steps in the progressor is consistent with the number of iterations performed in future_mice().

...

Arguments passed on to mice::mice

printFlag

If TRUE, mice will print history on console. Use print=FALSE for silent computation.

Value

Returns an S3 object of class mids

(multiply imputed data set)

Details

MICE is a method for creating multiple imputations (replacement values) for multivariate missing data. The method is based on Fully Conditional Specification (FCS), where each incomplete varaible is imputed by a separate model. The MICE algorithm can impute mixes of continuous, binary, unordered categorical and ordered categorical data. In addition, MICE can impute continuous two-level data and maintain consistency between imputations by means of passive imputation and post-processing. Many diagnostic plots are implemented to inspect the quality of the imputations. See the mice::mice() function and the vignettes on the {mice} package website for details.

future_mice() mimics the mice::mice() interface as closely as possible; however, some shared parameters have different defaults than their {mice} equivalents. Notably, the default maxit is much larger than in {mice}; this is because maxit is an upper bound in future_mice(), rather than an exact number of iterations, as in mice(). The default of 100 should be more than enough iterations for most problems; if you need more than 100 iterations for convergence, you may want to check your imputation model for circularity or other stability issues.

Additionally, future_mice() provides NULL defaults for all unset arguments; this is a best practice in R. Because of this, passing NULL to any argument without an explicit default is the same as not passing that argument, which differs from the behavior of mice() in some instances.

Finally, some output attributes are not identical to their equivalents in mice(). In particular, the call attribute contains the call to future_mice(), rather than a call to mice(). The lastSeedValue should be equivalent, but does not function identically in subsequent calls to mice.mids() and future_mids().

Examples

# Run imputations in parallel (just two to avoid hogging resources)
# Picking a number of workers that divides `m` evenly can help performance
future::plan("multisession", workers = pmin(2L, future::availableCores()))

# Use just like `mice::mice()` - examples from {mice} documentation
mids <- future_mice(mice::nhanes, m = 2L, maxit = 1L)
#> Warning: Sampling did not converge in 1 iteration
#> R-hat: NA

if (FALSE) {
# Run until convergence (`maxit = 100L` by default)
mids <- future_mice(mice::nhanes, m = 2L)
}

mids
#> Class: mids
#> Number of multiple imputations:  2 
#> Imputation methods:
#>   age   bmi   hyp   chl 
#>    "" "pmm" "pmm" "pmm" 
#> PredictorMatrix:
#>     age bmi hyp chl
#> age   0   1   1   1
#> bmi   1   0   1   1
#> hyp   1   1   0   1
#> chl   1   1   1   0

# List the actual imputations for BMI
mids$imp$bmi
#>       1    2
#> 1  30.1 35.3
#> 3  28.7 35.3
#> 4  21.7 21.7
#> 6  27.4 21.7
#> 10 28.7 27.4
#> 11 27.5 29.6
#> 12 29.6 22.7
#> 16 22.0 27.2
#> 21 28.7 33.2

# First completed data matrix
mice::complete(mids)
#>    age  bmi hyp chl
#> 1    1 30.1   1 187
#> 2    2 22.7   1 187
#> 3    1 28.7   1 187
#> 4    3 21.7   2 199
#> 5    1 20.4   1 113
#> 6    3 27.4   1 184
#> 7    1 22.5   1 118
#> 8    1 30.1   1 187
#> 9    2 22.0   1 238
#> 10   2 28.7   1 186
#> 11   1 27.5   1 187
#> 12   2 29.6   1 206
#> 13   3 21.7   1 206
#> 14   2 28.7   2 204
#> 15   1 29.6   1 187
#> 16   1 22.0   1 118
#> 17   3 27.2   2 284
#> 18   2 26.3   2 199
#> 19   1 35.3   1 218
#> 20   3 25.5   2 204
#> 21   1 28.7   1 187
#> 22   1 33.2   1 229
#> 23   1 27.5   1 131
#> 24   3 24.9   1 199
#> 25   2 27.4   1 186

# Reset future plan
future::plan("sequential")