From d7521fe8055ade76e8d938b35c30c2c6c282032c Mon Sep 17 00:00:00 2001 From: Michel Lang Date: Mon, 18 Aug 2025 21:41:14 +0200 Subject: [PATCH 1/6] docs: fix absolute urls in vigngette --- vignettes/batchtools.Rmd | 189 +++++++++++++++++---------------------- 1 file changed, 81 insertions(+), 108 deletions(-) diff --git a/vignettes/batchtools.Rmd b/vignettes/batchtools.Rmd index 560eafd0..8d4c2a5b 100644 --- a/vignettes/batchtools.Rmd +++ b/vignettes/batchtools.Rmd @@ -28,30 +28,30 @@ if (identical(Sys.getenv("IN_PKGDOWN"), "true")) { ## Cluster Functions The communication with the batch system is managed via so-called cluster functions. -They are created with the constructor [makeClusterFunctions](https://mlr-org.github.io/batchtools/reference/makeClusterFunctions) which defines how jobs are submitted on your system. +They are created with the constructor `makeClusterFunctions()` which defines how jobs are submitted on your system. Furthermore, you may provide functions to list queued/running jobs and to kill jobs. Usually you do not have to start from scratch but can just use one of the cluster functions which ship with the package: -* Interactive Cluster Functions (default): [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsInteractive), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsInteractive.R) -* Multicore Cluster Functions: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsMulticore), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsMulticore.R) -* Socket Cluster Functions: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSocket), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSocket.R) -* Makeshift SSH cluster: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSSH), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSSH.R) -* Docker Swarm: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsDocker), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsDocker.R) -* IBM Spectrum Load Sharing Facility (LSF): [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsLSF), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsLSF.R) -* OpenLava: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsOpenLava), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsOpenLava.R) -* Univa Grid Engine / Oracle Grid Engine (OGE) / Sun Grid Engine (SGE): [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSGE), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSGE.R) -* Slurm: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSlurm), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSlurm.R) -* TORQUE/OpenPBS: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsTORQUE), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsTORQUE.R) +* Interactive Cluster Functions (default): `makeClusterFunctionsInteractive()`, [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsInteractive.R) +* Multicore Cluster Functions: `makeClusterFunctionsMulticore()`, [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsMulticore.R) +* Socket Cluster Functions: `makeClusterFunctionsSocket()`, [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSocket.R) +* Makeshift SSH cluster: `makeClusterFunctionsSSH()`, [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSSH.R) +* Docker Swarm: `makeClusterFunctionsDocker()`, [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsDocker.R) +* IBM Spectrum Load Sharing Facility (LSF): `makeClusterFunctionsLSF()`, [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsLSF.R) +* OpenLava: `makeClusterFunctionsOpenLava()`, [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsOpenLava.R) +* Univa Grid Engine / Oracle Grid Engine (OGE) / Sun Grid Engine (SGE): `makeClusterFunctionsSGE()`, [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSGE.R) +* Slurm: `makeClusterFunctionsSlurm()`, [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSlurm.R) +* TORQUE/OpenPBS: `makeClusterFunctionsTORQUE()`, [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsTORQUE.R) -To use the package with the socket cluster functions, you would call the respective constructor [makeClusterFunctionsSocket()](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSocket): +To use the package with the socket cluster functions, you would call the respective constructor `makeClusterFunctionsSocket()`: ```{r, message=FALSE} reg = makeRegistry(NA) reg$cluster.functions = makeClusterFunctionsSocket(2) ``` -To make this selection permanent for this registry, save the Registry with [saveRegistry()](https://mlr-org.github.io/batchtools/reference/makeRegistry). +To make this selection permanent for this registry, save the Registry with `saveRegistry()`. To make your cluster function selection permanent for a specific system across R sessions for all new Registries, you can set up a configuration file (see below). If you have trouble debugging your cluster functions, you can enable the debug mode for extra output. @@ -69,30 +69,30 @@ These templates are used to communicate with the scheduler and contain placehold Internally, the [brew package](https://cran.r-project.org/package=brew) is used for this purpose. Some exemplary template files can be found [here](https://github.com/mlr-org/batchtools/tree/master/inst/templates). It would be great if you would help expand this collection to cover more exotic configurations. -To do so, please send your template via [mail](mailto:michellang@gmail.com) or open a new pull request. +To do so, open a new pull request. -Note that all variables defined in a [JobCollection](https://mlr-org.github.io/batchtools/reference/JobCollection) can be used inside the template. -If you need to pass extra variables, you can set them via the argument `resources` of [`submitJobs()`](https://mlr-org.github.io/batchtools/reference/submitJobs). +Note that all variables defined in a `JobCollection()` can be used inside the template. +If you need to pass extra variables, you can set them via the argument `resources` of `submitJobs()`. -If the flexibility which comes with templating is not sufficient, you can still construct a custom cluster function implementation yourself using the provided [constructor](https://mlr-org.github.io/batchtools/reference/makeClusterFunctions). +If the flexibility which comes with templating is not sufficient, you can still construct a custom cluster function implementation yourself using the provided `makeClusterFunctions()`. ## Configuration File The configuration file can be used to set system specific options. -Its default location depends on the operating system (see [Registry](https://mlr-org.github.io/batchtools/reference/makeRegistry)), but for the first time setup you can put one in the current working directory (as reported by `getwd()`). +Its default location depends on the operating system (see `makeRegistry()`), but for the first time setup you can put one in the current working directory (as reported by `getwd()`). In order to set the cluster function implementation, you would generate a file with the following content: ```{r,eval = FALSE} cluster.functions = makeClusterFunctionsInteractive() ``` -The configuration file is parsed whenever you create or load a [Registry](https://mlr-org.github.io/batchtools/reference/makeRegistry). -It is sourced inside of your registry which has the advantage that you can (a) access all of the parameters which are passed to [makeRegistry](https://mlr-org.github.io/batchtools/reference/makeRegistry) and (b) you can also directly change them. +The configuration file is parsed whenever you create or load a `Registry`. +It is sourced inside of your registry which has the advantage that you can (a) access all of the parameters which are passed to makeRegistry() and (b) you can also directly change them. Lets say you always want your working directory in your home directory and you always want to load the `checkmate` package on the nodes, you can just append these lines: ```{r, eval = FALSE} work.dir = "~" packages = union(packages, "checkmate") ``` -See the documentation on [Registry](https://mlr-org.github.io/batchtools/reference/makeRegistry) for a more complete list of supported configuration options. +See the documentation on `Registry` for a more complete list of supported configuration options. # Migration from `BatchJobs`/`Batchexperiments` @@ -102,14 +102,14 @@ The development of [BatchJobs](https://github.com/tudo-r/BatchJobs/) and [BatchE * Maintainability: The packages [BatchJobs](https://github.com/tudo-r/BatchJobs/) and [BatchExperiments](https://github.com/tudo-r/Batchexperiments) are tightly connected which makes maintaining difficult. Changes have to be synchronized and tested against the current CRAN versions for compatibility. Furthermore, BatchExperiments violates CRAN policies by calling internal functions of BatchJobs. * Data base issues: Although we invested weeks to mitigate issues with locks of the SQLite data base or file system (staged queries, file system timeouts, ...), BatchJobs kept working unreliable on some systems with high latency or specific file systems. This made BatchJobs unusable for many users. -[BatchJobs](https://github.com/tudo-r/BatchJobs/) and [BatchExperiments](https://github.com/tudo-r/Batchexperiments) will remain on CRAN, but new features are unlikely to be ported back. +[BatchJobs](https://github.com/tudo-r/BatchJobs/) and [BatchExperiments](https://github.com/tudo-r/Batchexperiments) will remain on CRAN or Github, but new features are unlikely to be ported back. ## Internal Changes * batchtools does not use SQLite anymore. Instead, all the information is stored directly in the registry using [data.tables](https://cran.r-project.org/package=data.table) acting as an in-memory database. As a side effect, many operations are much faster. * Nodes do not have to access the registry. - [submitJobs()](https://mlr-org.github.io/batchtools/reference/submitJobs) stores a temporary object of type [JobCollection](https://mlr-org.github.io/batchtools/reference/JobCollection) on the file system which holds all the information necessary to execute a chunk of jobs via [doJobCollection()](https://mlr-org.github.io/batchtools/reference/doJobCollection) on the node. + `submitJobs()` stores a temporary object of type `doJobCollection()` on the node. This avoids file system locks because each job accesses only one file exclusively. * `ClusterFunctionsMulticore` now uses the parallel package for multicore execution. * `ClusterFunctionsSSH` can still be used to emulate a scheduler-like system which respects the work load on the local machine. @@ -120,60 +120,33 @@ The development of [BatchJobs](https://github.com/tudo-r/BatchJobs/) and [BatchE This way, you do not need to pass the registry around anymore. If you need to work with multiple registries simultaneously on the other hand, you can still do so by explicitly passing registries to the functions. * Most functions now return a [data.table](https://cran.r-project.org/package=data.table) which is keyed with the `job.id`. - This way, return values can be joined together easily and efficient (see this [help page](https://mlr-org.github.io/batchtools/reference/JoinTables) for some examples). + This way, return values can be joined together easily and efficient (see `?JoinTables`) for some examples). * The building blocks of a problem has been renamed from `static` and `dynamic` to the more intuitive `data` and `fun`. Thus, algorithm function should have the formal arguments `job`, `data` and `instance`. * The function `makeDesign` has been removed. - Parameters can be defined by just passing a `data.frame` or `data.table` to [addExperiments](https://mlr-org.github.io/batchtools/reference/addExperiments). + Parameters can be defined by just passing a `data.frame` or `data.table` to `addExperiments()`. For exhaustive designs, use `data.table::CJ()`. ## Template changes * The scheduler should directly execute the command: + ``` Rscript -e 'batchtools::doJobCollection()' ``` + There is no intermediate R source file like there was in `BatchJobs`. -* All information stored in the object [`JobCollection`](https://mlr-org.github.io/batchtools/reference/JobCollection) can be accessed while brewing the template. -* Extra variables may be passed via the argument `resoures` of [submitJobs](https://mlr-org.github.io/batchtools/reference/submitJobs). +* All information stored in the object `?JobCollection` can be accessed while brewing the template. +* Extra variables may be passed via the argument `resoures` of `submitJobs()`. ## New features -* Support for Docker Swarm via `ClusterFunctionsDocker`. +* Support for Docker Swarm via `?ClusterFunctionsDocker`. * Jobs can now be tagged and untagged to provide an easy way to group them. * Some resources like the number of CPUs are now optionally passed to [parallelMap](https://cran.r-project.org/package=parallelMap). This eases nested parallelization, e.g. to use multicore parallelization on the slave by just setting a resource on the master. - See [submitJobs()](https://mlr-org.github.io/batchtools/reference/submitJobs) for an example. + See `submitJobs()` for an example. * `ClusterFunctions` are now more flexible in general as they can define hook functions which will be called at certain events. - [ClusterFunctionsDocker](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsDocker.R) is an example use case which implements a housekeeping routine. + `?ClusterFunctionsDocker` is an example use case which implements a housekeeping routine. This routine is called every time before a job is about to get submitted to the scheduler (in the case: the Docker Swarm) via the hook `pre.submit` and every time directly after the registry synchronized jobs stored on the file system via the hook `post.sync`. -* More new features are covered in the [NEWS](https://mlr-org.github.io/batchtools/news/index.html). - - -## Porting to `batchtools` - -The following table assists in porting to batchtools by mapping BatchJobs/BatchExperiments functions to their counterparts in batchtools. -The table does not cover functions which are (a) used only internally in BatchJobs and (b) functions which have not been renamed. - -| BatchJobs | batchtools | -| ------------- | :-------------: | -| `addRegistryPackages` | Set `reg$packages` or `reg$namespaces`, call [saveRegistry()](https://mlr-org.github.io/batchtools/reference/saveRegistry) | -| `addRegistrySourceDirs` | - | -| `addRegistrySourceFiles` | Set `reg$source`, call [saveRegistry()](https://mlr-org.github.io/batchtools/reference/saveRegistry) | -| `batchExpandGrid` | [batchMap](https://mlr-org.github.io/batchtools/reference/batchMap): `batchMap(..., args = CJ(x = 1:3, y = 1:10))` | -| `batchMapQuick` | [btmapply](https://mlr-org.github.io/batchtools/reference/btlapply) | -| `batchReduceResults` | - | -| `batchUnexport` | [batchExport](https://mlr-org.github.io/batchtools/reference/batchExport) | -| `filterResults` | - | -| `getJobIds` | [findJobs](https://mlr-org.github.io/batchtools/reference/findJobs) | -| `getJobInfo` | [getJobStatus](https://mlr-org.github.io/batchtools/reference/getJobTable) | -| `getJob` | [makeJob](https://mlr-org.github.io/batchtools/reference/JobExperiment) | -| `getJobParamDf` | [getJobPars](https://mlr-org.github.io/batchtools/reference/getJobTable) | -| `loadResults` | [reduceResultsList](https://mlr-org.github.io/batchtools/reference/reduceResultsList) | -| `reduceResultsDataFrame` | [reduceResultsDataTable](https://mlr-org.github.io/batchtools/reference/reduceResultsList) | -| `reduceResultsMatrix` | [reduceResultsList](https://mlr-org.github.io/batchtools/reference/reduceResultsList) + `do.call(rbind, res)` | -| `reduceResultsVector` | [reduceResultsDataTable](https://mlr-org.github.io/batchtools/reference/reduceResultsList) | -| `setJobFunction` | - | -| `setJobNames` | - | -| `showStatus` | [getStatus](https://mlr-org.github.io/batchtools/reference/getStatus) | @@ -182,8 +155,8 @@ To get a first insight into the usage of `batchtools`, we start with an exemplar For background information, see [Wikipedia](https://en.wikipedia.org/wiki/Monte_Carlo_method). First, a so-called registry object has to be created, which defines a directory where all relevant information, files and results of the computational jobs will be stored. -There are two different types of registry objects: First, a regular [`Registry`](https://mlr-org.github.io/batchtools/reference/makeRegistry) which we will use in this example. -Second, an [`ExperimentRegistry`](https://mlr-org.github.io/batchtools/reference/makeExperimentRegistry) which provides an alternative way to define computational jobs and thereby is tailored for a broad range of large scale computer experiments. +There are two different types of registry objects: First, a regular `?Registry` which we will use in this example. +Second, an `?ExperimentRegistry` which provides an alternative way to define computational jobs and thereby is tailored for a broad range of large scale computer experiments. Here, we use a temporary registry which is stored in the temp directory of the system and gets automatically deleted if you close the R session. ```{r, message = FALSE} @@ -195,8 +168,8 @@ It can then be reused later, e.g., when you login to the system again, by callin When a registry object is created or loaded, it is stored for the active R session as the default. Therefore the argument `reg` will be ignored in functions calls of this example, assuming the correct registry is set as default. -To get the current default registry, [`getDefaultRegistry`](https://mlr-org.github.io/batchtools/reference/getDefaultRegistry) can be used. -To switch to another registry, use [`setDefaultRegistry()`](https://mlr-org.github.io/batchtools/reference/getDefaultRegistry). +To get the current default registry, `getDefaultRegistry()` can be used. +To switch to another registry, use `setDefaultRegistry()`. First, we create a function which samples $n$ points $(x_i, y_i)$ whereas $x_i$ and $y_i$ are distributed uniformly, i.e. $x_i, y_i \sim \mathcal{U}(0,1)$. Next, the distance to the origin $(0, 0)$ is calculated and the fraction of points in the unit circle ($d \leq 1$) is returned. @@ -210,39 +183,39 @@ set.seed(42) piApprox(1000) ``` We now parallelize `piApprox()` with `batchtools`: We create 10 jobs, each doing a MC simulation with $10^5$ jobs. -We use [`batchMap()`](https://mlr-org.github.io/batchtools/reference/batchMap) to define the jobs (note that this does not yet start the calculation): +We use `batchMap()` to define the jobs (note that this does not yet start the calculation): ```{r} batchMap(fun = piApprox, n = rep(1e5, 10)) ``` The length of the vector or list defines how many different jobs are created, while the elements itself are used as arguments for the function. The function `batchMap(fun, ...)` works analogously to `Map(f, ...)` of the base package. -An overview over the jobs and their IDs can be retrieved with [`getJobTable()`](https://mlr-org.github.io/batchtools/reference/getJobTable) which returns a data.frame with all relevant information: +An overview over the jobs and their IDs can be retrieved with `getJobTable()` which returns a data.frame with all relevant information: ```{r} names(getJobTable()) ``` Note that a unique job ID is assigned to each job. These IDs can be used to restrict operations to subsets of jobs. -To actually start the calculation, call [`submitJobs()`](https://mlr-org.github.io/batchtools/reference/submitJobs). +To actually start the calculation, call `submitJobs()`. The registry and the selected job IDs can be taken as arguments as well as an arbitrary list of resource requirements, which are to be handled by the cluster back end. ```{r} submitJobs(resources = list(walltime = 3600, memory = 1024)) ``` In this example, a cap for the execution time (so-called walltime) and for the maximum memory requirements are set. -The progress of the submitted jobs can be checked with [`getStatus()`](https://mlr-org.github.io/batchtools/reference/getStatus). +The progress of the submitted jobs can be checked with `getStatus()`. ```{r} getStatus() ``` The resulting output includes the number of jobs in the registry, how many have been submitted, have started to execute on the batch system, are currently running, have successfully completed, and have terminated due to an R exception. After jobs have successfully terminated, we can load their results on the master. -This can be done in a simple fashion by using either [`loadResult()`](https://mlr-org.github.io/batchtools/reference/loadResult), which returns a single result exactly in the form it was calculated during mapping, or by using [`reduceResults()`](https://mlr-org.github.io/batchtools/reference/reduceResults), which is a version of `Reduce()` from the base package for registry objects. +This can be done in a simple fashion by using either `loadResult()`, which returns a single result exactly in the form it was calculated during mapping, or by using `reduceResults()`, which is a version of `Reduce()` from the base package for registry objects. ```{r} waitForJobs() mean(sapply(1:10, loadResult)) reduceResults(function(x, y) x + y) / 10 ``` -If you are absolutely sure that your function works, you can take a shortcut and use *batchtools* in an `lapply` fashion using [`btlapply()`](https://mlr-org.github.io/batchtools/reference/btlapply). -This function creates a temporary registry (but you may also pass one yourself), calls [`batchMap()`](https://mlr-org.github.io/batchtools/reference/reduceResultsList), wait for the jobs to terminate with [`waitForJobs()`](https://mlr-org.github.io/batchtools/reference/waitForJobs) and then uses [`reduceResultsList()`](https://mlr-org.github.io/batchtools/reference/reduceResultsList) to return the results. +If you are absolutely sure that your function works, you can take a shortcut and use *batchtools* in an `lapply` fashion using `btlapply()`. +This function creates a temporary registry (but you may also pass one yourself), calls `batchMap()`, wait for the jobs to terminate with `waitForJobs()` and then uses `reduceResultsList()` to return the results. ```{r, R.options=list(batchtools.verbose=FALSE)} res = btlapply(rep(1e5, 10), piApprox) @@ -254,7 +227,7 @@ mean(unlist(res)) We stick to a rather simple, but not unrealistic example to explain some further functionalities: Applying two classification learners to the famous iris data set (Anderson 1935), vary a few hyperparameters and evaluate the effect on the classification performance. First, we create a registry, the central meta-data object which records technical details and the setup of the experiments. -We use an [`ExperimentRegistry`](https://mlr-org.github.io/batchtools/reference/makeExperimentRegistry) where the job definition is split into creating problems and algorithms. +We use an `?ExperimentRegistry` where the job definition is split into creating problems and algorithms. See the paper on [BatchJobs and BatchExperiments](https://www.jstatsoft.org/article/view/v064i11) for a detailed explanation. Again, we use a temporary registry and make it the default registry. ```{r, message = FALSE} @@ -267,11 +240,11 @@ reg = makeExperimentRegistry(file.dir = NA, seed = 1) By adding a problem to the registry, we can define the data on which certain computational jobs shall work. This can be a matrix, data frame or array that always stays the same for all subsequent experiments. But it can also be of a more dynamic nature, e.g., subsamples of a dataset or random numbers drawn from a probability distribution . -Therefore the function [`addProblem()`](https://mlr-org.github.io/batchtools/reference/addProblem) accepts static parts in its `data` argument, which is passed to the argument `fun` which generates a (possibly stochastic) problem instance. +Therefore the function `addProblem()` accepts static parts in its `data` argument, which is passed to the argument `fun` which generates a (possibly stochastic) problem instance. For `data`, any R object can be used. If only `data` is given, the generated instance is `data`. The argument `fun` has to be a function with the arguments `data` and `job` (and optionally other arbitrary parameters). -The argument `job` is an object of type [`Job`](https://mlr-org.github.io/batchtools/reference/JobExperiment) which holds additional information about the job. +The argument `job` is an object of type `?Job` which holds additional information about the job. We want to split the iris data set into a training set and test set. In this example we use use subsampling which just randomly takes a fraction of the observations as training set. @@ -284,7 +257,7 @@ subsample = function(data, job, ratio, ...) { list(test = test, train = train) } ``` -[`addProblem()`](https://mlr-org.github.io/batchtools/reference/addProblem) files the problem to the file system and the problem gets recorded in the registry. +`addProblem()` files the problem to the file system and the problem gets recorded in the registry. ```{r} data("iris", package = "datasets") addProblem(name = "iris", data = iris, fun = subsample, seed = 42) @@ -294,7 +267,7 @@ In this process, the `data` part will be loaded and passed to the function. Note that we set a problem seed to synchronize the experiments in the sense that the same resampled training and test sets are used for the algorithm comparison in each distinct replication. The algorithms for the jobs are added to the registry in a similar manner. -When using [`addAlgorithm()`](https://mlr-org.github.io/batchtools/reference/addAlgorithm), an identifier as well as the algorithm to apply to are required arguments. +When using `addAlgorithm()`, an identifier as well as the algorithm to apply to are required arguments. The algorithm must be given as a function with arguments `job`, `data` and `instance`. Further arbitrary arguments (e.g., hyperparameters or strategy parameters) may be defined analogously as for the function in `addProblem`. The objects passed to the function via `job` and `data` are here the same as above, while via `instance` the return value of the evaluated problem function is passed. @@ -338,11 +311,11 @@ knitr::include_graphics("tikz_prob_algo_simple.png", auto_pdf = TRUE) ``` ## Creating jobs -[`addExperiments()`](https://mlr-org.github.io/batchtools/reference/addExperiments) is used to parametrize the jobs and thereby define computational jobs. -To do so, you have to pass named lists of parameters to [`addExperiments()`](https://mlr-org.github.io/batchtools/reference/addExperiments). +`addExperiments()` is used to parametrize the jobs and thereby define computational jobs. +To do so, you have to pass named lists of parameters to `addExperiments()`. The elements of the respective list (one for problems and one for algorithms) must be named after the problem or algorithm they refer to. The data frames contain parameter constellations for the problem or algorithm function where columns must have the same names as the target arguments. -When the problem design and the algorithm design are combined in [`addExperiments()`](https://mlr-org.github.io/batchtools/reference/addExperiments), each combination of the parameter sets of the two designs defines a distinct job. +When the problem design and the algorithm design are combined in `addExperiments()`, each combination of the parameter sets of the two designs defines a distinct job. How often each of these jobs should be computed can be determined with the argument `repls`. ```{r} @@ -359,7 +332,7 @@ ades = list( addExperiments(pdes, ades, repls = 5) ``` The jobs are now available in the registry with an individual job ID for each. -The function [`summarizeExperiments()`](https://mlr-org.github.io/batchtools/reference/summarizeExperiments) returns a table which gives a quick overview over all defined experiments. +The function `summarizeExperiments()` returns a table which gives a quick overview over all defined experiments. ```{r} summarizeExperiments() @@ -370,14 +343,14 @@ summarizeExperiments(by = c("problem", "algorithm", "ratio")) Before submitting all jobs to the batch system, we encourage you to test each algorithm individually. Or sometimes you want to submit only a subset of experiments because the jobs vastly differ in runtime. Another reoccurring task is the collection of results for only a subset of experiments. -For all these use cases, [`findExperiments()`](https://mlr-org.github.io/batchtools/reference/findJobs) can be employed to conveniently select a particular subset of jobs. +For all these use cases, `findExperiments()` can be employed to conveniently select a particular subset of jobs. It returns the IDs of all experiments that match the given criteria. Your selection can depend on substring matches of problem or algorithm IDs using `prob.name` or `algo.name`, respectively. You can also pass R expressions, which will be evaluated in your problem parameter setting (`prob.pars`) or algorithm parameter setting (`algo.pars`). The expression is then expected to evaluate to a Boolean value. Furthermore, you can restrict the experiments to specific replication numbers. -To illustrate [`findExperiments()`](https://mlr-org.github.io/batchtools/reference/findJobs), we will select two experiments, one with a support vector machine and the other with a random forest and the parameter `ntree = 1000`. +To illustrate `findExperiments()`, we will select two experiments, one with a support vector machine and the other with a random forest and the parameter `ntree = 1000`. The selected experiment IDs are then passed to testJob. ```{r} id1 = head(findExperiments(algo.name = "svm"), 1) @@ -393,19 +366,19 @@ If everything turns out fine, we can proceed with the calculation. ## Submitting and Collecting Results -To submit the jobs, we call [`submitJobs()`](https://mlr-org.github.io/batchtools/reference/submitJobs) and wait for all jobs to terminate using [`waitForJobs()`](https://mlr-org.github.io/batchtools/reference/waitForJobs). +To submit the jobs, we call `submitJobs()` and wait for all jobs to terminate using `waitForJobs()`. ```{r} submitJobs() waitForJobs() ``` -After jobs are finished, the results can be collected with [`reduceResultsDataTable()`](https://mlr-org.github.io/batchtools/reference/reduceResultsList) where we directly extract the mean misclassification error: +After jobs are finished, the results can be collected with `reduceResultsDataTable()` where we directly extract the mean misclassification error: ```{r} reduce = function(res) list(mce = (sum(res) - sum(diag(res))) / sum(res)) results = unwrap(reduceResultsDataTable(fun = reduce)) head(results) ``` -Next, we merge the results table with the table of job parameters using one of the [join helpers](https://mlr-org.github.io/batchtools/reference/JoinTables) provided by `batchtools` (here, we use an inner join): +Next, we merge the results table with the table of job parameters using one of the join helpers (see `?JoinTables`) provided by `batchtools` (here, we use an inner join): ```{r} pars = unwrap(getJobPars()) tab = ijoin(pars, results) @@ -426,7 +399,7 @@ In any large scale experiment many things can and will go wrong. The cluster might have an outage, jobs may run into resource limits or crash, subtle bugs in your code could be triggered or any other error condition might arise. In these situations it is important to quickly determine what went wrong and to recompute only the minimal number of required jobs. -Therefore, before you submit anything you should use [`testJob()`](https://mlr-org.github.io/batchtools/reference/testJob) to catch errors that are easy to spot because they are raised in many or all jobs. +Therefore, before you submit anything you should use `testJob()` to catch errors that are easy to spot because they are raised in many or all jobs. If `external` is set, this function runs the job without side effects in an independent R process on your local machine via `Rscript` similar as on the slave, redirects the output of the process to your R console, loads the job result and returns it. If you do not set `external`, the job is executed is in the currently running R session, with the drawback that you might be unable to catch missing variable declarations or missing package dependencies. @@ -459,17 +432,17 @@ We ignore the error here, and just assume everything looks fine and submit all j submitJobs() waitForJobs() ``` -After you have submitted jobs and suspect that something is going wrong, the first thing to do is to run [`getStatus()`](https://mlr-org.github.io/batchtools/reference/getStatus) to display a summary of the current state of the system. +After you have submitted jobs and suspect that something is going wrong, the first thing to do is to run `getStatus()` to display a summary of the current state of the system. ```{r} getStatus() ``` The status message shows that two of the jobs could not be executed successfully. -To get the IDs of all jobs that failed due to an error we can use [`findErrors()`](https://mlr-org.github.io/batchtools/reference/findJobs) and to retrieve the actual error message, we can use [`getErrorMessages()`](https://mlr-org.github.io/batchtools/reference/getErrorMessages). +To get the IDs of all jobs that failed due to an error we can use `findErrors()` and to retrieve the actual error message, we can use `getErrorMessages()`. ```{r} findErrors() getErrorMessages() ``` -If we want to peek into the R log file of a job to see more context for the error we can use [`showLog()`](https://mlr-org.github.io/batchtools/reference/showLog) which opens a pager or use [`getLog()`](https://mlr-org.github.io/batchtools/reference/showLog) to get the log as character vector: +If we want to peek into the R log file of a job to see more context for the error we can use `showLog()` which opens a pager or use `getLog()` to get the log as character vector: ```{r} tail(getLog(id = 9)) ``` @@ -484,21 +457,21 @@ grepLogs(pattern = "simple", ignore.case = TRUE) ## On the Local System -1. Create a Registry with [`makeRegistry()`](https://mlr-org.github.io/batchtools/reference/makeRegistry) (or [`makeExperimentRegistry()`](https://mlr-org.github.io/batchtools/reference/makeExperimentRegistry)) or load an existing from the file system with [`loadRegistry()`](https://mlr-org.github.io/batchtools/reference/loadRegistry). -2. Define computational jobs with [`batchMap()`](https://mlr-org.github.io/batchtools/reference/batchMap) or [`batchReduce()`](https://mlr-org.github.io/batchtools/reference/batchReduce) if you used [`makeRegistry()`](https://mlr-org.github.io/batchtools/reference/makeRegistry) or define with [`addAlgorithm()`](https://mlr-org.github.io/batchtools/reference/addAlgorithm), [`addProblem()`](https://mllg.github.io/batchtools/reference/addProblem) and [`addExperiments()`](https://mllg.github.io/batchtools/reference/addExperiments) if you started with [`makeExperimentRegistry()`](https://mllg.github.io/batchtools/reference/makeExperimentRegistry). - It is advised to test some jobs with [`testJob()`](https://mlr-org.github.io/batchtools/reference/testJob) in the interactive session and with `testJob(external = TRUE)` in a separate R process. - Note that you can add additional jobs if you are using an [`ExperimentRegistry`](https://mlr-org.github.io/batchtools/reference/makeExperimentRegistry). -3. If required, query the data base for job ids depending on their status, parameters or tags (see [`findJobs()`](https://mlr-org.github.io/batchtools/reference/findJobs)). - The returned tables can easily be combined in a set-like fashion with data base verbs: union ([`ojoin()`](https://mlr-org.github.io/batchtools/reference/JoinTables) for outer join), intersect ([`ijoin()`](https://mlr-org.github.io/batchtools/reference/JoinTables) for inner join), difference ([`ajoin()`](https://mlr-org.github.io/batchtools/reference/JoinTables) for anti join). -4. Submit jobs with [`submitJobs()`](https://mlr-org.github.io/batchtools/reference/submitJobs). You can specify job resources here. - If you have thousands of fast terminating jobs, you want to [`chunk()`](https://mlr-org.github.io/batchtools/reference/chunk) them first. - If some jobs already terminated, you can estimate the runtimes with [`estimateRuntimes()`](https://mlr-org.github.io/batchtools/reference/estimateRuntimes) and chunk jobs into heterogeneous groups with [`lpt()`](https://mlr-org.github.io/batchtools/reference/chunk) and [`binpack()`](https://mlr-org.github.io/batchtools/reference/chunk). -5. Monitor jobs. [`getStatus()`](https://mlr-org.github.io/batchtools/reference/getStatus) gives a summarizing overview. - Use [`showLog()`](https://mlr-org.github.io/batchtools/reference/showLog) and [`grepLogs()`](https://mlr-org.github.io/batchtools/reference/grepLogs) to investigate log file. - Run jobs in the currently running session with [`testJob()`](https://mlr-org.github.io/batchtools/reference/testJob) to get a `traceback()`. -6. Collect (partial) results. [`loadResult()`](https://mlr-org.github.io/batchtools/reference/loadResult) retrieves a single result from the file system. - [`reduceResults()`](https://mlr-org.github.io/batchtools/reference/reduceResults) mimics `Reduce()` and allows to apply a function to many files in an iterative fashion. - [`reduceResultsList()`](https://mlr-org.github.io/batchtools/reference/reduceResultsList) and [`reduceResultsDataTable()`](https://mlr-org.github.io/batchtools/reference/reduceResultsList) collect results into a `list` or `data.table`, respectively. +1. Create a Registry with `makeRegistry()` / `makeExperimentRegistry()` or load an existing from the file system with `loadRegistry()`. +2. Define computational jobs with `batchMap()` / `batchReduce()` if you used `makeRegistry()` or define an experiment with `addAlgorithm()`, `addProblem()` and `addExperiments()` if you started with `makeExperimentRegistry()`. + It is advised to test some jobs with `testJob()` in the interactive session and with `testJob(external = TRUE)` in a separate R process. + Note that you can add additional jobs if you are using an `?ExperimentRegistry`. +3. If required, query the data base for job ids depending on their status, parameters or tags (see `findJobs()`). + The returned tables can easily be combined in a set-like fashion with data base verbs: union (`ojoin()` for outer join, intersect `ijoin()` for inner join, difference `ajoin()` for anti join). +4. Submit jobs with `submitJobs()`. You can specify job resources here. + If you have thousands of fast terminating jobs, you want to `chunk()` them first. + If some jobs already terminated, you can estimate the runtimes with `estimateRuntimes()` and chunk jobs into heterogeneous groups with `lpt()` and `binpack()`. +5. Monitor jobs. `getStatus()` gives a summarizing overview. + Use `showLog()` and `grepLogs()` to investigate log file. + Run jobs in the currently running session with `testJob()` to get a `traceback()`. +6. Collect (partial) results. `loadResult()` retrieves a single result from the file system. + `reduceResults()` mimics `Reduce()` and allows to apply a function to many files in an iterative fashion. + `reduceResultsList()` and `reduceResultsDataTable()` collect results into a `list` or `data.table`, respectively. ```{r,echo=FALSE} knitr::include_graphics("function_overview.png", auto_pdf = TRUE) @@ -507,16 +480,16 @@ knitr::include_graphics("function_overview.png", auto_pdf = TRUE) ## On Multiple Systems Most users develop and prototype their experiments on a desktop box in their preferred IDE and later deploy to a large computing cluster. -This can be done by prototyping locally ([`testJob()`](https://mlr-org.github.io/batchtools/reference/testJob) or submit subsets via [`submitJobs()`](https://mlr-org.github.io/batchtools/reference/submitJobs)). +This can be done by prototyping locally (`testJob()`) or submit subsets via `submitJobs()`. To deploy to the cluster, just copy the file directory (as reported by `reg$file.dir`) to the remote system. Next, log in on the cluster (typically via `ssh`), `cd` to the copied directory and call `loadRegistry(", "", writeable = TRUE)`. This function will (a) source the local configuration file so that you can talk to the cluster (verify by checking the output of `reg$cluster.functions`) and (b) adjust the paths to the new system if argument `update.paths` is set. -After loading the Registry, it is advised to test some jobs again with [`testJob()`](https://mlr-org.github.io/batchtools/reference/testJob) before submitting all of them with `submitJobs(resources = list())` (remember you now need to set resources!). -After some jobs are finished, the `file.dir` can be copied back (do not merge with the previous directory!) and loaded again with [`loadRegistry()`](https://mlr-org.github.io/batchtools/reference/loadRegistry). +After loading the Registry, it is advised to test some jobs again with `testJob()` before submitting all of them with `submitJobs(resources = list())` (remember you now need to set resources!). +After some jobs are finished, the `file.dir` can be copied back (do not merge with the previous directory!) and loaded again with `loadRegistry()`. This approach is totally viable as long as some general rules are followed: -1. Make sure you have all packages installed. Package versions can be synchronized across machines with [`checkpoint`](https://cran.r-project.org/package=checkpoint) or [`packrat`](https://cran.r-project.org/package=packrat). +1. Make sure you have all packages installed. Package versions can be synchronized across machines with, e.g., [`checkpoint`](https://cran.r-project.org/package=checkpoint) or [`packrat`](https://cran.r-project.org/package=packrat). 2. Test jobs on the remote system prior to submit to ensure that paths are resolved correctly. 3. Make sure you have set the cluster functions in a configuration file, and stick to one backend as long as jobs are running. 4. The status can only be monitored on the remote system (for obvious reasons). @@ -524,7 +497,7 @@ This approach is totally viable as long as some general rules are followed: For the latter, you need to copy over the **complete** `file.dir` first. Overwriting/merging directories is not advised as this may lead to inconsistencies if you added or removed experiments on the remote. If you have to merge, use `rsync` with option `--delete`. - Load the registry locally with [`loadRegistry()`](https://mlr-org.github.io/batchtools/reference/loadRegistry) and collect results. Do not copy back and forth. + Load the registry locally with `loadRegistry()` and collect results. Do not copy back and forth. 6. Avoid accessing the `file.dir` with multiple sessions simultaneously. This includes accessing the registry via a mount! Simultaneous access may lead to inconsistencies and missing results. From b03e9e7e42d2aafb7483b63eabf4eba6fcd93b72 Mon Sep 17 00:00:00 2001 From: Michel Lang Date: Mon, 18 Aug 2025 21:48:26 +0200 Subject: [PATCH 2/6] add some missing links --- vignettes/batchtools.Rmd | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/vignettes/batchtools.Rmd b/vignettes/batchtools.Rmd index 8d4c2a5b..fc76783b 100644 --- a/vignettes/batchtools.Rmd +++ b/vignettes/batchtools.Rmd @@ -51,7 +51,7 @@ To use the package with the socket cluster functions, you would call the respect reg = makeRegistry(NA) reg$cluster.functions = makeClusterFunctionsSocket(2) ``` -To make this selection permanent for this registry, save the Registry with `saveRegistry()`. +To make this selection permanent for this registry, save the `?Registry` with `saveRegistry()`. To make your cluster function selection permanent for a specific system across R sessions for all new Registries, you can set up a configuration file (see below). If you have trouble debugging your cluster functions, you can enable the debug mode for extra output. @@ -71,7 +71,7 @@ Some exemplary template files can be found [here](https://github.com/mlr-org/bat It would be great if you would help expand this collection to cover more exotic configurations. To do so, open a new pull request. -Note that all variables defined in a `JobCollection()` can be used inside the template. +Note that all variables defined in a `?JobCollection` can be used inside the template. If you need to pass extra variables, you can set them via the argument `resources` of `submitJobs()`. If the flexibility which comes with templating is not sufficient, you can still construct a custom cluster function implementation yourself using the provided `makeClusterFunctions()`. @@ -85,14 +85,14 @@ In order to set the cluster function implementation, you would generate a file w ```{r,eval = FALSE} cluster.functions = makeClusterFunctionsInteractive() ``` -The configuration file is parsed whenever you create or load a `Registry`. +The configuration file is parsed whenever you create or load a `?Registry`. It is sourced inside of your registry which has the advantage that you can (a) access all of the parameters which are passed to makeRegistry() and (b) you can also directly change them. Lets say you always want your working directory in your home directory and you always want to load the `checkmate` package on the nodes, you can just append these lines: ```{r, eval = FALSE} work.dir = "~" packages = union(packages, "checkmate") ``` -See the documentation on `Registry` for a more complete list of supported configuration options. +See the documentation on `?Registry` for a more complete list of supported configuration options. # Migration from `BatchJobs`/`Batchexperiments` @@ -144,7 +144,7 @@ Rscript -e 'batchtools::doJobCollection()' * Some resources like the number of CPUs are now optionally passed to [parallelMap](https://cran.r-project.org/package=parallelMap). This eases nested parallelization, e.g. to use multicore parallelization on the slave by just setting a resource on the master. See `submitJobs()` for an example. -* `ClusterFunctions` are now more flexible in general as they can define hook functions which will be called at certain events. +* `?ClusterFunctions` are now more flexible in general as they can define hook functions which will be called at certain events. `?ClusterFunctionsDocker` is an example use case which implements a housekeeping routine. This routine is called every time before a job is about to get submitted to the scheduler (in the case: the Docker Swarm) via the hook `pre.submit` and every time directly after the registry synchronized jobs stored on the file system via the hook `post.sync`. From b9074f715848a3c3c4fd550c92a24afd7432cc5c Mon Sep 17 00:00:00 2001 From: Michel Lang Date: Tue, 19 Aug 2025 20:45:31 +0200 Subject: [PATCH 3/6] fix some more links in README --- README.Rmd | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/README.Rmd b/README.Rmd index 23177601..f216351d 100644 --- a/README.Rmd +++ b/README.Rmd @@ -14,7 +14,7 @@ Package website: [release](https://batchtools.mlr-org.com/) | [dev](https://batc [![Mattermost](https://img.shields.io/badge/chat-mattermost-orange.svg)](https://lmmisld-lmu-stats-slds.srv.mwn.de/mlr_invite/) -As a successor of the packages [BatchJobs](https://github.com/tudo-r/BatchJobs) and [BatchExperiments](https://github.com/tudo-r/Batchexperiments), batchtools provides a parallel implementation of Map for high performance computing systems managed by schedulers like Slurm, Sun Grid Engine, OpenLava, TORQUE/OpenPBS, Load Sharing Facility (LSF) or Docker Swarm (see the setup section in the [vignette](https://batchtools.mlr-org.com/batchtools/articles/batchtools.html)). +As a successor of the packages [BatchJobs](https://github.com/tudo-r/BatchJobs) and [BatchExperiments](https://github.com/tudo-r/Batchexperiments), batchtools provides a parallel implementation of Map for high performance computing systems managed by schedulers like Slurm, Sun Grid Engine, OpenLava, TORQUE/OpenPBS, Load Sharing Facility (LSF) or Docker Swarm (see the setup section in the [vignette](https://batchtools.mlr-org.com/articles/batchtools.html)). Main features: @@ -37,7 +37,7 @@ devtools::install_github("mlr-org/batchtools") ``` Next, you need to setup `batchtools` for your HPC (it will run sequentially otherwise). -See the [vignette](https://mlr-org.github.io/batchtools/articles/batchtools.html#setup) for instructions. +See the [vignette](https://batchtools.mlr-org.com/articles/batchtools.html) for instructions. ## Why batchtools? The development of [BatchJobs](https://github.com/tudo-r/BatchJobs/) and [BatchExperiments](https://github.com/tudo-r/Batchexperiments) is discontinued for the following reasons: @@ -46,15 +46,14 @@ The development of [BatchJobs](https://github.com/tudo-r/BatchJobs/) and [BatchE * Data base issues: Although we invested weeks to mitigate issues with locks of the SQLite data base or file system (staged queries, file system timeouts, ...), `BatchJobs` kept working unreliable on some systems with high latency under certain conditions. This made `BatchJobs` unusable for many users. [BatchJobs](https://github.com/tudo-r/BatchJobs/) and [BatchExperiments](https://github.com/tudo-r/Batchexperiments) will remain on CRAN, but new features are unlikely to be ported back. -The [vignette](https://batchtools.mlr-org.com/batchtools/articles/batchtools.html#migration) contains a section comparing the packages. +The [vignette](https://batchtools.mlr-org.com/articles/batchtools.html) contains a section comparing the packages. ## Resources -* [NEWS](https://batchtools.mlr-org.com/batchtools/news/) -* [Function reference](https://batchtools.mlr-org.com/batchtools/reference/) -* [Vignette](https://batchtools.mlr-org.com/batchtools/articles/batchtools.html) +* [Function reference](https://batchtools.mlr-org.com/reference/) +* [Vignette](https://batchtools.mlr-org.com/articles/batchtools.html) * [JOSS Paper](https://doi.org/10.21105/joss.00135): Short paper on batchtools. Please cite this if you use batchtools. -* [Paper on BatchJobs/BatchExperiments](https://www.jstatsoft.org/v64/i11): The described concept still holds for batchtools and most examples work analogously (see the [vignette](https://batchtools.mlr-org.com/batchtools/articles/batchtools.html#migration) for differences between the packages). +* [Paper on BatchJobs/BatchExperiments](https://www.jstatsoft.org/v64/i11): The described concept still holds for batchtools and most examples work analogously (see the [vignette](https://batchtools.mlr-org.com/articles/batchtools.html) for differences between the packages). ## Citation Please cite the [JOSS paper](https://doi.org/10.21105/joss.00135) using the following BibTeX entry: From f1df4ba24e9728b58ba174fe714e762b5a655f80 Mon Sep 17 00:00:00 2001 From: Michel Lang Date: Tue, 19 Aug 2025 20:50:26 +0200 Subject: [PATCH 4/6] fix links in description --- DESCRIPTION | 49 ++++++++++++++++++++++++++----------------------- 1 file changed, 26 insertions(+), 23 deletions(-) diff --git a/DESCRIPTION b/DESCRIPTION index dd09e18b..63043bd7 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -2,31 +2,30 @@ Package: batchtools Title: Tools for Computation on Batch Systems Version: 0.9.18 Authors@R: c( - person("Michel", "Lang", NULL, "michellang@gmail.com", - role = c("cre", "aut"), comment = c(ORCID = "0000-0001-9754-0393")), - person("Bernd", "Bischl", NULL, "bernd_bischl@gmx.net", role = "aut"), - person("Dirk", "Surmann", NULL, "surmann@statistik.tu-dortmund.de", - role = "ctb", comment = c(ORCID = "0000-0003-0873-137X")) - ) -Description: As a successor of the packages 'BatchJobs' and 'BatchExperiments', - this package provides a parallel implementation of the Map function for high - performance computing systems managed by schedulers 'IBM Spectrum LSF' - (), - 'OpenLava' (), 'Univa Grid Engine'/'Oracle Grid - Engine' (), 'Slurm' (), - 'TORQUE/PBS' + person("Michel", "Lang", , "michellang@gmail.com", role = c("cre", "aut"), + comment = c(ORCID = "0000-0001-9754-0393")), + person("Bernd", "Bischl", , "bernd_bischl@gmx.net", role = "aut"), + person("Dirk", "Surmann", , "surmann@statistik.tu-dortmund.de", role = "ctb", + comment = c(ORCID = "0000-0003-0873-137X")) + ) +Description: As a successor of the packages 'BatchJobs' and + 'BatchExperiments', this package provides a parallel implementation of + the Map function for high performance computing systems managed by + schedulers 'IBM Spectrum LSF' + (), 'Univa Grid + Engine'/'Oracle Grid Engine' + (), 'Slurm' + (), 'TORQUE/PBS' (), - or 'Docker Swarm' (). - A multicore and socket mode allow the parallelization on a local machines, - and multiple machines can be hooked up via SSH to create a makeshift - cluster. Moreover, the package provides an abstraction mechanism to define - large-scale computer experiments in a well-organized and reproducible way. + or 'Docker Swarm' (). A + multicore and socket mode allow the parallelization on a local + machines, and multiple machines can be hooked up via SSH to create a + makeshift cluster. Moreover, the package provides an abstraction + mechanism to define large-scale computer experiments in a + well-organized and reproducible way. License: LGPL-3 URL: https://github.com/mlr-org/batchtools, https://batchtools.mlr-org.com BugReports: https://github.com/mlr-org/batchtools/issues -NeedsCompilation: yes -ByteCompile: yes -Encoding: UTF-8 Depends: R (>= 3.0.0) Imports: @@ -47,8 +46,8 @@ Imports: withr (>= 2.0.0) Suggests: debugme, - doParallel, doMPI, + doParallel, e1071, foreach, future, @@ -61,6 +60,10 @@ Suggests: snow, testthat, tibble -VignetteBuilder: knitr +VignetteBuilder: + knitr +ByteCompile: yes +Encoding: UTF-8 +NeedsCompilation: yes Roxygen: list(r6 = FALSE) RoxygenNote: 7.3.2 From d1646b6114921d59a767d59a7a9ffad7b149db7e Mon Sep 17 00:00:00 2001 From: Michel Lang Date: Tue, 19 Aug 2025 20:56:14 +0200 Subject: [PATCH 5/6] add section on porting to vignette again --- vignettes/batchtools.Rmd | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/vignettes/batchtools.Rmd b/vignettes/batchtools.Rmd index fc76783b..2a5bfc96 100644 --- a/vignettes/batchtools.Rmd +++ b/vignettes/batchtools.Rmd @@ -149,6 +149,33 @@ Rscript -e 'batchtools::doJobCollection()' This routine is called every time before a job is about to get submitted to the scheduler (in the case: the Docker Swarm) via the hook `pre.submit` and every time directly after the registry synchronized jobs stored on the file system via the hook `post.sync`. +## Porting to `batchtools` + +The following table assists in porting to batchtools by mapping BatchJobs/BatchExperiments functions to their counterparts in batchtools. +The table does not cover functions which are (a) used only internally in BatchJobs and (b) functions which have not been renamed. + +| BatchJobs | batchtools | +| ------------- | :-------------: | +| `addRegistryPackages` | Set `reg$packages` or `reg$namespaces`, call saveRegistry()] | +| `addRegistrySourceDirs` | - | +| `addRegistrySourceFiles` | Set `reg$source`, call saveRegistry() | +| `batchExpandGrid` | `batchMap(..., args = CJ(x = 1:3, y = 1:10))` | +| `batchMapQuick` | `btmapply()` | +| `batchReduceResults` | - | +| `batchUnexport` | `batchExport()` | +| `filterResults` | - | +| `getJobIds` | `findJobs()` | +| `getJobInfo` | `getJobStatus()` | +| `getJob` | `makeJob()` | +| `getJobParamDf` | `getJobPars()` | +| `loadResults` | `reduceResultsList()` | +| `reduceResultsDataFrame` | `reduceResultsDataTable()` | +| `reduceResultsMatrix` | `reduceResultsList()` + `do.call(rbind, res)` | +| `reduceResultsVector` | `reduceResultsDataTable()` | +| `setJobFunction` | - | +| `setJobNames` | - | +| `showStatus` | `getStatus()` | + # Example 1: Approximation of $\pi$ To get a first insight into the usage of `batchtools`, we start with an exemplary Monte Carlo simulation to approximate $\pi$. From 3aeb20273fffeb8c718ad1b4b1f0b5182f6e5f99 Mon Sep 17 00:00:00 2001 From: Michel Lang Date: Tue, 19 Aug 2025 20:58:53 +0200 Subject: [PATCH 6/6] add articles to menu --- _pkgdown.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_pkgdown.yml b/_pkgdown.yml index 6c35b643..1d0a96e7 100644 --- a/_pkgdown.yml +++ b/_pkgdown.yml @@ -16,7 +16,7 @@ toc: navbar: structure: - left: [reference, news, book] + left: [reference, news, articles, book] right: [search, github, mattermost, stackoverflow, rss, lightswitch] components: home: ~