Zinbwave parallelization issue #38
Description
Hi,
I have been trying to run zinbwave on my dataset and haven't managed to finish it successfully. Specifically, I have not found a way to run it through with parallelization.
Even when subsetting the dataset to get only hundreds on genes/cells (but it is the same with larger subsets), after ~30 minutes the run is still not finished.
> filteredVar_scset_zinbclass: SummarizedExperiment
dim: 100 150
metadata(0):
assays(1): counts
rownames(100): HLA-DRA FTL ... PLEK IL2RG
rowData names(0):
colnames(150): ES1160017 ES1160022 ... ES1160312 ES1160316
colData names(9): biosampleID biosampleType ... subjectAge
> zinb <- zinbwave::zinbwave(filteredVar_scset_zinb,
+ K = 2,
+ BPPARAM=MulticoreParam(8)
+ )
When I kill the command, this is what I get
Warning messages:
1: In serialize(data, node$con, xdr = FALSE) :
'package:stats' may not be available when loading
2: In serialize(data, node$con, xdr = FALSE) :
'package:stats' may not be available when loading
3: In serialize(data, node$con, xdr = FALSE) :
'package:stats' may not be available when loading
4: In serialize(data, node$con, xdr = FALSE) :
'package:stats' may not be available when loading
5: In serialize(data, node$con, xdr = FALSE) :
'package:stats' may not be available when loading
6: In serialize(data, node$con, xdr = FALSE) :
'package:stats' may not be available when loading
7: In serialize(data, node$con, xdr = FALSE) :
'package:stats' may not be available when loading
8: In serialize(data, node$con, xdr = FALSE) :
'package:stats' may not be available when loading
This error is thrown also with zinbFit.
Instead, if I run:
> zinb <- zinbwave::zinbwave(filteredVar_scset_zinb,
+ K = 2,
+ BPPARAM=BiocParallel::SerialParam()
+ )
>
It finishes without problems.
I understand it is most likely a problem with the cluster/parallelization, but any clue on where I should start looking?
This also happen if I register cores before:
cores=8
BiocParallel::register(BiocParallel::MulticoreParam(workers=cores))
zinb <- zinbwave::zinbwave(filteredVar_scset_zinb,
K = 2
)Thank you very much for any suggestion,
Giovanni
R version 3.5.0 (2018-04-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server release 6.7 (Santiago)
Matrix products: default
BLAS/LAPACK: /usr/prog/OpenBLAS/0.2.8-gompi-1.5.14-NX-LAPACK-3.5.0/lib/libopenblas_nehalemp-r0.2.8.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] parallel stats4 stats graphics grDevices utils datasets
[8] methods base
other attached packages:
[1] bindrcpp_0.2.2 magrittr_1.5
[3] batchtools_0.9.11 data.table_1.11.8
[5] forcats_0.2.0 stringr_1.2.0
[7] dplyr_0.7.7 purrr_0.2.5
[9] readr_1.1.1 tidyr_0.8.2
[11] tibble_1.4.2 ggplot2_2.2.1
[13] tidyverse_1.2.1 biomaRt_2.38.0
[15] zinbwave_1.3.4 SingleCellExperiment_1.4.0
[17] SummarizedExperiment_1.12.0 DelayedArray_0.8.0
[19] BiocParallel_1.16.0 matrixStats_0.54.0
[21] Biobase_2.40.0 GenomicRanges_1.34.0
[23] GenomeInfoDb_1.16.0 IRanges_2.16.0
[25] S4Vectors_0.20.0 BiocGenerics_0.28.0
loaded via a namespace (and not attached):
[1] nlme_3.1-131 bitops_1.0-6 lubridate_1.7.4
[4] progress_1.2.0 httr_1.3.1 numDeriv_2016.8-1
[7] backports_1.1.2 tools_3.5.0 R6_2.3.0
[10] DBI_1.0.0 lazyeval_0.2.0 colorspace_1.3-2
[13] withr_2.1.2 tidyselect_0.2.5 prettyunits_1.0.2
[16] mnormt_1.5-5 compiler_3.5.0 cli_1.0.1
[19] glmnet_2.0-16 pspline_1.0-18 rvest_0.3.2
[22] xml2_1.1.1 checkmate_1.8.5 scales_0.4.1
[25] mvtnorm_1.0-8 psych_1.6.12 genefilter_1.64.0
[28] rappdirs_0.3.1 digest_0.6.12 foreign_0.8-67
[31] XVector_0.22.0 pkgconfig_2.0.2 stabledist_0.7-1
[34] ADGofTest_0.3 limma_3.38.2 rlang_0.3.0.1
[37] readxl_1.1.0 rstudioapi_0.8 RSQLite_1.1-2
[40] bindr_0.1.1 jsonlite_1.5 RCurl_1.95-4.8
[43] GenomeInfoDbData_1.1.0 Matrix_1.2-8 Rcpp_1.0.0
[46] munsell_0.4.3 stringi_1.1.3 edgeR_3.24.0
[49] zlibbioc_1.26.0 plyr_1.8.4 grid_3.5.0
[52] crayon_1.3.4 lattice_0.20-34 haven_1.1.2
[55] splines_3.5.0 annotate_1.60.0 hms_0.3
[58] locfit_1.5-9.1 pillar_1.3.0 base64url_1.4
[61] softImpute_1.4 reshape2_1.4.2 codetools_0.2-15
[64] XML_3.98-1.5 glue_1.3.0 modelr_0.1.2
[67] foreach_1.4.3 cellranger_1.1.0 gtable_0.2.0
[70] assertthat_0.2.0 xtable_1.8-2 broom_0.4.2
[73] survival_2.41-3 pcaPP_1.9-73 gsl_1.9-10.3
[76] copula_0.999-18 iterators_1.0.8 AnnotationDbi_1.44.0
[79] memoise_1.0.0 brew_1.0-6