CPU info:
    CPU Model Name: Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
    Hardware threads: 12
    Total Memory: 57700428 kB
-------------------------------------------------------------------
=== Running mpiexec -n 2 /home/ubuntu/workspace/build/gpu/release/bin/cntk configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/ParallelCrossValidation/cntkcv.cntk currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/ParallelCrossValidation OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu DeviceId=0 timestamping=true numCPUThreads=6 shareNodeValueMatrices=true stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu/stderr
CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:05:31

/home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/ParallelCrossValidation/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/ParallelCrossValidation  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu  DeviceId=0  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu/stderr
Changed current directory to /home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data
CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:05:31

/home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/ParallelCrossValidation/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/ParallelCrossValidation  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu  DeviceId=0  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu/stderr
Changed current directory to /home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data
--------------------------------------------------------------------------
[[47553,1],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: 7fee1579d8b2

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
ping [requestnodes (before change)]: 2 nodes pinging each other
ping [requestnodes (before change)]: 2 nodes pinging each other
ping [requestnodes (after change)]: 2 nodes pinging each other
requestnodes [MPIWrapperMpi]: using 2 out of 2 MPI nodes on a single host (2 requested); we (1) are in (participating)
ping [mpihelper]: 2 nodes pinging each other
ping [requestnodes (after change)]: 2 nodes pinging each other
requestnodes [MPIWrapperMpi]: using 2 out of 2 MPI nodes on a single host (2 requested); we (0) are in (participating)
ping [mpihelper]: 2 nodes pinging each other
01/16/2018 19:05:31: Redirecting stderr to file /tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu/stderr_speechTrain.logrank0
01/16/2018 19:05:32: Redirecting stderr to file /tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu/stderr_speechTrain.logrank1
[7fee1579d8b2:16199] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics
[7fee1579d8b2:16199] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
MPI Rank 0: CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:05:31
MPI Rank 0: 
MPI Rank 0: /home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/ParallelCrossValidation/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/ParallelCrossValidation  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu  DeviceId=0  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu/stderr
MPI Rank 0: 01/16/2018 19:05:31: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:05:31: Build info: 
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:31: 		Built time: Jan 16 2018 16:15:42
MPI Rank 0: 01/16/2018 19:05:31: 		Last modified date: Tue Jan 16 16:13:51 2018
MPI Rank 0: 01/16/2018 19:05:31: 		Build type: release
MPI Rank 0: 01/16/2018 19:05:31: 		Build target: GPU
MPI Rank 0: 01/16/2018 19:05:31: 		With ASGD: yes
MPI Rank 0: 01/16/2018 19:05:31: 		Math lib: mkl
MPI Rank 0: 01/16/2018 19:05:31: 		CUDA version: 9.0.0
MPI Rank 0: 01/16/2018 19:05:31: 		CUDNN version: 7.0.4
MPI Rank 0: 01/16/2018 19:05:31: 		Build Branch: HEAD
MPI Rank 0: 01/16/2018 19:05:31: 		Build SHA1: c4c2ce8c6e89b5c32e4d07523081283417bcfc6d
MPI Rank 0: 01/16/2018 19:05:31: 		MPI distribution: Open MPI
MPI Rank 0: 01/16/2018 19:05:31: 		MPI version: 1.10.7
MPI Rank 0: 01/16/2018 19:05:31: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:05:31: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:05:31: GPU info:
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:31: 		Device[0]: cores = 3072; computeCapability = 5.2; type = "Tesla M60"; total memory = 8123 MB; free memory = 8112 MB
MPI Rank 0: 01/16/2018 19:05:31: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:05:31: Using 6 CPU threads.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:31: ##############################################################################
MPI Rank 0: 01/16/2018 19:05:31: #                                                                            #
MPI Rank 0: 01/16/2018 19:05:31: # speechTrain command (train action)                                         #
MPI Rank 0: 01/16/2018 19:05:31: #                                                                            #
MPI Rank 0: 01/16/2018 19:05:31: ##############################################################################
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:31: 
MPI Rank 0: Creating virgin network.
MPI Rank 0: SimpleNetworkBuilder Using GPU 0
MPI Rank 0: Reading script file glob_0000.scp ... 948 entries
MPI Rank 0: HTKDeserializer: selected '948' utterances grouped into '3' chunks, average chunk size: 316.0 utterances, 84244.7 frames (for I/O: 316.0 utterances, 84244.7 frames)
MPI Rank 0: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 0: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 0: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 0: Reading script file glob_0000.cv.scp ... 300 entries
MPI Rank 0: HTKDeserializer: selected '300' utterances grouped into '1' chunks, average chunk size: 300.0 utterances, 83050.0 frames (for I/O: 300.0 utterances, 83050.0 frames)
MPI Rank 0: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 0: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 0: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 0: 01/16/2018 19:05:32: 
MPI Rank 0: Model has 25 nodes. Using GPU 0.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:32: Training criterion:   CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
MPI Rank 0: 01/16/2018 19:05:32: Evaluation criterion: EvalClassificationError = ClassificationError
MPI Rank 0: 
MPI Rank 0: 
MPI Rank 0: Allocating matrices for forward and/or backward propagation.
MPI Rank 0: 
MPI Rank 0: Gradient Memory Aliasing: 4 are aliased.
MPI Rank 0: 	W2*H1 (gradient) reuses HLast (gradient)
MPI Rank 0: 	W1*H1 (gradient) reuses W1*H1+B1 (gradient)
MPI Rank 0: 
MPI Rank 0: Memory Sharing: Out of 40 matrices, 21 are shared as 5, and 19 are not shared.
MPI Rank 0: 
MPI Rank 0: Here are the ones that share memory:
MPI Rank 0: 	{ PosteriorProb : [132 x 1 x *]
MPI Rank 0: 	  ScaledLogLikelihood : [132 x 1 x *] }
MPI Rank 0: 	{ HLast : [132 x 1 x *] (gradient)
MPI Rank 0: 	  W0 : [512 x 363] (gradient)
MPI Rank 0: 	  W0*features+B0 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  W1*H1 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  W1*H1+B1 : [512 x 1 x *]
MPI Rank 0: 	  W1*H1+B1 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  W2*H1 : [132 x 1 x *]
MPI Rank 0: 	  W2*H1 : [132 x 1 x *] (gradient) }
MPI Rank 0: 	{ B0 : [512 x 1] (gradient)
MPI Rank 0: 	  H1 : [512 x 1 x *] }
MPI Rank 0: 	{ H1 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  H2 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  HLast : [132 x 1 x *]
MPI Rank 0: 	  W0*features : [512 x *]
MPI Rank 0: 	  W0*features : [512 x *] (gradient) }
MPI Rank 0: 	{ H2 : [512 x 1 x *]
MPI Rank 0: 	  W0*features+B0 : [512 x 1 x *]
MPI Rank 0: 	  W1 : [512 x 512] (gradient)
MPI Rank 0: 	  W1*H1 : [512 x 1 x *] }
MPI Rank 0: 
MPI Rank 0: Here are the ones that don't share memory:
MPI Rank 0: 	{features : [363 x *]}
MPI Rank 0: 	{MeanOfFeatures : [363]}
MPI Rank 0: 	{InvStdOfFeatures : [363]}
MPI Rank 0: 	{W0 : [512 x 363]}
MPI Rank 0: 	{B0 : [512 x 1]}
MPI Rank 0: 	{W1 : [512 x 512]}
MPI Rank 0: 	{B1 : [512 x 1]}
MPI Rank 0: 	{W2 : [132 x 512]}
MPI Rank 0: 	{B2 : [132 x 1]}
MPI Rank 0: 	{labels : [132 x *]}
MPI Rank 0: 	{Prior : [132]}
MPI Rank 0: 	{EvalClassificationError : [1]}
MPI Rank 0: 	{CrossEntropyWithSoftmax : [1]}
MPI Rank 0: 	{W2 : [132 x 512] (gradient)}
MPI Rank 0: 	{LogOfPrior : [132]}
MPI Rank 0: 	{B1 : [512 x 1] (gradient)}
MPI Rank 0: 	{MVNormalizedFeatures : [363 x *]}
MPI Rank 0: 	{B2 : [132 x 1] (gradient)}
MPI Rank 0: 	{CrossEntropyWithSoftmax : [1] (gradient)}
MPI Rank 0: 
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:32: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:32: 	Node 'B0' (LearnableParameter operation) : [512 x 1]
MPI Rank 0: 01/16/2018 19:05:32: 	Node 'B1' (LearnableParameter operation) : [512 x 1]
MPI Rank 0: 01/16/2018 19:05:32: 	Node 'B2' (LearnableParameter operation) : [132 x 1]
MPI Rank 0: 01/16/2018 19:05:32: 	Node 'W0' (LearnableParameter operation) : [512 x 363]
MPI Rank 0: 01/16/2018 19:05:32: 	Node 'W1' (LearnableParameter operation) : [512 x 512]
MPI Rank 0: 01/16/2018 19:05:32: 	Node 'W2' (LearnableParameter operation) : [132 x 512]
MPI Rank 0: 
MPI Rank 0: Initializing dataParallelSGD with FP64 aggregation.
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:32: Precomputing --> 3 PreCompute nodes found.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:32: 	MeanOfFeatures = Mean()
MPI Rank 0: 01/16/2018 19:05:32: 	InvStdOfFeatures = InvStdDev()
MPI Rank 0: 01/16/2018 19:05:32: 	Prior = Mean()
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:36: Precomputing --> Completed.
MPI Rank 0: 
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:36: Starting Epoch 1: learning rate per sample = 0.015625  effective momentum = 0.900000  momentum as time constant = 607.4 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:36: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[   1-  10, 3.12%]: CrossEntropyWithSoftmax = 4.62512789 * 640; EvalClassificationError = 0.94062500 * 640; time = 0.0693s; samplesPerSecond = 9239.4
MPI Rank 0: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  11-  20, 6.25%]: CrossEntropyWithSoftmax = 4.35619366 * 640; EvalClassificationError = 0.92343750 * 640; time = 0.0619s; samplesPerSecond = 10331.1
MPI Rank 0: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  21-  30, 9.38%]: CrossEntropyWithSoftmax = 3.97911998 * 640; EvalClassificationError = 0.89531250 * 640; time = 0.0616s; samplesPerSecond = 10393.7
MPI Rank 0: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  31-  40, 12.50%]: CrossEntropyWithSoftmax = 3.73643568 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.0621s; samplesPerSecond = 10299.7
MPI Rank 0: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  41-  50, 15.62%]: CrossEntropyWithSoftmax = 3.83079081 * 640; EvalClassificationError = 0.88281250 * 640; time = 0.0621s; samplesPerSecond = 10309.7
MPI Rank 0: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  51-  60, 18.75%]: CrossEntropyWithSoftmax = 3.71437690 * 640; EvalClassificationError = 0.86875000 * 640; time = 0.0620s; samplesPerSecond = 10321.0
MPI Rank 0: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  61-  70, 21.88%]: CrossEntropyWithSoftmax = 3.42186231 * 640; EvalClassificationError = 0.79062500 * 640; time = 0.0620s; samplesPerSecond = 10328.9
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[  71-  80, 25.00%]: CrossEntropyWithSoftmax = 3.53658053 * 640; EvalClassificationError = 0.82031250 * 640; time = 0.0619s; samplesPerSecond = 10340.2
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[  81-  90, 28.12%]: CrossEntropyWithSoftmax = 3.49758018 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.0619s; samplesPerSecond = 10343.3
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[  91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.39996308 * 640; EvalClassificationError = 0.80468750 * 640; time = 0.0615s; samplesPerSecond = 10402.8
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.49445773 * 640; EvalClassificationError = 0.82500000 * 640; time = 0.0615s; samplesPerSecond = 10401.2
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.26676999 * 640; EvalClassificationError = 0.79218750 * 640; time = 0.0621s; samplesPerSecond = 10309.5
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 121- 130, 40.62%]: CrossEntropyWithSoftmax = 3.18870174 * 640; EvalClassificationError = 0.78906250 * 640; time = 0.0611s; samplesPerSecond = 10469.9
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.05687264 * 640; EvalClassificationError = 0.74687500 * 640; time = 0.0616s; samplesPerSecond = 10386.0
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.95594570 * 640; EvalClassificationError = 0.71875000 * 640; time = 0.0614s; samplesPerSecond = 10420.4
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.10219605 * 640; EvalClassificationError = 0.74062500 * 640; time = 0.0614s; samplesPerSecond = 10417.4
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 161- 170, 53.12%]: CrossEntropyWithSoftmax = 2.80745016 * 640; EvalClassificationError = 0.70625000 * 640; time = 0.0621s; samplesPerSecond = 10307.4
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.72061843 * 640; EvalClassificationError = 0.65468750 * 640; time = 0.0646s; samplesPerSecond = 9904.1
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.80425748 * 640; EvalClassificationError = 0.71718750 * 640; time = 0.0720s; samplesPerSecond = 8893.5
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.71253069 * 640; EvalClassificationError = 0.67812500 * 640; time = 0.0612s; samplesPerSecond = 10457.6
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 201- 210, 65.62%]: CrossEntropyWithSoftmax = 2.59360400 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.0619s; samplesPerSecond = 10340.9
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.60386650 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0614s; samplesPerSecond = 10419.6
MPI Rank 0: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.53706679 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0675s; samplesPerSecond = 9475.1
MPI Rank 0: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.56177344 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0620s; samplesPerSecond = 10317.2
MPI Rank 0: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 241- 250, 78.12%]: CrossEntropyWithSoftmax = 2.50118792 * 640; EvalClassificationError = 0.64218750 * 640; time = 0.0615s; samplesPerSecond = 10411.6
MPI Rank 0: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.40119789 * 640; EvalClassificationError = 0.62500000 * 640; time = 0.0621s; samplesPerSecond = 10309.2
MPI Rank 0: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27491504 * 640; EvalClassificationError = 0.58906250 * 640; time = 0.0618s; samplesPerSecond = 10358.2
MPI Rank 0: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.51724208 * 640; EvalClassificationError = 0.65781250 * 640; time = 0.0624s; samplesPerSecond = 10248.3
MPI Rank 0: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 281- 290, 90.62%]: CrossEntropyWithSoftmax = 2.27797543 * 640; EvalClassificationError = 0.59687500 * 640; time = 0.0620s; samplesPerSecond = 10316.0
MPI Rank 0: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26017741 * 640; EvalClassificationError = 0.60937500 * 640; time = 0.0615s; samplesPerSecond = 10401.1
MPI Rank 0: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24735343 * 640; EvalClassificationError = 0.58437500 * 640; time = 0.0616s; samplesPerSecond = 10385.7
MPI Rank 0: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.23665382 * 640; EvalClassificationError = 0.60625000 * 640; time = 0.0622s; samplesPerSecond = 10282.6
MPI Rank 0: 01/16/2018 19:05:38: Finished Epoch[ 1 of 3]: [Training] CrossEntropyWithSoftmax = 3.03815142 * 20480; EvalClassificationError = 0.73432617 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=2.00922s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:05:40: Final Results: Minibatch[1-1299]: CrossEntropyWithSoftmax = 2.24821048 * 83050; perplexity = 9.47077252; EvalClassificationError = 0.61623119 * 83050
MPI Rank 0: 01/16/2018 19:05:40: Finished Epoch[ 1 of 3]: [Validate] CrossEntropyWithSoftmax = 2.24821048 * 83050; EvalClassificationError = 0.61623119 * 83050
MPI Rank 0: 01/16/2018 19:05:40: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu/models/cntkSpeech.dnn.1'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:40: Starting Epoch 2: learning rate per sample = 0.001953  effective momentum = 0.656119  momentum as time constant = 607.5 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:40: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:05:40:  Epoch[ 2 of 3]-Minibatch[   1-  10, 12.50%]: CrossEntropyWithSoftmax = 2.13894071 * 2560; EvalClassificationError = 0.56992188 * 2560; time = 0.1212s; samplesPerSecond = 21124.3
MPI Rank 0: 01/16/2018 19:05:40:  Epoch[ 2 of 3]-Minibatch[  11-  20, 25.00%]: CrossEntropyWithSoftmax = 2.06106261 * 2560; EvalClassificationError = 0.55664062 * 2560; time = 0.1167s; samplesPerSecond = 21941.9
MPI Rank 0: 01/16/2018 19:05:40:  Epoch[ 2 of 3]-Minibatch[  21-  30, 37.50%]: CrossEntropyWithSoftmax = 2.04459475 * 2560; EvalClassificationError = 0.55039063 * 2560; time = 0.1129s; samplesPerSecond = 22677.1
MPI Rank 0: 01/16/2018 19:05:41:  Epoch[ 2 of 3]-Minibatch[  31-  40, 50.00%]: CrossEntropyWithSoftmax = 2.03347291 * 2560; EvalClassificationError = 0.55742187 * 2560; time = 0.1157s; samplesPerSecond = 22123.4
MPI Rank 0: 01/16/2018 19:05:41:  Epoch[ 2 of 3]-Minibatch[  41-  50, 62.50%]: CrossEntropyWithSoftmax = 2.02079287 * 2560; EvalClassificationError = 0.54414063 * 2560; time = 0.1135s; samplesPerSecond = 22562.7
MPI Rank 0: 01/16/2018 19:05:41:  Epoch[ 2 of 3]-Minibatch[  51-  60, 75.00%]: CrossEntropyWithSoftmax = 1.96950012 * 2560; EvalClassificationError = 0.53085938 * 2560; time = 0.1150s; samplesPerSecond = 22251.7
MPI Rank 0: 01/16/2018 19:05:41:  Epoch[ 2 of 3]-Minibatch[  61-  70, 87.50%]: CrossEntropyWithSoftmax = 1.95934863 * 2560; EvalClassificationError = 0.52812500 * 2560; time = 0.1122s; samplesPerSecond = 22820.6
MPI Rank 0: 01/16/2018 19:05:41:  Epoch[ 2 of 3]-Minibatch[  71-  80, 100.00%]: CrossEntropyWithSoftmax = 1.94070839 * 2560; EvalClassificationError = 0.53125000 * 2560; time = 0.1142s; samplesPerSecond = 22425.5
MPI Rank 0: 01/16/2018 19:05:41: Finished Epoch[ 2 of 3]: [Training] CrossEntropyWithSoftmax = 2.02105263 * 20480; EvalClassificationError = 0.54609375 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=0.92554s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:05:42: Final Results: Minibatch[1-326]: CrossEntropyWithSoftmax = 1.92733488 * 83050; perplexity = 6.87117334; EvalClassificationError = 0.53122216 * 83050
MPI Rank 0: 01/16/2018 19:05:42: Finished Epoch[ 2 of 3]: [Validate] CrossEntropyWithSoftmax = 1.92733488 * 83050; EvalClassificationError = 0.53122216 * 83050
MPI Rank 0: 01/16/2018 19:05:42: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu/models/cntkSpeech.dnn.2'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:42: Starting Epoch 3: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:42: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:05:43:  Epoch[ 3 of 3]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.94336420 * 10240; EvalClassificationError = 0.53056641 * 10240; time = 0.3680s; samplesPerSecond = 27825.2
MPI Rank 0: 01/16/2018 19:05:43:  Epoch[ 3 of 3]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.96525554 * 10240; EvalClassificationError = 0.54873047 * 10240; time = 0.3568s; samplesPerSecond = 28697.9
MPI Rank 0: 01/16/2018 19:05:43: Finished Epoch[ 3 of 3]: [Training] CrossEntropyWithSoftmax = 1.95430987 * 20480; EvalClassificationError = 0.53964844 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-05; epochTime=0.728932s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:05:44: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.90639119 * 83050; perplexity = 6.72876211; EvalClassificationError = 0.52304636 * 83050
MPI Rank 0: 01/16/2018 19:05:44: Finished Epoch[ 3 of 3]: [Validate] CrossEntropyWithSoftmax = 1.90639119 * 83050; EvalClassificationError = 0.52304636 * 83050
MPI Rank 0: 01/16/2018 19:05:44: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu/models/cntkSpeech.dnn'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:44: Action "train" complete.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:05:44: __COMPLETED__
MPI Rank 1: CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:05:31
MPI Rank 1: 
MPI Rank 1: /home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/ParallelCrossValidation/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/ParallelCrossValidation  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu  DeviceId=0  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_ParallelCrossValidation@release_gpu/stderr
MPI Rank 1: 01/16/2018 19:05:32: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:05:32: Build info: 
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:32: 		Built time: Jan 16 2018 16:15:42
MPI Rank 1: 01/16/2018 19:05:32: 		Last modified date: Tue Jan 16 16:13:51 2018
MPI Rank 1: 01/16/2018 19:05:32: 		Build type: release
MPI Rank 1: 01/16/2018 19:05:32: 		Build target: GPU
MPI Rank 1: 01/16/2018 19:05:32: 		With ASGD: yes
MPI Rank 1: 01/16/2018 19:05:32: 		Math lib: mkl
MPI Rank 1: 01/16/2018 19:05:32: 		CUDA version: 9.0.0
MPI Rank 1: 01/16/2018 19:05:32: 		CUDNN version: 7.0.4
MPI Rank 1: 01/16/2018 19:05:32: 		Build Branch: HEAD
MPI Rank 1: 01/16/2018 19:05:32: 		Build SHA1: c4c2ce8c6e89b5c32e4d07523081283417bcfc6d
MPI Rank 1: 01/16/2018 19:05:32: 		MPI distribution: Open MPI
MPI Rank 1: 01/16/2018 19:05:32: 		MPI version: 1.10.7
MPI Rank 1: 01/16/2018 19:05:32: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:05:32: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:05:32: GPU info:
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:32: 		Device[0]: cores = 3072; computeCapability = 5.2; type = "Tesla M60"; total memory = 8123 MB; free memory = 8017 MB
MPI Rank 1: 01/16/2018 19:05:32: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:05:32: Using 6 CPU threads.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:32: ##############################################################################
MPI Rank 1: 01/16/2018 19:05:32: #                                                                            #
MPI Rank 1: 01/16/2018 19:05:32: # speechTrain command (train action)                                         #
MPI Rank 1: 01/16/2018 19:05:32: #                                                                            #
MPI Rank 1: 01/16/2018 19:05:32: ##############################################################################
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:32: 
MPI Rank 1: Creating virgin network.
MPI Rank 1: SimpleNetworkBuilder Using GPU 0
MPI Rank 1: Reading script file glob_0000.scp ... 948 entries
MPI Rank 1: HTKDeserializer: selected '948' utterances grouped into '3' chunks, average chunk size: 316.0 utterances, 84244.7 frames (for I/O: 316.0 utterances, 84244.7 frames)
MPI Rank 1: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 1: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 1: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 1: Reading script file glob_0000.cv.scp ... 300 entries
MPI Rank 1: HTKDeserializer: selected '300' utterances grouped into '1' chunks, average chunk size: 300.0 utterances, 83050.0 frames (for I/O: 300.0 utterances, 83050.0 frames)
MPI Rank 1: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 1: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 1: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 1: 01/16/2018 19:05:32: 
MPI Rank 1: Model has 25 nodes. Using GPU 0.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:32: Training criterion:   CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
MPI Rank 1: 01/16/2018 19:05:32: Evaluation criterion: EvalClassificationError = ClassificationError
MPI Rank 1: 
MPI Rank 1: 
MPI Rank 1: Allocating matrices for forward and/or backward propagation.
MPI Rank 1: 
MPI Rank 1: Gradient Memory Aliasing: 4 are aliased.
MPI Rank 1: 	W2*H1 (gradient) reuses HLast (gradient)
MPI Rank 1: 	W1*H1 (gradient) reuses W1*H1+B1 (gradient)
MPI Rank 1: 
MPI Rank 1: Memory Sharing: Out of 40 matrices, 21 are shared as 5, and 19 are not shared.
MPI Rank 1: 
MPI Rank 1: Here are the ones that share memory:
MPI Rank 1: 	{ PosteriorProb : [132 x 1 x *]
MPI Rank 1: 	  ScaledLogLikelihood : [132 x 1 x *] }
MPI Rank 1: 	{ HLast : [132 x 1 x *] (gradient)
MPI Rank 1: 	  W0 : [512 x 363] (gradient)
MPI Rank 1: 	  W0*features+B0 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  W1*H1 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  W1*H1+B1 : [512 x 1 x *]
MPI Rank 1: 	  W1*H1+B1 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  W2*H1 : [132 x 1 x *]
MPI Rank 1: 	  W2*H1 : [132 x 1 x *] (gradient) }
MPI Rank 1: 	{ B0 : [512 x 1] (gradient)
MPI Rank 1: 	  H1 : [512 x 1 x *] }
MPI Rank 1: 	{ H2 : [512 x 1 x *]
MPI Rank 1: 	  W0*features+B0 : [512 x 1 x *]
MPI Rank 1: 	  W1 : [512 x 512] (gradient)
MPI Rank 1: 	  W1*H1 : [512 x 1 x *] }
MPI Rank 1: 	{ H1 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  H2 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  HLast : [132 x 1 x *]
MPI Rank 1: 	  W0*features : [512 x *]
MPI Rank 1: 	  W0*features : [512 x *] (gradient) }
MPI Rank 1: 
MPI Rank 1: Here are the ones that don't share memory:
MPI Rank 1: 	{features : [363 x *]}
MPI Rank 1: 	{MeanOfFeatures : [363]}
MPI Rank 1: 	{InvStdOfFeatures : [363]}
MPI Rank 1: 	{W0 : [512 x 363]}
MPI Rank 1: 	{B0 : [512 x 1]}
MPI Rank 1: 	{W1 : [512 x 512]}
MPI Rank 1: 	{B1 : [512 x 1]}
MPI Rank 1: 	{W2 : [132 x 512]}
MPI Rank 1: 	{B2 : [132 x 1]}
MPI Rank 1: 	{labels : [132 x *]}
MPI Rank 1: 	{Prior : [132]}
MPI Rank 1: 	{EvalClassificationError : [1]}
MPI Rank 1: 	{CrossEntropyWithSoftmax : [1]}
MPI Rank 1: 	{W2 : [132 x 512] (gradient)}
MPI Rank 1: 	{MVNormalizedFeatures : [363 x *]}
MPI Rank 1: 	{LogOfPrior : [132]}
MPI Rank 1: 	{B1 : [512 x 1] (gradient)}
MPI Rank 1: 	{B2 : [132 x 1] (gradient)}
MPI Rank 1: 	{CrossEntropyWithSoftmax : [1] (gradient)}
MPI Rank 1: 
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:32: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:32: 	Node 'B0' (LearnableParameter operation) : [512 x 1]
MPI Rank 1: 01/16/2018 19:05:32: 	Node 'B1' (LearnableParameter operation) : [512 x 1]
MPI Rank 1: 01/16/2018 19:05:32: 	Node 'B2' (LearnableParameter operation) : [132 x 1]
MPI Rank 1: 01/16/2018 19:05:32: 	Node 'W0' (LearnableParameter operation) : [512 x 363]
MPI Rank 1: 01/16/2018 19:05:32: 	Node 'W1' (LearnableParameter operation) : [512 x 512]
MPI Rank 1: 01/16/2018 19:05:32: 	Node 'W2' (LearnableParameter operation) : [132 x 512]
MPI Rank 1: 
MPI Rank 1: Initializing dataParallelSGD with FP64 aggregation.
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:32: Precomputing --> 3 PreCompute nodes found.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:32: 	MeanOfFeatures = Mean()
MPI Rank 1: 01/16/2018 19:05:32: 	InvStdOfFeatures = InvStdDev()
MPI Rank 1: 01/16/2018 19:05:32: 	Prior = Mean()
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:36: Precomputing --> Completed.
MPI Rank 1: 
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:36: Starting Epoch 1: learning rate per sample = 0.015625  effective momentum = 0.900000  momentum as time constant = 607.4 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:36: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[   1-  10, 3.12%]: CrossEntropyWithSoftmax = 4.62512789 * 640; EvalClassificationError = 0.94062500 * 640; time = 0.0688s; samplesPerSecond = 9302.8
MPI Rank 1: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  11-  20, 6.25%]: CrossEntropyWithSoftmax = 4.35619366 * 640; EvalClassificationError = 0.92343750 * 640; time = 0.0619s; samplesPerSecond = 10333.9
MPI Rank 1: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  21-  30, 9.38%]: CrossEntropyWithSoftmax = 3.97911998 * 640; EvalClassificationError = 0.89531250 * 640; time = 0.0616s; samplesPerSecond = 10393.9
MPI Rank 1: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  31-  40, 12.50%]: CrossEntropyWithSoftmax = 3.73643568 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.0621s; samplesPerSecond = 10300.6
MPI Rank 1: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  41-  50, 15.62%]: CrossEntropyWithSoftmax = 3.83079081 * 640; EvalClassificationError = 0.88281250 * 640; time = 0.0621s; samplesPerSecond = 10311.2
MPI Rank 1: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  51-  60, 18.75%]: CrossEntropyWithSoftmax = 3.71437690 * 640; EvalClassificationError = 0.86875000 * 640; time = 0.0629s; samplesPerSecond = 10176.4
MPI Rank 1: 01/16/2018 19:05:36:  Epoch[ 1 of 3]-Minibatch[  61-  70, 21.88%]: CrossEntropyWithSoftmax = 3.42186231 * 640; EvalClassificationError = 0.79062500 * 640; time = 0.0611s; samplesPerSecond = 10476.5
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[  71-  80, 25.00%]: CrossEntropyWithSoftmax = 3.53658053 * 640; EvalClassificationError = 0.82031250 * 640; time = 0.0619s; samplesPerSecond = 10340.3
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[  81-  90, 28.12%]: CrossEntropyWithSoftmax = 3.49758018 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.0619s; samplesPerSecond = 10344.3
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[  91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.39996308 * 640; EvalClassificationError = 0.80468750 * 640; time = 0.0615s; samplesPerSecond = 10404.6
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.49445773 * 640; EvalClassificationError = 0.82500000 * 640; time = 0.0616s; samplesPerSecond = 10386.3
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.26676999 * 640; EvalClassificationError = 0.79218750 * 640; time = 0.0618s; samplesPerSecond = 10363.1
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 121- 130, 40.62%]: CrossEntropyWithSoftmax = 3.18870174 * 640; EvalClassificationError = 0.78906250 * 640; time = 0.0614s; samplesPerSecond = 10431.2
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.05687264 * 640; EvalClassificationError = 0.74687500 * 640; time = 0.0616s; samplesPerSecond = 10387.4
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.95594570 * 640; EvalClassificationError = 0.71875000 * 640; time = 0.0614s; samplesPerSecond = 10421.7
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.10219605 * 640; EvalClassificationError = 0.74062500 * 640; time = 0.0614s; samplesPerSecond = 10418.1
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 161- 170, 53.12%]: CrossEntropyWithSoftmax = 2.80745016 * 640; EvalClassificationError = 0.70625000 * 640; time = 0.0621s; samplesPerSecond = 10303.6
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.72061843 * 640; EvalClassificationError = 0.65468750 * 640; time = 0.0651s; samplesPerSecond = 9825.5
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.80425748 * 640; EvalClassificationError = 0.71718750 * 640; time = 0.0715s; samplesPerSecond = 8951.2
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.71253069 * 640; EvalClassificationError = 0.67812500 * 640; time = 0.0612s; samplesPerSecond = 10458.5
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 201- 210, 65.62%]: CrossEntropyWithSoftmax = 2.59360400 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.0619s; samplesPerSecond = 10340.3
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.60386650 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0614s; samplesPerSecond = 10422.7
MPI Rank 1: 01/16/2018 19:05:37:  Epoch[ 1 of 3]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.53706679 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0675s; samplesPerSecond = 9477.3
MPI Rank 1: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.56177344 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0620s; samplesPerSecond = 10318.4
MPI Rank 1: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 241- 250, 78.12%]: CrossEntropyWithSoftmax = 2.50118792 * 640; EvalClassificationError = 0.64218750 * 640; time = 0.0623s; samplesPerSecond = 10264.8
MPI Rank 1: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.40119789 * 640; EvalClassificationError = 0.62500000 * 640; time = 0.0612s; samplesPerSecond = 10456.6
MPI Rank 1: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27491504 * 640; EvalClassificationError = 0.58906250 * 640; time = 0.0618s; samplesPerSecond = 10360.8
MPI Rank 1: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.51724208 * 640; EvalClassificationError = 0.65781250 * 640; time = 0.0624s; samplesPerSecond = 10248.2
MPI Rank 1: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 281- 290, 90.62%]: CrossEntropyWithSoftmax = 2.27797543 * 640; EvalClassificationError = 0.59687500 * 640; time = 0.0620s; samplesPerSecond = 10317.2
MPI Rank 1: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26017741 * 640; EvalClassificationError = 0.60937500 * 640; time = 0.0615s; samplesPerSecond = 10401.8
MPI Rank 1: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24735343 * 640; EvalClassificationError = 0.58437500 * 640; time = 0.0616s; samplesPerSecond = 10385.5
MPI Rank 1: 01/16/2018 19:05:38:  Epoch[ 1 of 3]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.23665382 * 640; EvalClassificationError = 0.60625000 * 640; time = 0.0622s; samplesPerSecond = 10284.4
MPI Rank 1: 01/16/2018 19:05:38: Finished Epoch[ 1 of 3]: [Training] CrossEntropyWithSoftmax = 3.03815142 * 20480; EvalClassificationError = 0.73432617 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=2.00879s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:05:40: Final Results: Minibatch[1-1299]: CrossEntropyWithSoftmax = 2.24821048 * 83050; perplexity = 9.47077252; EvalClassificationError = 0.61623119 * 83050
MPI Rank 1: 01/16/2018 19:05:40: Finished Epoch[ 1 of 3]: [Validate] CrossEntropyWithSoftmax = 2.24821048 * 83050; EvalClassificationError = 0.61623119 * 83050
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:40: Starting Epoch 2: learning rate per sample = 0.001953  effective momentum = 0.656119  momentum as time constant = 607.5 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:40: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:05:40:  Epoch[ 2 of 3]-Minibatch[   1-  10, 12.50%]: CrossEntropyWithSoftmax = 2.13894071 * 2560; EvalClassificationError = 0.56992188 * 2560; time = 0.1207s; samplesPerSecond = 21211.5
MPI Rank 1: 01/16/2018 19:05:40:  Epoch[ 2 of 3]-Minibatch[  11-  20, 25.00%]: CrossEntropyWithSoftmax = 2.06106261 * 2560; EvalClassificationError = 0.55664062 * 2560; time = 0.1166s; samplesPerSecond = 21946.9
MPI Rank 1: 01/16/2018 19:05:40:  Epoch[ 2 of 3]-Minibatch[  21-  30, 37.50%]: CrossEntropyWithSoftmax = 2.04459475 * 2560; EvalClassificationError = 0.55039063 * 2560; time = 0.1129s; samplesPerSecond = 22680.6
MPI Rank 1: 01/16/2018 19:05:41:  Epoch[ 2 of 3]-Minibatch[  31-  40, 50.00%]: CrossEntropyWithSoftmax = 2.03347291 * 2560; EvalClassificationError = 0.55742187 * 2560; time = 0.1156s; samplesPerSecond = 22145.7
MPI Rank 1: 01/16/2018 19:05:41:  Epoch[ 2 of 3]-Minibatch[  41-  50, 62.50%]: CrossEntropyWithSoftmax = 2.02079287 * 2560; EvalClassificationError = 0.54414063 * 2560; time = 0.1144s; samplesPerSecond = 22374.0
MPI Rank 1: 01/16/2018 19:05:41:  Epoch[ 2 of 3]-Minibatch[  51-  60, 75.00%]: CrossEntropyWithSoftmax = 1.96950012 * 2560; EvalClassificationError = 0.53085938 * 2560; time = 0.1142s; samplesPerSecond = 22419.4
MPI Rank 1: 01/16/2018 19:05:41:  Epoch[ 2 of 3]-Minibatch[  61-  70, 87.50%]: CrossEntropyWithSoftmax = 1.95934863 * 2560; EvalClassificationError = 0.52812500 * 2560; time = 0.1130s; samplesPerSecond = 22646.2
MPI Rank 1: 01/16/2018 19:05:41:  Epoch[ 2 of 3]-Minibatch[  71-  80, 100.00%]: CrossEntropyWithSoftmax = 1.94070839 * 2560; EvalClassificationError = 0.53125000 * 2560; time = 0.1133s; samplesPerSecond = 22585.2
MPI Rank 1: 01/16/2018 19:05:41: Finished Epoch[ 2 of 3]: [Training] CrossEntropyWithSoftmax = 2.02105263 * 20480; EvalClassificationError = 0.54609375 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=0.925111s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:05:42: Final Results: Minibatch[1-326]: CrossEntropyWithSoftmax = 1.92733488 * 83050; perplexity = 6.87117334; EvalClassificationError = 0.53122216 * 83050
MPI Rank 1: 01/16/2018 19:05:42: Finished Epoch[ 2 of 3]: [Validate] CrossEntropyWithSoftmax = 1.92733488 * 83050; EvalClassificationError = 0.53122216 * 83050
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:42: Starting Epoch 3: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:42: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:05:43:  Epoch[ 3 of 3]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.94336420 * 10240; EvalClassificationError = 0.53056641 * 10240; time = 0.3678s; samplesPerSecond = 27842.8
MPI Rank 1: 01/16/2018 19:05:43:  Epoch[ 3 of 3]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.96525554 * 10240; EvalClassificationError = 0.54873047 * 10240; time = 0.3568s; samplesPerSecond = 28701.8
MPI Rank 1: 01/16/2018 19:05:43: Finished Epoch[ 3 of 3]: [Training] CrossEntropyWithSoftmax = 1.95430987 * 20480; EvalClassificationError = 0.53964844 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-05; epochTime=0.728478s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:05:44: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.90639119 * 83050; perplexity = 6.72876211; EvalClassificationError = 0.52304636 * 83050
MPI Rank 1: 01/16/2018 19:05:44: Finished Epoch[ 3 of 3]: [Validate] CrossEntropyWithSoftmax = 1.90639119 * 83050; EvalClassificationError = 0.52304636 * 83050
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:44: Action "train" complete.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:05:44: __COMPLETED__