CPU info:
    CPU Model Name: Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
    Hardware threads: 12
    Total Memory: 57700428 kB
-------------------------------------------------------------------
=== Running mpiexec -n 2 /home/ubuntu/workspace/build/gpu/release/bin/cntk configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data RunDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion OutputDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu DeviceId=-1 timestamping=true numCPUThreads=6 shareNodeValueMatrices=true saveBestModelPerCriterion=true stderr=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr
CNTK 2.3.1+ (HEAD 294890, Jan  8 2018 16:47:50) at 2018/01/09 01:28:12

/home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DeviceId=-1  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr
CNTK 2.3.1+ (HEAD 294890, Jan  8 2018 16:47:50) at 2018/01/09 01:28:12

/home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DeviceId=-1  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr
Changed current directory to /home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data
Changed current directory to /home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data
--------------------------------------------------------------------------
[[4453,1],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: 17c29a606870

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
ping [requestnodes (before change)]: 2 nodes pinging each other
ping [requestnodes (before change)]: 2 nodes pinging each other
ping [requestnodes (after change)]: 2 nodes pinging each other
ping [requestnodes (after change)]: 2 nodes pinging each other
requestnodes [MPIWrapperMpi]: using 2 out of 2 MPI nodes on a single host (2 requested); we (0) are in (participating)
ping [mpihelper]: 2 nodes pinging each other
requestnodes [MPIWrapperMpi]: using 2 out of 2 MPI nodes on a single host (2 requested); we (1) are in (participating)
ping [mpihelper]: 2 nodes pinging each other
01/09/2018 01:28:12: Redirecting stderr to file /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr_speechTrain.logrank0
01/09/2018 01:28:13: Redirecting stderr to file /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr_speechTrain.logrank1
[17c29a606870:03409] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics
[17c29a606870:03409] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
MPI Rank 0: CNTK 2.3.1+ (HEAD 294890, Jan  8 2018 16:47:50) at 2018/01/09 01:28:12
MPI Rank 0: 
MPI Rank 0: /home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DeviceId=-1  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr
MPI Rank 0: 01/09/2018 01:28:12: -------------------------------------------------------------------
MPI Rank 0: 01/09/2018 01:28:12: Build info: 
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:28:12: 		Built time: Jan  8 2018 16:42:01
MPI Rank 0: 01/09/2018 01:28:12: 		Last modified date: Mon Jan  8 16:40:18 2018
MPI Rank 0: 01/09/2018 01:28:12: 		Build type: release
MPI Rank 0: 01/09/2018 01:28:12: 		Build target: GPU
MPI Rank 0: 01/09/2018 01:28:12: 		With ASGD: yes
MPI Rank 0: 01/09/2018 01:28:12: 		Math lib: mkl
MPI Rank 0: 01/09/2018 01:28:12: 		CUDA version: 9.0.0
MPI Rank 0: 01/09/2018 01:28:12: 		CUDNN version: 7.0.4
MPI Rank 0: 01/09/2018 01:28:12: 		Build Branch: HEAD
MPI Rank 0: 01/09/2018 01:28:12: 		Build SHA1: 294890cb1f83fc31a56bd2cc1fc1fec34894b71c
MPI Rank 0: 01/09/2018 01:28:12: 		MPI distribution: Open MPI
MPI Rank 0: 01/09/2018 01:28:12: 		MPI version: 1.10.7
MPI Rank 0: 01/09/2018 01:28:12: -------------------------------------------------------------------
MPI Rank 0: 01/09/2018 01:28:12: -------------------------------------------------------------------
MPI Rank 0: 01/09/2018 01:28:12: GPU info:
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:28:12: 		Device[0]: cores = 3072; computeCapability = 5.2; type = "Tesla M60"; total memory = 8123 MB; free memory = 8112 MB
MPI Rank 0: 01/09/2018 01:28:12: -------------------------------------------------------------------
MPI Rank 0: 01/09/2018 01:28:12: Using 6 CPU threads.
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:28:12: ##############################################################################
MPI Rank 0: 01/09/2018 01:28:12: #                                                                            #
MPI Rank 0: 01/09/2018 01:28:12: # speechTrain command (train action)                                         #
MPI Rank 0: 01/09/2018 01:28:12: #                                                                            #
MPI Rank 0: 01/09/2018 01:28:12: ##############################################################################
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:28:12: 
MPI Rank 0: Creating virgin network.
MPI Rank 0: SimpleNetworkBuilder Using CPU
MPI Rank 0: Reading script file glob_0000.scp ... 948 entries
MPI Rank 0: HTKDeserializer: selected '948' utterances grouped into '3' chunks, average chunk size: 316.0 utterances, 84244.7 frames (for I/O: 316.0 utterances, 84244.7 frames)
MPI Rank 0: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 0: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 0: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 0: Reading script file glob_0000.cv.scp ... 300 entries
MPI Rank 0: HTKDeserializer: selected '300' utterances grouped into '1' chunks, average chunk size: 300.0 utterances, 83050.0 frames (for I/O: 300.0 utterances, 83050.0 frames)
MPI Rank 0: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 0: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 0: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 0: 01/09/2018 01:28:13: 
MPI Rank 0: Model has 25 nodes. Using CPU.
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:28:13: Training criterion:   CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
MPI Rank 0: 01/09/2018 01:28:13: Evaluation criterion: EvalClassificationError = ClassificationError
MPI Rank 0: 
MPI Rank 0: 
MPI Rank 0: Allocating matrices for forward and/or backward propagation.
MPI Rank 0: 
MPI Rank 0: Gradient Memory Aliasing: 4 are aliased.
MPI Rank 0: 	W1*H1 (gradient) reuses W1*H1+B1 (gradient)
MPI Rank 0: 	W2*H1 (gradient) reuses HLast (gradient)
MPI Rank 0: 
MPI Rank 0: Memory Sharing: Out of 40 matrices, 21 are shared as 5, and 19 are not shared.
MPI Rank 0: 
MPI Rank 0: Here are the ones that share memory:
MPI Rank 0: 	{ PosteriorProb : [132 x 1 x *]
MPI Rank 0: 	  ScaledLogLikelihood : [132 x 1 x *] }
MPI Rank 0: 	{ HLast : [132 x 1 x *] (gradient)
MPI Rank 0: 	  W0 : [512 x 363] (gradient)
MPI Rank 0: 	  W0*features+B0 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  W1*H1 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  W1*H1+B1 : [512 x 1 x *]
MPI Rank 0: 	  W1*H1+B1 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  W2*H1 : [132 x 1 x *]
MPI Rank 0: 	  W2*H1 : [132 x 1 x *] (gradient) }
MPI Rank 0: 	{ B0 : [512 x 1] (gradient)
MPI Rank 0: 	  H1 : [512 x 1 x *] }
MPI Rank 0: 	{ H2 : [512 x 1 x *]
MPI Rank 0: 	  W0*features+B0 : [512 x 1 x *]
MPI Rank 0: 	  W1 : [512 x 512] (gradient)
MPI Rank 0: 	  W1*H1 : [512 x 1 x *] }
MPI Rank 0: 	{ H1 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  H2 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  HLast : [132 x 1 x *]
MPI Rank 0: 	  W0*features : [512 x *]
MPI Rank 0: 	  W0*features : [512 x *] (gradient) }
MPI Rank 0: 
MPI Rank 0: Here are the ones that don't share memory:
MPI Rank 0: 	{InvStdOfFeatures : [363]}
MPI Rank 0: 	{features : [363 x *]}
MPI Rank 0: 	{MeanOfFeatures : [363]}
MPI Rank 0: 	{W0 : [512 x 363]}
MPI Rank 0: 	{B0 : [512 x 1]}
MPI Rank 0: 	{W1 : [512 x 512]}
MPI Rank 0: 	{B1 : [512 x 1]}
MPI Rank 0: 	{W2 : [132 x 512]}
MPI Rank 0: 	{B2 : [132 x 1]}
MPI Rank 0: 	{labels : [132 x *]}
MPI Rank 0: 	{Prior : [132]}
MPI Rank 0: 	{CrossEntropyWithSoftmax : [1]}
MPI Rank 0: 	{EvalClassificationError : [1]}
MPI Rank 0: 	{LogOfPrior : [132]}
MPI Rank 0: 	{CrossEntropyWithSoftmax : [1] (gradient)}
MPI Rank 0: 	{MVNormalizedFeatures : [363 x *]}
MPI Rank 0: 	{B2 : [132 x 1] (gradient)}
MPI Rank 0: 	{B1 : [512 x 1] (gradient)}
MPI Rank 0: 	{W2 : [132 x 512] (gradient)}
MPI Rank 0: 
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:28:13: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:28:13: 	Node 'B0' (LearnableParameter operation) : [512 x 1]
MPI Rank 0: 01/09/2018 01:28:13: 	Node 'B1' (LearnableParameter operation) : [512 x 1]
MPI Rank 0: 01/09/2018 01:28:13: 	Node 'B2' (LearnableParameter operation) : [132 x 1]
MPI Rank 0: 01/09/2018 01:28:13: 	Node 'W0' (LearnableParameter operation) : [512 x 363]
MPI Rank 0: 01/09/2018 01:28:13: 	Node 'W1' (LearnableParameter operation) : [512 x 512]
MPI Rank 0: 01/09/2018 01:28:13: 	Node 'W2' (LearnableParameter operation) : [132 x 512]
MPI Rank 0: 
MPI Rank 0: Initializing dataParallelSGD with FP64 aggregation.
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:28:13: Precomputing --> 3 PreCompute nodes found.
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:28:13: 	MeanOfFeatures = Mean()
MPI Rank 0: 01/09/2018 01:28:13: 	InvStdOfFeatures = InvStdDev()
MPI Rank 0: 01/09/2018 01:28:13: 	Prior = Mean()
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:31:40: Precomputing --> Completed.
MPI Rank 0: 
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:31:42: Starting Epoch 1: learning rate per sample = 0.015625  effective momentum = 0.900000  momentum as time constant = 607.4 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:31:42: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:31:45:  Epoch[ 1 of 15]-Minibatch[   1-  10, 3.12%]: CrossEntropyWithSoftmax = 4.59755198 * 640; EvalClassificationError = 0.93125000 * 640; time = 3.3476s; samplesPerSecond = 191.2
MPI Rank 0: 01/09/2018 01:31:48:  Epoch[ 1 of 15]-Minibatch[  11-  20, 6.25%]: CrossEntropyWithSoftmax = 4.34610349 * 640; EvalClassificationError = 0.92031250 * 640; time = 3.1197s; samplesPerSecond = 205.1
MPI Rank 0: 01/09/2018 01:31:51:  Epoch[ 1 of 15]-Minibatch[  21-  30, 9.38%]: CrossEntropyWithSoftmax = 3.98222516 * 640; EvalClassificationError = 0.89062500 * 640; time = 3.0754s; samplesPerSecond = 208.1
MPI Rank 0: 01/09/2018 01:31:54:  Epoch[ 1 of 15]-Minibatch[  31-  40, 12.50%]: CrossEntropyWithSoftmax = 3.74152814 * 640; EvalClassificationError = 0.84531250 * 640; time = 2.9184s; samplesPerSecond = 219.3
MPI Rank 0: 01/09/2018 01:31:57:  Epoch[ 1 of 15]-Minibatch[  41-  50, 15.62%]: CrossEntropyWithSoftmax = 3.83818572 * 640; EvalClassificationError = 0.86718750 * 640; time = 3.1535s; samplesPerSecond = 203.0
MPI Rank 0: 01/09/2018 01:32:00:  Epoch[ 1 of 15]-Minibatch[  51-  60, 18.75%]: CrossEntropyWithSoftmax = 3.71641238 * 640; EvalClassificationError = 0.87500000 * 640; time = 3.0019s; samplesPerSecond = 213.2
MPI Rank 0: 01/09/2018 01:32:03:  Epoch[ 1 of 15]-Minibatch[  61-  70, 21.88%]: CrossEntropyWithSoftmax = 3.41802791 * 640; EvalClassificationError = 0.79687500 * 640; time = 2.8449s; samplesPerSecond = 225.0
MPI Rank 0: 01/09/2018 01:32:06:  Epoch[ 1 of 15]-Minibatch[  71-  80, 25.00%]: CrossEntropyWithSoftmax = 3.53832947 * 640; EvalClassificationError = 0.82812500 * 640; time = 2.9544s; samplesPerSecond = 216.6
MPI Rank 0: 01/09/2018 01:32:09:  Epoch[ 1 of 15]-Minibatch[  81-  90, 28.12%]: CrossEntropyWithSoftmax = 3.50628076 * 640; EvalClassificationError = 0.81718750 * 640; time = 3.1286s; samplesPerSecond = 204.6
MPI Rank 0: 01/09/2018 01:32:12:  Epoch[ 1 of 15]-Minibatch[  91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478252 * 640; EvalClassificationError = 0.80781250 * 640; time = 2.5697s; samplesPerSecond = 249.1
MPI Rank 0: 01/09/2018 01:32:15:  Epoch[ 1 of 15]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031210 * 640; EvalClassificationError = 0.82812500 * 640; time = 3.1050s; samplesPerSecond = 206.1
MPI Rank 0: 01/09/2018 01:32:18:  Epoch[ 1 of 15]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365485 * 640; EvalClassificationError = 0.79375000 * 640; time = 3.0150s; samplesPerSecond = 212.3
MPI Rank 0: 01/09/2018 01:32:21:  Epoch[ 1 of 15]-Minibatch[ 121- 130, 40.62%]: CrossEntropyWithSoftmax = 3.20932117 * 640; EvalClassificationError = 0.79531250 * 640; time = 3.2076s; samplesPerSecond = 199.5
MPI Rank 0: 01/09/2018 01:32:24:  Epoch[ 1 of 15]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460535 * 640; EvalClassificationError = 0.75468750 * 640; time = 3.0931s; samplesPerSecond = 206.9
MPI Rank 0: 01/09/2018 01:32:27:  Epoch[ 1 of 15]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97529104 * 640; EvalClassificationError = 0.72031250 * 640; time = 3.0740s; samplesPerSecond = 208.2
MPI Rank 0: 01/09/2018 01:32:30:  Epoch[ 1 of 15]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968883 * 640; EvalClassificationError = 0.74531250 * 640; time = 2.8239s; samplesPerSecond = 226.6
MPI Rank 0: 01/09/2018 01:32:33:  Epoch[ 1 of 15]-Minibatch[ 161- 170, 53.12%]: CrossEntropyWithSoftmax = 2.84172140 * 640; EvalClassificationError = 0.71093750 * 640; time = 3.1513s; samplesPerSecond = 203.1
MPI Rank 0: 01/09/2018 01:32:36:  Epoch[ 1 of 15]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031745 * 640; EvalClassificationError = 0.66093750 * 640; time = 2.9643s; samplesPerSecond = 215.9
MPI Rank 0: 01/09/2018 01:32:39:  Epoch[ 1 of 15]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83858085 * 640; EvalClassificationError = 0.72656250 * 640; time = 2.9031s; samplesPerSecond = 220.5
MPI Rank 0: 01/09/2018 01:32:42:  Epoch[ 1 of 15]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632253 * 640; EvalClassificationError = 0.69218750 * 640; time = 2.8496s; samplesPerSecond = 224.6
MPI Rank 0: 01/09/2018 01:32:45:  Epoch[ 1 of 15]-Minibatch[ 201- 210, 65.62%]: CrossEntropyWithSoftmax = 2.61033254 * 640; EvalClassificationError = 0.66250000 * 640; time = 2.9715s; samplesPerSecond = 215.4
MPI Rank 0: 01/09/2018 01:32:48:  Epoch[ 1 of 15]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330754 * 640; EvalClassificationError = 0.65000000 * 640; time = 3.1490s; samplesPerSecond = 203.2
MPI Rank 0: 01/09/2018 01:32:51:  Epoch[ 1 of 15]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591810 * 640; EvalClassificationError = 0.66406250 * 640; time = 2.8127s; samplesPerSecond = 227.5
MPI Rank 0: 01/09/2018 01:32:54:  Epoch[ 1 of 15]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566512 * 640; EvalClassificationError = 0.66093750 * 640; time = 2.8598s; samplesPerSecond = 223.8
MPI Rank 0: 01/09/2018 01:32:57:  Epoch[ 1 of 15]-Minibatch[ 241- 250, 78.12%]: CrossEntropyWithSoftmax = 2.49164945 * 640; EvalClassificationError = 0.63281250 * 640; time = 3.4485s; samplesPerSecond = 185.6
MPI Rank 0: 01/09/2018 01:33:00:  Epoch[ 1 of 15]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954797 * 640; EvalClassificationError = 0.62812500 * 640; time = 3.1269s; samplesPerSecond = 204.7
MPI Rank 0: 01/09/2018 01:33:04:  Epoch[ 1 of 15]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27034227 * 640; EvalClassificationError = 0.59375000 * 640; time = 3.2293s; samplesPerSecond = 198.2
MPI Rank 0: 01/09/2018 01:33:07:  Epoch[ 1 of 15]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112387 * 640; EvalClassificationError = 0.66093750 * 640; time = 3.0124s; samplesPerSecond = 212.5
MPI Rank 0: 01/09/2018 01:33:10:  Epoch[ 1 of 15]-Minibatch[ 281- 290, 90.62%]: CrossEntropyWithSoftmax = 2.27800991 * 640; EvalClassificationError = 0.59062500 * 640; time = 3.1804s; samplesPerSecond = 201.2
MPI Rank 0: 01/09/2018 01:33:13:  Epoch[ 1 of 15]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783634 * 640; EvalClassificationError = 0.61093750 * 640; time = 3.0816s; samplesPerSecond = 207.7
MPI Rank 0: 01/09/2018 01:33:16:  Epoch[ 1 of 15]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590355 * 640; EvalClassificationError = 0.58593750 * 640; time = 3.1528s; samplesPerSecond = 203.0
MPI Rank 0: 01/09/2018 01:33:19:  Epoch[ 1 of 15]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415615 * 640; EvalClassificationError = 0.59843750 * 640; time = 2.6257s; samplesPerSecond = 243.7
MPI Rank 0: 01/09/2018 01:33:19: Finished Epoch[ 1 of 15]: [Training] CrossEntropyWithSoftmax = 3.04696987 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=97.1236s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:34:28: Final Results: Minibatch[1-1299]: CrossEntropyWithSoftmax = 2.24737799 * 83050; perplexity = 9.46289145; EvalClassificationError = 0.61431668 * 83050
MPI Rank 0: 01/09/2018 01:34:28: Finished Epoch[ 1 of 15]: [Validate] CrossEntropyWithSoftmax = 2.24737799 * 83050; EvalClassificationError = 0.61431668 * 83050
MPI Rank 0: 01/09/2018 01:34:28: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 2.247378 (Epoch 1); EvalClassificationError = 0.614317 (Epoch 1)
MPI Rank 0: 01/09/2018 01:34:28: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.1'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:34:28: Starting Epoch 2: learning rate per sample = 0.001953  effective momentum = 0.656119  momentum as time constant = 607.5 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:34:28: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:34:32:  Epoch[ 2 of 15]-Minibatch[   1-  10, 12.50%]: CrossEntropyWithSoftmax = 2.14624416 * 2560; EvalClassificationError = 0.56953125 * 2560; time = 3.9849s; samplesPerSecond = 642.4
MPI Rank 0: 01/09/2018 01:34:36:  Epoch[ 2 of 15]-Minibatch[  11-  20, 25.00%]: CrossEntropyWithSoftmax = 2.06174352 * 2560; EvalClassificationError = 0.55742187 * 2560; time = 3.7072s; samplesPerSecond = 690.5
MPI Rank 0: 01/09/2018 01:34:39:  Epoch[ 2 of 15]-Minibatch[  21-  30, 37.50%]: CrossEntropyWithSoftmax = 2.04994567 * 2560; EvalClassificationError = 0.55351562 * 2560; time = 3.6754s; samplesPerSecond = 696.5
MPI Rank 0: 01/09/2018 01:34:43:  Epoch[ 2 of 15]-Minibatch[  31-  40, 50.00%]: CrossEntropyWithSoftmax = 2.03695762 * 2560; EvalClassificationError = 0.56132812 * 2560; time = 3.6440s; samplesPerSecond = 702.5
MPI Rank 0: 01/09/2018 01:34:46:  Epoch[ 2 of 15]-Minibatch[  41-  50, 62.50%]: CrossEntropyWithSoftmax = 2.03086449 * 2560; EvalClassificationError = 0.55664062 * 2560; time = 3.4407s; samplesPerSecond = 744.0
MPI Rank 0: 01/09/2018 01:34:50:  Epoch[ 2 of 15]-Minibatch[  51-  60, 75.00%]: CrossEntropyWithSoftmax = 1.97306418 * 2560; EvalClassificationError = 0.53671875 * 2560; time = 3.4599s; samplesPerSecond = 739.9
MPI Rank 0: 01/09/2018 01:34:53:  Epoch[ 2 of 15]-Minibatch[  61-  70, 87.50%]: CrossEntropyWithSoftmax = 1.96746291 * 2560; EvalClassificationError = 0.53164062 * 2560; time = 3.4767s; samplesPerSecond = 736.3
MPI Rank 0: 01/09/2018 01:34:57:  Epoch[ 2 of 15]-Minibatch[  71-  80, 100.00%]: CrossEntropyWithSoftmax = 1.95498387 * 2560; EvalClassificationError = 0.53750000 * 2560; time = 3.3188s; samplesPerSecond = 771.4
MPI Rank 0: 01/09/2018 01:34:57: Finished Epoch[ 2 of 15]: [Training] CrossEntropyWithSoftmax = 2.02765830 * 20480; EvalClassificationError = 0.55053711 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=28.7151s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:35:33: Final Results: Minibatch[1-326]: CrossEntropyWithSoftmax = 1.93559232 * 83050; perplexity = 6.92814655; EvalClassificationError = 0.53506321 * 83050
MPI Rank 0: 01/09/2018 01:35:33: Finished Epoch[ 2 of 15]: [Validate] CrossEntropyWithSoftmax = 1.93559232 * 83050; EvalClassificationError = 0.53506321 * 83050
MPI Rank 0: 01/09/2018 01:35:33: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.935592 (Epoch 2); EvalClassificationError = 0.535063 (Epoch 2)
MPI Rank 0: 01/09/2018 01:35:33: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.2'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:35:34: Starting Epoch 3: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:35:34: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:35:38:  Epoch[ 3 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.95358670 * 10240; EvalClassificationError = 0.53603516 * 10240; time = 4.6012s; samplesPerSecond = 2225.5
MPI Rank 0: 01/09/2018 01:35:43:  Epoch[ 3 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.97541130 * 10240; EvalClassificationError = 0.55253906 * 10240; time = 4.6819s; samplesPerSecond = 2187.2
MPI Rank 0: 01/09/2018 01:35:43: Finished Epoch[ 3 of 15]: [Training] CrossEntropyWithSoftmax = 1.96449900 * 20480; EvalClassificationError = 0.54428711 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-05; epochTime=9.46434s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:36:10: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.91503561 * 83050; perplexity = 6.78718045; EvalClassificationError = 0.52745334 * 83050
MPI Rank 0: 01/09/2018 01:36:10: Finished Epoch[ 3 of 15]: [Validate] CrossEntropyWithSoftmax = 1.91503561 * 83050; EvalClassificationError = 0.52745334 * 83050
MPI Rank 0: 01/09/2018 01:36:10: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.915036 (Epoch 3); EvalClassificationError = 0.527453 (Epoch 3)
MPI Rank 0: 01/09/2018 01:36:10: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.3'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:36:11: Starting Epoch 4: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:36:11: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:36:15:  Epoch[ 4 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.94063438 * 10240; EvalClassificationError = 0.53203125 * 10240; time = 4.5553s; samplesPerSecond = 2247.9
MPI Rank 0: 01/09/2018 01:36:19:  Epoch[ 4 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.92737921 * 10240; EvalClassificationError = 0.53046875 * 10240; time = 4.1900s; samplesPerSecond = 2443.9
MPI Rank 0: 01/09/2018 01:36:19: Finished Epoch[ 4 of 15]: [Training] CrossEntropyWithSoftmax = 1.93400680 * 20480; EvalClassificationError = 0.53125000 * 20480; totalSamplesSeen = 81920; learningRatePerSample = 9.7656251e-05; epochTime=8.95284s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:36:47: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.90598730 * 83050; perplexity = 6.72604500; EvalClassificationError = 0.52635762 * 83050
MPI Rank 0: 01/09/2018 01:36:47: Finished Epoch[ 4 of 15]: [Validate] CrossEntropyWithSoftmax = 1.90598730 * 83050; EvalClassificationError = 0.52635762 * 83050
MPI Rank 0: 01/09/2018 01:36:47: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.905987 (Epoch 4); EvalClassificationError = 0.526358 (Epoch 4)
MPI Rank 0: 01/09/2018 01:36:47: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.4'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:36:47: Starting Epoch 5: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:36:47: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:36:52:  Epoch[ 5 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.93993557 * 10240; EvalClassificationError = 0.53144531 * 10240; time = 4.7608s; samplesPerSecond = 2150.9
MPI Rank 0: 01/09/2018 01:36:56:  Epoch[ 5 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.91667918 * 10240; EvalClassificationError = 0.52070313 * 10240; time = 4.5411s; samplesPerSecond = 2255.0
MPI Rank 0: 01/09/2018 01:36:57: Finished Epoch[ 5 of 15]: [Training] CrossEntropyWithSoftmax = 1.92830738 * 20480; EvalClassificationError = 0.52607422 * 20480; totalSamplesSeen = 102400; learningRatePerSample = 9.7656251e-05; epochTime=9.4601s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:37:24: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.89836333 * 83050; perplexity = 6.67496079; EvalClassificationError = 0.52422637 * 83050
MPI Rank 0: 01/09/2018 01:37:24: Finished Epoch[ 5 of 15]: [Validate] CrossEntropyWithSoftmax = 1.89836333 * 83050; EvalClassificationError = 0.52422637 * 83050
MPI Rank 0: 01/09/2018 01:37:24: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.898363 (Epoch 5); EvalClassificationError = 0.524226 (Epoch 5)
MPI Rank 0: 01/09/2018 01:37:24: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.5'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:37:24: Starting Epoch 6: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:37:24: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:37:29:  Epoch[ 6 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.93109689 * 10240; EvalClassificationError = 0.53535156 * 10240; time = 4.8403s; samplesPerSecond = 2115.6
MPI Rank 0: 01/09/2018 01:37:33:  Epoch[ 6 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.91132106 * 10240; EvalClassificationError = 0.52353516 * 10240; time = 4.2349s; samplesPerSecond = 2418.0
MPI Rank 0: 01/09/2018 01:37:34: Finished Epoch[ 6 of 15]: [Training] CrossEntropyWithSoftmax = 1.92120897 * 20480; EvalClassificationError = 0.52944336 * 20480; totalSamplesSeen = 122880; learningRatePerSample = 9.7656251e-05; epochTime=9.27021s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:38:00: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.89149689 * 83050; perplexity = 6.62928457; EvalClassificationError = 0.52290187 * 83050
MPI Rank 0: 01/09/2018 01:38:00: Finished Epoch[ 6 of 15]: [Validate] CrossEntropyWithSoftmax = 1.89149689 * 83050; EvalClassificationError = 0.52290187 * 83050
MPI Rank 0: 01/09/2018 01:38:00: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.891497 (Epoch 6); EvalClassificationError = 0.522902 (Epoch 6)
MPI Rank 0: 01/09/2018 01:38:00: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.6'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:38:00: Starting Epoch 7: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:38:00: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:38:05:  Epoch[ 7 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.88512539 * 10240; EvalClassificationError = 0.51572266 * 10240; time = 4.6046s; samplesPerSecond = 2223.8
MPI Rank 0: 01/09/2018 01:38:09:  Epoch[ 7 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.91798861 * 10240; EvalClassificationError = 0.53535156 * 10240; time = 4.4755s; samplesPerSecond = 2288.0
MPI Rank 0: 01/09/2018 01:38:10: Finished Epoch[ 7 of 15]: [Training] CrossEntropyWithSoftmax = 1.90155700 * 20480; EvalClassificationError = 0.52553711 * 20480; totalSamplesSeen = 143360; learningRatePerSample = 9.7656251e-05; epochTime=9.16562s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:38:37: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.88459874 * 83050; perplexity = 6.58371212; EvalClassificationError = 0.52246839 * 83050
MPI Rank 0: 01/09/2018 01:38:37: Finished Epoch[ 7 of 15]: [Validate] CrossEntropyWithSoftmax = 1.88459874 * 83050; EvalClassificationError = 0.52246839 * 83050
MPI Rank 0: 01/09/2018 01:38:37: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.884599 (Epoch 7); EvalClassificationError = 0.522468 (Epoch 7)
MPI Rank 0: 01/09/2018 01:38:37: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.7'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:38:37: Starting Epoch 8: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:38:37: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:38:42:  Epoch[ 8 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.89139877 * 10240; EvalClassificationError = 0.52099609 * 10240; time = 4.7342s; samplesPerSecond = 2163.0
MPI Rank 0: 01/09/2018 01:38:46:  Epoch[ 8 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.87492662 * 10240; EvalClassificationError = 0.51923828 * 10240; time = 4.4726s; samplesPerSecond = 2289.5
MPI Rank 0: 01/09/2018 01:38:46: Finished Epoch[ 8 of 15]: [Training] CrossEntropyWithSoftmax = 1.88316269 * 20480; EvalClassificationError = 0.52011719 * 20480; totalSamplesSeen = 163840; learningRatePerSample = 9.7656251e-05; epochTime=9.27057s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:39:13: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.87932390 * 83050; perplexity = 6.54907556; EvalClassificationError = 0.52132450 * 83050
MPI Rank 0: 01/09/2018 01:39:13: Finished Epoch[ 8 of 15]: [Validate] CrossEntropyWithSoftmax = 1.87932390 * 83050; EvalClassificationError = 0.52132450 * 83050
MPI Rank 0: 01/09/2018 01:39:13: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.879324 (Epoch 8); EvalClassificationError = 0.521325 (Epoch 8)
MPI Rank 0: 01/09/2018 01:39:13: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.8'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:39:14: Starting Epoch 9: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:39:14: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:39:18:  Epoch[ 9 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.86868159 * 10240; EvalClassificationError = 0.51103516 * 10240; time = 4.3503s; samplesPerSecond = 2353.9
MPI Rank 0: 01/09/2018 01:39:22:  Epoch[ 9 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.86708001 * 10240; EvalClassificationError = 0.52001953 * 10240; time = 4.2125s; samplesPerSecond = 2430.9
MPI Rank 0: 01/09/2018 01:39:22: Finished Epoch[ 9 of 15]: [Training] CrossEntropyWithSoftmax = 1.86788080 * 20480; EvalClassificationError = 0.51552734 * 20480; totalSamplesSeen = 184320; learningRatePerSample = 9.7656251e-05; epochTime=8.76662s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:39:50: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.87278473 * 83050; perplexity = 6.50638972; EvalClassificationError = 0.52086695 * 83050
MPI Rank 0: 01/09/2018 01:39:50: Finished Epoch[ 9 of 15]: [Validate] CrossEntropyWithSoftmax = 1.87278473 * 83050; EvalClassificationError = 0.52086695 * 83050
MPI Rank 0: 01/09/2018 01:39:50: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.872785 (Epoch 9); EvalClassificationError = 0.520867 (Epoch 9)
MPI Rank 0: 01/09/2018 01:39:50: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.9'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:39:50: Starting Epoch 10: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:39:50: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:39:55:  Epoch[10 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.90473850 * 10240; EvalClassificationError = 0.52822266 * 10240; time = 4.7311s; samplesPerSecond = 2164.4
MPI Rank 0: 01/09/2018 01:39:59:  Epoch[10 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.85821242 * 10240; EvalClassificationError = 0.51484375 * 10240; time = 4.2475s; samplesPerSecond = 2410.8
MPI Rank 0: 01/09/2018 01:39:59: Finished Epoch[10 of 15]: [Training] CrossEntropyWithSoftmax = 1.88147546 * 20480; EvalClassificationError = 0.52153320 * 20480; totalSamplesSeen = 204800; learningRatePerSample = 9.7656251e-05; epochTime=9.03273s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:40:27: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.86655806 * 83050; perplexity = 6.46600244; EvalClassificationError = 0.51947020 * 83050
MPI Rank 0: 01/09/2018 01:40:27: Finished Epoch[10 of 15]: [Validate] CrossEntropyWithSoftmax = 1.86655806 * 83050; EvalClassificationError = 0.51947020 * 83050
MPI Rank 0: 01/09/2018 01:40:27: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.866558 (Epoch 10); EvalClassificationError = 0.519470 (Epoch 10)
MPI Rank 0: 01/09/2018 01:40:27: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.10'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:40:27: Starting Epoch 11: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:40:27: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:40:31:  Epoch[11 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.87411367 * 10240; EvalClassificationError = 0.51142578 * 10240; time = 4.4015s; samplesPerSecond = 2326.5
MPI Rank 0: 01/09/2018 01:40:36:  Epoch[11 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.87067306 * 10240; EvalClassificationError = 0.52158203 * 10240; time = 4.6717s; samplesPerSecond = 2191.9
MPI Rank 0: 01/09/2018 01:40:36: Finished Epoch[11 of 15]: [Training] CrossEntropyWithSoftmax = 1.87239337 * 20480; EvalClassificationError = 0.51650391 * 20480; totalSamplesSeen = 225280; learningRatePerSample = 9.7656251e-05; epochTime=9.16707s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:41:04: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.85980825 * 83050; perplexity = 6.42250511; EvalClassificationError = 0.51703793 * 83050
MPI Rank 0: 01/09/2018 01:41:04: Finished Epoch[11 of 15]: [Validate] CrossEntropyWithSoftmax = 1.85980825 * 83050; EvalClassificationError = 0.51703793 * 83050
MPI Rank 0: 01/09/2018 01:41:04: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.859808 (Epoch 11); EvalClassificationError = 0.517038 (Epoch 11)
MPI Rank 0: 01/09/2018 01:41:04: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.11'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:41:04: Starting Epoch 12: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:41:04: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:41:09:  Epoch[12 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.87570276 * 10240; EvalClassificationError = 0.52138672 * 10240; time = 4.8218s; samplesPerSecond = 2123.7
MPI Rank 0: 01/09/2018 01:41:13:  Epoch[12 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.84544781 * 10240; EvalClassificationError = 0.50859375 * 10240; time = 4.3397s; samplesPerSecond = 2359.6
MPI Rank 0: 01/09/2018 01:41:13: Finished Epoch[12 of 15]: [Training] CrossEntropyWithSoftmax = 1.86057528 * 20480; EvalClassificationError = 0.51499023 * 20480; totalSamplesSeen = 245760; learningRatePerSample = 9.7656251e-05; epochTime=9.16651s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:41:40: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.85346328 * 83050; perplexity = 6.38188356; EvalClassificationError = 0.51517158 * 83050
MPI Rank 0: 01/09/2018 01:41:40: Finished Epoch[12 of 15]: [Validate] CrossEntropyWithSoftmax = 1.85346328 * 83050; EvalClassificationError = 0.51517158 * 83050
MPI Rank 0: 01/09/2018 01:41:40: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.853463 (Epoch 12); EvalClassificationError = 0.515172 (Epoch 12)
MPI Rank 0: 01/09/2018 01:41:40: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.12'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:41:40: Starting Epoch 13: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:41:40: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:41:44:  Epoch[13 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.85293961 * 10046; EvalClassificationError = 0.52020705 * 10046; time = 4.5971s; samplesPerSecond = 2185.3
MPI Rank 0: 01/09/2018 01:41:49:  Epoch[13 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.88311779 * 10240; EvalClassificationError = 0.52031250 * 10240; time = 4.3981s; samplesPerSecond = 2328.3
MPI Rank 0: 01/09/2018 01:41:49: Finished Epoch[13 of 15]: [Training] CrossEntropyWithSoftmax = 1.86894139 * 20480; EvalClassificationError = 0.51992187 * 20480; totalSamplesSeen = 266240; learningRatePerSample = 9.7656251e-05; epochTime=9.22015s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:42:17: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.84713665 * 83050; perplexity = 6.34163517; EvalClassificationError = 0.51273931 * 83050
MPI Rank 0: 01/09/2018 01:42:17: Finished Epoch[13 of 15]: [Validate] CrossEntropyWithSoftmax = 1.84713665 * 83050; EvalClassificationError = 0.51273931 * 83050
MPI Rank 0: 01/09/2018 01:42:17: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.847137 (Epoch 13); EvalClassificationError = 0.512739 (Epoch 13)
MPI Rank 0: 01/09/2018 01:42:17: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.13'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:42:17: Starting Epoch 14: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:42:17: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:42:21:  Epoch[14 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.86634456 * 10240; EvalClassificationError = 0.50937500 * 10240; time = 4.6041s; samplesPerSecond = 2224.1
MPI Rank 0: 01/09/2018 01:42:26:  Epoch[14 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.85430186 * 10240; EvalClassificationError = 0.51328125 * 10240; time = 4.3315s; samplesPerSecond = 2364.1
MPI Rank 0: 01/09/2018 01:42:26: Finished Epoch[14 of 15]: [Training] CrossEntropyWithSoftmax = 1.86032321 * 20480; EvalClassificationError = 0.51132813 * 20480; totalSamplesSeen = 286720; learningRatePerSample = 9.7656251e-05; epochTime=9.02081s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:42:53: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.84154028 * 83050; perplexity = 6.30624417; EvalClassificationError = 0.51331728 * 83050
MPI Rank 0: 01/09/2018 01:42:53: Finished Epoch[14 of 15]: [Validate] CrossEntropyWithSoftmax = 1.84154028 * 83050; EvalClassificationError = 0.51331728 * 83050
MPI Rank 0: 01/09/2018 01:42:53: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.841540 (Epoch 14); EvalClassificationError = 0.512739 (Epoch 13)
MPI Rank 0: 01/09/2018 01:42:53: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.14'
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:42:53: Starting Epoch 15: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:42:53: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:42:58:  Epoch[15 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.82826227 * 10240; EvalClassificationError = 0.50820312 * 10240; time = 4.5743s; samplesPerSecond = 2238.6
MPI Rank 0: 01/09/2018 01:43:02:  Epoch[15 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.85323706 * 10240; EvalClassificationError = 0.51826172 * 10240; time = 4.3325s; samplesPerSecond = 2363.5
MPI Rank 0: 01/09/2018 01:43:03: Finished Epoch[15 of 15]: [Training] CrossEntropyWithSoftmax = 1.84074966 * 20480; EvalClassificationError = 0.51323242 * 20480; totalSamplesSeen = 307200; learningRatePerSample = 9.7656251e-05; epochTime=9.10099s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:43:30: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.83558729 * 83050; perplexity = 6.26881470; EvalClassificationError = 0.51134256 * 83050
MPI Rank 0: 01/09/2018 01:43:30: Finished Epoch[15 of 15]: [Validate] CrossEntropyWithSoftmax = 1.83558729 * 83050; EvalClassificationError = 0.51134256 * 83050
MPI Rank 0: 01/09/2018 01:43:30: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.835587 (Epoch 15); EvalClassificationError = 0.511343 (Epoch 15)
MPI Rank 0: 01/09/2018 01:43:31: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn'
MPI Rank 0: 01/09/2018 01:43:31: Best epoch for criterion 'CrossEntropyWithSoftmax' is 15 and model /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn copied to /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn_CrossEntropyWithSoftmax
MPI Rank 0: 01/09/2018 01:43:31: Best epoch for criterion 'EvalClassificationError' is 15 and model /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn copied to /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn_EvalClassificationError
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:43:31: Action "train" complete.
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:43:31: __COMPLETED__
MPI Rank 1: CNTK 2.3.1+ (HEAD 294890, Jan  8 2018 16:47:50) at 2018/01/09 01:28:12
MPI Rank 1: 
MPI Rank 1: /home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DeviceId=-1  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr
MPI Rank 1: 01/09/2018 01:28:13: -------------------------------------------------------------------
MPI Rank 1: 01/09/2018 01:28:13: Build info: 
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:28:13: 		Built time: Jan  8 2018 16:42:01
MPI Rank 1: 01/09/2018 01:28:13: 		Last modified date: Mon Jan  8 16:40:18 2018
MPI Rank 1: 01/09/2018 01:28:13: 		Build type: release
MPI Rank 1: 01/09/2018 01:28:13: 		Build target: GPU
MPI Rank 1: 01/09/2018 01:28:13: 		With ASGD: yes
MPI Rank 1: 01/09/2018 01:28:13: 		Math lib: mkl
MPI Rank 1: 01/09/2018 01:28:13: 		CUDA version: 9.0.0
MPI Rank 1: 01/09/2018 01:28:13: 		CUDNN version: 7.0.4
MPI Rank 1: 01/09/2018 01:28:13: 		Build Branch: HEAD
MPI Rank 1: 01/09/2018 01:28:13: 		Build SHA1: 294890cb1f83fc31a56bd2cc1fc1fec34894b71c
MPI Rank 1: 01/09/2018 01:28:13: 		MPI distribution: Open MPI
MPI Rank 1: 01/09/2018 01:28:13: 		MPI version: 1.10.7
MPI Rank 1: 01/09/2018 01:28:13: -------------------------------------------------------------------
MPI Rank 1: 01/09/2018 01:28:13: -------------------------------------------------------------------
MPI Rank 1: 01/09/2018 01:28:13: GPU info:
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:28:13: 		Device[0]: cores = 3072; computeCapability = 5.2; type = "Tesla M60"; total memory = 8123 MB; free memory = 8029 MB
MPI Rank 1: 01/09/2018 01:28:13: -------------------------------------------------------------------
MPI Rank 1: 01/09/2018 01:28:13: Using 6 CPU threads.
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:28:13: ##############################################################################
MPI Rank 1: 01/09/2018 01:28:13: #                                                                            #
MPI Rank 1: 01/09/2018 01:28:13: # speechTrain command (train action)                                         #
MPI Rank 1: 01/09/2018 01:28:13: #                                                                            #
MPI Rank 1: 01/09/2018 01:28:13: ##############################################################################
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:28:13: 
MPI Rank 1: Creating virgin network.
MPI Rank 1: SimpleNetworkBuilder Using CPU
MPI Rank 1: Reading script file glob_0000.scp ... 948 entries
MPI Rank 1: HTKDeserializer: selected '948' utterances grouped into '3' chunks, average chunk size: 316.0 utterances, 84244.7 frames (for I/O: 316.0 utterances, 84244.7 frames)
MPI Rank 1: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 1: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 1: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 1: Reading script file glob_0000.cv.scp ... 300 entries
MPI Rank 1: HTKDeserializer: selected '300' utterances grouped into '1' chunks, average chunk size: 300.0 utterances, 83050.0 frames (for I/O: 300.0 utterances, 83050.0 frames)
MPI Rank 1: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 1: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 1: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 1: 01/09/2018 01:28:13: 
MPI Rank 1: Model has 25 nodes. Using CPU.
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:28:13: Training criterion:   CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
MPI Rank 1: 01/09/2018 01:28:13: Evaluation criterion: EvalClassificationError = ClassificationError
MPI Rank 1: 
MPI Rank 1: 
MPI Rank 1: Allocating matrices for forward and/or backward propagation.
MPI Rank 1: 
MPI Rank 1: Gradient Memory Aliasing: 4 are aliased.
MPI Rank 1: 	W1*H1 (gradient) reuses W1*H1+B1 (gradient)
MPI Rank 1: 	W2*H1 (gradient) reuses HLast (gradient)
MPI Rank 1: 
MPI Rank 1: Memory Sharing: Out of 40 matrices, 21 are shared as 5, and 19 are not shared.
MPI Rank 1: 
MPI Rank 1: Here are the ones that share memory:
MPI Rank 1: 	{ PosteriorProb : [132 x 1 x *]
MPI Rank 1: 	  ScaledLogLikelihood : [132 x 1 x *] }
MPI Rank 1: 	{ HLast : [132 x 1 x *] (gradient)
MPI Rank 1: 	  W0 : [512 x 363] (gradient)
MPI Rank 1: 	  W0*features+B0 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  W1*H1 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  W1*H1+B1 : [512 x 1 x *]
MPI Rank 1: 	  W1*H1+B1 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  W2*H1 : [132 x 1 x *]
MPI Rank 1: 	  W2*H1 : [132 x 1 x *] (gradient) }
MPI Rank 1: 	{ B0 : [512 x 1] (gradient)
MPI Rank 1: 	  H1 : [512 x 1 x *] }
MPI Rank 1: 	{ H2 : [512 x 1 x *]
MPI Rank 1: 	  W0*features+B0 : [512 x 1 x *]
MPI Rank 1: 	  W1 : [512 x 512] (gradient)
MPI Rank 1: 	  W1*H1 : [512 x 1 x *] }
MPI Rank 1: 	{ H1 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  H2 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  HLast : [132 x 1 x *]
MPI Rank 1: 	  W0*features : [512 x *]
MPI Rank 1: 	  W0*features : [512 x *] (gradient) }
MPI Rank 1: 
MPI Rank 1: Here are the ones that don't share memory:
MPI Rank 1: 	{InvStdOfFeatures : [363]}
MPI Rank 1: 	{features : [363 x *]}
MPI Rank 1: 	{MeanOfFeatures : [363]}
MPI Rank 1: 	{W0 : [512 x 363]}
MPI Rank 1: 	{B0 : [512 x 1]}
MPI Rank 1: 	{W1 : [512 x 512]}
MPI Rank 1: 	{B1 : [512 x 1]}
MPI Rank 1: 	{W2 : [132 x 512]}
MPI Rank 1: 	{B2 : [132 x 1]}
MPI Rank 1: 	{labels : [132 x *]}
MPI Rank 1: 	{Prior : [132]}
MPI Rank 1: 	{CrossEntropyWithSoftmax : [1]}
MPI Rank 1: 	{EvalClassificationError : [1]}
MPI Rank 1: 	{LogOfPrior : [132]}
MPI Rank 1: 	{CrossEntropyWithSoftmax : [1] (gradient)}
MPI Rank 1: 	{MVNormalizedFeatures : [363 x *]}
MPI Rank 1: 	{B2 : [132 x 1] (gradient)}
MPI Rank 1: 	{B1 : [512 x 1] (gradient)}
MPI Rank 1: 	{W2 : [132 x 512] (gradient)}
MPI Rank 1: 
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:28:13: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:28:13: 	Node 'B0' (LearnableParameter operation) : [512 x 1]
MPI Rank 1: 01/09/2018 01:28:13: 	Node 'B1' (LearnableParameter operation) : [512 x 1]
MPI Rank 1: 01/09/2018 01:28:13: 	Node 'B2' (LearnableParameter operation) : [132 x 1]
MPI Rank 1: 01/09/2018 01:28:13: 	Node 'W0' (LearnableParameter operation) : [512 x 363]
MPI Rank 1: 01/09/2018 01:28:13: 	Node 'W1' (LearnableParameter operation) : [512 x 512]
MPI Rank 1: 01/09/2018 01:28:13: 	Node 'W2' (LearnableParameter operation) : [132 x 512]
MPI Rank 1: 
MPI Rank 1: Initializing dataParallelSGD with FP64 aggregation.
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:28:13: Precomputing --> 3 PreCompute nodes found.
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:28:13: 	MeanOfFeatures = Mean()
MPI Rank 1: 01/09/2018 01:28:13: 	InvStdOfFeatures = InvStdDev()
MPI Rank 1: 01/09/2018 01:28:13: 	Prior = Mean()
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:31:42: Precomputing --> Completed.
MPI Rank 1: 
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:31:42: Starting Epoch 1: learning rate per sample = 0.015625  effective momentum = 0.900000  momentum as time constant = 607.4 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:31:42: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:31:45:  Epoch[ 1 of 15]-Minibatch[   1-  10, 3.12%]: CrossEntropyWithSoftmax = 4.59755198 * 640; EvalClassificationError = 0.93125000 * 640; time = 3.3133s; samplesPerSecond = 193.2
MPI Rank 1: 01/09/2018 01:31:48:  Epoch[ 1 of 15]-Minibatch[  11-  20, 6.25%]: CrossEntropyWithSoftmax = 4.34610349 * 640; EvalClassificationError = 0.92031250 * 640; time = 2.9384s; samplesPerSecond = 217.8
MPI Rank 1: 01/09/2018 01:31:51:  Epoch[ 1 of 15]-Minibatch[  21-  30, 9.38%]: CrossEntropyWithSoftmax = 3.98222516 * 640; EvalClassificationError = 0.89062500 * 640; time = 3.2864s; samplesPerSecond = 194.7
MPI Rank 1: 01/09/2018 01:31:54:  Epoch[ 1 of 15]-Minibatch[  31-  40, 12.50%]: CrossEntropyWithSoftmax = 3.74152814 * 640; EvalClassificationError = 0.84531250 * 640; time = 2.9056s; samplesPerSecond = 220.3
MPI Rank 1: 01/09/2018 01:31:57:  Epoch[ 1 of 15]-Minibatch[  41-  50, 15.62%]: CrossEntropyWithSoftmax = 3.83818572 * 640; EvalClassificationError = 0.86718750 * 640; time = 3.1430s; samplesPerSecond = 203.6
MPI Rank 1: 01/09/2018 01:32:00:  Epoch[ 1 of 15]-Minibatch[  51-  60, 18.75%]: CrossEntropyWithSoftmax = 3.71641238 * 640; EvalClassificationError = 0.87500000 * 640; time = 2.9966s; samplesPerSecond = 213.6
MPI Rank 1: 01/09/2018 01:32:03:  Epoch[ 1 of 15]-Minibatch[  61-  70, 21.88%]: CrossEntropyWithSoftmax = 3.41802791 * 640; EvalClassificationError = 0.79687500 * 640; time = 2.8643s; samplesPerSecond = 223.4
MPI Rank 1: 01/09/2018 01:32:06:  Epoch[ 1 of 15]-Minibatch[  71-  80, 25.00%]: CrossEntropyWithSoftmax = 3.53832947 * 640; EvalClassificationError = 0.82812500 * 640; time = 3.0562s; samplesPerSecond = 209.4
MPI Rank 1: 01/09/2018 01:32:09:  Epoch[ 1 of 15]-Minibatch[  81-  90, 28.12%]: CrossEntropyWithSoftmax = 3.50628076 * 640; EvalClassificationError = 0.81718750 * 640; time = 3.0122s; samplesPerSecond = 212.5
MPI Rank 1: 01/09/2018 01:32:12:  Epoch[ 1 of 15]-Minibatch[  91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.41478252 * 640; EvalClassificationError = 0.80781250 * 640; time = 2.5458s; samplesPerSecond = 251.4
MPI Rank 1: 01/09/2018 01:32:15:  Epoch[ 1 of 15]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.51031210 * 640; EvalClassificationError = 0.82812500 * 640; time = 3.1404s; samplesPerSecond = 203.8
MPI Rank 1: 01/09/2018 01:32:18:  Epoch[ 1 of 15]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.28365485 * 640; EvalClassificationError = 0.79375000 * 640; time = 2.9965s; samplesPerSecond = 213.6
MPI Rank 1: 01/09/2018 01:32:21:  Epoch[ 1 of 15]-Minibatch[ 121- 130, 40.62%]: CrossEntropyWithSoftmax = 3.20932117 * 640; EvalClassificationError = 0.79531250 * 640; time = 3.2124s; samplesPerSecond = 199.2
MPI Rank 1: 01/09/2018 01:32:24:  Epoch[ 1 of 15]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.07460535 * 640; EvalClassificationError = 0.75468750 * 640; time = 3.0748s; samplesPerSecond = 208.1
MPI Rank 1: 01/09/2018 01:32:27:  Epoch[ 1 of 15]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.97529104 * 640; EvalClassificationError = 0.72031250 * 640; time = 3.0959s; samplesPerSecond = 206.7
MPI Rank 1: 01/09/2018 01:32:30:  Epoch[ 1 of 15]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.11968883 * 640; EvalClassificationError = 0.74531250 * 640; time = 2.8343s; samplesPerSecond = 225.8
MPI Rank 1: 01/09/2018 01:32:33:  Epoch[ 1 of 15]-Minibatch[ 161- 170, 53.12%]: CrossEntropyWithSoftmax = 2.84172140 * 640; EvalClassificationError = 0.71093750 * 640; time = 3.1388s; samplesPerSecond = 203.9
MPI Rank 1: 01/09/2018 01:32:36:  Epoch[ 1 of 15]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.74031745 * 640; EvalClassificationError = 0.66093750 * 640; time = 2.9926s; samplesPerSecond = 213.9
MPI Rank 1: 01/09/2018 01:32:39:  Epoch[ 1 of 15]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.83858085 * 640; EvalClassificationError = 0.72656250 * 640; time = 2.8856s; samplesPerSecond = 221.8
MPI Rank 1: 01/09/2018 01:32:42:  Epoch[ 1 of 15]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.74632253 * 640; EvalClassificationError = 0.69218750 * 640; time = 2.8575s; samplesPerSecond = 224.0
MPI Rank 1: 01/09/2018 01:32:45:  Epoch[ 1 of 15]-Minibatch[ 201- 210, 65.62%]: CrossEntropyWithSoftmax = 2.61033254 * 640; EvalClassificationError = 0.66250000 * 640; time = 2.9533s; samplesPerSecond = 216.7
MPI Rank 1: 01/09/2018 01:32:48:  Epoch[ 1 of 15]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.61330754 * 640; EvalClassificationError = 0.65000000 * 640; time = 3.1042s; samplesPerSecond = 206.2
MPI Rank 1: 01/09/2018 01:32:51:  Epoch[ 1 of 15]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.54591810 * 640; EvalClassificationError = 0.66406250 * 640; time = 2.8483s; samplesPerSecond = 224.7
MPI Rank 1: 01/09/2018 01:32:54:  Epoch[ 1 of 15]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.57566512 * 640; EvalClassificationError = 0.66093750 * 640; time = 3.0934s; samplesPerSecond = 206.9
MPI Rank 1: 01/09/2018 01:32:57:  Epoch[ 1 of 15]-Minibatch[ 241- 250, 78.12%]: CrossEntropyWithSoftmax = 2.49164945 * 640; EvalClassificationError = 0.63281250 * 640; time = 3.0426s; samplesPerSecond = 210.3
MPI Rank 1: 01/09/2018 01:33:00:  Epoch[ 1 of 15]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.39954797 * 640; EvalClassificationError = 0.62812500 * 640; time = 3.4721s; samplesPerSecond = 184.3
MPI Rank 1: 01/09/2018 01:33:04:  Epoch[ 1 of 15]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27034227 * 640; EvalClassificationError = 0.59375000 * 640; time = 3.0671s; samplesPerSecond = 208.7
MPI Rank 1: 01/09/2018 01:33:07:  Epoch[ 1 of 15]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.52112387 * 640; EvalClassificationError = 0.66093750 * 640; time = 3.0013s; samplesPerSecond = 213.2
MPI Rank 1: 01/09/2018 01:33:10:  Epoch[ 1 of 15]-Minibatch[ 281- 290, 90.62%]: CrossEntropyWithSoftmax = 2.27800991 * 640; EvalClassificationError = 0.59062500 * 640; time = 3.1885s; samplesPerSecond = 200.7
MPI Rank 1: 01/09/2018 01:33:13:  Epoch[ 1 of 15]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26783634 * 640; EvalClassificationError = 0.61093750 * 640; time = 3.0056s; samplesPerSecond = 212.9
MPI Rank 1: 01/09/2018 01:33:16:  Epoch[ 1 of 15]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24590355 * 640; EvalClassificationError = 0.58593750 * 640; time = 3.2113s; samplesPerSecond = 199.3
MPI Rank 1: 01/09/2018 01:33:19:  Epoch[ 1 of 15]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.24415615 * 640; EvalClassificationError = 0.59843750 * 640; time = 2.6318s; samplesPerSecond = 243.2
MPI Rank 1: 01/09/2018 01:33:19: Finished Epoch[ 1 of 15]: [Training] CrossEntropyWithSoftmax = 3.04696987 * 20480; EvalClassificationError = 0.73583984 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=97.0738s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:34:28: Final Results: Minibatch[1-1299]: CrossEntropyWithSoftmax = 2.24737799 * 83050; perplexity = 9.46289145; EvalClassificationError = 0.61431668 * 83050
MPI Rank 1: 01/09/2018 01:34:28: Finished Epoch[ 1 of 15]: [Validate] CrossEntropyWithSoftmax = 2.24737799 * 83050; EvalClassificationError = 0.61431668 * 83050
MPI Rank 1: 01/09/2018 01:34:28: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 2.247378 (Epoch 1); EvalClassificationError = 0.614317 (Epoch 1)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:34:28: Starting Epoch 2: learning rate per sample = 0.001953  effective momentum = 0.656119  momentum as time constant = 607.5 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:34:28: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:34:32:  Epoch[ 2 of 15]-Minibatch[   1-  10, 12.50%]: CrossEntropyWithSoftmax = 2.14624416 * 2560; EvalClassificationError = 0.56953125 * 2560; time = 3.9773s; samplesPerSecond = 643.6
MPI Rank 1: 01/09/2018 01:34:36:  Epoch[ 2 of 15]-Minibatch[  11-  20, 25.00%]: CrossEntropyWithSoftmax = 2.06174352 * 2560; EvalClassificationError = 0.55742187 * 2560; time = 3.7327s; samplesPerSecond = 685.8
MPI Rank 1: 01/09/2018 01:34:39:  Epoch[ 2 of 15]-Minibatch[  21-  30, 37.50%]: CrossEntropyWithSoftmax = 2.04994567 * 2560; EvalClassificationError = 0.55351562 * 2560; time = 3.6602s; samplesPerSecond = 699.4
MPI Rank 1: 01/09/2018 01:34:43:  Epoch[ 2 of 15]-Minibatch[  31-  40, 50.00%]: CrossEntropyWithSoftmax = 2.03695762 * 2560; EvalClassificationError = 0.56132812 * 2560; time = 3.6725s; samplesPerSecond = 697.1
MPI Rank 1: 01/09/2018 01:34:46:  Epoch[ 2 of 15]-Minibatch[  41-  50, 62.50%]: CrossEntropyWithSoftmax = 2.03086449 * 2560; EvalClassificationError = 0.55664062 * 2560; time = 3.3028s; samplesPerSecond = 775.1
MPI Rank 1: 01/09/2018 01:34:50:  Epoch[ 2 of 15]-Minibatch[  51-  60, 75.00%]: CrossEntropyWithSoftmax = 1.97306418 * 2560; EvalClassificationError = 0.53671875 * 2560; time = 3.5433s; samplesPerSecond = 722.5
MPI Rank 1: 01/09/2018 01:34:53:  Epoch[ 2 of 15]-Minibatch[  61-  70, 87.50%]: CrossEntropyWithSoftmax = 1.96746291 * 2560; EvalClassificationError = 0.53164062 * 2560; time = 3.5805s; samplesPerSecond = 715.0
MPI Rank 1: 01/09/2018 01:34:57:  Epoch[ 2 of 15]-Minibatch[  71-  80, 100.00%]: CrossEntropyWithSoftmax = 1.95498387 * 2560; EvalClassificationError = 0.53750000 * 2560; time = 3.2397s; samplesPerSecond = 790.2
MPI Rank 1: 01/09/2018 01:34:57: Finished Epoch[ 2 of 15]: [Training] CrossEntropyWithSoftmax = 2.02765830 * 20480; EvalClassificationError = 0.55053711 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=28.7151s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:35:33: Final Results: Minibatch[1-326]: CrossEntropyWithSoftmax = 1.93559232 * 83050; perplexity = 6.92814655; EvalClassificationError = 0.53506321 * 83050
MPI Rank 1: 01/09/2018 01:35:33: Finished Epoch[ 2 of 15]: [Validate] CrossEntropyWithSoftmax = 1.93559232 * 83050; EvalClassificationError = 0.53506321 * 83050
MPI Rank 1: 01/09/2018 01:35:33: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.935592 (Epoch 2); EvalClassificationError = 0.535063 (Epoch 2)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:35:34: Starting Epoch 3: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:35:34: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:35:38:  Epoch[ 3 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.95358670 * 10240; EvalClassificationError = 0.53603516 * 10240; time = 4.6210s; samplesPerSecond = 2216.0
MPI Rank 1: 01/09/2018 01:35:43:  Epoch[ 3 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.97541130 * 10240; EvalClassificationError = 0.55253906 * 10240; time = 4.6282s; samplesPerSecond = 2212.5
MPI Rank 1: 01/09/2018 01:35:43: Finished Epoch[ 3 of 15]: [Training] CrossEntropyWithSoftmax = 1.96449900 * 20480; EvalClassificationError = 0.54428711 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-05; epochTime=9.45658s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:36:10: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.91503561 * 83050; perplexity = 6.78718045; EvalClassificationError = 0.52745334 * 83050
MPI Rank 1: 01/09/2018 01:36:10: Finished Epoch[ 3 of 15]: [Validate] CrossEntropyWithSoftmax = 1.91503561 * 83050; EvalClassificationError = 0.52745334 * 83050
MPI Rank 1: 01/09/2018 01:36:10: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.915036 (Epoch 3); EvalClassificationError = 0.527453 (Epoch 3)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:36:11: Starting Epoch 4: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:36:11: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:36:15:  Epoch[ 4 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.94063438 * 10240; EvalClassificationError = 0.53203125 * 10240; time = 4.5342s; samplesPerSecond = 2258.4
MPI Rank 1: 01/09/2018 01:36:19:  Epoch[ 4 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.92737921 * 10240; EvalClassificationError = 0.53046875 * 10240; time = 4.2183s; samplesPerSecond = 2427.5
MPI Rank 1: 01/09/2018 01:36:19: Finished Epoch[ 4 of 15]: [Training] CrossEntropyWithSoftmax = 1.93400680 * 20480; EvalClassificationError = 0.53125000 * 20480; totalSamplesSeen = 81920; learningRatePerSample = 9.7656251e-05; epochTime=8.95284s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:36:47: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.90598730 * 83050; perplexity = 6.72604500; EvalClassificationError = 0.52635762 * 83050
MPI Rank 1: 01/09/2018 01:36:47: Finished Epoch[ 4 of 15]: [Validate] CrossEntropyWithSoftmax = 1.90598730 * 83050; EvalClassificationError = 0.52635762 * 83050
MPI Rank 1: 01/09/2018 01:36:47: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.905987 (Epoch 4); EvalClassificationError = 0.526358 (Epoch 4)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:36:47: Starting Epoch 5: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:36:47: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:36:52:  Epoch[ 5 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.93993557 * 10240; EvalClassificationError = 0.53144531 * 10240; time = 4.7756s; samplesPerSecond = 2144.2
MPI Rank 1: 01/09/2018 01:36:56:  Epoch[ 5 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.91667918 * 10240; EvalClassificationError = 0.52070313 * 10240; time = 4.5151s; samplesPerSecond = 2267.9
MPI Rank 1: 01/09/2018 01:36:57: Finished Epoch[ 5 of 15]: [Training] CrossEntropyWithSoftmax = 1.92830738 * 20480; EvalClassificationError = 0.52607422 * 20480; totalSamplesSeen = 102400; learningRatePerSample = 9.7656251e-05; epochTime=9.4375s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:37:24: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.89836333 * 83050; perplexity = 6.67496079; EvalClassificationError = 0.52422637 * 83050
MPI Rank 1: 01/09/2018 01:37:24: Finished Epoch[ 5 of 15]: [Validate] CrossEntropyWithSoftmax = 1.89836333 * 83050; EvalClassificationError = 0.52422637 * 83050
MPI Rank 1: 01/09/2018 01:37:24: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.898363 (Epoch 5); EvalClassificationError = 0.524226 (Epoch 5)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:37:24: Starting Epoch 6: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:37:24: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:37:29:  Epoch[ 6 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.93109689 * 10240; EvalClassificationError = 0.53535156 * 10240; time = 4.7022s; samplesPerSecond = 2177.7
MPI Rank 1: 01/09/2018 01:37:33:  Epoch[ 6 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.91132106 * 10240; EvalClassificationError = 0.52353516 * 10240; time = 4.3784s; samplesPerSecond = 2338.8
MPI Rank 1: 01/09/2018 01:37:34: Finished Epoch[ 6 of 15]: [Training] CrossEntropyWithSoftmax = 1.92120897 * 20480; EvalClassificationError = 0.52944336 * 20480; totalSamplesSeen = 122880; learningRatePerSample = 9.7656251e-05; epochTime=9.27821s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:38:00: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.89149689 * 83050; perplexity = 6.62928457; EvalClassificationError = 0.52290187 * 83050
MPI Rank 1: 01/09/2018 01:38:00: Finished Epoch[ 6 of 15]: [Validate] CrossEntropyWithSoftmax = 1.89149689 * 83050; EvalClassificationError = 0.52290187 * 83050
MPI Rank 1: 01/09/2018 01:38:00: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.891497 (Epoch 6); EvalClassificationError = 0.522902 (Epoch 6)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:38:00: Starting Epoch 7: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:38:00: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:38:05:  Epoch[ 7 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.88512539 * 10240; EvalClassificationError = 0.51572266 * 10240; time = 4.6126s; samplesPerSecond = 2220.0
MPI Rank 1: 01/09/2018 01:38:10:  Epoch[ 7 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.91798861 * 10240; EvalClassificationError = 0.53535156 * 10240; time = 4.5073s; samplesPerSecond = 2271.9
MPI Rank 1: 01/09/2018 01:38:10: Finished Epoch[ 7 of 15]: [Training] CrossEntropyWithSoftmax = 1.90155700 * 20480; EvalClassificationError = 0.52553711 * 20480; totalSamplesSeen = 143360; learningRatePerSample = 9.7656251e-05; epochTime=9.16563s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:38:37: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.88459874 * 83050; perplexity = 6.58371212; EvalClassificationError = 0.52246839 * 83050
MPI Rank 1: 01/09/2018 01:38:37: Finished Epoch[ 7 of 15]: [Validate] CrossEntropyWithSoftmax = 1.88459874 * 83050; EvalClassificationError = 0.52246839 * 83050
MPI Rank 1: 01/09/2018 01:38:37: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.884599 (Epoch 7); EvalClassificationError = 0.522468 (Epoch 7)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:38:37: Starting Epoch 8: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:38:37: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:38:42:  Epoch[ 8 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.89139877 * 10240; EvalClassificationError = 0.52099609 * 10240; time = 4.7392s; samplesPerSecond = 2160.7
MPI Rank 1: 01/09/2018 01:38:46:  Epoch[ 8 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.87492662 * 10240; EvalClassificationError = 0.51923828 * 10240; time = 4.4849s; samplesPerSecond = 2283.2
MPI Rank 1: 01/09/2018 01:38:46: Finished Epoch[ 8 of 15]: [Training] CrossEntropyWithSoftmax = 1.88316269 * 20480; EvalClassificationError = 0.52011719 * 20480; totalSamplesSeen = 163840; learningRatePerSample = 9.7656251e-05; epochTime=9.28938s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:39:13: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.87932390 * 83050; perplexity = 6.54907556; EvalClassificationError = 0.52132450 * 83050
MPI Rank 1: 01/09/2018 01:39:13: Finished Epoch[ 8 of 15]: [Validate] CrossEntropyWithSoftmax = 1.87932390 * 83050; EvalClassificationError = 0.52132450 * 83050
MPI Rank 1: 01/09/2018 01:39:13: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.879324 (Epoch 8); EvalClassificationError = 0.521325 (Epoch 8)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:39:14: Starting Epoch 9: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:39:14: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:39:18:  Epoch[ 9 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.86868159 * 10240; EvalClassificationError = 0.51103516 * 10240; time = 4.3910s; samplesPerSecond = 2332.0
MPI Rank 1: 01/09/2018 01:39:22:  Epoch[ 9 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.86708001 * 10240; EvalClassificationError = 0.52001953 * 10240; time = 4.1556s; samplesPerSecond = 2464.1
MPI Rank 1: 01/09/2018 01:39:22: Finished Epoch[ 9 of 15]: [Training] CrossEntropyWithSoftmax = 1.86788080 * 20480; EvalClassificationError = 0.51552734 * 20480; totalSamplesSeen = 184320; learningRatePerSample = 9.7656251e-05; epochTime=8.75933s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:39:50: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.87278473 * 83050; perplexity = 6.50638972; EvalClassificationError = 0.52086695 * 83050
MPI Rank 1: 01/09/2018 01:39:50: Finished Epoch[ 9 of 15]: [Validate] CrossEntropyWithSoftmax = 1.87278473 * 83050; EvalClassificationError = 0.52086695 * 83050
MPI Rank 1: 01/09/2018 01:39:50: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.872785 (Epoch 9); EvalClassificationError = 0.520867 (Epoch 9)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:39:50: Starting Epoch 10: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:39:50: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:39:55:  Epoch[10 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.90473850 * 10240; EvalClassificationError = 0.52822266 * 10240; time = 4.7446s; samplesPerSecond = 2158.2
MPI Rank 1: 01/09/2018 01:39:59:  Epoch[10 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.85821242 * 10240; EvalClassificationError = 0.51484375 * 10240; time = 4.2523s; samplesPerSecond = 2408.1
MPI Rank 1: 01/09/2018 01:39:59: Finished Epoch[10 of 15]: [Training] CrossEntropyWithSoftmax = 1.88147546 * 20480; EvalClassificationError = 0.52153320 * 20480; totalSamplesSeen = 204800; learningRatePerSample = 9.7656251e-05; epochTime=9.03281s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:40:27: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.86655806 * 83050; perplexity = 6.46600244; EvalClassificationError = 0.51947020 * 83050
MPI Rank 1: 01/09/2018 01:40:27: Finished Epoch[10 of 15]: [Validate] CrossEntropyWithSoftmax = 1.86655806 * 83050; EvalClassificationError = 0.51947020 * 83050
MPI Rank 1: 01/09/2018 01:40:27: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.866558 (Epoch 10); EvalClassificationError = 0.519470 (Epoch 10)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:40:27: Starting Epoch 11: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:40:27: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:40:31:  Epoch[11 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.87411367 * 10240; EvalClassificationError = 0.51142578 * 10240; time = 4.4127s; samplesPerSecond = 2320.6
MPI Rank 1: 01/09/2018 01:40:36:  Epoch[11 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.87067306 * 10240; EvalClassificationError = 0.52158203 * 10240; time = 4.6539s; samplesPerSecond = 2200.3
MPI Rank 1: 01/09/2018 01:40:36: Finished Epoch[11 of 15]: [Training] CrossEntropyWithSoftmax = 1.87239337 * 20480; EvalClassificationError = 0.51650391 * 20480; totalSamplesSeen = 225280; learningRatePerSample = 9.7656251e-05; epochTime=9.191s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:41:04: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.85980825 * 83050; perplexity = 6.42250511; EvalClassificationError = 0.51703793 * 83050
MPI Rank 1: 01/09/2018 01:41:04: Finished Epoch[11 of 15]: [Validate] CrossEntropyWithSoftmax = 1.85980825 * 83050; EvalClassificationError = 0.51703793 * 83050
MPI Rank 1: 01/09/2018 01:41:04: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.859808 (Epoch 11); EvalClassificationError = 0.517038 (Epoch 11)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:41:04: Starting Epoch 12: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:41:04: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:41:09:  Epoch[12 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.87570276 * 10240; EvalClassificationError = 0.52138672 * 10240; time = 4.8065s; samplesPerSecond = 2130.4
MPI Rank 1: 01/09/2018 01:41:13:  Epoch[12 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.84544781 * 10240; EvalClassificationError = 0.50859375 * 10240; time = 4.3362s; samplesPerSecond = 2361.5
MPI Rank 1: 01/09/2018 01:41:13: Finished Epoch[12 of 15]: [Training] CrossEntropyWithSoftmax = 1.86057528 * 20480; EvalClassificationError = 0.51499023 * 20480; totalSamplesSeen = 245760; learningRatePerSample = 9.7656251e-05; epochTime=9.16654s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:41:40: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.85346328 * 83050; perplexity = 6.38188356; EvalClassificationError = 0.51517158 * 83050
MPI Rank 1: 01/09/2018 01:41:40: Finished Epoch[12 of 15]: [Validate] CrossEntropyWithSoftmax = 1.85346328 * 83050; EvalClassificationError = 0.51517158 * 83050
MPI Rank 1: 01/09/2018 01:41:40: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.853463 (Epoch 12); EvalClassificationError = 0.515172 (Epoch 12)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:41:40: Starting Epoch 13: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:41:40: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:41:44:  Epoch[13 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.85293961 * 10046; EvalClassificationError = 0.52020705 * 10046; time = 4.6216s; samplesPerSecond = 2173.7
MPI Rank 1: 01/09/2018 01:41:49:  Epoch[13 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.88311779 * 10240; EvalClassificationError = 0.52031250 * 10240; time = 4.4128s; samplesPerSecond = 2320.5
MPI Rank 1: 01/09/2018 01:41:49: Finished Epoch[13 of 15]: [Training] CrossEntropyWithSoftmax = 1.86894139 * 20480; EvalClassificationError = 0.51992187 * 20480; totalSamplesSeen = 266240; learningRatePerSample = 9.7656251e-05; epochTime=9.20683s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:42:17: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.84713665 * 83050; perplexity = 6.34163517; EvalClassificationError = 0.51273931 * 83050
MPI Rank 1: 01/09/2018 01:42:17: Finished Epoch[13 of 15]: [Validate] CrossEntropyWithSoftmax = 1.84713665 * 83050; EvalClassificationError = 0.51273931 * 83050
MPI Rank 1: 01/09/2018 01:42:17: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.847137 (Epoch 13); EvalClassificationError = 0.512739 (Epoch 13)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:42:17: Starting Epoch 14: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:42:17: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:42:21:  Epoch[14 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.86634456 * 10240; EvalClassificationError = 0.50937500 * 10240; time = 4.6362s; samplesPerSecond = 2208.7
MPI Rank 1: 01/09/2018 01:42:26:  Epoch[14 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.85430186 * 10240; EvalClassificationError = 0.51328125 * 10240; time = 4.1784s; samplesPerSecond = 2450.7
MPI Rank 1: 01/09/2018 01:42:26: Finished Epoch[14 of 15]: [Training] CrossEntropyWithSoftmax = 1.86032321 * 20480; EvalClassificationError = 0.51132813 * 20480; totalSamplesSeen = 286720; learningRatePerSample = 9.7656251e-05; epochTime=9.01662s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:42:53: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.84154028 * 83050; perplexity = 6.30624417; EvalClassificationError = 0.51331728 * 83050
MPI Rank 1: 01/09/2018 01:42:53: Finished Epoch[14 of 15]: [Validate] CrossEntropyWithSoftmax = 1.84154028 * 83050; EvalClassificationError = 0.51331728 * 83050
MPI Rank 1: 01/09/2018 01:42:53: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.841540 (Epoch 14); EvalClassificationError = 0.512739 (Epoch 13)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:42:53: Starting Epoch 15: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:42:53: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:42:58:  Epoch[15 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.82826227 * 10240; EvalClassificationError = 0.50820312 * 10240; time = 4.5965s; samplesPerSecond = 2227.8
MPI Rank 1: 01/09/2018 01:43:02:  Epoch[15 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.85323706 * 10240; EvalClassificationError = 0.51826172 * 10240; time = 4.2995s; samplesPerSecond = 2381.6
MPI Rank 1: 01/09/2018 01:43:03: Finished Epoch[15 of 15]: [Training] CrossEntropyWithSoftmax = 1.84074966 * 20480; EvalClassificationError = 0.51323242 * 20480; totalSamplesSeen = 307200; learningRatePerSample = 9.7656251e-05; epochTime=9.09234s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:43:30: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.83558729 * 83050; perplexity = 6.26881470; EvalClassificationError = 0.51134256 * 83050
MPI Rank 1: 01/09/2018 01:43:30: Finished Epoch[15 of 15]: [Validate] CrossEntropyWithSoftmax = 1.83558729 * 83050; EvalClassificationError = 0.51134256 * 83050
MPI Rank 1: 01/09/2018 01:43:30: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.835587 (Epoch 15); EvalClassificationError = 0.511343 (Epoch 15)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:43:31: Action "train" complete.
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:43:31: __COMPLETED__
=== Deleting last epoch data
==== Re-running from checkpoint
=== Running mpiexec -n 2 /home/ubuntu/workspace/build/gpu/release/bin/cntk configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data RunDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion OutputDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu DeviceId=-1 timestamping=true makeMode=true numCPUThreads=6 shareNodeValueMatrices=true saveBestModelPerCriterion=true stderr=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr
CNTK 2.3.1+ (HEAD 294890, Jan  8 2018 16:47:50) at 2018/01/09 01:43:31

/home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DeviceId=-1  timestamping=true  makeMode=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr
CNTK 2.3.1+ (HEAD 294890, Jan  8 2018 16:47:50) at 2018/01/09 01:43:31

/home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DeviceId=-1  timestamping=true  makeMode=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr
Changed current directory to /home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data
Changed current directory to /home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data
--------------------------------------------------------------------------
[[8211,1],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: 17c29a606870

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
ping [requestnodes (before change)]: 2 nodes pinging each other
ping [requestnodes (before change)]: 2 nodes pinging each other
ping [requestnodes (after change)]: 2 nodes pinging each other
ping [requestnodes (after change)]: 2 nodes pinging each other
requestnodes [MPIWrapperMpi]: using 2 out of 2 MPI nodes on a single host (2 requested); we (0) are in (participating)
ping [mpihelper]: 2 nodes pinging each other
requestnodes [MPIWrapperMpi]: using 2 out of 2 MPI nodes on a single host (2 requested); we (1) are in (participating)
ping [mpihelper]: 2 nodes pinging each other
01/09/2018 01:43:31: Redirecting stderr to file /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr_speechTrain.logrank0
01/09/2018 01:43:31: Redirecting stderr to file /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr_speechTrain.logrank1
[17c29a606870:15399] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics
[17c29a606870:15399] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
MPI Rank 0: CNTK 2.3.1+ (HEAD 294890, Jan  8 2018 16:47:50) at 2018/01/09 01:43:31
MPI Rank 0: 
MPI Rank 0: /home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DeviceId=-1  timestamping=true  makeMode=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr
MPI Rank 0: 01/09/2018 01:43:31: -------------------------------------------------------------------
MPI Rank 0: 01/09/2018 01:43:31: Build info: 
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:43:31: 		Built time: Jan  8 2018 16:42:01
MPI Rank 0: 01/09/2018 01:43:31: 		Last modified date: Mon Jan  8 16:40:18 2018
MPI Rank 0: 01/09/2018 01:43:31: 		Build type: release
MPI Rank 0: 01/09/2018 01:43:31: 		Build target: GPU
MPI Rank 0: 01/09/2018 01:43:31: 		With ASGD: yes
MPI Rank 0: 01/09/2018 01:43:31: 		Math lib: mkl
MPI Rank 0: 01/09/2018 01:43:31: 		CUDA version: 9.0.0
MPI Rank 0: 01/09/2018 01:43:31: 		CUDNN version: 7.0.4
MPI Rank 0: 01/09/2018 01:43:31: 		Build Branch: HEAD
MPI Rank 0: 01/09/2018 01:43:31: 		Build SHA1: 294890cb1f83fc31a56bd2cc1fc1fec34894b71c
MPI Rank 0: 01/09/2018 01:43:31: 		MPI distribution: Open MPI
MPI Rank 0: 01/09/2018 01:43:31: 		MPI version: 1.10.7
MPI Rank 0: 01/09/2018 01:43:31: -------------------------------------------------------------------
MPI Rank 0: 01/09/2018 01:43:31: -------------------------------------------------------------------
MPI Rank 0: 01/09/2018 01:43:31: GPU info:
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:43:31: 		Device[0]: cores = 3072; computeCapability = 5.2; type = "Tesla M60"; total memory = 8123 MB; free memory = 8112 MB
MPI Rank 0: 01/09/2018 01:43:31: -------------------------------------------------------------------
MPI Rank 0: 01/09/2018 01:43:31: Using 6 CPU threads.
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:43:31: ##############################################################################
MPI Rank 0: 01/09/2018 01:43:31: #                                                                            #
MPI Rank 0: 01/09/2018 01:43:31: # speechTrain command (train action)                                         #
MPI Rank 0: 01/09/2018 01:43:31: #                                                                            #
MPI Rank 0: 01/09/2018 01:43:31: ##############################################################################
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:43:31: 
MPI Rank 0: Starting from checkpoint. Loading network from '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.14'.
MPI Rank 0: SimpleNetworkBuilder Using CPU
MPI Rank 0: Reading script file glob_0000.scp ... 948 entries
MPI Rank 0: HTKDeserializer: selected '948' utterances grouped into '3' chunks, average chunk size: 316.0 utterances, 84244.7 frames (for I/O: 316.0 utterances, 84244.7 frames)
MPI Rank 0: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 0: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 0: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 0: Reading script file glob_0000.cv.scp ... 300 entries
MPI Rank 0: HTKDeserializer: selected '300' utterances grouped into '1' chunks, average chunk size: 300.0 utterances, 83050.0 frames (for I/O: 300.0 utterances, 83050.0 frames)
MPI Rank 0: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 0: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 0: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 0: 01/09/2018 01:43:31: 
MPI Rank 0: Model has 25 nodes. Using CPU.
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:43:31: Training criterion:   CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
MPI Rank 0: 01/09/2018 01:43:31: Evaluation criterion: EvalClassificationError = ClassificationError
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:43:31: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:43:31: 	Node 'B0' (LearnableParameter operation) : [512 x 1]
MPI Rank 0: 01/09/2018 01:43:31: 	Node 'B1' (LearnableParameter operation) : [512 x 1]
MPI Rank 0: 01/09/2018 01:43:31: 	Node 'B2' (LearnableParameter operation) : [132 x 1]
MPI Rank 0: 01/09/2018 01:43:31: 	Node 'W0' (LearnableParameter operation) : [512 x 363]
MPI Rank 0: 01/09/2018 01:43:31: 	Node 'W1' (LearnableParameter operation) : [512 x 512]
MPI Rank 0: 01/09/2018 01:43:31: 	Node 'W2' (LearnableParameter operation) : [132 x 512]
MPI Rank 0: 
MPI Rank 0: Initializing dataParallelSGD with FP64 aggregation.
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:43:32: No PreCompute nodes found, or all already computed. Skipping pre-computation step.
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:43:32: Starting Epoch 15: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:43:32: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/09/2018 01:43:37:  Epoch[15 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.82826227 * 10240; EvalClassificationError = 0.50820312 * 10240; time = 4.9004s; samplesPerSecond = 2089.6
MPI Rank 0: 01/09/2018 01:43:41:  Epoch[15 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.85323706 * 10240; EvalClassificationError = 0.51826172 * 10240; time = 4.2451s; samplesPerSecond = 2412.2
MPI Rank 0: 01/09/2018 01:43:41: Finished Epoch[15 of 15]: [Training] CrossEntropyWithSoftmax = 1.84074966 * 20480; EvalClassificationError = 0.51323242 * 20480; totalSamplesSeen = 307200; learningRatePerSample = 9.7656251e-05; epochTime=9.33997s
MPI Rank 0: NcclComm: disabled, at least one rank using CPU device
MPI Rank 0: 01/09/2018 01:44:08: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.83558729 * 83050; perplexity = 6.26881470; EvalClassificationError = 0.51134256 * 83050
MPI Rank 0: 01/09/2018 01:44:08: Finished Epoch[15 of 15]: [Validate] CrossEntropyWithSoftmax = 1.83558729 * 83050; EvalClassificationError = 0.51134256 * 83050
MPI Rank 0: 01/09/2018 01:44:08: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.835587 (Epoch 15); EvalClassificationError = 0.511343 (Epoch 15)
MPI Rank 0: 01/09/2018 01:44:08: SGD: Saving checkpoint model '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn'
MPI Rank 0: 01/09/2018 01:44:08: Best epoch for criterion 'CrossEntropyWithSoftmax' is 15 and model /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn copied to /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn_CrossEntropyWithSoftmax
MPI Rank 0: 01/09/2018 01:44:08: Best epoch for criterion 'EvalClassificationError' is 15 and model /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn copied to /tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn_EvalClassificationError
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:44:08: Action "train" complete.
MPI Rank 0: 
MPI Rank 0: 01/09/2018 01:44:08: __COMPLETED__
MPI Rank 1: CNTK 2.3.1+ (HEAD 294890, Jan  8 2018 16:47:50) at 2018/01/09 01:43:31
MPI Rank 1: 
MPI Rank 1: /home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu  DeviceId=-1  timestamping=true  makeMode=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/stderr
MPI Rank 1: 01/09/2018 01:43:31: -------------------------------------------------------------------
MPI Rank 1: 01/09/2018 01:43:31: Build info: 
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:43:31: 		Built time: Jan  8 2018 16:42:01
MPI Rank 1: 01/09/2018 01:43:31: 		Last modified date: Mon Jan  8 16:40:18 2018
MPI Rank 1: 01/09/2018 01:43:31: 		Build type: release
MPI Rank 1: 01/09/2018 01:43:31: 		Build target: GPU
MPI Rank 1: 01/09/2018 01:43:31: 		With ASGD: yes
MPI Rank 1: 01/09/2018 01:43:31: 		Math lib: mkl
MPI Rank 1: 01/09/2018 01:43:31: 		CUDA version: 9.0.0
MPI Rank 1: 01/09/2018 01:43:31: 		CUDNN version: 7.0.4
MPI Rank 1: 01/09/2018 01:43:31: 		Build Branch: HEAD
MPI Rank 1: 01/09/2018 01:43:31: 		Build SHA1: 294890cb1f83fc31a56bd2cc1fc1fec34894b71c
MPI Rank 1: 01/09/2018 01:43:31: 		MPI distribution: Open MPI
MPI Rank 1: 01/09/2018 01:43:31: 		MPI version: 1.10.7
MPI Rank 1: 01/09/2018 01:43:31: -------------------------------------------------------------------
MPI Rank 1: 01/09/2018 01:43:31: -------------------------------------------------------------------
MPI Rank 1: 01/09/2018 01:43:31: GPU info:
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:43:31: 		Device[0]: cores = 3072; computeCapability = 5.2; type = "Tesla M60"; total memory = 8123 MB; free memory = 8029 MB
MPI Rank 1: 01/09/2018 01:43:31: -------------------------------------------------------------------
MPI Rank 1: 01/09/2018 01:43:31: Using 6 CPU threads.
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:43:31: ##############################################################################
MPI Rank 1: 01/09/2018 01:43:31: #                                                                            #
MPI Rank 1: 01/09/2018 01:43:31: # speechTrain command (train action)                                         #
MPI Rank 1: 01/09/2018 01:43:31: #                                                                            #
MPI Rank 1: 01/09/2018 01:43:31: ##############################################################################
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:43:31: 
MPI Rank 1: Starting from checkpoint. Loading network from '/tmp/cntk-test-20180109012214.493694/Speech/DNN_SaveBestModelPerCriterion@release_cpu/models/cntkSpeech.dnn.14'.
MPI Rank 1: SimpleNetworkBuilder Using CPU
MPI Rank 1: Reading script file glob_0000.scp ... 948 entries
MPI Rank 1: HTKDeserializer: selected '948' utterances grouped into '3' chunks, average chunk size: 316.0 utterances, 84244.7 frames (for I/O: 316.0 utterances, 84244.7 frames)
MPI Rank 1: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 1: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 1: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 1: Reading script file glob_0000.cv.scp ... 300 entries
MPI Rank 1: HTKDeserializer: selected '300' utterances grouped into '1' chunks, average chunk size: 300.0 utterances, 83050.0 frames (for I/O: 300.0 utterances, 83050.0 frames)
MPI Rank 1: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 1: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 1: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 1: 01/09/2018 01:43:32: 
MPI Rank 1: Model has 25 nodes. Using CPU.
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:43:32: Training criterion:   CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
MPI Rank 1: 01/09/2018 01:43:32: Evaluation criterion: EvalClassificationError = ClassificationError
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:43:32: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:43:32: 	Node 'B0' (LearnableParameter operation) : [512 x 1]
MPI Rank 1: 01/09/2018 01:43:32: 	Node 'B1' (LearnableParameter operation) : [512 x 1]
MPI Rank 1: 01/09/2018 01:43:32: 	Node 'B2' (LearnableParameter operation) : [132 x 1]
MPI Rank 1: 01/09/2018 01:43:32: 	Node 'W0' (LearnableParameter operation) : [512 x 363]
MPI Rank 1: 01/09/2018 01:43:32: 	Node 'W1' (LearnableParameter operation) : [512 x 512]
MPI Rank 1: 01/09/2018 01:43:32: 	Node 'W2' (LearnableParameter operation) : [132 x 512]
MPI Rank 1: 
MPI Rank 1: Initializing dataParallelSGD with FP64 aggregation.
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:43:32: No PreCompute nodes found, or all already computed. Skipping pre-computation step.
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:43:32: Starting Epoch 15: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:43:32: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/09/2018 01:43:37:  Epoch[15 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.82826227 * 10240; EvalClassificationError = 0.50820312 * 10240; time = 4.8970s; samplesPerSecond = 2091.1
MPI Rank 1: 01/09/2018 01:43:41:  Epoch[15 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.85323706 * 10240; EvalClassificationError = 0.51826172 * 10240; time = 4.2418s; samplesPerSecond = 2414.1
MPI Rank 1: 01/09/2018 01:43:41: Finished Epoch[15 of 15]: [Training] CrossEntropyWithSoftmax = 1.84074966 * 20480; EvalClassificationError = 0.51323242 * 20480; totalSamplesSeen = 307200; learningRatePerSample = 9.7656251e-05; epochTime=9.33998s
MPI Rank 1: NcclComm: disabled, at least one rank using CPU device
MPI Rank 1: 01/09/2018 01:44:08: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.83558729 * 83050; perplexity = 6.26881470; EvalClassificationError = 0.51134256 * 83050
MPI Rank 1: 01/09/2018 01:44:08: Finished Epoch[15 of 15]: [Validate] CrossEntropyWithSoftmax = 1.83558729 * 83050; EvalClassificationError = 0.51134256 * 83050
MPI Rank 1: 01/09/2018 01:44:08: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.835587 (Epoch 15); EvalClassificationError = 0.511343 (Epoch 15)
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:44:08: Action "train" complete.
MPI Rank 1: 
MPI Rank 1: 01/09/2018 01:44:08: __COMPLETED__