CPU info:
    CPU Model Name: Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
    Hardware threads: 12
    Total Memory: 57700428 kB
-------------------------------------------------------------------
=== Running mpiexec -n 2 /home/ubuntu/workspace/build/gpu/release/bin/cntk configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu DeviceId=0 timestamping=true numCPUThreads=6 shareNodeValueMatrices=true saveBestModelPerCriterion=true stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr
CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:06:04

/home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DeviceId=0  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr
CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:06:04

/home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DeviceId=0  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr
Changed current directory to /home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data
Changed current directory to /home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data
--------------------------------------------------------------------------
[[22444,1],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: 7fee1579d8b2

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
ping [requestnodes (before change)]: 2 nodes pinging each other
ping [requestnodes (before change)]: 2 nodes pinging each other
ping [requestnodes (after change)]: 2 nodes pinging each other
requestnodes [MPIWrapperMpi]: using 2 out of 2 MPI nodes on a single host (2 requested); we (1) are in (participating)
ping [mpihelper]: 2 nodes pinging each other
ping [requestnodes (after change)]: 2 nodes pinging each other
requestnodes [MPIWrapperMpi]: using 2 out of 2 MPI nodes on a single host (2 requested); we (0) are in (participating)
ping [mpihelper]: 2 nodes pinging each other
01/16/2018 19:06:04: Redirecting stderr to file /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr_speechTrain.logrank0
01/16/2018 19:06:04: Redirecting stderr to file /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr_speechTrain.logrank1
[7fee1579d8b2:53546] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics
[7fee1579d8b2:53546] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
MPI Rank 0: CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:06:04
MPI Rank 0: 
MPI Rank 0: /home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DeviceId=0  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr
MPI Rank 0: 01/16/2018 19:06:04: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:06:04: Build info: 
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:04: 		Built time: Jan 16 2018 16:15:42
MPI Rank 0: 01/16/2018 19:06:04: 		Last modified date: Tue Jan 16 16:13:51 2018
MPI Rank 0: 01/16/2018 19:06:04: 		Build type: release
MPI Rank 0: 01/16/2018 19:06:04: 		Build target: GPU
MPI Rank 0: 01/16/2018 19:06:04: 		With ASGD: yes
MPI Rank 0: 01/16/2018 19:06:04: 		Math lib: mkl
MPI Rank 0: 01/16/2018 19:06:04: 		CUDA version: 9.0.0
MPI Rank 0: 01/16/2018 19:06:04: 		CUDNN version: 7.0.4
MPI Rank 0: 01/16/2018 19:06:04: 		Build Branch: HEAD
MPI Rank 0: 01/16/2018 19:06:04: 		Build SHA1: c4c2ce8c6e89b5c32e4d07523081283417bcfc6d
MPI Rank 0: 01/16/2018 19:06:04: 		MPI distribution: Open MPI
MPI Rank 0: 01/16/2018 19:06:04: 		MPI version: 1.10.7
MPI Rank 0: 01/16/2018 19:06:04: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:06:04: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:06:04: GPU info:
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:04: 		Device[0]: cores = 3072; computeCapability = 5.2; type = "Tesla M60"; total memory = 8123 MB; free memory = 8112 MB
MPI Rank 0: 01/16/2018 19:06:04: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:06:04: Using 6 CPU threads.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:04: ##############################################################################
MPI Rank 0: 01/16/2018 19:06:04: #                                                                            #
MPI Rank 0: 01/16/2018 19:06:04: # speechTrain command (train action)                                         #
MPI Rank 0: 01/16/2018 19:06:04: #                                                                            #
MPI Rank 0: 01/16/2018 19:06:04: ##############################################################################
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:04: 
MPI Rank 0: Creating virgin network.
MPI Rank 0: SimpleNetworkBuilder Using GPU 0
MPI Rank 0: Reading script file glob_0000.scp ... 948 entries
MPI Rank 0: HTKDeserializer: selected '948' utterances grouped into '3' chunks, average chunk size: 316.0 utterances, 84244.7 frames (for I/O: 316.0 utterances, 84244.7 frames)
MPI Rank 0: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 0: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 0: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 0: Reading script file glob_0000.cv.scp ... 300 entries
MPI Rank 0: HTKDeserializer: selected '300' utterances grouped into '1' chunks, average chunk size: 300.0 utterances, 83050.0 frames (for I/O: 300.0 utterances, 83050.0 frames)
MPI Rank 0: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 0: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 0: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 0: 01/16/2018 19:06:04: 
MPI Rank 0: Model has 25 nodes. Using GPU 0.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:04: Training criterion:   CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
MPI Rank 0: 01/16/2018 19:06:04: Evaluation criterion: EvalClassificationError = ClassificationError
MPI Rank 0: 
MPI Rank 0: 
MPI Rank 0: Allocating matrices for forward and/or backward propagation.
MPI Rank 0: 
MPI Rank 0: Gradient Memory Aliasing: 4 are aliased.
MPI Rank 0: 	W2*H1 (gradient) reuses HLast (gradient)
MPI Rank 0: 	W1*H1 (gradient) reuses W1*H1+B1 (gradient)
MPI Rank 0: 
MPI Rank 0: Memory Sharing: Out of 40 matrices, 21 are shared as 5, and 19 are not shared.
MPI Rank 0: 
MPI Rank 0: Here are the ones that share memory:
MPI Rank 0: 	{ PosteriorProb : [132 x 1 x *]
MPI Rank 0: 	  ScaledLogLikelihood : [132 x 1 x *] }
MPI Rank 0: 	{ H2 : [512 x 1 x *]
MPI Rank 0: 	  W0*features+B0 : [512 x 1 x *]
MPI Rank 0: 	  W1 : [512 x 512] (gradient)
MPI Rank 0: 	  W1*H1 : [512 x 1 x *] }
MPI Rank 0: 	{ B0 : [512 x 1] (gradient)
MPI Rank 0: 	  H1 : [512 x 1 x *] }
MPI Rank 0: 	{ HLast : [132 x 1 x *] (gradient)
MPI Rank 0: 	  W0 : [512 x 363] (gradient)
MPI Rank 0: 	  W0*features+B0 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  W1*H1 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  W1*H1+B1 : [512 x 1 x *]
MPI Rank 0: 	  W1*H1+B1 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  W2*H1 : [132 x 1 x *]
MPI Rank 0: 	  W2*H1 : [132 x 1 x *] (gradient) }
MPI Rank 0: 	{ H1 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  H2 : [512 x 1 x *] (gradient)
MPI Rank 0: 	  HLast : [132 x 1 x *]
MPI Rank 0: 	  W0*features : [512 x *]
MPI Rank 0: 	  W0*features : [512 x *] (gradient) }
MPI Rank 0: 
MPI Rank 0: Here are the ones that don't share memory:
MPI Rank 0: 	{features : [363 x *]}
MPI Rank 0: 	{MeanOfFeatures : [363]}
MPI Rank 0: 	{InvStdOfFeatures : [363]}
MPI Rank 0: 	{W0 : [512 x 363]}
MPI Rank 0: 	{B0 : [512 x 1]}
MPI Rank 0: 	{W1 : [512 x 512]}
MPI Rank 0: 	{B1 : [512 x 1]}
MPI Rank 0: 	{W2 : [132 x 512]}
MPI Rank 0: 	{B2 : [132 x 1]}
MPI Rank 0: 	{labels : [132 x *]}
MPI Rank 0: 	{Prior : [132]}
MPI Rank 0: 	{CrossEntropyWithSoftmax : [1]}
MPI Rank 0: 	{EvalClassificationError : [1]}
MPI Rank 0: 	{W2 : [132 x 512] (gradient)}
MPI Rank 0: 	{LogOfPrior : [132]}
MPI Rank 0: 	{B1 : [512 x 1] (gradient)}
MPI Rank 0: 	{B2 : [132 x 1] (gradient)}
MPI Rank 0: 	{CrossEntropyWithSoftmax : [1] (gradient)}
MPI Rank 0: 	{MVNormalizedFeatures : [363 x *]}
MPI Rank 0: 
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:04: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:04: 	Node 'B0' (LearnableParameter operation) : [512 x 1]
MPI Rank 0: 01/16/2018 19:06:04: 	Node 'B1' (LearnableParameter operation) : [512 x 1]
MPI Rank 0: 01/16/2018 19:06:04: 	Node 'B2' (LearnableParameter operation) : [132 x 1]
MPI Rank 0: 01/16/2018 19:06:04: 	Node 'W0' (LearnableParameter operation) : [512 x 363]
MPI Rank 0: 01/16/2018 19:06:04: 	Node 'W1' (LearnableParameter operation) : [512 x 512]
MPI Rank 0: 01/16/2018 19:06:04: 	Node 'W2' (LearnableParameter operation) : [132 x 512]
MPI Rank 0: 
MPI Rank 0: Initializing dataParallelSGD with FP64 aggregation.
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:05: Precomputing --> 3 PreCompute nodes found.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:05: 	MeanOfFeatures = Mean()
MPI Rank 0: 01/16/2018 19:06:05: 	InvStdOfFeatures = InvStdDev()
MPI Rank 0: 01/16/2018 19:06:05: 	Prior = Mean()
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:08: Precomputing --> Completed.
MPI Rank 0: 
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:08: Starting Epoch 1: learning rate per sample = 0.015625  effective momentum = 0.900000  momentum as time constant = 607.4 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:08: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:08:  Epoch[ 1 of 15]-Minibatch[   1-  10, 3.12%]: CrossEntropyWithSoftmax = 4.62512789 * 640; EvalClassificationError = 0.94062500 * 640; time = 0.0672s; samplesPerSecond = 9522.9
MPI Rank 0: 01/16/2018 19:06:08:  Epoch[ 1 of 15]-Minibatch[  11-  20, 6.25%]: CrossEntropyWithSoftmax = 4.35619366 * 640; EvalClassificationError = 0.92343750 * 640; time = 0.0617s; samplesPerSecond = 10370.3
MPI Rank 0: 01/16/2018 19:06:08:  Epoch[ 1 of 15]-Minibatch[  21-  30, 9.38%]: CrossEntropyWithSoftmax = 3.97911998 * 640; EvalClassificationError = 0.89531250 * 640; time = 0.0626s; samplesPerSecond = 10219.8
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  31-  40, 12.50%]: CrossEntropyWithSoftmax = 3.73643568 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.0621s; samplesPerSecond = 10302.4
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  41-  50, 15.62%]: CrossEntropyWithSoftmax = 3.83079081 * 640; EvalClassificationError = 0.88281250 * 640; time = 0.0622s; samplesPerSecond = 10283.1
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  51-  60, 18.75%]: CrossEntropyWithSoftmax = 3.71437690 * 640; EvalClassificationError = 0.86875000 * 640; time = 0.0624s; samplesPerSecond = 10261.4
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  61-  70, 21.88%]: CrossEntropyWithSoftmax = 3.42186231 * 640; EvalClassificationError = 0.79062500 * 640; time = 0.0622s; samplesPerSecond = 10282.8
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  71-  80, 25.00%]: CrossEntropyWithSoftmax = 3.53658053 * 640; EvalClassificationError = 0.82031250 * 640; time = 0.0619s; samplesPerSecond = 10338.9
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  81-  90, 28.12%]: CrossEntropyWithSoftmax = 3.49758018 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.0618s; samplesPerSecond = 10359.1
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.39996308 * 640; EvalClassificationError = 0.80468750 * 640; time = 0.0627s; samplesPerSecond = 10205.4
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.49445773 * 640; EvalClassificationError = 0.82500000 * 640; time = 0.0619s; samplesPerSecond = 10341.9
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.26676999 * 640; EvalClassificationError = 0.79218750 * 640; time = 0.0635s; samplesPerSecond = 10071.0
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 121- 130, 40.62%]: CrossEntropyWithSoftmax = 3.18870174 * 640; EvalClassificationError = 0.78906250 * 640; time = 0.0633s; samplesPerSecond = 10111.7
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.05687264 * 640; EvalClassificationError = 0.74687500 * 640; time = 0.0632s; samplesPerSecond = 10127.8
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.95594570 * 640; EvalClassificationError = 0.71875000 * 640; time = 0.0651s; samplesPerSecond = 9834.4
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.10219605 * 640; EvalClassificationError = 0.74062500 * 640; time = 0.0691s; samplesPerSecond = 9267.9
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 161- 170, 53.12%]: CrossEntropyWithSoftmax = 2.80745016 * 640; EvalClassificationError = 0.70625000 * 640; time = 0.0746s; samplesPerSecond = 8574.8
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.72061843 * 640; EvalClassificationError = 0.65468750 * 640; time = 0.0619s; samplesPerSecond = 10341.4
MPI Rank 0: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.80425748 * 640; EvalClassificationError = 0.71718750 * 640; time = 0.0612s; samplesPerSecond = 10454.6
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.71253069 * 640; EvalClassificationError = 0.67812500 * 640; time = 0.0624s; samplesPerSecond = 10254.2
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 201- 210, 65.62%]: CrossEntropyWithSoftmax = 2.59360400 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.0622s; samplesPerSecond = 10287.0
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.60386650 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0615s; samplesPerSecond = 10411.2
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.53706679 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0618s; samplesPerSecond = 10350.2
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.56177344 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0622s; samplesPerSecond = 10287.5
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 241- 250, 78.12%]: CrossEntropyWithSoftmax = 2.50118792 * 640; EvalClassificationError = 0.64218750 * 640; time = 0.0673s; samplesPerSecond = 9507.0
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.40119789 * 640; EvalClassificationError = 0.62500000 * 640; time = 0.0628s; samplesPerSecond = 10196.2
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27491504 * 640; EvalClassificationError = 0.58906250 * 640; time = 0.0618s; samplesPerSecond = 10360.0
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.51724208 * 640; EvalClassificationError = 0.65781250 * 640; time = 0.0619s; samplesPerSecond = 10342.4
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 281- 290, 90.62%]: CrossEntropyWithSoftmax = 2.27797543 * 640; EvalClassificationError = 0.59687500 * 640; time = 0.0622s; samplesPerSecond = 10297.7
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26017741 * 640; EvalClassificationError = 0.60937500 * 640; time = 0.0617s; samplesPerSecond = 10380.3
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24735343 * 640; EvalClassificationError = 0.58437500 * 640; time = 0.0630s; samplesPerSecond = 10158.3
MPI Rank 0: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.23665382 * 640; EvalClassificationError = 0.60625000 * 640; time = 0.0623s; samplesPerSecond = 10275.7
MPI Rank 0: 01/16/2018 19:06:10: Finished Epoch[ 1 of 15]: [Training] CrossEntropyWithSoftmax = 3.03815142 * 20480; EvalClassificationError = 0.73432617 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=2.02959s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:12: Final Results: Minibatch[1-1299]: CrossEntropyWithSoftmax = 2.24821048 * 83050; perplexity = 9.47077252; EvalClassificationError = 0.61623119 * 83050
MPI Rank 0: 01/16/2018 19:06:12: Finished Epoch[ 1 of 15]: [Validate] CrossEntropyWithSoftmax = 2.24821048 * 83050; EvalClassificationError = 0.61623119 * 83050
MPI Rank 0: 01/16/2018 19:06:12: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 2.248210 (Epoch 1); EvalClassificationError = 0.616231 (Epoch 1)
MPI Rank 0: 01/16/2018 19:06:12: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.1'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:12: Starting Epoch 2: learning rate per sample = 0.001953  effective momentum = 0.656119  momentum as time constant = 607.5 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:12: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[   1-  10, 12.50%]: CrossEntropyWithSoftmax = 2.13894071 * 2560; EvalClassificationError = 0.56992188 * 2560; time = 0.1196s; samplesPerSecond = 21411.0
MPI Rank 0: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  11-  20, 25.00%]: CrossEntropyWithSoftmax = 2.06106261 * 2560; EvalClassificationError = 0.55664062 * 2560; time = 0.1152s; samplesPerSecond = 22221.7
MPI Rank 0: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  21-  30, 37.50%]: CrossEntropyWithSoftmax = 2.04459475 * 2560; EvalClassificationError = 0.55039063 * 2560; time = 0.1218s; samplesPerSecond = 21025.6
MPI Rank 0: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  31-  40, 50.00%]: CrossEntropyWithSoftmax = 2.03347291 * 2560; EvalClassificationError = 0.55742187 * 2560; time = 0.1170s; samplesPerSecond = 21877.1
MPI Rank 0: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  41-  50, 62.50%]: CrossEntropyWithSoftmax = 2.02079287 * 2560; EvalClassificationError = 0.54414063 * 2560; time = 0.1138s; samplesPerSecond = 22493.7
MPI Rank 0: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  51-  60, 75.00%]: CrossEntropyWithSoftmax = 1.96950012 * 2560; EvalClassificationError = 0.53085938 * 2560; time = 0.1151s; samplesPerSecond = 22242.2
MPI Rank 0: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  61-  70, 87.50%]: CrossEntropyWithSoftmax = 1.95934863 * 2560; EvalClassificationError = 0.52812500 * 2560; time = 0.1136s; samplesPerSecond = 22535.4
MPI Rank 0: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  71-  80, 100.00%]: CrossEntropyWithSoftmax = 1.94070839 * 2560; EvalClassificationError = 0.53125000 * 2560; time = 0.1134s; samplesPerSecond = 22575.3
MPI Rank 0: 01/16/2018 19:06:13: Finished Epoch[ 2 of 15]: [Training] CrossEntropyWithSoftmax = 2.02105263 * 20480; EvalClassificationError = 0.54609375 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=0.933586s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:14: Final Results: Minibatch[1-326]: CrossEntropyWithSoftmax = 1.92733488 * 83050; perplexity = 6.87117334; EvalClassificationError = 0.53122216 * 83050
MPI Rank 0: 01/16/2018 19:06:14: Finished Epoch[ 2 of 15]: [Validate] CrossEntropyWithSoftmax = 1.92733488 * 83050; EvalClassificationError = 0.53122216 * 83050
MPI Rank 0: 01/16/2018 19:06:14: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.927335 (Epoch 2); EvalClassificationError = 0.531222 (Epoch 2)
MPI Rank 0: 01/16/2018 19:06:14: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.2'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:15: Starting Epoch 3: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:15: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:15:  Epoch[ 3 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.94336420 * 10240; EvalClassificationError = 0.53056641 * 10240; time = 0.3724s; samplesPerSecond = 27499.3
MPI Rank 0: 01/16/2018 19:06:15:  Epoch[ 3 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.96525554 * 10240; EvalClassificationError = 0.54873047 * 10240; time = 0.3533s; samplesPerSecond = 28986.4
MPI Rank 0: 01/16/2018 19:06:15: Finished Epoch[ 3 of 15]: [Training] CrossEntropyWithSoftmax = 1.95430987 * 20480; EvalClassificationError = 0.53964844 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-05; epochTime=0.729696s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:16: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.90639119 * 83050; perplexity = 6.72876211; EvalClassificationError = 0.52304636 * 83050
MPI Rank 0: 01/16/2018 19:06:16: Finished Epoch[ 3 of 15]: [Validate] CrossEntropyWithSoftmax = 1.90639119 * 83050; EvalClassificationError = 0.52304636 * 83050
MPI Rank 0: 01/16/2018 19:06:16: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.906391 (Epoch 3); EvalClassificationError = 0.523046 (Epoch 3)
MPI Rank 0: 01/16/2018 19:06:16: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.3'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:16: Starting Epoch 4: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:16: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:17:  Epoch[ 4 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.92960398 * 10240; EvalClassificationError = 0.52734375 * 10240; time = 0.3570s; samplesPerSecond = 28682.0
MPI Rank 0: 01/16/2018 19:06:17:  Epoch[ 4 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.91791093 * 10240; EvalClassificationError = 0.52138672 * 10240; time = 0.3611s; samplesPerSecond = 28361.6
MPI Rank 0: 01/16/2018 19:06:17: Finished Epoch[ 4 of 15]: [Training] CrossEntropyWithSoftmax = 1.92375746 * 20480; EvalClassificationError = 0.52436523 * 20480; totalSamplesSeen = 81920; learningRatePerSample = 9.7656251e-05; epochTime=0.722412s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:18: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.89723688 * 83050; perplexity = 6.66744604; EvalClassificationError = 0.52192655 * 83050
MPI Rank 0: 01/16/2018 19:06:18: Finished Epoch[ 4 of 15]: [Validate] CrossEntropyWithSoftmax = 1.89723688 * 83050; EvalClassificationError = 0.52192655 * 83050
MPI Rank 0: 01/16/2018 19:06:18: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.897237 (Epoch 4); EvalClassificationError = 0.521927 (Epoch 4)
MPI Rank 0: 01/16/2018 19:06:18: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.4'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:18: Starting Epoch 5: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:18: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:18:  Epoch[ 5 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.93213905 * 10240; EvalClassificationError = 0.52744141 * 10240; time = 0.3625s; samplesPerSecond = 28252.0
MPI Rank 0: 01/16/2018 19:06:19:  Epoch[ 5 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.91008045 * 10240; EvalClassificationError = 0.52197266 * 10240; time = 0.3572s; samplesPerSecond = 28665.4
MPI Rank 0: 01/16/2018 19:06:19: Finished Epoch[ 5 of 15]: [Training] CrossEntropyWithSoftmax = 1.92110975 * 20480; EvalClassificationError = 0.52470703 * 20480; totalSamplesSeen = 102400; learningRatePerSample = 9.7656251e-05; epochTime=0.724175s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:20: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.88941575 * 83050; perplexity = 6.61550243; EvalClassificationError = 0.52039735 * 83050
MPI Rank 0: 01/16/2018 19:06:20: Finished Epoch[ 5 of 15]: [Validate] CrossEntropyWithSoftmax = 1.88941575 * 83050; EvalClassificationError = 0.52039735 * 83050
MPI Rank 0: 01/16/2018 19:06:20: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.889416 (Epoch 5); EvalClassificationError = 0.520397 (Epoch 5)
MPI Rank 0: 01/16/2018 19:06:20: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.5'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:20: Starting Epoch 6: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:20: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:20:  Epoch[ 6 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.92107601 * 10240; EvalClassificationError = 0.52783203 * 10240; time = 0.3589s; samplesPerSecond = 28534.9
MPI Rank 0: 01/16/2018 19:06:20:  Epoch[ 6 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.90118051 * 10240; EvalClassificationError = 0.52031250 * 10240; time = 0.3552s; samplesPerSecond = 28829.1
MPI Rank 0: 01/16/2018 19:06:20: Finished Epoch[ 6 of 15]: [Training] CrossEntropyWithSoftmax = 1.91112826 * 20480; EvalClassificationError = 0.52407227 * 20480; totalSamplesSeen = 122880; learningRatePerSample = 9.7656251e-05; epochTime=0.717771s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:21: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.88230716 * 83050; perplexity = 6.56864231; EvalClassificationError = 0.51898856 * 83050
MPI Rank 0: 01/16/2018 19:06:21: Finished Epoch[ 6 of 15]: [Validate] CrossEntropyWithSoftmax = 1.88230716 * 83050; EvalClassificationError = 0.51898856 * 83050
MPI Rank 0: 01/16/2018 19:06:21: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.882307 (Epoch 6); EvalClassificationError = 0.518989 (Epoch 6)
MPI Rank 0: 01/16/2018 19:06:21: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.6'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:21: Starting Epoch 7: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:21: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:22:  Epoch[ 7 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.87751809 * 10240; EvalClassificationError = 0.51201172 * 10240; time = 0.3572s; samplesPerSecond = 28663.6
MPI Rank 0: 01/16/2018 19:06:22:  Epoch[ 7 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.90589643 * 10240; EvalClassificationError = 0.53007812 * 10240; time = 0.3568s; samplesPerSecond = 28701.2
MPI Rank 0: 01/16/2018 19:06:22: Finished Epoch[ 7 of 15]: [Training] CrossEntropyWithSoftmax = 1.89170726 * 20480; EvalClassificationError = 0.52104492 * 20480; totalSamplesSeen = 143360; learningRatePerSample = 9.7656251e-05; epochTime=0.71829s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:23: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.87533201 * 83050; perplexity = 6.52298444; EvalClassificationError = 0.51865141 * 83050
MPI Rank 0: 01/16/2018 19:06:23: Finished Epoch[ 7 of 15]: [Validate] CrossEntropyWithSoftmax = 1.87533201 * 83050; EvalClassificationError = 0.51865141 * 83050
MPI Rank 0: 01/16/2018 19:06:23: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.875332 (Epoch 7); EvalClassificationError = 0.518651 (Epoch 7)
MPI Rank 0: 01/16/2018 19:06:23: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.7'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:23: Starting Epoch 8: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:23: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:23:  Epoch[ 8 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.88190523 * 10240; EvalClassificationError = 0.51777344 * 10240; time = 0.3531s; samplesPerSecond = 29002.2
MPI Rank 0: 01/16/2018 19:06:24:  Epoch[ 8 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.86655063 * 10240; EvalClassificationError = 0.51562500 * 10240; time = 0.3679s; samplesPerSecond = 27835.6
MPI Rank 0: 01/16/2018 19:06:24: Finished Epoch[ 8 of 15]: [Training] CrossEntropyWithSoftmax = 1.87422793 * 20480; EvalClassificationError = 0.51669922 * 20480; totalSamplesSeen = 163840; learningRatePerSample = 9.7656251e-05; epochTime=0.725738s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:25: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.86996773 * 83050; perplexity = 6.48808705; EvalClassificationError = 0.51725467 * 83050
MPI Rank 0: 01/16/2018 19:06:25: Finished Epoch[ 8 of 15]: [Validate] CrossEntropyWithSoftmax = 1.86996773 * 83050; EvalClassificationError = 0.51725467 * 83050
MPI Rank 0: 01/16/2018 19:06:25: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.869968 (Epoch 8); EvalClassificationError = 0.517255 (Epoch 8)
MPI Rank 0: 01/16/2018 19:06:25: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.8'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:25: Starting Epoch 9: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:25: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:25:  Epoch[ 9 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.85947921 * 10240; EvalClassificationError = 0.50673828 * 10240; time = 0.3612s; samplesPerSecond = 28349.7
MPI Rank 0: 01/16/2018 19:06:26:  Epoch[ 9 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.85700426 * 10240; EvalClassificationError = 0.51582031 * 10240; time = 0.3547s; samplesPerSecond = 28872.5
MPI Rank 0: 01/16/2018 19:06:26: Finished Epoch[ 9 of 15]: [Training] CrossEntropyWithSoftmax = 1.85824174 * 20480; EvalClassificationError = 0.51127930 * 20480; totalSamplesSeen = 184320; learningRatePerSample = 9.7656251e-05; epochTime=0.720714s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:27: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.86323873 * 83050; perplexity = 6.44457525; EvalClassificationError = 0.51674895 * 83050
MPI Rank 0: 01/16/2018 19:06:27: Finished Epoch[ 9 of 15]: [Validate] CrossEntropyWithSoftmax = 1.86323873 * 83050; EvalClassificationError = 0.51674895 * 83050
MPI Rank 0: 01/16/2018 19:06:27: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.863239 (Epoch 9); EvalClassificationError = 0.516749 (Epoch 9)
MPI Rank 0: 01/16/2018 19:06:27: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.9'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:27: Starting Epoch 10: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:27: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:27:  Epoch[10 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.89317989 * 10240; EvalClassificationError = 0.52548828 * 10240; time = 0.3581s; samplesPerSecond = 28596.9
MPI Rank 0: 01/16/2018 19:06:27:  Epoch[10 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.84631301 * 10240; EvalClassificationError = 0.50986328 * 10240; time = 0.3461s; samplesPerSecond = 29588.8
MPI Rank 0: 01/16/2018 19:06:27: Finished Epoch[10 of 15]: [Training] CrossEntropyWithSoftmax = 1.86974645 * 20480; EvalClassificationError = 0.51767578 * 20480; totalSamplesSeen = 204800; learningRatePerSample = 9.7656251e-05; epochTime=0.708342s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:28: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.85695611 * 83050; perplexity = 6.40421333; EvalClassificationError = 0.51576159 * 83050
MPI Rank 0: 01/16/2018 19:06:28: Finished Epoch[10 of 15]: [Validate] CrossEntropyWithSoftmax = 1.85695611 * 83050; EvalClassificationError = 0.51576159 * 83050
MPI Rank 0: 01/16/2018 19:06:28: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.856956 (Epoch 10); EvalClassificationError = 0.515762 (Epoch 10)
MPI Rank 0: 01/16/2018 19:06:28: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.10'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:28: Starting Epoch 11: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:28: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:29:  Epoch[11 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.86460008 * 10240; EvalClassificationError = 0.50751953 * 10240; time = 0.3603s; samplesPerSecond = 28422.2
MPI Rank 0: 01/16/2018 19:06:29:  Epoch[11 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.86031159 * 10240; EvalClassificationError = 0.51816406 * 10240; time = 0.3577s; samplesPerSecond = 28628.2
MPI Rank 0: 01/16/2018 19:06:29: Finished Epoch[11 of 15]: [Training] CrossEntropyWithSoftmax = 1.86245583 * 20480; EvalClassificationError = 0.51284180 * 20480; totalSamplesSeen = 225280; learningRatePerSample = 9.7656251e-05; epochTime=0.721709s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:30: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.85008405 * 83050; perplexity = 6.36035408; EvalClassificationError = 0.51326911 * 83050
MPI Rank 0: 01/16/2018 19:06:30: Finished Epoch[11 of 15]: [Validate] CrossEntropyWithSoftmax = 1.85008405 * 83050; EvalClassificationError = 0.51326911 * 83050
MPI Rank 0: 01/16/2018 19:06:30: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.850084 (Epoch 11); EvalClassificationError = 0.513269 (Epoch 11)
MPI Rank 0: 01/16/2018 19:06:30: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.11'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:30: Starting Epoch 12: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:30: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:30:  Epoch[12 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.86700752 * 10240; EvalClassificationError = 0.51181641 * 10240; time = 0.3577s; samplesPerSecond = 28630.6
MPI Rank 0: 01/16/2018 19:06:31:  Epoch[12 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.83390766 * 10240; EvalClassificationError = 0.50585938 * 10240; time = 0.3541s; samplesPerSecond = 28915.6
MPI Rank 0: 01/16/2018 19:06:31: Finished Epoch[12 of 15]: [Training] CrossEntropyWithSoftmax = 1.85045759 * 20480; EvalClassificationError = 0.50883789 * 20480; totalSamplesSeen = 245760; learningRatePerSample = 9.7656251e-05; epochTime=0.71565s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:32: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.84352145 * 83050; perplexity = 6.31875031; EvalClassificationError = 0.51169175 * 83050
MPI Rank 0: 01/16/2018 19:06:32: Finished Epoch[12 of 15]: [Validate] CrossEntropyWithSoftmax = 1.84352145 * 83050; EvalClassificationError = 0.51169175 * 83050
MPI Rank 0: 01/16/2018 19:06:32: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.843521 (Epoch 12); EvalClassificationError = 0.511692 (Epoch 12)
MPI Rank 0: 01/16/2018 19:06:32: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.12'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:32: Starting Epoch 13: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:32: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:32:  Epoch[13 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.84005490 * 10046; EvalClassificationError = 0.51542903 * 10046; time = 0.4029s; samplesPerSecond = 24931.3
MPI Rank 0: 01/16/2018 19:06:32:  Epoch[13 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.87225994 * 10240; EvalClassificationError = 0.51484375 * 10240; time = 0.3425s; samplesPerSecond = 29899.0
MPI Rank 0: 01/16/2018 19:06:33: Finished Epoch[13 of 15]: [Training] CrossEntropyWithSoftmax = 1.85713955 * 20480; EvalClassificationError = 0.51479492 * 20480; totalSamplesSeen = 266240; learningRatePerSample = 9.7656251e-05; epochTime=0.759888s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:33: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.83713385 * 83050; perplexity = 6.27851730; EvalClassificationError = 0.50862131 * 83050
MPI Rank 0: 01/16/2018 19:06:33: Finished Epoch[13 of 15]: [Validate] CrossEntropyWithSoftmax = 1.83713385 * 83050; EvalClassificationError = 0.50862131 * 83050
MPI Rank 0: 01/16/2018 19:06:33: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.837134 (Epoch 13); EvalClassificationError = 0.508621 (Epoch 13)
MPI Rank 0: 01/16/2018 19:06:33: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.13'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:33: Starting Epoch 14: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:33: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:34:  Epoch[14 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.85347546 * 10240; EvalClassificationError = 0.50312500 * 10240; time = 0.3467s; samplesPerSecond = 29539.4
MPI Rank 0: 01/16/2018 19:06:34:  Epoch[14 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.84170081 * 10240; EvalClassificationError = 0.50791016 * 10240; time = 0.3433s; samplesPerSecond = 29832.2
MPI Rank 0: 01/16/2018 19:06:34: Finished Epoch[14 of 15]: [Training] CrossEntropyWithSoftmax = 1.84758814 * 20480; EvalClassificationError = 0.50551758 * 20480; totalSamplesSeen = 286720; learningRatePerSample = 9.7656251e-05; epochTime=0.694005s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:35: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.83143597 * 83050; perplexity = 6.24284478; EvalClassificationError = 0.50930765 * 83050
MPI Rank 0: 01/16/2018 19:06:35: Finished Epoch[14 of 15]: [Validate] CrossEntropyWithSoftmax = 1.83143597 * 83050; EvalClassificationError = 0.50930765 * 83050
MPI Rank 0: 01/16/2018 19:06:35: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.831436 (Epoch 14); EvalClassificationError = 0.508621 (Epoch 13)
MPI Rank 0: 01/16/2018 19:06:35: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.14'
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:35: Starting Epoch 15: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:35: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:36:  Epoch[15 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.81729821 * 10240; EvalClassificationError = 0.50380859 * 10240; time = 0.3525s; samplesPerSecond = 29052.1
MPI Rank 0: 01/16/2018 19:06:36:  Epoch[15 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.84154546 * 10240; EvalClassificationError = 0.51152344 * 10240; time = 0.3463s; samplesPerSecond = 29568.3
MPI Rank 0: 01/16/2018 19:06:36: Finished Epoch[15 of 15]: [Training] CrossEntropyWithSoftmax = 1.82942183 * 20480; EvalClassificationError = 0.50766602 * 20480; totalSamplesSeen = 307200; learningRatePerSample = 9.7656251e-05; epochTime=0.702998s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:37: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.82545027 * 83050; perplexity = 6.20558856; EvalClassificationError = 0.50745334 * 83050
MPI Rank 0: 01/16/2018 19:06:37: Finished Epoch[15 of 15]: [Validate] CrossEntropyWithSoftmax = 1.82545027 * 83050; EvalClassificationError = 0.50745334 * 83050
MPI Rank 0: 01/16/2018 19:06:37: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.825450 (Epoch 15); EvalClassificationError = 0.507453 (Epoch 15)
MPI Rank 0: 01/16/2018 19:06:37: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn'
MPI Rank 0: 01/16/2018 19:06:37: Best epoch for criterion 'CrossEntropyWithSoftmax' is 15 and model /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn copied to /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn_CrossEntropyWithSoftmax
MPI Rank 0: 01/16/2018 19:06:37: Best epoch for criterion 'EvalClassificationError' is 15 and model /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn copied to /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn_EvalClassificationError
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:37: Action "train" complete.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:37: __COMPLETED__
MPI Rank 1: CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:06:04
MPI Rank 1: 
MPI Rank 1: /home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DeviceId=0  timestamping=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr
MPI Rank 1: 01/16/2018 19:06:04: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:06:04: Build info: 
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:04: 		Built time: Jan 16 2018 16:15:42
MPI Rank 1: 01/16/2018 19:06:04: 		Last modified date: Tue Jan 16 16:13:51 2018
MPI Rank 1: 01/16/2018 19:06:04: 		Build type: release
MPI Rank 1: 01/16/2018 19:06:04: 		Build target: GPU
MPI Rank 1: 01/16/2018 19:06:04: 		With ASGD: yes
MPI Rank 1: 01/16/2018 19:06:04: 		Math lib: mkl
MPI Rank 1: 01/16/2018 19:06:04: 		CUDA version: 9.0.0
MPI Rank 1: 01/16/2018 19:06:04: 		CUDNN version: 7.0.4
MPI Rank 1: 01/16/2018 19:06:04: 		Build Branch: HEAD
MPI Rank 1: 01/16/2018 19:06:04: 		Build SHA1: c4c2ce8c6e89b5c32e4d07523081283417bcfc6d
MPI Rank 1: 01/16/2018 19:06:04: 		MPI distribution: Open MPI
MPI Rank 1: 01/16/2018 19:06:04: 		MPI version: 1.10.7
MPI Rank 1: 01/16/2018 19:06:04: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:06:04: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:06:04: GPU info:
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:04: 		Device[0]: cores = 3072; computeCapability = 5.2; type = "Tesla M60"; total memory = 8123 MB; free memory = 8017 MB
MPI Rank 1: 01/16/2018 19:06:04: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:06:04: Using 6 CPU threads.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:04: ##############################################################################
MPI Rank 1: 01/16/2018 19:06:04: #                                                                            #
MPI Rank 1: 01/16/2018 19:06:04: # speechTrain command (train action)                                         #
MPI Rank 1: 01/16/2018 19:06:04: #                                                                            #
MPI Rank 1: 01/16/2018 19:06:04: ##############################################################################
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:04: 
MPI Rank 1: Creating virgin network.
MPI Rank 1: SimpleNetworkBuilder Using GPU 0
MPI Rank 1: Reading script file glob_0000.scp ... 948 entries
MPI Rank 1: HTKDeserializer: selected '948' utterances grouped into '3' chunks, average chunk size: 316.0 utterances, 84244.7 frames (for I/O: 316.0 utterances, 84244.7 frames)
MPI Rank 1: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 1: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 1: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 1: Reading script file glob_0000.cv.scp ... 300 entries
MPI Rank 1: HTKDeserializer: selected '300' utterances grouped into '1' chunks, average chunk size: 300.0 utterances, 83050.0 frames (for I/O: 300.0 utterances, 83050.0 frames)
MPI Rank 1: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 1: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 1: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 1: 01/16/2018 19:06:05: 
MPI Rank 1: Model has 25 nodes. Using GPU 0.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:05: Training criterion:   CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
MPI Rank 1: 01/16/2018 19:06:05: Evaluation criterion: EvalClassificationError = ClassificationError
MPI Rank 1: 
MPI Rank 1: 
MPI Rank 1: Allocating matrices for forward and/or backward propagation.
MPI Rank 1: 
MPI Rank 1: Gradient Memory Aliasing: 4 are aliased.
MPI Rank 1: 	W2*H1 (gradient) reuses HLast (gradient)
MPI Rank 1: 	W1*H1 (gradient) reuses W1*H1+B1 (gradient)
MPI Rank 1: 
MPI Rank 1: Memory Sharing: Out of 40 matrices, 21 are shared as 5, and 19 are not shared.
MPI Rank 1: 
MPI Rank 1: Here are the ones that share memory:
MPI Rank 1: 	{ PosteriorProb : [132 x 1 x *]
MPI Rank 1: 	  ScaledLogLikelihood : [132 x 1 x *] }
MPI Rank 1: 	{ B0 : [512 x 1] (gradient)
MPI Rank 1: 	  H1 : [512 x 1 x *] }
MPI Rank 1: 	{ H2 : [512 x 1 x *]
MPI Rank 1: 	  W0*features+B0 : [512 x 1 x *]
MPI Rank 1: 	  W1 : [512 x 512] (gradient)
MPI Rank 1: 	  W1*H1 : [512 x 1 x *] }
MPI Rank 1: 	{ H1 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  H2 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  HLast : [132 x 1 x *]
MPI Rank 1: 	  W0*features : [512 x *]
MPI Rank 1: 	  W0*features : [512 x *] (gradient) }
MPI Rank 1: 	{ HLast : [132 x 1 x *] (gradient)
MPI Rank 1: 	  W0 : [512 x 363] (gradient)
MPI Rank 1: 	  W0*features+B0 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  W1*H1 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  W1*H1+B1 : [512 x 1 x *]
MPI Rank 1: 	  W1*H1+B1 : [512 x 1 x *] (gradient)
MPI Rank 1: 	  W2*H1 : [132 x 1 x *]
MPI Rank 1: 	  W2*H1 : [132 x 1 x *] (gradient) }
MPI Rank 1: 
MPI Rank 1: Here are the ones that don't share memory:
MPI Rank 1: 	{features : [363 x *]}
MPI Rank 1: 	{MeanOfFeatures : [363]}
MPI Rank 1: 	{InvStdOfFeatures : [363]}
MPI Rank 1: 	{W0 : [512 x 363]}
MPI Rank 1: 	{B0 : [512 x 1]}
MPI Rank 1: 	{W1 : [512 x 512]}
MPI Rank 1: 	{B1 : [512 x 1]}
MPI Rank 1: 	{W2 : [132 x 512]}
MPI Rank 1: 	{B2 : [132 x 1]}
MPI Rank 1: 	{labels : [132 x *]}
MPI Rank 1: 	{Prior : [132]}
MPI Rank 1: 	{EvalClassificationError : [1]}
MPI Rank 1: 	{CrossEntropyWithSoftmax : [1]}
MPI Rank 1: 	{W2 : [132 x 512] (gradient)}
MPI Rank 1: 	{LogOfPrior : [132]}
MPI Rank 1: 	{MVNormalizedFeatures : [363 x *]}
MPI Rank 1: 	{B1 : [512 x 1] (gradient)}
MPI Rank 1: 	{B2 : [132 x 1] (gradient)}
MPI Rank 1: 	{CrossEntropyWithSoftmax : [1] (gradient)}
MPI Rank 1: 
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:05: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:05: 	Node 'B0' (LearnableParameter operation) : [512 x 1]
MPI Rank 1: 01/16/2018 19:06:05: 	Node 'B1' (LearnableParameter operation) : [512 x 1]
MPI Rank 1: 01/16/2018 19:06:05: 	Node 'B2' (LearnableParameter operation) : [132 x 1]
MPI Rank 1: 01/16/2018 19:06:05: 	Node 'W0' (LearnableParameter operation) : [512 x 363]
MPI Rank 1: 01/16/2018 19:06:05: 	Node 'W1' (LearnableParameter operation) : [512 x 512]
MPI Rank 1: 01/16/2018 19:06:05: 	Node 'W2' (LearnableParameter operation) : [132 x 512]
MPI Rank 1: 
MPI Rank 1: Initializing dataParallelSGD with FP64 aggregation.
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:05: Precomputing --> 3 PreCompute nodes found.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:05: 	MeanOfFeatures = Mean()
MPI Rank 1: 01/16/2018 19:06:05: 	InvStdOfFeatures = InvStdDev()
MPI Rank 1: 01/16/2018 19:06:05: 	Prior = Mean()
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:08: Precomputing --> Completed.
MPI Rank 1: 
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:08: Starting Epoch 1: learning rate per sample = 0.015625  effective momentum = 0.900000  momentum as time constant = 607.4 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:08: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:08:  Epoch[ 1 of 15]-Minibatch[   1-  10, 3.12%]: CrossEntropyWithSoftmax = 4.62512789 * 640; EvalClassificationError = 0.94062500 * 640; time = 0.0668s; samplesPerSecond = 9586.5
MPI Rank 1: 01/16/2018 19:06:08:  Epoch[ 1 of 15]-Minibatch[  11-  20, 6.25%]: CrossEntropyWithSoftmax = 4.35619366 * 640; EvalClassificationError = 0.92343750 * 640; time = 0.0627s; samplesPerSecond = 10203.5
MPI Rank 1: 01/16/2018 19:06:08:  Epoch[ 1 of 15]-Minibatch[  21-  30, 9.38%]: CrossEntropyWithSoftmax = 3.97911998 * 640; EvalClassificationError = 0.89531250 * 640; time = 0.0616s; samplesPerSecond = 10383.9
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  31-  40, 12.50%]: CrossEntropyWithSoftmax = 3.73643568 * 640; EvalClassificationError = 0.84531250 * 640; time = 0.0629s; samplesPerSecond = 10173.0
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  41-  50, 15.62%]: CrossEntropyWithSoftmax = 3.83079081 * 640; EvalClassificationError = 0.88281250 * 640; time = 0.0622s; samplesPerSecond = 10293.6
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  51-  60, 18.75%]: CrossEntropyWithSoftmax = 3.71437690 * 640; EvalClassificationError = 0.86875000 * 640; time = 0.0623s; samplesPerSecond = 10264.9
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  61-  70, 21.88%]: CrossEntropyWithSoftmax = 3.42186231 * 640; EvalClassificationError = 0.79062500 * 640; time = 0.0612s; samplesPerSecond = 10453.6
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  71-  80, 25.00%]: CrossEntropyWithSoftmax = 3.53658053 * 640; EvalClassificationError = 0.82031250 * 640; time = 0.0619s; samplesPerSecond = 10339.7
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  81-  90, 28.12%]: CrossEntropyWithSoftmax = 3.49758018 * 640; EvalClassificationError = 0.81718750 * 640; time = 0.0627s; samplesPerSecond = 10214.0
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[  91- 100, 31.25%]: CrossEntropyWithSoftmax = 3.39996308 * 640; EvalClassificationError = 0.80468750 * 640; time = 0.0618s; samplesPerSecond = 10356.3
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 101- 110, 34.38%]: CrossEntropyWithSoftmax = 3.49445773 * 640; EvalClassificationError = 0.82500000 * 640; time = 0.0628s; samplesPerSecond = 10191.1
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 111- 120, 37.50%]: CrossEntropyWithSoftmax = 3.26676999 * 640; EvalClassificationError = 0.79218750 * 640; time = 0.0630s; samplesPerSecond = 10154.0
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 121- 130, 40.62%]: CrossEntropyWithSoftmax = 3.18870174 * 640; EvalClassificationError = 0.78906250 * 640; time = 0.0638s; samplesPerSecond = 10032.6
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 131- 140, 43.75%]: CrossEntropyWithSoftmax = 3.05687264 * 640; EvalClassificationError = 0.74687500 * 640; time = 0.0626s; samplesPerSecond = 10230.9
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 141- 150, 46.88%]: CrossEntropyWithSoftmax = 2.95594570 * 640; EvalClassificationError = 0.71875000 * 640; time = 0.0657s; samplesPerSecond = 9737.9
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 151- 160, 50.00%]: CrossEntropyWithSoftmax = 3.10219605 * 640; EvalClassificationError = 0.74062500 * 640; time = 0.0682s; samplesPerSecond = 9379.9
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 161- 170, 53.12%]: CrossEntropyWithSoftmax = 2.80745016 * 640; EvalClassificationError = 0.70625000 * 640; time = 0.0746s; samplesPerSecond = 8578.5
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 171- 180, 56.25%]: CrossEntropyWithSoftmax = 2.72061843 * 640; EvalClassificationError = 0.65468750 * 640; time = 0.0619s; samplesPerSecond = 10347.6
MPI Rank 1: 01/16/2018 19:06:09:  Epoch[ 1 of 15]-Minibatch[ 181- 190, 59.38%]: CrossEntropyWithSoftmax = 2.80425748 * 640; EvalClassificationError = 0.71718750 * 640; time = 0.0621s; samplesPerSecond = 10306.5
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 191- 200, 62.50%]: CrossEntropyWithSoftmax = 2.71253069 * 640; EvalClassificationError = 0.67812500 * 640; time = 0.0615s; samplesPerSecond = 10400.3
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 201- 210, 65.62%]: CrossEntropyWithSoftmax = 2.59360400 * 640; EvalClassificationError = 0.66093750 * 640; time = 0.0622s; samplesPerSecond = 10287.5
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 211- 220, 68.75%]: CrossEntropyWithSoftmax = 2.60386650 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0615s; samplesPerSecond = 10412.3
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 221- 230, 71.88%]: CrossEntropyWithSoftmax = 2.53706679 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0618s; samplesPerSecond = 10350.2
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 231- 240, 75.00%]: CrossEntropyWithSoftmax = 2.56177344 * 640; EvalClassificationError = 0.65625000 * 640; time = 0.0622s; samplesPerSecond = 10287.6
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 241- 250, 78.12%]: CrossEntropyWithSoftmax = 2.50118792 * 640; EvalClassificationError = 0.64218750 * 640; time = 0.0682s; samplesPerSecond = 9386.8
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 251- 260, 81.25%]: CrossEntropyWithSoftmax = 2.40119789 * 640; EvalClassificationError = 0.62500000 * 640; time = 0.0619s; samplesPerSecond = 10339.3
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 261- 270, 84.38%]: CrossEntropyWithSoftmax = 2.27491504 * 640; EvalClassificationError = 0.58906250 * 640; time = 0.0618s; samplesPerSecond = 10360.4
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 271- 280, 87.50%]: CrossEntropyWithSoftmax = 2.51724208 * 640; EvalClassificationError = 0.65781250 * 640; time = 0.0619s; samplesPerSecond = 10342.3
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 281- 290, 90.62%]: CrossEntropyWithSoftmax = 2.27797543 * 640; EvalClassificationError = 0.59687500 * 640; time = 0.0622s; samplesPerSecond = 10297.6
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 291- 300, 93.75%]: CrossEntropyWithSoftmax = 2.26017741 * 640; EvalClassificationError = 0.60937500 * 640; time = 0.0625s; samplesPerSecond = 10234.0
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 301- 310, 96.88%]: CrossEntropyWithSoftmax = 2.24735343 * 640; EvalClassificationError = 0.58437500 * 640; time = 0.0621s; samplesPerSecond = 10303.6
MPI Rank 1: 01/16/2018 19:06:10:  Epoch[ 1 of 15]-Minibatch[ 311- 320, 100.00%]: CrossEntropyWithSoftmax = 2.23665382 * 640; EvalClassificationError = 0.60625000 * 640; time = 0.0623s; samplesPerSecond = 10278.7
MPI Rank 1: 01/16/2018 19:06:10: Finished Epoch[ 1 of 15]: [Training] CrossEntropyWithSoftmax = 3.03815142 * 20480; EvalClassificationError = 0.73432617 * 20480; totalSamplesSeen = 20480; learningRatePerSample = 0.015625; epochTime=2.02917s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:12: Final Results: Minibatch[1-1299]: CrossEntropyWithSoftmax = 2.24821048 * 83050; perplexity = 9.47077252; EvalClassificationError = 0.61623119 * 83050
MPI Rank 1: 01/16/2018 19:06:12: Finished Epoch[ 1 of 15]: [Validate] CrossEntropyWithSoftmax = 2.24821048 * 83050; EvalClassificationError = 0.61623119 * 83050
MPI Rank 1: 01/16/2018 19:06:12: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 2.248210 (Epoch 1); EvalClassificationError = 0.616231 (Epoch 1)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:12: Starting Epoch 2: learning rate per sample = 0.001953  effective momentum = 0.656119  momentum as time constant = 607.5 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:12: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[   1-  10, 12.50%]: CrossEntropyWithSoftmax = 2.13894071 * 2560; EvalClassificationError = 0.56992188 * 2560; time = 0.1191s; samplesPerSecond = 21497.7
MPI Rank 1: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  11-  20, 25.00%]: CrossEntropyWithSoftmax = 2.06106261 * 2560; EvalClassificationError = 0.55664062 * 2560; time = 0.1152s; samplesPerSecond = 22227.4
MPI Rank 1: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  21-  30, 37.50%]: CrossEntropyWithSoftmax = 2.04459475 * 2560; EvalClassificationError = 0.55039063 * 2560; time = 0.1223s; samplesPerSecond = 20938.5
MPI Rank 1: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  31-  40, 50.00%]: CrossEntropyWithSoftmax = 2.03347291 * 2560; EvalClassificationError = 0.55742187 * 2560; time = 0.1174s; samplesPerSecond = 21804.6
MPI Rank 1: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  41-  50, 62.50%]: CrossEntropyWithSoftmax = 2.02079287 * 2560; EvalClassificationError = 0.54414063 * 2560; time = 0.1129s; samplesPerSecond = 22666.4
MPI Rank 1: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  51-  60, 75.00%]: CrossEntropyWithSoftmax = 1.96950012 * 2560; EvalClassificationError = 0.53085938 * 2560; time = 0.1151s; samplesPerSecond = 22245.7
MPI Rank 1: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  61-  70, 87.50%]: CrossEntropyWithSoftmax = 1.95934863 * 2560; EvalClassificationError = 0.52812500 * 2560; time = 0.1136s; samplesPerSecond = 22538.9
MPI Rank 1: 01/16/2018 19:06:13:  Epoch[ 2 of 15]-Minibatch[  71-  80, 100.00%]: CrossEntropyWithSoftmax = 1.94070839 * 2560; EvalClassificationError = 0.53125000 * 2560; time = 0.1134s; samplesPerSecond = 22577.9
MPI Rank 1: 01/16/2018 19:06:13: Finished Epoch[ 2 of 15]: [Training] CrossEntropyWithSoftmax = 2.02105263 * 20480; EvalClassificationError = 0.54609375 * 20480; totalSamplesSeen = 40960; learningRatePerSample = 0.001953125; epochTime=0.933153s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:14: Final Results: Minibatch[1-326]: CrossEntropyWithSoftmax = 1.92733488 * 83050; perplexity = 6.87117334; EvalClassificationError = 0.53122216 * 83050
MPI Rank 1: 01/16/2018 19:06:14: Finished Epoch[ 2 of 15]: [Validate] CrossEntropyWithSoftmax = 1.92733488 * 83050; EvalClassificationError = 0.53122216 * 83050
MPI Rank 1: 01/16/2018 19:06:14: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.927335 (Epoch 2); EvalClassificationError = 0.531222 (Epoch 2)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:15: Starting Epoch 3: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:15: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:15:  Epoch[ 3 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.94336420 * 10240; EvalClassificationError = 0.53056641 * 10240; time = 0.3722s; samplesPerSecond = 27515.7
MPI Rank 1: 01/16/2018 19:06:15:  Epoch[ 3 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.96525554 * 10240; EvalClassificationError = 0.54873047 * 10240; time = 0.3532s; samplesPerSecond = 28990.7
MPI Rank 1: 01/16/2018 19:06:15: Finished Epoch[ 3 of 15]: [Training] CrossEntropyWithSoftmax = 1.95430987 * 20480; EvalClassificationError = 0.53964844 * 20480; totalSamplesSeen = 61440; learningRatePerSample = 9.7656251e-05; epochTime=0.729244s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:16: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.90639119 * 83050; perplexity = 6.72876211; EvalClassificationError = 0.52304636 * 83050
MPI Rank 1: 01/16/2018 19:06:16: Finished Epoch[ 3 of 15]: [Validate] CrossEntropyWithSoftmax = 1.90639119 * 83050; EvalClassificationError = 0.52304636 * 83050
MPI Rank 1: 01/16/2018 19:06:16: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.906391 (Epoch 3); EvalClassificationError = 0.523046 (Epoch 3)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:16: Starting Epoch 4: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:16: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:17:  Epoch[ 4 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.92960398 * 10240; EvalClassificationError = 0.52734375 * 10240; time = 0.3576s; samplesPerSecond = 28638.6
MPI Rank 1: 01/16/2018 19:06:17:  Epoch[ 4 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.91791093 * 10240; EvalClassificationError = 0.52138672 * 10240; time = 0.3602s; samplesPerSecond = 28428.4
MPI Rank 1: 01/16/2018 19:06:17: Finished Epoch[ 4 of 15]: [Training] CrossEntropyWithSoftmax = 1.92375746 * 20480; EvalClassificationError = 0.52436523 * 20480; totalSamplesSeen = 81920; learningRatePerSample = 9.7656251e-05; epochTime=0.721968s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:18: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.89723688 * 83050; perplexity = 6.66744604; EvalClassificationError = 0.52192655 * 83050
MPI Rank 1: 01/16/2018 19:06:18: Finished Epoch[ 4 of 15]: [Validate] CrossEntropyWithSoftmax = 1.89723688 * 83050; EvalClassificationError = 0.52192655 * 83050
MPI Rank 1: 01/16/2018 19:06:18: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.897237 (Epoch 4); EvalClassificationError = 0.521927 (Epoch 4)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:18: Starting Epoch 5: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:18: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:18:  Epoch[ 5 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.93213905 * 10240; EvalClassificationError = 0.52744141 * 10240; time = 0.3621s; samplesPerSecond = 28277.4
MPI Rank 1: 01/16/2018 19:06:19:  Epoch[ 5 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.91008045 * 10240; EvalClassificationError = 0.52197266 * 10240; time = 0.3581s; samplesPerSecond = 28596.9
MPI Rank 1: 01/16/2018 19:06:19: Finished Epoch[ 5 of 15]: [Training] CrossEntropyWithSoftmax = 1.92110975 * 20480; EvalClassificationError = 0.52470703 * 20480; totalSamplesSeen = 102400; learningRatePerSample = 9.7656251e-05; epochTime=0.724601s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:20: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.88941575 * 83050; perplexity = 6.61550243; EvalClassificationError = 0.52039735 * 83050
MPI Rank 1: 01/16/2018 19:06:20: Finished Epoch[ 5 of 15]: [Validate] CrossEntropyWithSoftmax = 1.88941575 * 83050; EvalClassificationError = 0.52039735 * 83050
MPI Rank 1: 01/16/2018 19:06:20: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.889416 (Epoch 5); EvalClassificationError = 0.520397 (Epoch 5)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:20: Starting Epoch 6: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:20: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:20:  Epoch[ 6 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.92107601 * 10240; EvalClassificationError = 0.52783203 * 10240; time = 0.3594s; samplesPerSecond = 28490.7
MPI Rank 1: 01/16/2018 19:06:20:  Epoch[ 6 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.90118051 * 10240; EvalClassificationError = 0.52031250 * 10240; time = 0.3543s; samplesPerSecond = 28898.8
MPI Rank 1: 01/16/2018 19:06:20: Finished Epoch[ 6 of 15]: [Training] CrossEntropyWithSoftmax = 1.91112826 * 20480; EvalClassificationError = 0.52407227 * 20480; totalSamplesSeen = 122880; learningRatePerSample = 9.7656251e-05; epochTime=0.71817s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:21: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.88230716 * 83050; perplexity = 6.56864231; EvalClassificationError = 0.51898856 * 83050
MPI Rank 1: 01/16/2018 19:06:21: Finished Epoch[ 6 of 15]: [Validate] CrossEntropyWithSoftmax = 1.88230716 * 83050; EvalClassificationError = 0.51898856 * 83050
MPI Rank 1: 01/16/2018 19:06:21: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.882307 (Epoch 6); EvalClassificationError = 0.518989 (Epoch 6)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:21: Starting Epoch 7: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:21: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:22:  Epoch[ 7 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.87751809 * 10240; EvalClassificationError = 0.51201172 * 10240; time = 0.3576s; samplesPerSecond = 28633.0
MPI Rank 1: 01/16/2018 19:06:22:  Epoch[ 7 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.90589643 * 10240; EvalClassificationError = 0.53007812 * 10240; time = 0.3568s; samplesPerSecond = 28696.5
MPI Rank 1: 01/16/2018 19:06:22: Finished Epoch[ 7 of 15]: [Training] CrossEntropyWithSoftmax = 1.89170726 * 20480; EvalClassificationError = 0.52104492 * 20480; totalSamplesSeen = 143360; learningRatePerSample = 9.7656251e-05; epochTime=0.718722s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:23: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.87533201 * 83050; perplexity = 6.52298444; EvalClassificationError = 0.51865141 * 83050
MPI Rank 1: 01/16/2018 19:06:23: Finished Epoch[ 7 of 15]: [Validate] CrossEntropyWithSoftmax = 1.87533201 * 83050; EvalClassificationError = 0.51865141 * 83050
MPI Rank 1: 01/16/2018 19:06:23: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.875332 (Epoch 7); EvalClassificationError = 0.518651 (Epoch 7)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:23: Starting Epoch 8: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:23: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:23:  Epoch[ 8 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.88190523 * 10240; EvalClassificationError = 0.51777344 * 10240; time = 0.3526s; samplesPerSecond = 29039.7
MPI Rank 1: 01/16/2018 19:06:24:  Epoch[ 8 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.86655063 * 10240; EvalClassificationError = 0.51562500 * 10240; time = 0.3687s; samplesPerSecond = 27773.4
MPI Rank 1: 01/16/2018 19:06:24: Finished Epoch[ 8 of 15]: [Training] CrossEntropyWithSoftmax = 1.87422793 * 20480; EvalClassificationError = 0.51669922 * 20480; totalSamplesSeen = 163840; learningRatePerSample = 9.7656251e-05; epochTime=0.725308s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:25: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.86996773 * 83050; perplexity = 6.48808705; EvalClassificationError = 0.51725467 * 83050
MPI Rank 1: 01/16/2018 19:06:25: Finished Epoch[ 8 of 15]: [Validate] CrossEntropyWithSoftmax = 1.86996773 * 83050; EvalClassificationError = 0.51725467 * 83050
MPI Rank 1: 01/16/2018 19:06:25: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.869968 (Epoch 8); EvalClassificationError = 0.517255 (Epoch 8)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:25: Starting Epoch 9: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:25: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:25:  Epoch[ 9 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.85947921 * 10240; EvalClassificationError = 0.50673828 * 10240; time = 0.3616s; samplesPerSecond = 28317.2
MPI Rank 1: 01/16/2018 19:06:26:  Epoch[ 9 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.85700426 * 10240; EvalClassificationError = 0.51582031 * 10240; time = 0.3547s; samplesPerSecond = 28869.8
MPI Rank 1: 01/16/2018 19:06:26: Finished Epoch[ 9 of 15]: [Training] CrossEntropyWithSoftmax = 1.85824174 * 20480; EvalClassificationError = 0.51127930 * 20480; totalSamplesSeen = 184320; learningRatePerSample = 9.7656251e-05; epochTime=0.720311s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:27: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.86323873 * 83050; perplexity = 6.44457525; EvalClassificationError = 0.51674895 * 83050
MPI Rank 1: 01/16/2018 19:06:27: Finished Epoch[ 9 of 15]: [Validate] CrossEntropyWithSoftmax = 1.86323873 * 83050; EvalClassificationError = 0.51674895 * 83050
MPI Rank 1: 01/16/2018 19:06:27: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.863239 (Epoch 9); EvalClassificationError = 0.516749 (Epoch 9)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:27: Starting Epoch 10: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:27: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:27:  Epoch[10 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.89317989 * 10240; EvalClassificationError = 0.52548828 * 10240; time = 0.3585s; samplesPerSecond = 28560.7
MPI Rank 1: 01/16/2018 19:06:27:  Epoch[10 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.84631301 * 10240; EvalClassificationError = 0.50986328 * 10240; time = 0.3452s; samplesPerSecond = 29660.4
MPI Rank 1: 01/16/2018 19:06:27: Finished Epoch[10 of 15]: [Training] CrossEntropyWithSoftmax = 1.86974645 * 20480; EvalClassificationError = 0.51767578 * 20480; totalSamplesSeen = 204800; learningRatePerSample = 9.7656251e-05; epochTime=0.707901s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:28: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.85695611 * 83050; perplexity = 6.40421333; EvalClassificationError = 0.51576159 * 83050
MPI Rank 1: 01/16/2018 19:06:28: Finished Epoch[10 of 15]: [Validate] CrossEntropyWithSoftmax = 1.85695611 * 83050; EvalClassificationError = 0.51576159 * 83050
MPI Rank 1: 01/16/2018 19:06:28: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.856956 (Epoch 10); EvalClassificationError = 0.515762 (Epoch 10)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:28: Starting Epoch 11: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:28: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:29:  Epoch[11 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.86460008 * 10240; EvalClassificationError = 0.50751953 * 10240; time = 0.3607s; samplesPerSecond = 28390.5
MPI Rank 1: 01/16/2018 19:06:29:  Epoch[11 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.86031159 * 10240; EvalClassificationError = 0.51816406 * 10240; time = 0.3568s; samplesPerSecond = 28696.8
MPI Rank 1: 01/16/2018 19:06:29: Finished Epoch[11 of 15]: [Training] CrossEntropyWithSoftmax = 1.86245583 * 20480; EvalClassificationError = 0.51284180 * 20480; totalSamplesSeen = 225280; learningRatePerSample = 9.7656251e-05; epochTime=0.722115s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:30: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.85008405 * 83050; perplexity = 6.36035408; EvalClassificationError = 0.51326911 * 83050
MPI Rank 1: 01/16/2018 19:06:30: Finished Epoch[11 of 15]: [Validate] CrossEntropyWithSoftmax = 1.85008405 * 83050; EvalClassificationError = 0.51326911 * 83050
MPI Rank 1: 01/16/2018 19:06:30: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.850084 (Epoch 11); EvalClassificationError = 0.513269 (Epoch 11)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:30: Starting Epoch 12: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:30: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:30:  Epoch[12 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.86700752 * 10240; EvalClassificationError = 0.51181641 * 10240; time = 0.3581s; samplesPerSecond = 28597.7
MPI Rank 1: 01/16/2018 19:06:31:  Epoch[12 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.83390766 * 10240; EvalClassificationError = 0.50585938 * 10240; time = 0.3533s; samplesPerSecond = 28985.9
MPI Rank 1: 01/16/2018 19:06:31: Finished Epoch[12 of 15]: [Training] CrossEntropyWithSoftmax = 1.85045759 * 20480; EvalClassificationError = 0.50883789 * 20480; totalSamplesSeen = 245760; learningRatePerSample = 9.7656251e-05; epochTime=0.716038s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:32: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.84352145 * 83050; perplexity = 6.31875031; EvalClassificationError = 0.51169175 * 83050
MPI Rank 1: 01/16/2018 19:06:32: Finished Epoch[12 of 15]: [Validate] CrossEntropyWithSoftmax = 1.84352145 * 83050; EvalClassificationError = 0.51169175 * 83050
MPI Rank 1: 01/16/2018 19:06:32: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.843521 (Epoch 12); EvalClassificationError = 0.511692 (Epoch 12)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:32: Starting Epoch 13: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:32: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:32:  Epoch[13 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.84005490 * 10046; EvalClassificationError = 0.51542903 * 10046; time = 0.4025s; samplesPerSecond = 24960.9
MPI Rank 1: 01/16/2018 19:06:32:  Epoch[13 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.87225994 * 10240; EvalClassificationError = 0.51484375 * 10240; time = 0.3425s; samplesPerSecond = 29901.1
MPI Rank 1: 01/16/2018 19:06:33: Finished Epoch[13 of 15]: [Training] CrossEntropyWithSoftmax = 1.85713955 * 20480; EvalClassificationError = 0.51479492 * 20480; totalSamplesSeen = 266240; learningRatePerSample = 9.7656251e-05; epochTime=0.759483s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:33: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.83713385 * 83050; perplexity = 6.27851730; EvalClassificationError = 0.50862131 * 83050
MPI Rank 1: 01/16/2018 19:06:33: Finished Epoch[13 of 15]: [Validate] CrossEntropyWithSoftmax = 1.83713385 * 83050; EvalClassificationError = 0.50862131 * 83050
MPI Rank 1: 01/16/2018 19:06:33: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.837134 (Epoch 13); EvalClassificationError = 0.508621 (Epoch 13)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:33: Starting Epoch 14: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:33: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:34:  Epoch[14 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.85347546 * 10240; EvalClassificationError = 0.50312500 * 10240; time = 0.3472s; samplesPerSecond = 29492.1
MPI Rank 1: 01/16/2018 19:06:34:  Epoch[14 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.84170081 * 10240; EvalClassificationError = 0.50791016 * 10240; time = 0.3433s; samplesPerSecond = 29829.5
MPI Rank 1: 01/16/2018 19:06:34: Finished Epoch[14 of 15]: [Training] CrossEntropyWithSoftmax = 1.84758814 * 20480; EvalClassificationError = 0.50551758 * 20480; totalSamplesSeen = 286720; learningRatePerSample = 9.7656251e-05; epochTime=0.694436s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:35: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.83143597 * 83050; perplexity = 6.24284478; EvalClassificationError = 0.50930765 * 83050
MPI Rank 1: 01/16/2018 19:06:35: Finished Epoch[14 of 15]: [Validate] CrossEntropyWithSoftmax = 1.83143597 * 83050; EvalClassificationError = 0.50930765 * 83050
MPI Rank 1: 01/16/2018 19:06:35: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.831436 (Epoch 14); EvalClassificationError = 0.508621 (Epoch 13)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:35: Starting Epoch 15: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:35: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:36:  Epoch[15 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.81729821 * 10240; EvalClassificationError = 0.50380859 * 10240; time = 0.3530s; samplesPerSecond = 29008.3
MPI Rank 1: 01/16/2018 19:06:36:  Epoch[15 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.84154546 * 10240; EvalClassificationError = 0.51152344 * 10240; time = 0.3464s; samplesPerSecond = 29563.5
MPI Rank 1: 01/16/2018 19:06:36: Finished Epoch[15 of 15]: [Training] CrossEntropyWithSoftmax = 1.82942183 * 20480; EvalClassificationError = 0.50766602 * 20480; totalSamplesSeen = 307200; learningRatePerSample = 9.7656251e-05; epochTime=0.703433s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:37: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.82545027 * 83050; perplexity = 6.20558856; EvalClassificationError = 0.50745334 * 83050
MPI Rank 1: 01/16/2018 19:06:37: Finished Epoch[15 of 15]: [Validate] CrossEntropyWithSoftmax = 1.82545027 * 83050; EvalClassificationError = 0.50745334 * 83050
MPI Rank 1: 01/16/2018 19:06:37: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.825450 (Epoch 15); EvalClassificationError = 0.507453 (Epoch 15)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:37: Action "train" complete.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:37: __COMPLETED__
=== Deleting last epoch data
==== Re-running from checkpoint
=== Running mpiexec -n 2 /home/ubuntu/workspace/build/gpu/release/bin/cntk configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu DeviceId=0 timestamping=true makeMode=true numCPUThreads=6 shareNodeValueMatrices=true saveBestModelPerCriterion=true stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr
CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:06:37

/home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DeviceId=0  timestamping=true  makeMode=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr
Changed current directory to /home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data
CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:06:37

/home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DeviceId=0  timestamping=true  makeMode=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr
Changed current directory to /home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data
--------------------------------------------------------------------------
[[31088,1],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: 7fee1579d8b2

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
ping [requestnodes (before change)]: 2 nodes pinging each other
ping [requestnodes (before change)]: 2 nodes pinging each other
ping [requestnodes (after change)]: 2 nodes pinging each other
requestnodes [MPIWrapperMpi]: using 2 out of 2 MPI nodes on a single host (2 requested); we (1) are in (participating)
ping [mpihelper]: 2 nodes pinging each other
ping [requestnodes (after change)]: 2 nodes pinging each other
requestnodes [MPIWrapperMpi]: using 2 out of 2 MPI nodes on a single host (2 requested); we (0) are in (participating)
ping [mpihelper]: 2 nodes pinging each other
01/16/2018 19:06:37: Redirecting stderr to file /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr_speechTrain.logrank0
01/16/2018 19:06:38: Redirecting stderr to file /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr_speechTrain.logrank1
[7fee1579d8b2:65526] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics
[7fee1579d8b2:65526] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
MPI Rank 0: CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:06:37
MPI Rank 0: 
MPI Rank 0: /home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DeviceId=0  timestamping=true  makeMode=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr
MPI Rank 0: 01/16/2018 19:06:37: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:06:37: Build info: 
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:37: 		Built time: Jan 16 2018 16:15:42
MPI Rank 0: 01/16/2018 19:06:37: 		Last modified date: Tue Jan 16 16:13:51 2018
MPI Rank 0: 01/16/2018 19:06:37: 		Build type: release
MPI Rank 0: 01/16/2018 19:06:37: 		Build target: GPU
MPI Rank 0: 01/16/2018 19:06:37: 		With ASGD: yes
MPI Rank 0: 01/16/2018 19:06:37: 		Math lib: mkl
MPI Rank 0: 01/16/2018 19:06:37: 		CUDA version: 9.0.0
MPI Rank 0: 01/16/2018 19:06:37: 		CUDNN version: 7.0.4
MPI Rank 0: 01/16/2018 19:06:37: 		Build Branch: HEAD
MPI Rank 0: 01/16/2018 19:06:37: 		Build SHA1: c4c2ce8c6e89b5c32e4d07523081283417bcfc6d
MPI Rank 0: 01/16/2018 19:06:37: 		MPI distribution: Open MPI
MPI Rank 0: 01/16/2018 19:06:37: 		MPI version: 1.10.7
MPI Rank 0: 01/16/2018 19:06:37: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:06:37: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:06:37: GPU info:
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:37: 		Device[0]: cores = 3072; computeCapability = 5.2; type = "Tesla M60"; total memory = 8123 MB; free memory = 8112 MB
MPI Rank 0: 01/16/2018 19:06:37: -------------------------------------------------------------------
MPI Rank 0: 01/16/2018 19:06:37: Using 6 CPU threads.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:37: ##############################################################################
MPI Rank 0: 01/16/2018 19:06:37: #                                                                            #
MPI Rank 0: 01/16/2018 19:06:37: # speechTrain command (train action)                                         #
MPI Rank 0: 01/16/2018 19:06:37: #                                                                            #
MPI Rank 0: 01/16/2018 19:06:37: ##############################################################################
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:37: 
MPI Rank 0: Starting from checkpoint. Loading network from '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.14'.
MPI Rank 0: SimpleNetworkBuilder Using GPU 0
MPI Rank 0: Reading script file glob_0000.scp ... 948 entries
MPI Rank 0: HTKDeserializer: selected '948' utterances grouped into '3' chunks, average chunk size: 316.0 utterances, 84244.7 frames (for I/O: 316.0 utterances, 84244.7 frames)
MPI Rank 0: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 0: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 0: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 0: Reading script file glob_0000.cv.scp ... 300 entries
MPI Rank 0: HTKDeserializer: selected '300' utterances grouped into '1' chunks, average chunk size: 300.0 utterances, 83050.0 frames (for I/O: 300.0 utterances, 83050.0 frames)
MPI Rank 0: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 0: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 0: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 0: 01/16/2018 19:06:37: 
MPI Rank 0: Model has 25 nodes. Using GPU 0.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:37: Training criterion:   CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
MPI Rank 0: 01/16/2018 19:06:37: Evaluation criterion: EvalClassificationError = ClassificationError
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:37: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:37: 	Node 'B0' (LearnableParameter operation) : [512 x 1]
MPI Rank 0: 01/16/2018 19:06:37: 	Node 'B1' (LearnableParameter operation) : [512 x 1]
MPI Rank 0: 01/16/2018 19:06:37: 	Node 'B2' (LearnableParameter operation) : [132 x 1]
MPI Rank 0: 01/16/2018 19:06:37: 	Node 'W0' (LearnableParameter operation) : [512 x 363]
MPI Rank 0: 01/16/2018 19:06:37: 	Node 'W1' (LearnableParameter operation) : [512 x 512]
MPI Rank 0: 01/16/2018 19:06:37: 	Node 'W2' (LearnableParameter operation) : [132 x 512]
MPI Rank 0: 
MPI Rank 0: Initializing dataParallelSGD with FP64 aggregation.
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:38: No PreCompute nodes found, or all already computed. Skipping pre-computation step.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:38: Starting Epoch 15: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:38: Starting minibatch loop, DataParallelSGD training (myRank = 0, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 0: 01/16/2018 19:06:39:  Epoch[15 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.81729821 * 10240; EvalClassificationError = 0.50380859 * 10240; time = 1.0405s; samplesPerSecond = 9841.3
MPI Rank 0: 01/16/2018 19:06:39:  Epoch[15 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.84154546 * 10240; EvalClassificationError = 0.51152344 * 10240; time = 0.3553s; samplesPerSecond = 28817.2
MPI Rank 0: 01/16/2018 19:06:39: Finished Epoch[15 of 15]: [Training] CrossEntropyWithSoftmax = 1.82942183 * 20480; EvalClassificationError = 0.50766602 * 20480; totalSamplesSeen = 307200; learningRatePerSample = 9.7656251e-05; epochTime=1.42926s
MPI Rank 0: NcclComm: disabled, same device used by more than one rank
MPI Rank 0: 01/16/2018 19:06:41: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.82545027 * 83050; perplexity = 6.20558856; EvalClassificationError = 0.50745334 * 83050
MPI Rank 0: 01/16/2018 19:06:41: Finished Epoch[15 of 15]: [Validate] CrossEntropyWithSoftmax = 1.82545027 * 83050; EvalClassificationError = 0.50745334 * 83050
MPI Rank 0: 01/16/2018 19:06:41: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.825450 (Epoch 15); EvalClassificationError = 0.507453 (Epoch 15)
MPI Rank 0: 01/16/2018 19:06:41: SGD: Saving checkpoint model '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn'
MPI Rank 0: 01/16/2018 19:06:41: Best epoch for criterion 'CrossEntropyWithSoftmax' is 15 and model /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn copied to /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn_CrossEntropyWithSoftmax
MPI Rank 0: 01/16/2018 19:06:41: Best epoch for criterion 'EvalClassificationError' is 15 and model /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn copied to /tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn_EvalClassificationError
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:41: Action "train" complete.
MPI Rank 0: 
MPI Rank 0: 01/16/2018 19:06:41: __COMPLETED__
MPI Rank 1: CNTK 2.3.1+ (HEAD c4c2ce, Jan 16 2018 16:21:59) at 2018/01/16 19:06:37
MPI Rank 1: 
MPI Rank 1: /home/ubuntu/workspace/build/gpu/release/bin/cntk  configFile=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion/cntkcv.cntk  currentDirectory=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  RunDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DataDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data  ConfigDir=/home/ubuntu/workspace/Tests/EndToEndTests/Speech/DNN/SaveBestModelPerCriterion  OutputDir=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu  DeviceId=0  timestamping=true  makeMode=true  numCPUThreads=6  shareNodeValueMatrices=true  saveBestModelPerCriterion=true  stderr=/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/stderr
MPI Rank 1: 01/16/2018 19:06:38: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:06:38: Build info: 
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:38: 		Built time: Jan 16 2018 16:15:42
MPI Rank 1: 01/16/2018 19:06:38: 		Last modified date: Tue Jan 16 16:13:51 2018
MPI Rank 1: 01/16/2018 19:06:38: 		Build type: release
MPI Rank 1: 01/16/2018 19:06:38: 		Build target: GPU
MPI Rank 1: 01/16/2018 19:06:38: 		With ASGD: yes
MPI Rank 1: 01/16/2018 19:06:38: 		Math lib: mkl
MPI Rank 1: 01/16/2018 19:06:38: 		CUDA version: 9.0.0
MPI Rank 1: 01/16/2018 19:06:38: 		CUDNN version: 7.0.4
MPI Rank 1: 01/16/2018 19:06:38: 		Build Branch: HEAD
MPI Rank 1: 01/16/2018 19:06:38: 		Build SHA1: c4c2ce8c6e89b5c32e4d07523081283417bcfc6d
MPI Rank 1: 01/16/2018 19:06:38: 		MPI distribution: Open MPI
MPI Rank 1: 01/16/2018 19:06:38: 		MPI version: 1.10.7
MPI Rank 1: 01/16/2018 19:06:38: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:06:38: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:06:38: GPU info:
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:38: 		Device[0]: cores = 3072; computeCapability = 5.2; type = "Tesla M60"; total memory = 8123 MB; free memory = 8019 MB
MPI Rank 1: 01/16/2018 19:06:38: -------------------------------------------------------------------
MPI Rank 1: 01/16/2018 19:06:38: Using 6 CPU threads.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:38: ##############################################################################
MPI Rank 1: 01/16/2018 19:06:38: #                                                                            #
MPI Rank 1: 01/16/2018 19:06:38: # speechTrain command (train action)                                         #
MPI Rank 1: 01/16/2018 19:06:38: #                                                                            #
MPI Rank 1: 01/16/2018 19:06:38: ##############################################################################
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:38: 
MPI Rank 1: Starting from checkpoint. Loading network from '/tmp/cntk-test-20180116190516.17566/Speech/DNN_SaveBestModelPerCriterion@release_gpu/models/cntkSpeech.dnn.14'.
MPI Rank 1: SimpleNetworkBuilder Using GPU 0
MPI Rank 1: Reading script file glob_0000.scp ... 948 entries
MPI Rank 1: HTKDeserializer: selected '948' utterances grouped into '3' chunks, average chunk size: 316.0 utterances, 84244.7 frames (for I/O: 316.0 utterances, 84244.7 frames)
MPI Rank 1: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 1: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 1: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 1: Reading script file glob_0000.cv.scp ... 300 entries
MPI Rank 1: HTKDeserializer: selected '300' utterances grouped into '1' chunks, average chunk size: 300.0 utterances, 83050.0 frames (for I/O: 300.0 utterances, 83050.0 frames)
MPI Rank 1: HTKDeserializer: determined feature kind as '33'-dimensional 'USER' with frame shift 10.0 ms
MPI Rank 1: Total (133) state names in state list '/home/ubuntu/workspace/Tests/EndToEndTests/Speech/Data/state.list'
MPI Rank 1: MLFDeserializer: '948' utterances with '252734' frames
MPI Rank 1: 01/16/2018 19:06:38: 
MPI Rank 1: Model has 25 nodes. Using GPU 0.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:38: Training criterion:   CrossEntropyWithSoftmax = CrossEntropyWithSoftmax
MPI Rank 1: 01/16/2018 19:06:38: Evaluation criterion: EvalClassificationError = ClassificationError
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:38: Training 516740 parameters in 6 out of 6 parameter tensors and 15 nodes with gradient:
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:38: 	Node 'B0' (LearnableParameter operation) : [512 x 1]
MPI Rank 1: 01/16/2018 19:06:38: 	Node 'B1' (LearnableParameter operation) : [512 x 1]
MPI Rank 1: 01/16/2018 19:06:38: 	Node 'B2' (LearnableParameter operation) : [132 x 1]
MPI Rank 1: 01/16/2018 19:06:38: 	Node 'W0' (LearnableParameter operation) : [512 x 363]
MPI Rank 1: 01/16/2018 19:06:38: 	Node 'W1' (LearnableParameter operation) : [512 x 512]
MPI Rank 1: 01/16/2018 19:06:38: 	Node 'W2' (LearnableParameter operation) : [132 x 512]
MPI Rank 1: 
MPI Rank 1: Initializing dataParallelSGD with FP64 aggregation.
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:38: No PreCompute nodes found, or all already computed. Skipping pre-computation step.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:38: Starting Epoch 15: learning rate per sample = 0.000098  effective momentum = 0.656119  momentum as time constant = 2429.9 samples
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:38: Starting minibatch loop, DataParallelSGD training (myRank = 1, numNodes = 2, numGradientBits = 64), distributed reading is ENABLED.
MPI Rank 1: 01/16/2018 19:06:39:  Epoch[15 of 15]-Minibatch[   1-  10, 50.00%]: CrossEntropyWithSoftmax = 1.81729821 * 10240; EvalClassificationError = 0.50380859 * 10240; time = 1.0415s; samplesPerSecond = 9832.4
MPI Rank 1: 01/16/2018 19:06:39:  Epoch[15 of 15]-Minibatch[  11-  20, 100.00%]: CrossEntropyWithSoftmax = 1.84154546 * 10240; EvalClassificationError = 0.51152344 * 10240; time = 0.3545s; samplesPerSecond = 28883.6
MPI Rank 1: 01/16/2018 19:06:39: Finished Epoch[15 of 15]: [Training] CrossEntropyWithSoftmax = 1.82942183 * 20480; EvalClassificationError = 0.50766602 * 20480; totalSamplesSeen = 307200; learningRatePerSample = 9.7656251e-05; epochTime=1.42967s
MPI Rank 1: NcclComm: disabled, same device used by more than one rank
MPI Rank 1: 01/16/2018 19:06:41: Final Results: Minibatch[1-83]: CrossEntropyWithSoftmax = 1.82545027 * 83050; perplexity = 6.20558856; EvalClassificationError = 0.50745334 * 83050
MPI Rank 1: 01/16/2018 19:06:41: Finished Epoch[15 of 15]: [Validate] CrossEntropyWithSoftmax = 1.82545027 * 83050; EvalClassificationError = 0.50745334 * 83050
MPI Rank 1: 01/16/2018 19:06:41: Best epoch per criterion so far: [Validate] CrossEntropyWithSoftmax = 1.825450 (Epoch 15); EvalClassificationError = 0.507453 (Epoch 15)
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:41: Action "train" complete.
MPI Rank 1: 
MPI Rank 1: 01/16/2018 19:06:41: __COMPLETED__