Skip to content

Conversation

jyegerlehner
Copy link
Contributor

The problem is that in ConvolutionLayer::Backward_[gpu,cpu], the pointer "weight" is set only if the this->param_propagate_down_[0] is true, otherwise it is left NULL. However "weight" is used if either this->param_propagate_down_[0] or propagate_down[i] is true. So in the case where the first bool is false but the second is true, we end up adding weight offset to a NULL "weight". This caused a seg fault running on a CPU and cuda error 13 running on GPU.

I hit this in a scenario where I was doing layerwise training of a convolutional autoencoder. The learning rate for the already-trained layers was set to zero, and the newly-added layers had non-zero learning rate.

@jeffdonahue
Copy link
Contributor

Eep, nice catch, and thanks for the fix!

jeffdonahue added a commit that referenced this pull request Sep 8, 2014
Conv layer: fix crash by setting weight pointer
@jeffdonahue jeffdonahue merged commit 5ab3d97 into BVLC:dev Sep 8, 2014
@jyegerlehner jyegerlehner deleted the conv_layer-init-weight branch September 8, 2014 06:28
mitmul pushed a commit to mitmul/caffe that referenced this pull request Sep 30, 2014
Conv layer: fix crash by setting weight pointer
shelhamer added a commit that referenced this pull request Oct 1, 2014
fix NULL pointer when back-propagating to the data for fixed i.e. not
back-propagated weights. same issue as #1048
@shelhamer
Copy link
Member

Fixed for CuDNNConvolutionLayer in a02ec65.

mitmul pushed a commit to mitmul/caffe that referenced this pull request Oct 11, 2014
fix NULL pointer when back-propagating to the data for fixed i.e. not
back-propagated weights. same issue as BVLC#1048
RazvanRanca pushed a commit to RazvanRanca/caffe that referenced this pull request Nov 4, 2014
Conv layer: fix crash by setting weight pointer
RazvanRanca pushed a commit to RazvanRanca/caffe that referenced this pull request Nov 4, 2014
fix NULL pointer when back-propagating to the data for fixed i.e. not
back-propagated weights. same issue as BVLC#1048
cbfinn pushed a commit to cbfinn/caffe that referenced this pull request Feb 26, 2015
fix NULL pointer when back-propagating to the data for fixed i.e. not
back-propagated weights. same issue as BVLC#1048
slayton58 pushed a commit to slayton58/caffe that referenced this pull request Mar 4, 2015
fix NULL pointer when back-propagating to the data for fixed i.e. not
back-propagated weights. same issue as BVLC#1048
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants