After double check, I found the current matcaffe #2505 can perform arbitrary size input at the cost that you reshape the net in matlab every time you forward.

e.g.
net.blobs('data').reshape([W H C N]);

So it's user's choice. Personally, I prefer the way of the caffe-fast-rcnn matcaffe. So I'll keep this PR open :)

Personally I prefer the current way since it is consistent with pycaffe.

Note: after calling net.blobs('data').reshape([W H C N]);, you are required to call net.reshape() to reshape all subsequent layers it is ready to call net.forward({..}).

In the previous matcaffe, the input image can be of arbitrary size

I don't remember that. Maybe it is supported in a custom branch?

When we want to get a response map of the input data, matcaffe sends the input data to Caffe and receives output data from Caffe, so
net.blobs('data').reshape([W H C N]) is enough. net.reshape() is not necessary.

So it is in matlab/+caffe/private/caffe_.cpp

I don't remember that. Maybe it is supported in a custom branch?

Sorry I made it confusing. It was caffe-fast-rcnn branch.

When we want to get a response map of the input data, matcaffe sends the input data to Caffe and receives output data from Caffe, so net.blobs('data').reshape([W H C N]) is enough. net.reshape() is not necessary.

Yes, you are right. I see that all layers get reshaped automatically when you call net.forward(), since in https://github.com/BVLC/caffe/blob/master/include/caffe/layer.hpp#L409 every layer reshape itself before running forward.