I'm getting segfaults from write proto with large models. It seems that protobuf simply doesn't support writing messages greater than 2gb (marbl/harvest-tools#3). Does anyone have a workaround they would suggest? It would be nice if we added a check on the message size to give a nicer error message if nothing else.

Here are the details:

I0228 14:22:15.306717 23672 solver.cpp:355] Snapshotting to /snapshots/caffe_iter_10.caffemodel

Program received signal SIGSEGV, Segmentation fault.
0x000000000047fa74 in caffe::LayerParameter::GetCachedSize (
this=0x903d621f263c2284) at .build_debug/src/caffe/proto/caffe.pb.h:2063

The backtrace from gdb:
#0 0x000000000047fa74 in caffe::LayerParameter::GetCachedSize (

this=0x903d621f263c2284) at .build_debug/src/caffe/proto/caffe.pb.h:2063

#1 0x000000000048cedd in google::protobuf::internal::WireFormatLite::WriteMessageNoVirtualToArraycaffe::LayerParameter (field_number=2, value=...,

target=0x2305c18b "<\v\a\237\274>\025\254\274r\376\245<g̪<\003`\332<n\345\061=\241\374\202\273>ֶ\274J2|=\201\300\326\067\203\211.=\212\035\371\274\224\337B\275\025&ɼ") at /usr/include/google/protobuf/wire_format_lite_inl.h:708

#2 0x0000000000433cbe in caffe::NetParameter::SerializeWithCachedSizesToArray

(this=0x7fffffffd160, 
target=0x2305c18a "\022<\v\a\237\274>\025\254\274r\376\245<g̪<\003`\332<n\345\061=\241\374\202\273>ֶ\274J2|=\201\300\326\067\203\211.=\212\035\371\274\224\337B\275\025&ɼ") at .build_debug/src/caffe/proto/caffe.pb.cc:3530

#3 0x00007ffff2094d0c in google::protobuf::MessageLite::SerializePartialToCodedStream(google::protobuf::io::CodedOutputStream*) const ()

from /usr/lib/x86_64-linux-gnu/libprotobuf.so.8
#4 0x00007ffff2094dc5 in google::protobuf::MessageLite::SerializeToCodedStream(google::protobuf::io::CodedOutputStream*) const ()

from /usr/lib/x86_64-linux-gnu/libprotobuf.so.8
#5 0x00007ffff2094f01 in google::protobuf::MessageLite::SerializeToZeroCopyStream(google::protobuf::io::ZeroCopyOutputStream*) const ()

from /usr/lib/x86_64-linux-gnu/libprotobuf.so.8
#6 0x00007ffff20ea20b in google::protobuf::Message::SerializeToOstream(std::ostream*) const () from /usr/lib/x86_64-linux-gnu/libprotobuf.so.8

---Type to continue, or q to quit---
#7 0x00000000004a98fe in caffe::WriteProtoToBinaryFile (proto=...,

filename=0x193f2c68 "/snapshots/caffe_iter_10.caffemodel")
at src/caffe/util/io.cpp:66

#8 0x0000000000497249 in caffe::Solver::Snapshot (this=0x50f9e90)

at src/caffe/solver.cpp:356