Skip to content

Use gflags to parse command line arguments for convert_imageset #857

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Aug 28, 2014
Merged

Use gflags to parse command line arguments for convert_imageset #857

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a0a5bb1
convert_imageset now uses gflags; optional arguments can be positioned
netheril96 Aug 5, 2014
42ebee5
Handles gflags's change of namespace
netheril96 Aug 5, 2014
e1d6655
Fix lint errors
netheril96 Aug 5, 2014
75f2938
Reordering of header includes for convert_imageset.cpp
netheril96 Aug 6, 2014
2288d28
Use gflags to show help when the arguments not correct
netheril96 Aug 10, 2014
df7fe6f
create_imagenet.sh updated to new syntax
netheril96 Aug 28, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions examples/imagenet/create_imagenet.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,17 +36,21 @@ fi
echo "Creating train leveldb..."

GLOG_logtostderr=1 $TOOLS/convert_imageset.bin \
--resize_height=$RESIZE_HEIGHT \
--resize_width=$RESIZE_WIDTH \
--shuffle \
$TRAIN_DATA_ROOT \
$DATA/train.txt \
ilsvrc12_train_leveldb 1
$RESIZE_HEIGHT $RESIZE_WIDTH
ilsvrc12_train_leveldb

echo "Creating val leveldb..."

GLOG_logtostderr=1 $TOOLS/convert_imageset.bin \
--resize_height=$RESIZE_HEIGHT \
--resize_width=$RESIZE_WIDTH \
--shuffle \
$VAL_DATA_ROOT \
$DATA/val.txt \
ilsvrc12_val_leveldb 1
$RESIZE_HEIGHT $RESIZE_WIDTH
ilsvrc12_val_leveldb

echo "Done."
70 changes: 36 additions & 34 deletions tools/convert_imageset.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
// single-channel grayscale. If omitted, grayscale images will be
// converted to color.

#include <gflags/gflags.h>
#include <glog/logging.h>
#include <leveldb/db.h>
#include <leveldb/write_batch.h>
Expand All @@ -34,53 +35,54 @@ using namespace caffe; // NOLINT(build/namespaces)
using std::pair;
using std::string;

DEFINE_bool(gray, false,
"When this option is on, treat images as grayscale ones");
DEFINE_bool(shuffle, false,
"Randomly shuffle the order of images and their labels");
DEFINE_string(backend, "leveldb", "The backend for storing the result");
DEFINE_int32(resize_width, 0, "Width images are resized to");
DEFINE_int32(resize_height, 0, "Height images are resized to");

int main(int argc, char** argv) {
::google::InitGoogleLogging(argv[0]);
if (argc < 4 || argc > 9) {
printf("Convert a set of images to the leveldb format used\n"
"as input for Caffe.\n"

#ifndef GFLAGS_GFLAGS_H_
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may no longer be needed after the most recent bugfix.

namespace gflags = google;
#endif

gflags::SetUsageMessage("Convert a set of images to the leveldb/lmdb\n"
"format used as input for Caffe.\n"
"Usage:\n"
" convert_imageset [-g] ROOTFOLDER/ LISTFILE DB_NAME"
" RANDOM_SHUFFLE_DATA[0 or 1] DB_BACKEND[leveldb or lmdb]"
" [resize_height] [resize_width]\n"
" convert_imageset [FLAGS] ROOTFOLDER/ LISTFILE DB_NAME\n"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of keeping some of the names still hard-coded, you can use gflags for them too. It would make the command longer but is usually acceptable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ROOTFOLDER/ LISTFILE DB_NAME seems to me mandatory arguments, and thus not quite fit to be parsed by gflags altogether.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand. However using gflags is still better - we are thinking of merging everything into a single caffe binary, and different utilities having different mandatory arguments implemented as argv will still be hard to maintain (also, they will become non-mandatory in the single caffe binary :)).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Yangqing
I personally think that this utility does not belong to caffe.bin since currently that utility deals with the network (training, testing, etc.) while this utility simply prepares images into certain format.

However, I can integrate this change into caffe.bin if you insist.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@netheril96 I think keeping this a separate utility, that also uses gflags,
makes sense as the caffe tool is for taking actions with nets.

@Yangqing thoughts?

On Sunday, August 10, 2014, netheril96 [email protected] wrote:

In tools/convert_imageset.cpp:

     "Usage:\n"

  •    "    convert_imageset [-g] ROOTFOLDER/ LISTFILE DB_NAME"

  •    " RANDOM_SHUFFLE_DATA[0 or 1] DB_BACKEND[leveldb or lmdb]"

  •    " [resize_height] [resize_width]\n"

  •    "    convert_imageset [FLAGS] ROOTFOLDER/ LISTFILE DB_NAME\n"

@Yangqing https://github.com/Yangqing
I personally think that this utility does not belong to caffe.bin since
currently that utility deals with the network (training, testing, etc.)
while this utility simply prepares images into certain format.

However, I can integrate this change into caffe.bin if you insist.

—
Reply to this email directly or view it on GitHub
https://github.com/BVLC/caffe/pull/857/files#r16031584.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am OK both ways. Also, I don't have a strong opinion against default arguments although I personally like everything to be with gflags. It makes bookkeeping easier in the future (as we have already experienced pain of unnamed arguments).

"The ImageNet dataset for the training demo is at\n"
" http://www.image-net.org/download-images\n");
gflags::ParseCommandLineFlags(&argc, &argv, true);

if (argc != 4) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(see above - if we use all gflags, we won't need to set a custom help, which may be simpler.)

gflags::ShowUsageWithFlagsRestrict(argv[0], "tools/convert_imageset");
return 1;
}

// Test whether argv[1] == "-g"
bool is_color= !(string("-g") == string(argv[1]));
int arg_offset = (is_color ? 0 : 1);
std::ifstream infile(argv[arg_offset+2]);
bool is_color = !FLAGS_gray;
std::ifstream infile(argv[2]);
std::vector<std::pair<string, int> > lines;
string filename;
int label;
while (infile >> filename >> label) {
lines.push_back(std::make_pair(filename, label));
}
if (argc >= (arg_offset+5) && argv[arg_offset+4][0] == '1') {
if (FLAGS_shuffle) {
// randomly shuffle data
LOG(INFO) << "Shuffling data";
shuffle(lines.begin(), lines.end());
}
LOG(INFO) << "A total of " << lines.size() << " images.";

string db_backend = "leveldb";
if (argc >= (arg_offset+6)) {
db_backend = string(argv[arg_offset+5]);
if (!(db_backend == "leveldb") && !(db_backend == "lmdb")) {
LOG(FATAL) << "Unknown db backend " << db_backend;
}
}
const string& db_backend = FLAGS_backend;
const char* db_path = argv[3];

int resize_height = 0;
int resize_width = 0;
if (argc >= (arg_offset+7)) {
resize_height = atoi(argv[arg_offset+6]);
}
if (argc >= (arg_offset+8)) {
resize_width = atoi(argv[arg_offset+7]);
}
int resize_height = std::max<int>(0, FLAGS_resize_height);
int resize_width = std::max<int>(0, FLAGS_resize_width);

// Open new db
// lmdb
Expand All @@ -98,19 +100,19 @@ int main(int argc, char** argv) {

// Open db
if (db_backend == "leveldb") { // leveldb
LOG(INFO) << "Opening leveldb " << argv[arg_offset+3];
LOG(INFO) << "Opening leveldb " << db_path;
leveldb::Status status = leveldb::DB::Open(
options, argv[arg_offset+3], &db);
CHECK(status.ok()) << "Failed to open leveldb " << argv[arg_offset+3];
options, db_path, &db);
CHECK(status.ok()) << "Failed to open leveldb " << db_path;
batch = new leveldb::WriteBatch();
} else if (db_backend == "lmdb") { // lmdb
LOG(INFO) << "Opening lmdb " << argv[arg_offset+3];
CHECK_EQ(mkdir(argv[arg_offset+3], 0744), 0)
<< "mkdir " << argv[arg_offset+3] << "failed";
LOG(INFO) << "Opening lmdb " << db_path;
CHECK_EQ(mkdir(db_path, 0744), 0)
<< "mkdir " << db_path << "failed";
CHECK_EQ(mdb_env_create(&mdb_env), MDB_SUCCESS) << "mdb_env_create failed";
CHECK_EQ(mdb_env_set_mapsize(mdb_env, 1099511627776), MDB_SUCCESS) // 1TB
<< "mdb_env_set_mapsize failed";
CHECK_EQ(mdb_env_open(mdb_env, argv[3], 0, 0664), MDB_SUCCESS)
CHECK_EQ(mdb_env_open(mdb_env, db_path, 0, 0664), MDB_SUCCESS)
<< "mdb_env_open failed";
CHECK_EQ(mdb_txn_begin(mdb_env, NULL, 0, &mdb_txn), MDB_SUCCESS)
<< "mdb_txn_begin failed";
Expand All @@ -121,7 +123,7 @@ int main(int argc, char** argv) {
}

// Storing to db
string root_folder(argv[arg_offset+1]);
string root_folder(argv[1]);
Datum datum;
int count = 0;
const int kMaxKeyLength = 256;
Expand Down