As discussed in #64 I am integrating RS code into cam + IMU calibration.

One of the first things I noted is a possible error in the code. At https://github.com/othlu/kalibr/blob/master/aslam_offline_calibration/kalibr/python/kalibr_rs_camera_calibration/RsCalibrator.py#L316 we see there is a test for successful observation, which only checks if there is at least one successful corner observation (see GridCalibrationTargetObservation::hasSuccessfulObservation).

But above at https://github.com/othlu/kalibr/blob/master/aslam_offline_calibration/kalibr/python/kalibr_rs_camera_calibration/RsCalibrator.py#L276 it was stated that the order of observations has to be preserved. As an observation is only a vector of corners (see GridCalibrationTargetObservation::getCornersImageFrame), if there is any missing corner it means the indexes of the following corners will be changed, in the worst case (first corner missing), all of them will be changed and all the errors for the frame will be setup wrong in the jacobian.

The confirmation of this can be seen at https://github.com/ethz-asl/kalibr/blob/master/aslam_cv/aslam_cameras/src/GridCalibrationTargetObservation.cpp#L75-L78

I think the fix is either:

• the test should discard the frame if any of the observations are missing;
• the observations should be passed in a hash so that their order is not impacted by missing detections at a certain frame.

That would make the dataset creation and camera calibration much more robust.

Do you think it makes sense?