Libraries and applications for typical computer vision and perception processes.

You need librealsense: https://github.com/IntelRealSense/librealsense/releases/tag/v2.56.3

Rest of deps can be installed with:

sudo apt install cmake libopencv-dev libyaml-cpp-dev liblcm-dev libglfw3 libgtk-3-dev libcli11-dev -y

For testing (optional but recommended):

sudo apt install libgtest-dev libgmock-dev -y

# TODO: Maybe we should standardize to just copying the drake include like the linux-arm64 steps below?

Drake can be installed on debian based systems with https://drake.mit.edu/apt.html#stable-releases.

Build spdlog on your system: https://github.com/gabime/spdlog?tab=readme-ov-file#compiled-version-recommended---much-faster-compile-times. Ex:

git clone https://github.com/gabime/spdlog.git && cd spdlog && mkdir build && cd build &&  cmake .. && cmake --build .

linux-arm64 setup:

Drake isn't supported on linux arm yet, but we only need the lcm types. If on arm based systems where it isn't supported, you can run the following (from within this dir):

cp -rf drake /opt/drake

Note that we do not attempt a find_package call, and are explicitly expecting that it is in the /opt/drake/include path. Currently, the 1.38.0-1 version of drake-dev is the only version that has been tested. Also, you need to cp the spdlog include to your /usr/include path.

When you have the necessary dependencies installed, you can use the run_build script to build.

For example, to build release, just run it without any args:

./run_build

Build in debug mode:

./run_build Debug

Build and run tests:

./run_build --test

Build in debug mode with tests:

./run_build Debug --test

Run the test suite to verify your build:

cd build
ctest --output-on-failure

Or use the convenient build script with --test flag:

./run_build --test

For more details on testing, see tests/README.md.

To build docker images, you need docker buildx:

sudo apt install docker-buildx -y

If you want to build an x86_64 image on a x86_64 machine:

./run_x86_64_docker_build

If you want to build an aarch64 image on a aarch64 machine:

./run_aarch64_docker_build

Note, if you are using realsense cameras, you need librealsense2-dkms installed on your host machine (as it is a kernel module). Note that you need to provide <ABSOLUTE_CONFIG_FOLDER_PATH> and <YOUR_SENSOR_FILE> below. You have to mount an entire dir, so it is recommended you create a folder with your sensor configs, such as /opt/sensor_configs. Also note that you can replace the x86_64 image with the image for whatever architecture you are running.

docker run --rm -v <ABSOLUTE_CONFIG_FOLDER_PATH>:<ABSOLUTE_CONFIG_FOLDER_PATH> --privileged --network=host theia_sensor_service_x86_64:latest -c <ABSOLUTE_CONFIG_FOLDER_PATH>/<YOUR_SENSOR_FILE> -l trace

After building, you can simply run:

./build/theia_sensor_service -c <PATH_TO_CONFIG_YAML_FILE>

Example config for librealsense camera:

type: realsense
serial: 102422070700
color_stream_channel: /realsense/0/color
depth_stream_channel: /realsense/0/depth
enable_color: true
enable_depth: true

Example config for v4l2 camera:

type: v4l2 # this is default if not provided
color_stream_channel: /webcam/0/color
device: 6
# TODO: should support having update rate be separately configurable from fps
# rather than having fps govern update rate and camera setting
fps: 25
width: 800
height: 600
# TODO: MJPG should be treated same as MJPEG 
pixel_format: MJPEG
compress_as_jpeg: true

Running a channel with lcm logs enabled:

LCM_DBG=lc_msg  ./build/apps/visualize_stream /devpi/webcam/0/color

Setup conda env:

conda env create -f environment.yml

You can use the v4l2-ctl tool (provided by the v4l-utils package) to query your webcam's capabilities.

In all below examples you can use the syntax -d <PATH_TO_YOUR_VIDEO_DEVICE> to specify a specific webcam. Ex. -d /dev/video or just -d 0.

For example, to list all supported formats, frame sizes, and pixel formats, run:

v4l2-ctl --list-formats-ext

You can also see the current video format your webcam is in (below is assuming you have a /dev/video0):

v4l2-ctl --get-fmt-video

You can set your webcam's video format like so (confirm supported settings from the --list-formats-ext command):

v4l2-ctl --set-fmt-video=width=640,height=480,pixelformat=YUYV

To see all the available controls (like brightness, contrast, exposure, etc.) along with their current values and ranges, use:

v4l2-ctl --list-ctrls

For a comprehensive dump of all information about the device, you can run:

v4l2-ctl --all

These commands will help you determine what configurations your webcam supports and what values you can use.

You can disable auto features (like auto exposure and auto white balance) by using v4l2-ctl to set the corresponding control values to their manual settings. For example, many UVC cameras support controls such as exposure_auto and white_balance_temperature_auto. First, list the controls to see what values they accept:

v4l2-ctl --list-ctrls

In the output you might see something like:

User Controls

                     brightness 0x00980900 (int)    : min=0 max=255 step=1 default=128 value=128
                       contrast 0x00980901 (int)    : min=0 max=255 step=1 default=32 value=32
                     saturation 0x00980902 (int)    : min=0 max=100 step=1 default=64 value=64
                            hue 0x00980903 (int)    : min=-180 max=180 step=1 default=0 value=0
        white_balance_automatic 0x0098090c (bool)   : default=1 value=1
                          gamma 0x00980910 (int)    : min=90 max=150 step=1 default=120 value=120
           power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=1 value=1 (50 Hz)
      white_balance_temperature 0x0098091a (int)    : min=2800 max=6500 step=10 default=4600 value=4600 flags=inactive
                      sharpness 0x0098091b (int)    : min=0 max=7 step=1 default=0 value=0
         backlight_compensation 0x0098091c (int)    : min=0 max=2 step=1 default=1 value=1

Camera Controls

                  auto_exposure 0x009a0901 (menu)   : min=0 max=3 default=3 value=1 (Manual Mode)
         exposure_time_absolute 0x009a0902 (int)    : min=2 max=1250 step=1 default=156 value=156
     exposure_dynamic_framerate 0x009a0903 (bool)   : default=0 value=1

To update webcam settings you can use the following syntac. Ex. to set auto_exposure from above to 1:

v4l2-ctl --set-ctrl auto_exposure=1

Keep in mind that the actual values depend on your camera’s driver. If your camera uses different enumerations (for example, manual might be 1), adjust the values accordingly. Use the output from --list-ctrls to determine the correct settings for your device.