Copyright 2012, Reagents of The University Of Michigan

This is a readme for the Michigan Embedded Vision Benchmark (MEVBench).

What is MEVBench?

It is a set of benchmarks that are targeted towards embedded computer vision.


Platforms Tested On:
	Ubuntu 11.04 on x86 with OpenCV 2.4.2 and ARM A9 with OpenCV2.3.1 ( OpenCV 2.4.2 should also work on ARM but I have not tested it)


Required libraries:

OpenCV 2.4.2 *^
Pthreads




* Note that it runs with OpenCV2.3.1 also. 
^This release officially drops OpenCV 2.2 support.  Partially because it is a pain getting 2.2 to compile on 11.10+.  If using 2.2 on ARM you will need to enable SIFT by modifying the sift.cpp in the OpenCV source and then building with CMAKE.  You need to comment out the lines that disable SIFT when ARM is detected.  You will also need to make sure XTRA_PARAMS has -DOPENCV_VER_2_2



This package includes:


	Feature Extraction:

		SIFT               (N* Threads)
		SURF               (N Theads)
		FAST with Brief    (N Threads)
		HoG                (N Threads)


	Feature Classification:

		SVM                 (N THreads)
		K-nearest neighbor  (N threads)
		Adaboost            (N Threads)


	Application Examples:

		Object Localization (Feature extraction and classification) (N Threads)
		Augmented Reality                                           (1 Thread)
		Face Detection                                              (1 Thread)

*The base build MEVBench structure only has config files for 1,2,4 or 8 threads.  In order to use a different number of threads, a new config file needs to be made.


Directory Structure:
	MEVBench
            |
            |
	    |- Benchmarks - Holds the Binaries and Sources
            |      |
                   |- Applications
                          |
                          |-AugmentedReality - Source
                                    |
                                    |-bin  - Binaries
					|-arm - armv7a Binaries
                                    |-obj  - obj files
                          |-FaceDetection - Source
                                    |
                                    |-bin  - Binaries
					|-arm - armv7a Binaries
                                    |-obj  - obj files
                          |-ObjectRecogition - Source
                                    |
                                    |-bin  - Binaries
					|-arm - armv7a Binaries
                                    |-obj  - obj files
                   |- Feature Classification - Source
                          |-bin  - Binaries
				|-arm - armv7a Binaries
                          |-obj  - obj files

                   |- Feature Extraction - Source
                          |-bin  - Binaries
				|-arm - armv7a Binaries
                          |-obj  - obj files

	    |- Common - Holds commond scripts like run all
            |      

	    |- Configs - Holds the Config Files for the algorithms
                   |- Applications
                          |
                          |-AugmentedReality - Config Files
                                    |
                          |-FaceDetection - Config Files
                                    |
                          |-ObjectRecogition - Config Files
                                    
                   |- Feature Classification - 
                          |
                          |-BOOST - Config Files
                                    |
                          |-KNN - Config Files
                                    |
                          |-SVM - Config Files
                   |- Feature Extraction 
                          |
                          |-SIFT - Config Files
                                    |
                          |-HoG - Config Files
                                    |
                          |-SURF - Config Files
                                    |
                          |-FAST - Config Files

            |      

	    |- Data - Holds the Data for the benchmarks
            |      

	    |- Licenses - Holds License Files
            |      

	    |- Results - This is where timing results are stored
            |      

	    |- Tools - This is where timing results are stored
                          |- AddFeatureClassificationToDescFile - Tool to agument feature descriptor file with classifications
                          |
                          | - CameraCalibration - Tool to create a calibration file for augmented reality if you want to add your own images
                          | - pattern.png - pattern for performing calibration
                          | - ARMarker.pdf - Marker for Augmented Reality
                                             

Building MEVBench:


Build OpenCV 2.4.2:
	Download OpenCV 2.4.2 and use cmake to cross compile for Arm (Directions can be found at opencv.willowgarage.com)
		*This has been tested with the native builds static and dynamic.  Arm has not been tested but is expected to work.  To use on ARM add the -DOPENCV_2_4 to the ARM_XTRA_PARAMS in build.sh to be safe.
	Note: When building static OpenCV for native ubuntu, you will need build 3rd party libraries.  In particular using cmake set it to build: jasper, jpeg, png, zlib, and tiff libraries.  Also build with 1394, eigen, jasper, jpeg, png, tiff, unicap.  Make sure to uncheck Build shared libs and build package.


Build OpenCV 2.3.1:
	Download OpenCV 2.3.1 and use cmake to cross compile for Arm (Directions can be found at opencv.willowgarage.com and opencv.org)
		*You will need to remove the -DOPENCV_2_4 from XTRA_PARAMS in build.sh


<No Longer Supported>
Build OpenCV 2.2:
	Download OpenCV 2.2 and use cmake to build (Directions can be found at opencv.willowgarage.com)
		If you plan on doing static builds or runs please build a separate static set of OpenCV Libraries



Setup Required Variables in build.sh:
	Define OpenCV_DIR as the OpenCV Dynamically Linked Install Directory
		export OpenCV_DIR=~/Documents/<OpenCV2.4.2InstallBase>		
	Define OpenCV_STATIC_DIR as the OpenCV Statically Linked Install Directory
		export OpenCV_DIR=~/Documents/<OpenCV2.4.2StaticInstallBase
	Define OpenCV_ARM_DIR as the OpenCV Armv7a Dynamically Linked Install Directory
		export OpenCV_ARM_DIR="$HOME/Documents/OpenCV/<OpenCV2.3.1ArmInstallBase>"
	Define OpenCV_ARM_STATIC_DIR as the OpenCV Armv7a Statically Linked Install Directory
		export OpenCV_ARM_STATIC_DIR="$HOME/Documents/OpenCV/<OpenCV2.3.1ArmStaticInstallBase>"

	Add OpenCV Library Directory to the LD_LIBRARY_PATH before running
		export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$OpenCV_DIR/lib


Build the Benchmark:
	In the MEVBench Root:
		./build.sh static
		  or
		./build.sh dynamic

	For more information run build.sh with no parameters

Run the Benchmark(Does not work for cross compiled versions because we assume you will run them in a simulator or on target other than where the cross compiled versions were built):
	In the MEVBench Root:
   		./run.sh all
                 or ./run.sh for a list of options.
	


Next Update Will Include:
	Automatic Checker For results.

How to check results:
	
	First let me say I am aware of the need to check results.  The result checking was not automated because of time constraints.  Instead for each algorithm I would compare the single threaded result with any multithreaded.
	I would determine if it was reasonable and if they matched well.  This can be done visually by turning the windows on for single threaded.  
	
	Feature Extraction:  There are quite a few feature points.  Thus comparing to the list from the single threaded execuation was how I did it.  

	Feature Classification:  Once again comparison to the single threaded

	Augmented Reality:  A red cube should be projected on the marker.  Turn on the window and check.

	Face Detection:  Typcally only 1 face although it can have 2 false alarms as well.

	Object Recognition:  Look at the output window.  Also the calculated transform or center point will help.

	Once again I will be adding automatic checking in the future.  But for now I have been using knowledge of the algorithms to check them

Download the Paper:  http://www.eecs.umich.edu/MEVBench

Contact Us: [email protected]


Special Thanks to:

OpenCV Community
GSRC
NSF
SD-VBS  and Dr Taylor
Rob Hess for allowing us the use of his SIFT Library  (Coming Soon).
Marius Muja et al. for allowing the use of their FLANN library.




Release Log:


v.03 - Update to OpenCV Support
	- Adds OpenCV 2.4.2 support
	- Drops OpenCV 2.2 support due to difficulty building on Ubuntu 11.10+.  If you want to use 2.2, you will need to use MEVBench v.02


v.02 - Bug Fixes and Cross Compiling
	- Fixes directory missing build bug
	- Fixes augmented reality smallest size config error
	- Adds in cross compiling for armv7a
	- Adds in initial suppot for GEM5 Simulator operations

v.01 - Initial Release