Extracting stereo pairs from 180 VR camerasWritten by Paul Bourke
Image example courtesy of Ed Bates
In the current climate of interest in virtual reality and head mounted displays there are a number of cameras on the market designed to capture video for these devices. One category does not attempt to create full 360 stereoscopic panoramas, since that is frankly rather difficult to do well, but rather a left and right eye fisheye pair. Within a head mounted display one gets a stereoscopic 3D depth effect including peripheral vision but without the ability to look around.
While these cameras are perfectly suited to creating content for head mounted displays, the fisheye projections are not suitable for viewing on flat screen stereoscopic projection systems or televisions displays. However the fisheyes so captured obviously include the necessary material for the limited field of view of these flat screen stereoscopic systems. All that is required is to extract the perspective projection at the desired field of view.
The remainder of this document presents an example of this and the workflow one might engage. It is based upon the fish2persp utility by the author that can be implemented in a number of ways, including CPU based off-line processing, GPU as a GLSL shader, or as lookup remap images for ffmpeg or Nuke.
The fish2persp command line utility allows one to extract a perspective projection of any reasonable horizontal field of view (-t option), and orientated in any direction within the fisheye field of view. For example, to extract a perspective view from the center one might use:
fish2persp -w 2100 -h 1200 -s 190 -r 1480 -c 1448 1436 -t 100 cam_left.jpg fish2persp -w 2100 -h 1200 -s 190 -r 1480 -c 1414 1426 -t 100 cam_right.jpg
For the meaning of the other command line options the reader should refer to the documentation on the fish2persp web page.
Once the fisheye is extracted then the stereoscopic pairs need to be aligned, in much the same way as one would align images from a standard stereoscopic camera. Any vertical misalignment corrected, this is typically due to inaccuracies in the camera/lens/sensor mechanical alignment. Secondly the zero parallax distance needs to be set, that is, the distance that will appear at the screen depth. This is typically an object close to the camera to reduce negative parallax objects cutting the display boundary. After selecting a feature to reside at the screen depth, one translates the stereo pairs so that this feature has zero parallax, in aligned in the two stereo pair images. After these two procedures the images would typically be cropped to the final desired resolution, in this case 1920x1080, note this alignment and cropping is the reason for choosing slightly larger perspective views of 2100x1200 in the command options (-w and -h).
An example looking down on the bench.
fish2persp -w 2100 -h 1200 -s 190 -r 1480 -c 1448 1436 -t 100 -x 30 cam_left.jpg fish2persp -w 2100 -h 1200 -s 190 -r 1480 -c 1414 1426 -t 100 -x 30 cam_right.jpg
While the example above is using the CPU post processing approach, with a GLSL shader the above could readily be performed in real time allowing the viewer to interactively navigate within the fisheye bounds.Notes