Extracting stereo pairs from 180 VR cameras

Introduction

In the current climate of interest in virtual reality and head mounted displays there are a number of cameras on the market designed to capture video for these devices. One category does not attempt to create full 360 stereoscopic panoramas, since that is frankly rather difficult to do well, but rather a left and right eye fisheye pair. Within a head mounted display one gets a stereoscopic 3D depth effect including peripheral vision but without the ability to look around.

While these cameras are perfectly suited to creating content for head mounted displays, the fisheye projections are not suitable for viewing on flat screen stereoscopic projection systems or televisions displays. However the fisheyes so captured obviously include the necessary material for the limited field of view of these flat screen stereoscopic systems. All that is required is to extract the perspective projection at the desired field of view.

Example

The remainder of this document presents an example of this and the workflow one might engage. It is based upon the fish2persp utility by the author that can be implemented in a number of ways, including CPU based off-line processing, GPU as a GLSL shader, or as lookup remap images for ffmpeg or Nuke.

Left camera

Right camera

The fish2persp command line utility allows one to extract a perspective projection of any reasonable horizontal field of view (-t option), and orientated in any direction within the fisheye field of view. For example, to extract a perspective view from the center one might use:

   fish2persp -w 2100 -h 1200 -s 190 -r 1480 -c 1448 1436 -t 100 cam_left.jpg
   fish2persp -w 2100 -h 1200 -s 190 -r 1480 -c 1414 1426 -t 100 cam_right.jpg

For the meaning of the other command line options the reader should refer to the documentation on the fish2persp web page.

Once the fisheye is extracted then the stereoscopic pairs need to be aligned, in much the same way as one would align images from a standard stereoscopic camera. Any vertical misalignment corrected, this is typically due to inaccuracies in the camera/lens/sensor mechanical alignment. Secondly the zero parallax distance needs to be set, that is, the distance that will appear at the screen depth. This is typically an object close to the camera to reduce negative parallax objects cutting the display boundary. After selecting a feature to reside at the screen depth, one translates the stereo pairs so that this feature has zero parallax, in aligned in the two stereo pair images. After these two procedures the images would typically be cropped to the final desired resolution, in this case 1920x1080, note this alignment and cropping is the reason for choosing slightly larger perspective views of 2100x1200 in the command options (-w and -h).

Left eye

Right eye

An example looking down on the bench.

   fish2persp -w 2100 -h 1200 -s 190 -r 1480 -c 1448 1436 -t 100 -x 30 cam_left.jpg
   fish2persp -w 2100 -h 1200 -s 190 -r 1480 -c 1414 1426 -t 100 -x 30 cam_right.jpg

Left eye

Right eye

While the example above is using the CPU post processing approach, with a GLSL shader the above could readily be performed in real time allowing the viewer to interactively navigate within the fisheye bounds.

• The correct perspective field of view depends on the distance of the viewer from the projection screen and the width of the projection screen. That is, the viewers field of view to the screen should match the extracted field of view of the perspective projection.

• One is generally going to want a high resolution fisheye capture system since the final resolution of the perspective image will only be a fraction of that, depending on the perspective field of view extracted.

• Towards the boundary of the fisheye view vertical parallax and image feature compression may occur due to non-linearities in the physical fisheye lens. These are readily corrected for, see fisheye lens correction.