Conversion of images from OneShot camera

Photographic 360 degree capture devices such as the "360 One VR" manufactured by EyeSee360 and Kaidan are based upon conical mirrors mounted in front of the lens of an ordinary still or video camera. They allow one to capture the whole 360 horizontal field of view and a limited vertical field of view, typically around 100 degrees. The resulting image looks not unlike a panoramic image but with a different radial distortion. Obviously one wants to undistort these images to create more conventional views, in the discussion here that includes panoramic images, and cubic (perspective) maps.

The suppliers of these capture devices typically provide stand alone software or plug-ins for other commercial image manipulation packages, eg: PhotoShop. They don't generally provide source code so that the undistortion can be integrated into a customers software. The goal of this exercise was to verify the geometric undistortion required was correctly understood and to demonstrate that with efficient and portable algorithms written in C/C++.

Transformation to a panoramic image is relatively straightforward in theory, lines of longitude in the panoramic are concentric circles in the oneshot image. Lines of latitude in the panoramic are radial lines in the oneshot image. In practice the oneshot image is not necessary centered and the inner and outer ray rims need to be specified.

Original image

Supersampling antialiasing of 3 oneshot2pan -a 3

Specify center of cone oneshot2pan -a 3 -c 330 250

Specify inner and outer radius oneshot2pan -a 3 -c 330 250 -r1 70 -r2 224

Change direction oneshot2pan -a 3 -c 330 250 -r1 70 -r2 224 -d

Force the height of the panoramic oneshot2pan -a 3 -c 330 250 -r1 70 -r2 224 -w 800 -h 200

Subset of longitude oneshot2pan -a 3 -c 330 250 -r1 70 -r2 224 -long1 -45 -long2 90

Antialising is achieved using supersampling with a rectangular weighted window. The right horizontal axis of the panoramic is taken as 0 degrees longitude. Options are also provided to create a subset of the panoramic image, namely, the panoramic image between two lines of longitude.

oneshot2pan [options] tgafile
Options:
   -long1  n   start longitude value
   -long2  n   start longitude value
   -r1     n   inner radius
   -r2     n   outer radius
   -c    x y   center of cone
   -a      n   set antialias level
   -w      n   width of the output image
   -h      n   height of the output image
   -i          apply vertical flip
   -d          change direction

oneshot2cube -a 3 -c 330 250 -r1 70 -r2 224 -lat1 -60 -lat2 0

Command line usage

oneshot2cube [options] tgafile
Options:
   -r1     n   inner radius
   -r2     n   outer radius
   -c    x y   center of cone
   -a      n   set antialias level
   -w      n   width of the output image
   -h      n   height of the output image
   -l1     n   latitude at r1
   -l2     n   latitude at r2
   -i          apply vertical flip

Standard perspective projections can be considered to be longitude subsets (less than 90 degrees wide) taken from the cubic map. Ideally the cubic map would be aligned so that edges didn't not fall within the perspective frustum (not yet implemented). Of course one could derive the perspective views directly without using this cubic map approach.

oneshot2cube2 -a 3 -c 330 250 -r1 70 -r2 224 -lat1 -60 -lat2 0 -long1 270 -long2 360

oneshot2cube2 -a 3 -c 330 250 -r1 70 -r2 224 -lat1 -60 -lat2 0 -long1 180 -long2 360

oneshot2cube2 [options] tgafile
Options:
   -r1     n   inner radius
   -r2     n   outer radius
   -c    x y   center of cone
   -a      n   set antialias level
   -w      n   width of the output image
   -h      n   height of the output image
   -lat1   n   latitude at r1
   -lat2   n   latitude at r2
   -long1  n   start longitude
   -long2  n   stop longitude
   -i          apply vertical flip

A Catadioptric Sensor with Multiple Viewpoints
Libor Spacek
Department of Computer Science, University of Essex

Single viewpoint catadioptric cameras.
Baker, S., Nayar, S., 2001.
PV01. pp. 39-71

Omnidirectional vision: Theory and algorithms.
Daniilidis, K., Geyer, C.,
In: ICPR00. Vol. 1. 2000

A unifying theory for central panoramic systems and practical applications.
Geyer, C., Daniilidis, K.,
ECCV00. 2000

Non-metric calibration of wide-angle lenses and polycameras.
Swaminathan, R., Nayar, S. K.,
IEEE Transactions on Pattern Analysis and Machine Intelligence

Introduction The lomography fisheye camera claims to be the worlds cheapest fisheye (film) camera, the following discusses the suitability of using it to create images for planetarium style domes. One shouldn't be expecting a professional level camera/optics, the question is what can be achieved with a US$40 camera.		Lomography fisheye camera
Example The first thing one notices is the fisheye is truncated at the top and bottom of the frame. A real shame since it wouldn't seem like it is a requirement, one would imagine it might be just as easy to arrange the lens so it is inset with the 35mm image frame which would greatly improve the product. Aperture: 170 degrees.
Most of the fisheye tools expect, at least will be easier to use, if the fisheye is centered in the image frame. Unfortunately due to variations in the film processing the fisheye wasn't always in the same place so this process can't readily be automated. The discussion here is targeted at planetarium style domes, for front facing domes (eg: VisionStation) the processing of the panorama will be different since these domes are often truncated at the bottom of the fisheye.
For a uni-direction planetarium (seating mostly pointing in the same direction) it isn't critical if the part of the fisheye at the back of the dome is missing. This means that one can virtually rotate the fisheye so as to create a correct view around the front horizon. Notice that in the image on the right the bottom of the fisheye (camera upright) is truncated more than the top. For fisheye rotating it is better to use the camera upside-down which means less rotation is necessary. An alternative is to use the camera on its side which means the result will be clipped on the left and right of the dome, less desirable in my opinion.
Many fisheye projection systems truncate the back 1/4 of the fisheye anyway, in those cases this fisheye rotating results in even less image loss. Unfortunately this camera is pretty much limited to daylight (or at least very well lit) spaces, the flash is located behind the lens so the lens casts a shadow into the scene. This is described on the lomography site, indeed they "try" to suggest it gives interesting results.
Panoramic images Using a fisheye camera, it is possible to form a 360 degree panoramic by just capturing 4 images (see below). The aperture of this camera is just enough to get some overlap between the 4 images with the camera in used on its side. The process involves converting the fisheye images into limited angle panoramic images and then stitching them together to form the full panoramic image (see below). The panoramic can then be used in applications such as QuickTime VR, see right. It is easy to find the artefacts in the panoramic shown here, to create these successfully one really needs a good tripod in order to keep the camera level (this camera doesn't have a tripod mount) and one needs good control over the lighting.		QuickTime VR
The 4 individual fisheye images.
Stitched panoramic image (Melbourne Museum and Royal Exhibition Building).
Further examples
Melbourne Library		Melbourne Central

The following is a discussion of an evaluation of the suitability of the recently released Canon video camera coupled with a fisheye lens for filming footage for projection into a hemispherical dome, for example, a planetarium. While high resolution stills are readily achievable with a digital still camera and fisheye lens, it is more difficult to achieve a high quality result without resorting to film or high priced specialist camera rigs.

Please note in what follows all examples are using the video mode of the Canon HV20 not the photographic mode. While the still image capture mode is more suited to fisheye capture since it uses a 4:3 aspect ratio, if stills are what one wants then a higher resolution still digital camera and fisheye lens can give higher quality results.

The camera is the Canon HV20, a (true) 1080p digital video camera. That is, 1080 lines and progressive instead of interlaced. The lens is the Nikon FC-E9, a 185 degree circular fisheye lens. The two are matched together with a single 43mm to 46mm step-up ring.

The first thing one notices is that the circular fisheye, with the minimum zoom, does not fit within the sensor and it also isn't centered. The details (which may vary slightly with each camera) are shown below. While this may seem like a serious problem, for many applications it isn't. Many small planetarium systems employing a single projector and fisheye lens often only use 3/4 of the fisheye image, the portion at the "back" of the dome isn't used. Alternatively for upright domes it is common for the lower portion of the dome to be truncated (see examples later).

One solution to this is to digitally tilt the fisheye so as to remove the truncation at the bottom of the fisheye image. This technique applied to the image above is shown below, the green line shows the truncated region employed by many fisheye projection systems. Note that the image in the top half of the fisheye now does not fill the circular fisheye area, blue circle outline. This too can be corrected although unlike the tilting of the fisheye, this additional correction introduces distortion to the the upper half of the image which may or may not be objectionable, remembering that it is behind the viewer (at least in a directional dome).

Complete example

Original frame off the Canon HV20 (movie sequence)

Further examples

Two further examples are shown below, single frames extracted from a movie. These are rendered at XGA resolution (1024x768), that is, the lowest resolution fulldome projection employing a fisheye lens. It is my estimation that the quality in this case is limited by the the projection system as much as by the footage. By comparison, the quality on a HD projection system employing a spherical mirror is clearly limited by the quality of the footage. I suspect carefully captured, processed footage would be a good match for a SXGA+ based system.

Upright dome

It turns out that the camera and fisheye position is more suited to truncated style upright domes. When the camera is operated upsidedown the larger part of the fisheye image missing isn't required anyway since it is below the hemisphere truncation. The smaller missing piece at the other end of the fisheye results in a small section of missing image at the very top of the dome, or it may be filled by using the same fisheye tilting procedure discussed above.

Other lenses on the same camera

Nikon FC-E8:
This gives a similar circle but doesn't focus towards the rim where the image fades to black. Also seems the field of view is somewhat less than 180 degrees.

Phoenix Super Fisheye Lens 0.25X:
This gives a smaller circle than the Nikon FC-E9 and results in a field of view I suspect between 160 and 170 degrees. Sample image below.

This combination, Nikon D300 and Sunex 185 degree (5.6mm) fisheye lens, gives a full 180 degree fisheye image using more of the full frame than most other fisheye lens tested (notably Sigma lens). After cropping to 180 degree the resulting fisheye is in the order of 2580 pixels square.

The lens is certainly very solid. Extremely good focus, no manual focus required since it is fixed. I experienced very few problems even when shooting into the sun. It is not a perfect f theta lens, but then what fisheye lens is.

Perth city centre

University of Western Australia

20 second exposure

Contribution by Alexander Mitchell

A low cost option using a Lumix camera and fisheye lens results in a 1080 square fisheye at a very low price point. Of particular note is the high bit rate of 176mb/s by changing the camera firmware (by donation). Total cost is around $600 for the camera, $800 for the lens, and $20 for the Nikon m43 adaptor.

Example of the Canon 5D and Canon EF8-15mm fisheye lens

Results in almost a 3600 pixel square fisheye. On a full frame sensor the zoom allows a full fisheye, a partial fisheye with 180 degrees horizontally, or zoomed all the way in approximately a 170 degree diagonal.

When recording at 4K there is a slight clipping of the fisheye at top and bottom. Resulting fisheye video is in a 2320 circle.

Two sample frames from the video are shown below, each recorded at 4K at 30fps and 8:1 compression ratio.

The following outlines some photography and filming options I have been experimenting with. They generally involve a camera (still or video) along with a fisheye lens. This is certainly not intended to be an exhaustive survey, only the combinations I have recently employed. The date above should be noted, as with most technology reviews they can become out of date as time progresses.

The easiest is stills, there are a number of fisheye lens options for SLR cameras but many lenses are out of productions leaving one to troll through ebay. Sunex have a 185 degree fisheye for Nikon and Canon APS-C sensors, they are fairly "soft" and it seems the quality also varies on a per lens basis. My favourite lens at the moment is the (newish) Canon 8-15mm fisheye for a full frame sensor. It is unusual for a fisheye to be able to zoom, this lens goes from a full fisheye to about 170 degree diagonal (similar to the GoPro lens). Example from the Moodabidri temple near Manipal, fisheye diameter is about 3600 pixels.

I use a Canon 5D MKII and in combination with this lens it is ideal for capturing 360 full spherical panoramas, some examples in a virtual environment related to rock art in the Pilbara can be found here.

The Canon 5D (and other) can record HD video but since they use only a portion of the sensor the image is truncated top and bottom, and of course the fisheye result is only around 1K diameter.

Another option is the LadyBug-3 video camera. This was used in the production of "Dark", see example frame below. The result is about 2.4K fisheye but of course it (like most video cameras) uses a lossy video compression so the effective resolution is somewhat less. One of the nice things about the LadyBug-3 is that it captures 360 degrees horizontally by about 150 degrees vertically, this means the exact angle of the fisheye can be adjusted in post production.

And finally there is the Red series of cameras, I am currently exploring the Red Scarlet and the Sigma 4.5mm fisheye. The Scarlet has an APS-C sensor as does the Sigma lens but the full sensor size is only available at 5K recording mode which is limited to 12fps. At 30fps the highest resolution is 4Kx2K pixels and since it uses a portion of the sensor there is a slight truncation top and bottom of the fisheye. There are of course ways around this, slightly rotating the fisheye in post or using the camera on the side, or cheating and just cropping to the largest inscribed fisheye. Camera and fisheye shown below.

A sample frame from the video is given below to illustrate the degree of clipping, please note this is a single frame from a video, not a still shot. The fisheye circle is 2320 pixels in diameter.

Hope some of this is of interest. Filming for fulldome is certainly still a challenge. In my case much of this is for the iDome so fisheye resolutions around the 2K range are adequate. The next test is with the Coastal Optics fisheye lens, but it priced such that purchasing for testing is a little tricky, doesn't anyone have one they would consider loaning?