Follow Me on:

• Facebook
• Twitter
• Instagram
• Google+

Today we officially open-sourced the specs for Surround 360, our high-quality 3D-360 hardware and software video capture system. The open source project includes the hardware camera design and software stitching code that makes end-to-end 3D-360 video capture possible in one system — from shooting to video processing.

We believe making the camera design and stitching code freely available on GitHub will accelerate the growth of the 3D-360 ecosystem — developers will be able to leverage the code, and content creators can use the camera in their productions. Anyone will be able to contribute to, build on top of, improve, or distribute the camera based on these specs.

Both the hardware design and software play a role in achieving a high-quality output. We've previously written about the hardware design of the camera and its importance in decreasing the processing time for the footage. Even though we use high-quality optics and machine-fabricated rigs, there is still enough variation between camera lenses that without software corrections, results are not viewable in stereo (you see double vision and not 3D). Our stitching software takes the images captured by the 17 cameras in Surround 360 and transforms them into a stereoscopic 360 panorama suitable for viewing in VR. This software vastly reduces the typical 3D-360 processing time while maintaining the 8K-per-eye quality we think is optimal for the best VR viewing experience. This post goes into some depth on the rendering software we created to solve this problem, covering the goals we set, the challenges we took on, and the decisions we made in developing this technology.

Rendering stereo 360 video is a hard problem for a variety of reasons:

Traditional mono stitching software won't solve these problems. It can apply relatively simple transforms to the source images, then blend them together at the edges, but this approach is not sufficient for stereo. It produces stereo inconsistencies that break perception of 3D and cause discomfort.

We tried a lot of camera/software combinations as we developed our prototypes for a 3D-360 camera. Our first attempt was four pairs of cameras pointing out in a square, spaced roughly eye-distance apart. Then we tried to stitch the left and right eye images into a seamless panorama with alpha blending, which is a basic technique for mixing images together to create a smooth transition. It looked great when pointing straight out in the direction of the real cameras, but the stereo was off at the corners. Another issue with this approach is that as you rotate around, your eyes have to point in different directions to look at objects a constant distance away, which causes eyestrain.

There are more sophisticated ways to construct stereo panoramas than basic alpha blending and stitching. One approach is to take slices of the images captured by a camera that rotates around a fixed axis. This approach produces panoramas that have proper vergence and creates more consistent perception of depth than basic stitching. However, it cannot be applied to scenes where anything is moving. Going one step further, we use optical flow to simulate a rotating camera, by "view interpolation" from a small number of cameras arranged in a ring; this approach can capture scenes with motion, but optical flow is a challenging problem in computer vision research.

The Surround 360 rendering code is designed to be fast so it can practically render video while maintaining image quality, accurate perception of depth and scale, comfortable stereo viewing, and increased immersion by incorporating top and bottom cameras to provide full 360-degree x 180-degree coverage and making the tripod pole invisible.

Equirectangular projections are one technique commonly used for encoding photos and video in VR. An equirectangular projection is just a way of wrapping a sphere with a rectangular texture, like a world map. Each column of pixels in the left or right equirect image corresponds to a particular orientation of the user's head, and the pixels are what should be seen by the left and right eyes in that head orientation.

So our goal is to generate equirectangular panoramas for the left/right eyes, such that each column of pixels corresponds to a different head orientation. For a given head orientation, there are two virtual eyes spaced 6.4 cm apart (or whatever we specify), which rotate around the center of the head. From each virtual eye, we consider a ray that goes out in the direction of the nose, and we want to know what color the light from the real world is along that ray. That is the color we should use for the corresponding pixel in the equirectangular image.

Now imagine all this is happening inside the camera (from a 2D top-down perspective); the center of the head is the center of the camera. The ray we constructed through each pixel starts at a virtual eye and eventually exits the circle of cameras. If that ray passed right through a real camera, we would know what color it should be from some pixel in the camera's image. But usually the ray exits the circle where there is no real camera. In this case, we use view interpolation to generate a virtual camera in between the real cameras, and get the ray color from a pixel in the interpolated view.

The description above is a model of how the stereo equirectangular images that we render correspond to the color of light along rays in the real world, which explains why it feels realistic (although it is a rough approximation).

In practice, we implement the above model in a pipeline of image processing stages. There are many steps in the pipeline, but it is mostly automatic, so the time to render is still considerably less than manual stitching.

First, cameras output RAW bayer pattern images. The Image Signal Processor (ISP) part of the Surround 360 rendering code converts the RAW sensor data to a standard RGB format .png image, applying gamma and color correction. Then, the rendering system reads the camera images and builds equirectangular projections from each. The equirectangular projection covers a whole sphere with a rectangular texture. Each camera captures only a portion of the full sphere, but this can still be represented on an equirectangular projection. We found it convenient to use this projection throughout all subsequent stages of the pipeline.

While projecting the original image to equirectangular, we apply corrections for lens distortion. We calibrate models for lens distortion by taking many pictures of checkerboards, which is a common approach in computer vision.

We also correct for rotational or translational misalignment of the camera/lens/mounting system, which would cause vertical parallax (which causes double vision and breaks 3D). We do this by finding matching key points in images from all adjacent cameras, and optimizing a perspective transform for each camera that rectifies all the cameras jointly, to minimize vertical parallax (the vertical difference between where an object appears in the left eye's image, and in the right eye's image).

All the cameras are projected to equirectangular in such a way that objects at a distance of infinity from the camera are aligned with zero displacement, while any difference in object position between different cameras in the equirectangular projection corresponds to parallax from being closer than infinity. This helps optical flow, because it typically starts with an initial guess of 0 flow, which is the right answer for things that are far away.

Overlapping regions of the images from neighboring cameras are extracted, and optical flow is computed between them.

Optical flow is used to do view interpolation, so we can get the rays we need from virtual cameras.

So far, we have described how to render the side stereo parts of the panorama. We also incorporate top and bottom cameras to make it more immersive by providing full 360-degree x 180-degree coverage. The side cameras are wide-angle lenses with roughly 90-degree horizontal and vertical FOV (which decreases to 77 degrees after correcting for barrel distortion). So that covers a 77-degree strip at the horizon, out of the full 180-degree vertical range. We have cameras with different 180-degree fisheye lenses on the top and bottom to fill in those holes with mono. It actually has to be mono at the very top and bottom because there is no way to bake correct stereo for all head orientations into a left/right equirectangular image pair, so we make it gradually transition from stereo at the horizon to mono at the poles. With different lenses, we could increase the stereo range at the horizon, but that is a trade-off with resolution/pixel density.

To seamlessly stitch the top camera with the side camera left/right images in a way that produces comfortable stereo, we use optical flow to match the top image to the side panoramas. Then the top image is warped according to the flow and alpha-blended (with deghosting) to composite the images.

There are two cameras on the bottom, so we can automatically remove the tripod pole. The primary bottom camera is in the center of the rig, symmetric with the top camera. The problem is, a big chunk of its image is blocked by the tripod pole. The secondary camera is on the other side of the pole, so it sees the part of the scene that is blocked by the pole in the primary camera's image. We build a synthetic image that would have been seen by the primary bottom camera but with the pole removed. This is done by specifying a mask of the pole in both images, then using optical flow to warp the secondary image onto the primary image, and mixing the two such that the primary pixels are used unless they are in the pole mask, and otherwise the warped secondary pixels are used. After the two bottom images are fused to remove the pole, the result is stitched to the side left/right eye panoramas similar to the top camera.

All this happens in multiple threads to take advantage of multi-CPU systems. For example, optical flow can be computed between all image pairs simultaneously. Beside that, each step of the processing pipeline is scheduled according to its dependencies on other steps, to keep all the CPUs busy for as much of the rendering time as possible.

This rendering pipeline produces high-resolution stereo 3D-360 panoramas that occupy the full 360-degree x 180-degree sphere for a highly immersive experience. We have taken great care to preserve image quality at every step of the pipeline, to optimize for speed, and to produce the most accurate approximation to reality that is practical with off-the-shelf hardware. We are very excited to contribute all the hardware designs and software as open source and to continue to iterate on the design, the code, and the whole 360 pipeline in the future. We hope this will inspire others to build their own stereo 360 rigs and, most of all, to capture truly immersive experiences. Please let us know what you think and what you contribute back to the community.

Today 1.6 billion people live in regions that don't have access to a mobile broadband network. Connecting these remote parts of the world with existing technologies such as buried optical fiber or microwave links on towers is often cost-prohibitive. As part of our commitment to Internet.org, the Facebook Connectivity Lab is developing many new technologies to bring affordable internet access to more people, more quickly. One of the technologies we are building is a fleet of solar-powered aircraft called Aquila. Once they are fully operational, these high-altitude planes will stay airborne for up to 90 days at a time and beam broadband coverage to a 60-mile-wide area on the ground, helping to open the opportunities of the internet to people in underconnected regions.

After several months of flying scale models, our team has reached an important milestone: We successfully completed the first full-scale test flight. The low-altitude test flight lasted for 96 minutes — more than three times our planned mission length — and provided our aeronautics team with data on numerous aspects of Aquila's performance, including the autopilot, motors, batteries, radios, ground station, displays, basic aerodynamic handling, structural viability, and crew training. This post details some of our early learnings from the test flight and the data we collected, and looks ahead to some of the challenges we're working on next.

• Takeoff: Aquila's design is optimized for minimal mass, so it does not include traditional takeoff and landing gear. The first part of our test involved orchestrating a new kind of takeoff. We attached the airplane to a dolly structure using four straps, then accelerated the dolly to takeoff speed. Once the autopilot sensed that the plane had reached the right speed, the straps were cut simultaneously by pyrotechnic cable cutters known as “squibs.” While the ground-based crew can command the plane's heading, altitude, airspeed, and GPS-based route from a control computer, there are no joysticks involved — takeoff is on autopilot. A great deal of simulation and analysis were performed prior to first flight to understand the dynamics of takeoff and to choose the aircraft's pitch on the dolly, the speed at which it would be released, and the initial elevon angle. The specifications based on our simulations resulted in a successful takeoff.
• Aerodynamic model: Aquila will need to operate in both cold, thin air at high altitude and warm, thick air at low altitude. The air is 10x more dense at sea level than at cruising altitude, and the aerodynamics of the plane's wings and propellers vary greatly over that range. When designing the aircraft, we built computer models based on computational fluid dynamics that helped us understand how much power is required to propel the plane for the different altitudes and speeds at which it will operate. We then ran the model through thousands of tests, using different parameters, to ensure the plane would meet our performance, stability, and control requirements. We also performed a coarse verification of our drag model, which measures the force of air opposing the moving aircraft. Both the climb rate and the battery usage were close to the predicted values. We tested the autopilot and the aeroservoelastic model on which it is based when we flew our 1/5-scale aircraft earlier this year. Our initial analysis from the full-scale flight suggests our aerodynamic models are in line with what we observed.
• Battery and power performance: While Aquila's power system will eventually include solar cells, our first few test aircraft are powered only by batteries to give us early indications about their aerodynamic performance, handling qualities, and autopilot performance, and to verify our structural models. Given the low drag and slow speed of 25 mph, Aquila flew on less than 2,000 W of power during its first flight. This matched well with our prediction, which suggests that the propeller efficiency, motor efficiency, and drag were also in line with predicted values.
• Autopilot performance model: Like many modern UAVs, Aquila is controlled by a full-time autopilot. To make our large, flexible aircraft controllable requires that we understand how the aircraft deforms under the aerodynamic and inertial loads, and what the structural modes and frequencies look like. Our first flight verified that the autopilot design is correct at low altitude and low true airspeed. During all maneuvers, the autopilot was stable and accurate, and it brought the aircraft to the commanded condition in the predicted amount of time. The automatic landing algorithm also performed well, tracking the glide path and centerline with expected accuracy.
• Real-world conditions: Our simulations and analysis predict how the aircraft will respond to wind, turbulence, and vertical gusts. Our first flight lasted three times longer than the minimum mission length, so we were able to gather data on how the structure and autopilot responded under a range of real-world conditions to help verify these predictions. We are still analyzing the results of the extended test, including a structural failure we experienced just before landing. We hope to share more details on this and other structural tests in the future. To prove out the full capacity of the design, we will continue to push the plane to its limits under more extreme conditions in a lengthy series of tests.

Challenges ahead

• Challenge #1 — Getting enough sun. For a solar-powered airplane to work, enough energy must be collected during daylight hours to operate for the full 24-hour day. The amount of energy the airplane requires to power its propulsion, payload, avionics, heaters, lights, communications, and other systems has to be less than or equal to what the solar panels can collect. At its nighttime altitude of 60,000 feet, Aquila will require approximately 5,000 W of power — about as much as three hair dryers. During the winter, the solar panels must be able to collect enough energy during the short days to keep the batteries charged over the long nights, up to 14 hours at a time.
• Challenge #2 — Batteries. Keeping the airplane aloft on a long winter night requires a lot of stored energy. Given current and projected battery performance, that means batteries will account for roughly half the mass of the airplane. The rest of Aquila's mass consists of motors, payload, avionics, and structure. We're pushing the edge of high energy-density batteries while exploring the best designs to ensure we have enough resilience in the system.
• Challenge #3 — Size and speed. Aquila has a wingspan comparable to a commercial airliner's but weighs only one-third as much as a car. This ratio of weight to surface area means that Aquila is able to fly much slower than a typical airplane of the same size — about 25 mph at sea level, compared with 50 mph for a glider or more than 200 mph for a commercial airliner. Such stark differences challenge many of the assumptions relied upon in aeronautical engineering. There is little to no precedent to guide us as we continue to optimize Aquila's performance.
• Challenge #4 — New cost paradigms. For Aquila to succeed, it needs to be an economically viable alternative to current network infrastructure. We need to develop more efficient onboard power and communication systems; ensure the aircraft are resilient to structural damage to reduce maintenance costs, and able to stay aloft for long periods of time to keep fleet numbers low; and minimize the amount of human supervision associated with their operation. We're working with the industry on ways to accelerate the development of new technologies that can drastically change the economics of deploying internet infrastructure.