Facebook simulation to teach robots to reality

It is necessary to teach the robot that works as an assistant in the house what the real world is like. Facebook has been developing Habitat since 2019, which trains robots in virtual space. Recently, Habitat 2.0 has been announced, which is significantly different from existing simulations and realizes high speed and unprecedented interactive simulation.

To develop a robot that works as a secretary in an ordinary household, the robot must learn in advance by simulating the movements required in ordinary households. However, simulation in a real environment requires a lot of time. So, Facebook devised a way to run simulations in virtual space and complete training that would take thousands of hours in minutes. It is an open source simulator named Habitat, which was published in a paper in 2019.

Facebook announced Habitat 2.0 on June 30, 2021. According to Facebook, Habitat 2.0 expands Habitat capabilities and is significantly faster than existing 3D simulators. Thus, Habitat 2.0 is said to be capable of performing billions of tasks in real life, including complex tasks such as placing food on a table or putting food in the refrigerator. The advantage of Habitat 2.0 is that it can perform interactive simulations as well as simulations in a static 3D environment.

The Habitat dataset ReplicaCAD was originally static 3D scan data, but Habitat 2.0 includes physical parameters and semantic annotations to enable manipulation and exercise training. In addition, in order to have a change in the setting environment, 111 patterns are prepared for the layout of the living space, and objects such as kitchenware, books, and furniture are also provided. Although these objects are created by the artist, they are free to use for non-commercial purposes for the CC By-NC 4.0 license.

Facebook is also collaborating with 3D data platform Matterport to open source the largest ever indoor 3D scan data used in Habitat 2.0. Related information can be found here.