A trip into augmented reality

(!!! translated by machine !!!)

The concepts of virtual and augmented reality are related, based on the same technologies, yet fundamentally different. Both concepts are concerned with changing a person's perception. The term virtual reality (virtual reality, or VR for short) refers to the complete replacement of sensory perception with any conceivable (virtual) reality, which may be entirely artificial. The term augmented reality (AR) specifically describes the enrichment of the natural environment with mostly real-time computer-generated graphic elements. The person can perceive information that is not directly present in reality, i.e. as an extension of his or her reality.

Both approaches have in common that for the experience of this changed reality devices are used, which can measure the position and movement of a person in the area, as well as into a three-dimensional picture produce can. In the case of virtual reality, for example, this is done using a headset (a pair of closed "glasses") that can project a three-dimensional image into the user's eyes, alongside which it detects the orientation of the head and the wearer's position in space. The user is no longer aware of the real environment, all sensory impressions are projected, you move in a reality next to the real one.

In contrast, the extended reality means the projection of additional information into the real reality. For example, the user uses a smartphone equipped with motion sensors and a camera, films his surroundings with it, and sees the same on the screen, as well as additional objects projected into the room. Also conceivable is a pair of (non-closed) glasses through which the user sees his environment along with extensions.

In both cases, sensor data is used to control the viewing angle of the altered reality. Optionally, interaction with the altered reality is also possible. Sensor data or input via additional devices is used for this purpose, often realized in VR via specially equipped gloves.

In the following article, we focus mainly on the topic of augmented reality. We use the abbreviation XR when we talk about VR and AR at the same time.

macos/deepLFree.translatedWithDeepL.text

(!!! translated by machine !!!)

The applications of AR are many, ranging from useful tools, such as the tape measure app included with Apple iOS or the Google Translate translation tool, to applications in education (inter-active textbook), entertainment, gaming (Nintendo Pokémon Go, Ingress), and social media (Snapchat), to commercial applications, such as those offered by Apple or IKEA.

We look at a few examples of AR in action.

We look at a few of these examples:

The smartphone app Google Translate makes it possible to translate words or entire texts from one language to another. Relevant in this context is the possibility to use the camera. For example, you can point the camera at an English newspaper article and see the same newspaper on the screen, but with the translation instead of the original text.

A popular social media app is Snapchat. The original idea of the app, unlike Facebook, is to allow only ephemeral content. That is, a user's posts are only available on the network for a certain amount of time, after which they disappear. The content that can be shared in the app are camera snapshots. These can be enriched with AR content, you can change your appearance or decorate yourself with items. In the meantime, it is also made possible for third-party providers to develop their own "lenses", which can then be used in Snapchat, think of an eyewear manufacturer whose models you can put on in the app.

As a final example, we would like to talk about the app IKEA Place, which is currently available as a test version. In the app, you can not only view the furniture offered by the company in photos, but also place it in your own home right away.

The app is a great way to get a feel for the furniture.

After outlining the possibilities of AR using native smartphone apps in the previous section, we will now look at AR applications on the web, i.e. without the need to install an ap- plication. We also explicitly go into more detail here about more technical aspects of implementing AR, or VR as well.

(!!! translated by machine !!!)

A basic requirement for the implementation of an AR application is the output of 3D graphics. The support for this is well covered by WebGL and mature frameworks based on it, such as three.js or A-Frame. In addition to graphics output, it is necessary to retrieve user input, sensor and camera data, and possibly geo-information. All of this is possible with modern browsers, whether on a desktop or mobile device.

With these prerequisites, it is possible to create fancy VR applications, but it becomes problematic in AR projects where, for example, techniques such as object tracking or face tracking are used. In an AR application it is necessary to understand or "see" the environment. By object tracking, we generally mean tracking an object in the real world, as is necessary in the Google Translate example. The application must be able to recognize the text in a magazine and also determine its position relative to the device. Face tracking specifically refers to tracking a human face, as is needed in Snapchat, for example.

The SDKs provided for native Android and iOS app development already offer frameworks for AR app development, but web browser development currently lags behind. With Scene Viewer and AR Quick Look, the operating systems each offer a way to provide a website visitor with 3D models that can then in turn be viewed embedded in the real world, but more complex AR applications are more complex to implement.

The fundamental technologies are widely available on the web with WebGL (output of 3D graphics), WebRTC (including access to camera) and the WebXR Device API (sensing, positioning). However, for example, computationally intensive algorithms are required to implement object tracking,

which are not available in public web-ready implementations. In addition to commercial providers of AR solutions, however, there are initial frame- works such as AR.js or libraries offered by the company Jeeliz, which can be used here.

The first AR solutions are available in the form of AR.js.

Among other things, some use cases can already be covered with these: AR.js, for example, offers the possibility of aligning 3D models with special markers in the real world, and the Jeeliz Face Filter API can be used, similar to Snapchat, to align objects with a face or to react to the recognized facial expression. macos/deepLFree.translatedWithDeepL.text

We wunderwerker have researched and acquired some knowledge about 3D graphics and AR on the web. We now have the tools to implement many 3D and AR applications. If you have acquired a taste for this, we will be happy to advise you when it comes to implementing your application in a meaningful and efficient way.