Seeing is believing: How augmented and mixed reality are transforming the human experience

Image: https://tinyurl.com/y2su64n8

Introduction

Since its origin, augmented reality (AR) has been incorporated into a wide range of real-world applications (Figs 1-2). AR technology, whether wearable computers or mobile apps, promises to transform how we communicate, use our smart devices, play games, enjoy entertainment, and even how we do work in our professional lives. Despite this, AR is inaccessible to many people around the world and still has a long way to go in becoming viable for the everyday consumer. This essay will trace the evolution and applications of AR, with a focus on wearable and mobile AR technologies (for the purposes of this essay, MR and AR will be referred to collectively as “AR”). The author contends that AR is a promising medium; one that has the potential to transform and revolutionise the human experience by opening up the physical realm to endless possibilities.

 

Definitions

The term “augmented reality” (AR) can be applied to a range of interactive systems “that attempt to merge computer graphics and real imagery into a single, coherent perception of an enhanced world around the user” (Fuchs et al., 1998, p. 935). Although AR is related to virtual reality (VR), the latter “attempts to create an entirely different reality that is separate from real life” while the former “adds to the real world and does not create a unique world” (Mohn, 2018, p. 17). More recently, the term “mixed reality” (MR) has been used to describe technologies and processes that not only “merge the digital with the physical” (as AR does), but also incorporate human physicality and gesture as inputs (Lindgren & Johnson-Glenberg, 2013, p. 445) (Fig 3).

 

Fig 1. Applications of AR. Image: https://tinyurl.com/y2x2wqpe

 

Fig 2. The Microsoft HoloLens interface. Image: https://tinyurl.com/y3gjcxg3

 

Fig 3. Differentiating between virtual, augmented and mixed realities. Image: https://tinyurl.com/yxwav7e7

 

The genesis of AR

Although AR is embedded deep within humankind’s early desire to manipulate, alter, improve and add symbolic meaning to our physical environment, its genesis lies in the twentieth century with the advent of the computer, which allowed information to be stored, modified, represented and retrieved digitally (Craig, 2013, p. 6). Ivan Sutherland invented the first see-through head-mount in 1968 (Fig 4), which is considered to be the first true augmented-reality interface (Poupyrev et al., 2002, p. 45). In 1992, Louis Rosenberg created his “Virtual Fixtures”; a remote operator system that integrated the human senses with a headset and became the first recognised MR setup. In the early twenty-first century, advancements in computerised displays, trackers, 3D graphics and software meant that AR was well on its way to becoming a commercial product (Feiner, 2002, p. 50).

 

Fig 4. Ivan Sutherland demonstrating the HMD (1968). Video: https://www.youtube.com/watch?v=NtwZXGprxag

 

AR as wearable technology

One of the major milestones in AR technology was the commercialisation of the Head Mounted Display (HMD), which signalled a new era in wearable computers. In 1999, Steve Mann’s EyeTap Digital Eye Glass allowed the human eye “to function as both a camera and display” (Mann, 2012, p. 10). In the 2010s, head mounts became smaller yet more powerful, making it possible to “display information hands-free in smartphone-like format [and] interact with the Internet via natural language voice commands” (Hyman, 2013, p. 18). Today, HMDs are equipped with sensors, connectivity capabilities, and a see-through display (Hernandez & Picard, 2014, p. 1). Google Glass (2013) and Microsoft HoloLens (2016) integrate smartphone functionalities into their devices, including a HD display, spatial mapping of objects, gesture and gaze interface, and voice recognition (Evans, et al., 2017, p. 5), as well as affect sensing and emotion recognition (Hernandez & Picard, 2014, p. 1) (Fig 5). Today, companies such as Magic Leap are working on new aspects of AR technology, such as beaming images and information directly onto users’ retinas (Mohn, 2018, p. 18).

 

Fig 5. Microsoft HoloLens 2 live demonstration. Video: https://www.youtube.com/watch?v=r0ubiU3PRHw

 

AR in smartphone applications

Aside from wearable computers, AR technologies have been implemented into the ubiquitous network of interconnected hand-held “smart” devices such as computers, laptops, tablets, smartphones, and other gadgets (Craig, 2013, p. 6). AR apps are designed for a range of needs and work by overlaying virtual (computer-generated) content or information onto the physical environment, which can be accessed using the built-in functionalities of a device (Rese et al., 2017, p. 306). Examples include the IKEA Place App, L’Oreal’s Makeup Genius App, Apple’s ARKit, Google’s ARCore, Google Translate, and Night Sky, among others. AR is also being integrated in social media platforms and apps. Snapchat’s Lens Studio, for examples, allows artists, animators, developers and other creative individuals to create, publish and share augmented reality lenses to the platform (Snapchat, 2019).

 

AR in gaming

Another dynamic application of AR is in the area of gaming. AR technologies both “enhance existing game styles and open up new ones” (Nilsen, Linton, & Looser, 2004, p. 2). Pokemon GO (Fig 6), released by Niantic in 2016, is a location-based AR app that uses GPS tracking capabilities and allows players to “capture” virtual creatures with their phone cameras. AR in gaming consoles and personal computers often makes use of peripheral devices such as webcams and in-built cameras, which merge the physical setting of the room with the virtual setting (Kochi et al., 2017, p. 8). Sony’s Playstation Camera and Microsoft’s Xbox Kinect allow players to interact with virtual characters and play games in the space of their own homes. Innovations in AR gaming also include the development of holographic versions of traditional and tabletop games, such as the Tilt Five system.

 

Fig 6. Pokemon GO (Niantic). Image: https://tinyurl.com/yxbs4wec

 

Key players in the AR industry

AR is being implemented across a variety of sectors and industries. The key players are the companies and organisations that create AR technologies, develop the platforms, and produce the content; as well as industry leaders and policy makers that approve and/or reject these novel technologies. Tech giants, global multinational companies, private enterprises, start-ups, entrepreneurs, investors, researchers and developers have all taken an interest in the AR market, which is estimated to be worth $80 billion by 2021 (three times more than virtual reality), and that has the potential to reach millions of customers (Evans et al., 2017, p. 4) (Fig 7).

 

Fig 7. Key players and market predictions for AR & VR. Infographic: https://tinyurl.com/y2vdkncx

 

The transformative effects of AR

The transformative potential of AR is evident in the key areas of art, entertainment, media and communications, business, education, medicine, science and engineering, military, among other sectors (Mohn, 2018, p. 17) (Fig 8). As a visual medium, AR can transform the artistic, creative and collaborative workspace needed for effective communication. The new HoloLens creates a virtual space where people who use their hands for work (e.g. engineers, designers, doctors) can turn their visualisations and ideas into 4D objects. One major example where AR technology is being adapted is in medicine. In laparoscopic surgery, for example, tracking technologies are being developed to “display a merged real and synthetic image in the surgeon’s video-see-through head-mounted display” (Fuchs et al., 1998, p. 934). Another area where the transformative effects of AR are taking place is in education, especially teaching and learning (Diegmann et al., 2015). An example of an educational AR app is TombSeer, which “aims to immerse the wearer in a museum space… through a holographic heads-up interface that brings virtual, historical artifacts back to life” (Pedersen et al., 2017). Barfield argues that AR can “benefit humanity in ways we are just now beginning to realise” (Barfield, 2015, p. 5), which is clear from the broad range of industries and fields where the technology is being applied (Fig 9).

 

Fig 8. Families experience AR in a zoo. Video: https://www.youtube.com/watch?v=Xmpe1uYTDgI

 

Fig 9. The potential of AR in the future, from Minecraft Earth to everyday applications. Video: https://www.youtube.com/watch?v=yuRV27c8HDw

 

The limitations of AR

Despite the proliferation and promise of AR, the technology is still inaccessible to large numbers of people around the world. Devices such as the HoloLens are “not commercially available for purchase [or] not stable enough to distribute to consumers” (Evans et al., 2017, p. 4), while the price tag of several thousand dollars means they are reserved for specialised businesses. AR apps, while more accessible and affordable, also have their limitations. The companies that develop AR still have far to go in terms of improving the technology, visualisation, interaction (i.e. user perception and cognition), design, displays, interfaces (i.e. task performance) and algorithms (Rese et al., 2017, p. 306). A major area of concern with AR devices is privacy and security – due to their “always on, always-sensing” status (Roesner, Kohno, & Molnar, 2014, p. 2), users may be threatened by both internal and external sources. For example, malicious applications or viruses can be used to hijack wearable AR devices (Fig 10). Another potential threat to using AR comes from the intimate access to users’ sensory data (including biometric, visual and aural feeds) that parent companies have, giving them intimate access to users’ state of mind, interests, behavioural patterns, locations and activities. Concerns have also been raised in terms of how AR will affect traditional job roles and employment rates. As new AR technologies and applications emerge, it is thus imperative that new privacy and security-enhancing measures are developed in conjunction with them (Roesner et al., 2014, p. 8).

 

Fig 10. The potential for malicious attacks in AR is highlighted in Matsuda’s short film, “Hyper Reality”. Video: https://www.youtube.com/watch?v=YJg02ivYzSs&t=2s

 

Conclusion

AR allows us to experience worlds and entities “that seem real yet are actually either photographed or synthesized imagery” (Craig, 2013, p. 6). AR technologies, ranging from wearable headsets and glasses to smartphones (Barfield, 2015, p. 3), are broad and wide-ranging. Although there is a growing market for AR in home and business applications, AR still has a long way to go in terms of accessibility and technological viability. Once these limitations are addressed, AR’s potential and reach will become global – changing how humankind perceive and interact with the world, and giving all of us access to information whenever, and wherever, we need it.

 

Word count: 1683 (including title and all captions, excluding references)

References

Barfield, W. (2015). Fundamentals of wearable computers and augmented reality: CRC Press.

Craig, A. B. (2013). Understanding augmented reality: Concepts and applications. Amsterdam: Morgan Kaufmann.

Diegmann, P., Schmidt-Kraepelin, M., Eynden, S., & Basten, D. (2015). Benefits of augmented reality in educational environments. Benefits, 3(6), 1542-1556.

Evans, G., Miller, J., Pena, M. I., MacAllister, A., & Winer, E. (2017). Evaluating the Microsoft HoloLens through an augmented reality assembly application. Paper presented at the Degraded Environments: Sensing, Processing, and Display 2017.

Feiner, S. K. (2002). Augmented reality: A new way of seeing. Scientific American, 286(4), 48-55.

Fuchs, H., Livingston, M. A., Raskar, R., Keller, K., Crawford, J. R., Rademacher, P., . . . Meyer, A. A. (1998). Augmented reality visualization for laparoscopic surgery. Paper presented at the International Conference on Medical Image Computing and Computer-Assisted Intervention.

Hernandez, J., & Picard, R. W. (2014). SenseGlass: Using google glass to sense daily emotions. Paper presented at the Proceedings of the adjunct publication of the 27th annual ACM symposium on User interface software and technology.

Hyman, P. (2013). Augmented-reality glasses bring cloud security into sharp focus. Communications of the ACM, 56(6), 18-20. doi:10.1145/2461256.2461264

Kochi, M., Harding, R., Campbell, D. A., Ranyard, D., & Hocking, I. M. (2017). Apparatus and method for augmented reality. In: Google Patents.

Lindgren, R., & Johnson-Glenberg, M. (2013). Emboldened by Embodiment: Six Precepts for Research on Embodied Learning and Mixed Reality. Educational Researcher, 42(8), 445–452. https://doi.org/10.3102/0013189X135116

Mann, S. (2012). Through the glass, lightly. IEEE Technology and Society Magazine, 31(3), 10-14.

Mohn, E. (2018). Augmented Reality. In D. R. Franceschetti (Ed.), Principles of robotics and artificial intelligence (pp. 17-19).

Nilsen, T., Linton, S., & Looser, J. (2004). Motivations for augmented reality gaming. Proceedings of FUSE, 4, 86-93.

Pedersen, I., Gale, N., Mirza-Babaei, P., & Reid, S. (2017). More than meets the eye: The benefits of augmented reality and holographic displays for digital cultural heritage. Journal on Computing and Cultural Heritage (JOCCH), 10(2), 11.

Poupyrev, I., Tan, D. S., Billinghurst, M., Kato, H., Regenbrecht, H., & Tetsutani, N. (2002). Developing a generic augmented-reality interface. Computer, 35(3), 44-50.

Rese, A., Baier, D., Geyer-Schulz, A., & Schreiber, S. (2017). How augmented reality apps are accepted by consumers: A comparative analysis using scales and opinions. Technological Forecasting & Social Change, 124, 306-319. doi:10.1016/j.techfore.2016.10.010

Roesner, F., Kohno, T., & Molnar, D. (2014). Security and privacy for augmented reality systems. Commun. ACM, 57(4), 88-96.

Snapchat (Team Lens Studio). “Making AR has never been easier”. Accessed 12/10/2019. Available at: https://lensstudio.snapchat.com/news/making-ar-has-never-been-easier/

Be the first to comment

Leave a Reply

Your email address will not be published.


*