I’m curious – when I talk about “augmented reality,” do you know what that is? I feel like it might not be part of the general consciousness yet. I don’t have any new development to point to, nobody introduced new products last week or anything like that. But it’s time for you to pay attention, because AR is going to be a Very Big Deal before long, and because it’s very cool, the best kind of technology, both science fictional and very useful.
I’m going to give you a couple of quick examples to get you oriented. Mostly, though, I want to call your attention to how important AR is to really smart people, so you’re not surprised when this turns out to change our lives in the next few years.
Augmented reality is easy to describe, hard to imagine. In the simplest terms, AR adds extra stuff to what you see as you look around. Right now it’s done by holding your phone up so the phone screen shows you what’s in front of you – and also adds something useful.
Here are two examples.
When you get walking directions from Google Maps (not driving directions, just walking), you can switch to Live View. Hold your phone up in front of you in Live View. You’ll see the street in front of you. You’ll also see giant arrows that look like they’re floating in the air to show you which way to walk and where to turn. If you haven’t tried it, Google’s instructions are here. It’s awesome. It’s also the perfect demonstration of augmented reality.
Google Lens uses your phone camera to focus on something and give you information about it. It is almost absurdly useful. Use it for the QR codes that are turning up more frequently at restaurants to take you to the online menu. Focus on the street and Google Lens will show you info about nearby businesses, like the photo above. Point at a plant and Google Lens will tell you what it is. Point at a painting in a museum and you’ll get details and links. You can use Google Lens for almost anything. It’s a very intuitive way to become comfortable with augmented reality.
On iPhones, Google Lens is built into the Google app and Google Photos. On Android phones, Google Lens is built into the search bar on the home screen, plus it’s a mode in the camera, plus it’s in the Google app and Google Photos, plus it’s a separate app if that feels more natural.
Those are just examples. There are lots more. Our world is filling up with augmented reality in some form or another. AR is common in sports on TV now – the yellow first down line in football games, the three point line for basketball, the flags and countries that appear to be at the bottom of Olympic swimming pools. If you (or your child) ever use Snapchat filters, it’s a kind of augmented reality. Five years ago AR swept the world when people held up their phones and went chasing after Pokemon characters that appeared to be on the grass in front of them.
Holding up a phone is a bit clumsy. Imagine if you could see useful info all the time through, say, a special pair of glasses that get information from online about what you’re looking at. You could look at your living room and see what an Ikea chair would look like in the corner. You could see an arrow pointing to your car in the parking lot. A surgeon could see an overlay that shows them where to operate. Students could view the solar system in the middle of the classroom.
The classic example with immediate appeal is being able to see a sign over each person’s head to remind you of their names. Yes, please, that would be great! A moment’s thought will tell you there are some pretty profound privacy issues, but putting that aside, it would be transformational for those of us who are the teensiest bit challenged by names, faces, and details of other people’s lives.
Here’s the dilemma.
Making that pair of AR glasses is one of the most difficult computer science problems ever faced by mankind.
Designing the iPhone was easy compared to bringing augmented reality to a pair of glasses.
AR requires strapping a supercomputer to your head, powerful enough to use artificial intelligence at a very high level, with a fast Internet connection and batteries that last all day – and doing it in glasses that have to be lightweight and look a lot like today’s typical eyeglasses.
Have you seen the movie Free Guy? Great movie. Streaming on Apple TV, rent or buy it on Amazon, very funny. Ryan Reynolds picks up a pair of glasses that look completely normal, but when he looks through them, he can see all the AR elements of the world around him. Without the glasses, he’s strolling down a city street. With the glasses, he can see cars racing around and crashing, spaceships flying between the buildings, giant signs identifying HOSPITAL and COURTHOUSE.
It’s hard to make those glasses.
You know who’s working on those glasses?
Apple. Tim Cook thinks augmented reality is “one of these very few profound technologies that we will look back on one day and, ‘How did we live without it?’” He’s been talking about the game-changing potential of AR for five years, taking every opportunity to let us know that he has committed huge resources at Apple to developing AR glasses or a headset that have an even greater potential to change the world than the iPhone or iPad. In a recent interview with Kara Swisher, Cook called AR a “critically important” part of Apple’s future. Here’s the website for Apple’s AR division.
Google. Google has done more to bring augmented reality to consumers than anyone else, with Google Lens, AR additions to Google Maps, and much more. Google was the first to market with AR-enhanced glasses, Google Glass. The initial reaction was unfavorable as people thought for the first time about the privacy implications of AR and dealt for the first time with glasses that looked peculiar, so Google Glass is now being marketed to enterprises. Google doesn’t talk about consumer AR glasses today but they’re working on them, I know they are. Here’s the website for Google’s AR division.
Microsoft. Hololens are AR-enhanced glasses that aren’t just a research project; they exist today and they are starting to be widely used. The first Hololens was released more than five years ago; almost immediately Microsoft decided to focus on enterprise and military applications. Hololens is being used in manufacturing, healthcare, and education, and Microsoft just signed a $22 billion contract to supply Hololens to the US Army. Here’s the website for Microsoft Hololens.
Facebook. Mark Zuckerberg said a few months ago that building AR glasses that can gain mainsteam acceptance will be “one of the hardest technical challenges of the decade.” He went on:
“I believe that that augmented and virtual reality are going to enable a deeper sense of presence and social connection than any existing platform. And they’re going to be an important part of how we will interact with computers in the future. So we’re gonna keep investing heavily in building out the best experiences here. And this accounts for a major part of our overall R&D budget growth.”
Zuckerberg spoke at a Facebook developers conference in 2018 about why AR would render televisions and smartphones obsolete. A few months ago Facebook described its ten-year vision for augmented reality glasses, and showed off a wrist-based wearable device with sensors to translate signals traveling through your nervous system into digital commands for interacting with AR glasses. Here’s the website for Facebook Reality Labs.
No one knows when an AR device will arrive that breaks through to the consumer market. Apple’s rumored AR glasses might turn up in 2022, but that depends on progress in AI, miniaturization, battery life, connectivity, and power, things that are not completely in Apple’s control. It’s easy to imagine delays.
Sounds like science fiction, doesn’t it? Nope, these are all things that tech companies are working on right now, today. The real science fiction would be if we could get past the glasses by having nanobots sending signals to our brains from directly behind our eyes. Somebody ought to write an SF novel about that. Maybe I will.
In the meantime, make an effort to notice when you see a computer putting an overlay on top of the real world. AR is everywhere. You’ll be better prepared for the AR glasses, wristbands, contact lenses, and everything else coming at us soon.