©2014. Published in Landslide, Vol. 6, No. 3, January/February 2014, by the American Bar Association. Reproduced with permission. All rights reserved. This information or any portion thereof may not be copied or disseminated in any form or by any means or stored in an electronic database or retrieval system without the express written consent of the American Bar Association or the copyright holder.
As amazing as digital technology has become, it has always suffered from one fundamental limitation: it can only be experienced within the four corners of a computer screen. Suppose, however, that—just like the Kryptonian criminals in 1980’s Superman II—these flat images and data were suddenly liberated from their two-dimensional confinement and were given the ability to move in three dimensions, interacting with other physical objects and people as they pleased.
That is the promise of augmented reality (AR). It creates at least the illusion of digital data having physical substance and interacting with the three-dimensional world. The result is that we experience, and can take advantage of, digital content in entirely new ways. When fully realized, this technology could shape society as radically as the Internet itself has done. Any development that so significantly changes the way we behave and interact cannot help but also affect the laws governing our rights and our behavior—especially, in this case, the intellectual property laws.
What Is Augmented Reality?
As in any good legal analysis, let’s start by defining our terms. To “augment” something means to make something greater by adding to it. “Reality,” in this context, is what we humans experience with our five senses. “Augmented reality,” therefore, is technology that adds digital content to our experience of the physical world. Another apt and more colloquial description is that AR makes the world “clickable,” inserting digital hyperlinks into physical space.
A few examples will help make the concept more tangible. There are projects underway dedicated to augmenting each of the five senses. Several groups are doing interesting work in “haptic” technology, which augments the sense of touch. The Finnish company Senseg,1 for example, is working on turning touchscreens into “feel” screens, able to digitally mimic any number of textures by means of touch pixels, or Tixels™. A Dutch scientist is developing a device that would turn sound waves into synthetic vision for the blind.2 There are also those working on digitally enhancing the senses of taste and smell.3
When people talk about AR today, however, they are almost always referring to enhancing the sense of sight. And with good reason—the primary and fastest way we humans gather information is by visual means. Software designers and hardware engineers are hard at work on a number of different ways to digitally augment our fields of vision. The most abundant examples are the hundreds of currently available mobile apps that overlay digital games, tourist information, advertisements, and more on top of what we see through our device’s video camera.
The concept of visual AR is nothing new. The Terminator and RoboCop movies of the 1980s and 1990s, for example, let viewers see through the eyes of a robot, with all manner of digital readouts floating in the air around the faces and objects the robot viewed. The more recent Iron Man movies demonstrate how innovators can make use of digital images that can be viewed and manipulated just like physical objects. The digital scrimmage and first down lines that have appeared on the field in televised football games for decades are among the earliest examples of this concept being provided to a mass audience.
But this is the point in history when mobile computing has become sufficiently powerful and ubiquitous that consumers everywhere can experience truly immersive and effective AR. The enthusiasm and interest sparked by Google Glass™ has ensured that digital eyewear is now a category of mobile devices that is here to stay, and many of those devices are being designed with AR in mind.
Where there is commercial activity and rapid innovation, there will be intellectual property disputes galore. The following is a brief summary of the intellectual property issues that are most likely to present themselves in the near future.
Patent infringement litigation is already so commonplace in the mobile device industry that it has become an inevitable cost of doing business. It should be no surprise, therefore, that we have already seen a number of patent infringement lawsuits related to AR technology.
Perhaps the first AR-related patent lawsuit was Tomita Technologies USA, LLC v. Nintendo Co., filed in June 2011. The claimed infringement involved a sliding control button on the Nintendo 3DS handheld console, one of the first attempts at AR gaming. The slider adjusts the three-dimensional image displayed in the 3DS’s “AR mode.” A jury in the Southern District of New York returned a verdict in Tomita’s favor and awarded it $30.2 million in damages, although the court reduced that amount by half in an August 2013 opinion.4
More significant to the AR industry, however, is the patent enforcement campaign being pursued by nonpracticing entity Lennon Image Technologies LLC. It has targeted a number of retailers with websites featuring “virtual try-on” technology. This form of AR uses a computer’s webcam to recognize a customer’s hand, arm, face, or entire body, and superimpose on it such apparel as rings, watches, eyeglasses, and dresses. This technique had been spurring a lot of interest from retailers and customers alike—until Lennon got involved.
Lennon owns a patent issued in 2003 entitled “Customer Image Capture and Use Thereof in a Retailing System.” The patent’s abstract describes a method of capturing a customer’s image “at [a] retailer’s place of business” and blending it with prerecorded “models wearing apparel items” to allow the customer “to virtually assess the selected merchandise without actually having to try on the apparel.”5 The several obvious gaps between this description and the AR websites did not deter Lennon from filing six identical infringement lawsuits in 2012, followed by another seven in March 2013. Among the targets were well-known brands such as Mattel, Macy’s, Bloomingdale’s, Luxottica, and Tacori.
Each targeted retailer immediately removed the feature from its website, and many have since quietly settled. But the damage to the industry remains. This sort of virtual try-on technology was the primary business model for several AR startups. Even those that were not forced to indemnify their customers from these suits—itself a potential death sentence to a fragile startup—still face the daunting challenge of convincing new potential customers to experiment with the technology and risk liability. As AR innovations expand into various sectors, others are sure to face similar litigation roadblocks.
Because so much of what makes AR compelling is its ability to display creative text and images in new ways, the potential for copyright issues is obvious.
Chief among these are copyright owners’ exclusive rights of reproduction and alteration.6 Before digital eyewear is able to add digital content to our view of the world, the devices must first be able to know what we’re looking at—hence the video cameras that come standard on all such devices that have been announced. Most of the public discourse to date around these cameras has centered on their privacy implications. But making audiovisual recordings of our everyday surroundings creates an obvious potential for reproducing copyrighted visual art, text, and music.
Of course, the potential for such reproduction is already present in today’s mobile devices. But once AR software begins to make this captured footage “greater” by adding digital content to it, there is a potential for creating derivative works. This could be as subtle as taking pictures or video of our surroundings with static digital imagery overlaid on top. After all, just as desktop computer users have long customized the appearance of their screen’s wallpaper, so too will AR users eventually be able to customize the look and feel of the “clickable world” around them. Or it could be as complex as the recent mobile AR app designed to recognize the face of Captain Barbossa in the Pirates of the Caribbean: On Stranger Tides movie poster, and cause it to morph before the user’s eyes into the face of the CEO of Goldman Sachs. Because nothing actually changes on the physical poster, the question of whether that sort of functionality infringes depends on such facts as how much of the original poster was reproduced within the app in order to create the effect, and whether the end result can be fixed into tangible form and distributed.
The fair use doctrine will also take on new dimensions. The political statement behind the Pirates app (that this investment banker is “the real pirate”) raises an obvious fair use question, as will the ongoing debate (in such cases as Cariou v. Prince7) over whether “mashups” and other appropriation art are transformative uses.
Public display and performance rights will also be at issue, in sometimes novel ways. The British Museum released a mobile app designed to show users historical London photos in the actual, public location where they were taken. (Sort of a digital version of DearPhotograph.com, which itself could be called “analog AR.”) The photo itself never leaves the confines of the mobile device, but its display is only triggered by the user’s physical location. Is that a “public” display? And if so, has the app developer purchased or licensed the appropriate rights for that public display from the copyright owner? Such questions will grow in importance as our surroundings become populated with triggers for all sorts of digital data.
Enforcement will also be a major challenge in this context. The mass file-sharing lawsuits of the past decade have required a significant amount of detective work and discovery to connect individual users to allegedly infringing downloads. One can imagine that it will become even more difficult to prove that a particular user viewed a particular work when the “display” occurred entirely within a mobile headset. I expect that many litigators will soon be conducting “v-discovery,” in which they must determine not only the device to which virtual data was routed, but also where individual users were located, and in what direction they were looking, when the data was displayed.
On the other hand, AR eyewear could also be used as a copyright enforcement mechanism. The YouTube video A Read-Only Future8 depicts life through the eyes of someone wearing digital eyewear that is regulated by the entertainment industry. His glasses recognize copyrighted content in the user’s field of view—such as a photo hanging on the wall or a song being played on the sidewalk—and obscures it unless he agrees to a micro-license payment. Just as in concept videos for actual digital headsets, the eyewear in this video is able to share content directly to Facebook, but will refuse to do so if it detects unlicensed content. It even alerts the authorities if the user stumbles across an unauthorized reproduction published by someone else. Excerpts from Larry Lessig feature prominently in A Read-Only Future, which plays out as if it were Lessig’s nightmare.
We can be certain that, as digital content gets published in augmented media, trademark-laden commercial content will follow. Perhaps the most extreme (and disturbingly plausible) depiction of “sponsored” AR can be found in Keiichi Matsuda’s short video Augmented (hyper)Reality: Domestic Robocop.9 The AR user in this video sees literally every flat surface in his modest kitchenette digitally plastered with branded advertisements. At one point he even manually raises the “advertising level” of his eyewear, suggesting that he’s receiving micropayments or subsidized services for each ad he sees.
Certainly, trademarks displayed in this manner could be infringed or diluted just as they are online today. But AR’s signature blending of the digital with the physical offers avenues of unlawfulness that have never existed before. Eyes were opened to this fact in 2010 by the iPhone app “The Leak in Your Home Town.” Through this app, one could view a physical sign bearing the BP logo at a local gas station, and see superimposed on that logo a digital broken pipe spewing oil, exactly like the one responsible for the then-current spill in the Gulf of Mexico.
The political commentary motivating this app may have made a strong argument for fair use if it had ever been challenged, but that didn’t stop a flurry of legal commentary on the specter of “augmented trademark infringement”10—although the much more obvious concern in this context is dilution. The concern centers on the use of trademarked logos as triggers for augmented content. In one way or another, AR programs need to recognize physical objects in order to properly superimpose digital content on them. Early AR apps relied on printed QR codes as “targets,” but corporate logos—which are always displayed in a uniform manner and with maximum visibility—are the perfect substitute.
Hundreds of companies—including Starbucks, Lego, Moosejaw, Maxim, and more—have already run successful marketing campaigns based on consumers using a mobile app to find augmented content on branded physical goods. As branded products become more widely recognized as platforms for receiving digital content, though, those brands will need to be more concerned about who else may be augmenting their content. Trademark law exists in order to maintain the purity of the message conveyed by a mark. The more successful third parties become at repurposing popular marks as triggers for subversive digital commentary, the more concerned the brand owner will be that the distinctive quality of its mark is being diluted. How far the First Amendment’s freedom of speech will allow brand owners to go in regulating such content, however, remains to be seen.
Competitors are also likely to begin augmenting branded goods in order to gain market advantage. For more than a decade, we have seen litigation over keyword and sponsored advertising on Internet search engine pages. AR eyewear will not only relocate the search function from a computer screen to a user’s heads-up display, but it will also allow visual recognition of physical objects to trigger search results and advertising. When that happens, today’s advertising techniques will migrate into augmented space as well. For example, what is to stop Burger King from paying to have its ad pop into a user’s field of view whenever the user merely looks at a McDonald’s road sign? Just as in today’s keyword cases, this may become a generally accepted means of comparative advertising in the augmented age. But even that will not stop brand owners from aggressively policing augmentations of their marks that they believe go too far in creating a likelihood of confusion.
As the BP example shows, single-purpose apps already exist for purposes like this. It is when large groups of users start using the same, multipurpose platform for all their AR consumption that we will see these concerns arise in earnest. That will happen sooner rather than later, just as a few clear favorites rose above the pack in the Internet search and social media industries. Competitive forces and advertising dollars guarantee it. And when that happens, you can be sure that advertisers will be clamoring for laws and judicial decisions that protect their ability to control the message conveyed by their brand in every facet of reality.
Just as trademarked objects can easily serve as triggers for digital content, so too could the physical characteristics of individual people. The simmering debate over facial recognition technology and privacy is a preview of the concerns we will face when a large segment of the population is wearing eyewear capable of recognizing the faces of others. To date, industry-leading companies have shown remarkable restraint in implementing such features. Google has disallowed facial recognition apps on its Glass headset, and Facebook has refrained from rolling out the technology to the degree that it could.
But as AR hardware proliferates, it will be impossible to keep this genie in the bottle. The potential commercial applications are just too numerous and profitable to expect such restraint from all service providers. AR concept videos are chock full of examples in which digital data—including links to social media profiles, dating service information, and even whether he or she is a registered sex offender—is seen hovering in the air over a person’s head. Facial recognition is by far the easiest and most direct means by which to associate such displays with a particular person.
The main concern voiced about this technology to date has been “privacy,” although society in general seems to have no consensus about what that word actually means. The right of publicity—that weird, state-law transitional species between the common law of privacy and intellectual property—will play an increasingly prominent role in this debate going forward. To see why, just follow the money. Privacy law is about keeping things out of the public view. Publicity rights, on the other hand, allow the person whose likeness is being exploited to profit from it.
Before long, someone is going to make the argument that facial recognition technology infringes the publicity rights of the person being scanned. It is surprising that no one seems to have made this argument in court yet. Right of publicity law regulates the commercial exploitation of a person’s identity, which is generally thought to include at least his or her physical appearance. The same commercial forces that guarantee the expansion of facial recognition will also provide plenty of evidence demonstrating the commercial value of the data. It will not take a scholar to connect the dots and argue that the people scanned should recoup a portion of any money made from their biometric data.
Whether this argument gains any traction is another matter. Biometric data is already widely used for entirely utilitarian (and especially security) purposes—witness, for example, the fingerprint scanner in the new iPhone 5S. Allowing people to own intellectual property rights in that data might complicate matters too much for that technology to remain useful, to the detriment of society as a whole.
Facial recognition, however, may not be the only way in which AR-enabling technologies exploit biometric information. Mass-market devices like Microsoft’s Kinect are already designed to recognize entire bodies. A few years ago, artists in Spain set up a booth that used three Kinect cameras to scan individuals from head to toe. That data was relayed to a 3D printer in order to make a personalized figurine of the person right there on the street. Today, there are companies simultaneously using more than 60 sensors more precise than the Kinect to digitally render individuals in real time with amazing accuracy.
AR applications will take advantage of such capabilities in order to superimpose digital data on a person’s entire body. Personally, I’m waiting for the day when I don’t have to worry about digging the right clothes out of my closet; instead I’ll press a few buttons and the people nearby will see on me whichever outfit I’ve programmed.
Other applications, however, will go beyond augmented display to image capture and repurposing. In an age where sexting is an epidemic among teens and states like California are forced to outlaw the repurposing of such content (e.g., “revenge porn”), it does not require much imagination to conceive of the unsavory uses to which 3D personal imaging technologies could be put. To date, in courts across the country, one of the most frequent reasons for invoking the right of publicity has been to enjoin the prurient use of girls’ and women’s images, which are often recorded unwittingly. It is logical to expect the same laws to be applied when those images are collected and manipulated by new digital media.
How effective this right will be in these new augmented realms remains to be seen. The right of publicity has always existed in tension with the First Amendment’s protection of free speech, and often finds itself preempted by the Copyright Act as well. Both of these more-established bodies of law are likely to keep publicity rights from expanding too broadly. But there is still quite a bit of conduct that falls within the gray area between these areas of law, where the boundaries have yet to be definitively drawn.
This article has only begun to describe the amazing potential of augmented reality technology, and the intellectual property considerations that come along with it. To be sure, those currently innovating in this area will find boundaries to push that no one has yet anticipated. At the very least, however, it is safe to say that the practice of intellectual property law is not likely to get boring any time soon.
3. See, e.g., Adrian David Cheok, Digital Taste Interface, Mixed Reality Lab, http://mixedrealitylab.org/projects/all-projects/digital-taste-interface (last visited Oct. 25, 2013).
4. Tomita Techs. USA, LLC v. Nintendo Co., No. 11 Civ. 4256 (JSR), 2013 U.S. Dist. LEXIS 116486 (S.D.N.Y. Aug. 14, 2013).
5. U.S. Patent No. 6,624,843 B2 (filed Dec. 8, 2000).
6. See 17 U.S.C. § 106.
7. 714 F.3d 694 (2d Cir. 2013).
10. See, e.g., Chris Cameron, Are We Entering the Age of Augmented Trademark Infringement?, ReadWrite (July 6, 2010), Article no longer available.