The Power of Experimentation in the Tech Sector
Randomized experiments have gone mainstream in the technology sector, serving as a critical tool in evaluating the impact of an invention. In the search for continuous product improvements, technology companies extensively rely on experimental evidence to make informed business decisions aimed at refining products, platforms, and technologies. The systematic testing and experimentation of ideas enables tech companies to test hypotheses quickly, resulting in frequent product improvements. As Jeff Bezos has stated, “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.” Based on the findings from randomized experiments, firms can answer questions like:
- Should we incorporate this new feature into our product?
- How much are customers willing to pay for this feature?
- Do specific user groups value this technology more than others?
For example, consider a technology firm that wants to determine whether a different version of a product feature is more effective in increasing user engagement on its mobile app than the current version. The firm could employ a randomized experiment and show the current version to a randomly selected group A and the new version to group B, with group A and group B being as similar as possible in terms of demographics, interests, engagement, etc. The firm could then measure which version has higher engagement rates based on metrics such as time spent on the app or number of clicks on the feature. The version that performs better is then used for future releases.
The process of running such an experiment typically involves several steps, including (1) defining the hypothesis, (2) designing the experiment, (3) selecting and randomizing the user groups, (4) measuring engagement rates, and (5) analyzing the results. The number of people involved in each step can vary depending on the size of the company and the complexity of the experiment. For example, a small startup may have one person responsible for designing and running an experiment, while a larger company may have a team of people working on different aspects of the experiment. The cost and length of time it takes to run an experiment can also vary depending on the complexity of the feature being tested and the size of the user groups being tested. A simple randomized experiment can be run in a matter of days or weeks, while more complex experiments can take several months. A user experience copywriter explained the speed and autonomy that he experienced at Booking.com: “I can come up with an idea over breakfast, bike into work, and have it implemented and live well before lunch.” In general, randomized experiments are relatively inexpensive to run for technology companies. In fact, given the advancements in data processing capabilities of technology firms, and the relevant infrastructure, the marginal cost of running digital experiments is ever approaching zero. Such real-time experiments replace labor-intensive customer surveys to understand customer behavior and the impact of innovations on core business metrics.
As discussed below, evaluating the impact of an invention through experimentation is particularly relevant in the context of IP litigation where the core challenge is determining the value of a technology. Randomized experiments can provide empirical evidence of a technology’s value, based on its impact on user behavior and willingness to pay. Thus, the practice of randomized experimentation not only drives innovation in the technology sector but also can play a key role in addressing the core challenges in IP matters.
A Core Challenge in IP Matters
The computation of damages in IP matters focuses in large part on the value of the IP to the infringer and the owner. Whether as a part of lost profits, unjust enrichment, disgorgement, or a reasonable royalty, damages experts are required by the courts to focus on the value of the specific technology in question. This is often separate and a subset of the technology of the accused product as a whole. Indeed, expert testimony is often excluded due to a lack of apportionment to the technology at issue.
For example, in litigation involving patent infringement, the teachings of the asserted patent may represent a minute segment of the accused product. Consider an extreme example: in 2012, RPX estimated that “there are more than 250,000 active patents relevant to today’s smartphones.” Consideration of the value of any patent or group of patents relevant to the accused product often requires the damages expert to perform apportionment from the entire value of the accused product down to the value specifically tied to the teachings of the asserted patent. Such apportionment can be done in any number of ways, including, but not limited to, comparing products with and without the patented technology, reliance on a technical expert or feature counting, conjoint surveys, and hedonic regressions. Often, a damages expert will determine the best method based on available evidence. However, evidence probative of the value of the specific technology in question can be difficult to obtain. Even if such evidence is available, it may be interpretable in various ways, leading opposing experts to look at the same documents and come to different conclusions.
Economists and other analytical practitioners are well-versed in the application of randomized experiments to infer causal effects. Indeed, the 2021 Nobel Prize in Economic Sciences was awarded to David Card, Joshua Angrist, and Guido Imbens for their contributions in clarifying conclusions that can be drawn about cause and effect through the use of natural experiments. Central to such studies are the analysis of significance level of results and the power of such analysis. These concepts are often neglected by damages experts in valuing asserted technology; there simply is not a natural experiment that can be leveraged for such a robust analysis.
Linking Experimentation with the Value of IP
Just as technology firms increasingly rely on fast, inexpensive, randomized experimentations to drive product innovation, a similar approach could be beneficial in the realm of IP damages. Currently, damages experts rely on the best available information, requiring assumptions to arrive at an apportioned value of a technology. Courts, litigators, and expert witnesses alike could benefit from leveraging randomized experiments as a more robust and reliable tool to assess the value of a technology within IP litigation.
Randomized experiments can address many of the challenging questions that arise in IP litigation in a manner that is accessible to practitioners in the field at relatively low cost. In particular, well-designed and well-implemented causal experimentation can provide accurate and reliable indicators of the value of an invention to either party. In a properly performed randomized experiment, the invention can be designated as the only difference between the control and treatment group, allowing for an understanding of the invention’s value.
There are numerous examples of experiments that could be run by IP litigation defendants in the technology industry to answer questions such as:
- Does an alternative payment method lead to higher or lower conversion rates across consumers?
- What impact does the use of a new algorithm have on social media use (and thus advertisement efficacy)?
- How does the use of a specific trademark or copyright influence purchasing decisions among consumers?
While surveys have traditionally been used to answer some of these questions, they can be time-consuming, costly, and limited by small sample sizes. In contrast, the experimentation methods employed by technology firms offer a viable alternative that is capable of facilitating larger, more efficient, and less expensive studies that can answer a wider range of questions than surveys. Randomized experiments can strengthen the evidence in a case by providing large-scale evidence using actual customers as subjects to credibly answer questions in patent litigation matters.
Technology companies are already running experiments like these every day and have in-house, easy-to-use experimental infrastructure that could be leveraged for the purpose of their legal proceedings. As the then senior product director at Booking.com, Andrea Carini, said: “We have a philosophy of testing as much as we can live with customers. . . . Everything is tested, from entire redesigns and infrastructure changes to small bug fixes.” Yet, the use of experimentation capabilities within litigation seems to be limited at the moment.
In certain situations, the use of randomized experiments might present an improvement over current ad hoc methods in both precision and understanding. When concerned with the value placed on an innovation, practitioners and litigators generally rely on available production combined with economic methods to accomplish a valuation for intellectual assets. However, these resources are not always designed to establish a causal link between the accused innovation and the value users attribute to it. But under certain conditions, experimentation can help courts, attorneys, and expert witnesses to shed light on the value of intellectual assets.
Randomized experiments are helpful when:
- applying the income approach to determine the incremental value to the alleged infringer—in cases where there are highly comparable agreements available as a basis for a royalty opinion under the market approach, experiments may serve as additional evidence rather than as primary value indicators;
- there is limited documentation regarding the alleged infringer’s valuation of the technology in question—by conducting controlled experiments, it becomes possible to generate empirical data that sheds light on the value associated with the asserted technology;
- the asserted technology is software-based as opposed to hardware-based—for example, when assessing the value of a specific type of camera in a smartphone, it could be challenging and expensive to gather data on user smartphone purchases but less challenging to modify a checkout process for a group of consumers;
- the metric used for assessing the economic value of an invention is sufficiently sensitive to the specific feature at issue, enabling the timely detection of the impact—for example, a certain checkout process may lead to incremental follow-through, but attempting to measure the impact of a new customer service feature on total user purchases may be difficult; and
- the time frame in question (e.g., a hypothetical negotiation) is closer to the present—the proximity in time ensures that the experiment’s results align with current market conditions and provide more accurate insights into the economic value associated with the technology in dispute.
Conclusion
Randomized experiments have gone mainstream in the technology sector, serving as a critical tool in evaluating the impact of an invention. The questions that arise in IP litigation parallel those addressed by experimentation in technology firms, suggesting a natural alignment of these methodologies. Yet within IP litigation, damages experts often rely on ad hoc methods that may lack the precision and robustness of experimental approaches. Randomized experiments may be used to assess the value of IP within legal proceedings. Courts, attorneys, and expert witnesses would benefit from embracing the power of experimentation, leveraging existing experimental infrastructure within technology companies. Experimental evidence not only offers a level of accessibility to fact finders comparable to that of survey evidence but also might provide a more precise and efficient approach in assessing the value attributable to an invention. The field of IP litigation could enhance its precision, efficiency, and evidentiary robustness, thereby better serving the interests of justice.