Below you will find answers to questions you may have about field experiments and working with economists.
WHAT IS A FIELD EXPERIMENT? Field experiments apply the scientific method to examine the effect of an intervention in the real world. Economic experiments randomize subjects into treatment and control groups and compare outcomes between these groups. Randomization guarantees that if we observe a difference between a treatment and a control group, that difference must be caused by the treatment. Randomization also allows you to control for several potential explanatory factors such as age, gender, and income level. This enables you not only to understand what works, but also why. Field experiments allow you to discover the most efficient way to allocate your resources.
WILL THE NAME OF OUR ORGANIZATION BE ON THE ACADEMIC PAPER? It is your choice whether the name of your organization will or will not appear in an academic paper.
WILL YOU ACQUIRE PERSONAL IDENTIFIABLE INFORMATION? No. Like medical experiments, economic experiments need to be approved by an Institutional Review Board (or Ethical Review Board). The IRB process is aimed at guaranteeing the safety of participants. In the case of economic experiments, safety typically means privacy. To receive IRB approval, two general conditions need to hold: 1) the experiment does not involve deception of any kind; 2) no personally identifiable information is retained. We abide by what academics call data de-identification: while we retain individual-level data that is useful for research purposes (e.g. age, gender, income, level of education etc.), we try to design our experiments so that no personal identifiable information is collected (e.g. names, addresses, phone numbers, social security numbers). If the nature of the experiment requires acquiring some sensitive information, we will ensure that the data is de-identified before any dataset is built. IBE academic fellows routinely work with sensitive data and we have established practices aimed at guaranteeing that sensitive information always remains within the organization. IBE fellows regularly sign nondisclosure agreements and data usage agreements according to the specific needs of the organization. If your legal team is interested in more details, please reach out to us and we will provide you with more information.
IF WE RUN AN EXPERIMENT, WILL THE SCHOLARS OWN THE DATA? Absolutely not. IBE academic fellows routinely prepare with our partners a Data Usage Agreement— a contract that specifies that the organization remains the sole owner of all data generated. The partner University, however, will own the results and analysis performed on the data generated by the experiment. This is to guarantee the academic freedom of publishing novel results in scientific journals.
HOW MUCH DOES IT COST TO COLLABORATE WITH IFE? The Institute for Behavioral Economics does not charge its members. However, organizations should be prepared to cover some costs for the experiments, if jointly deemed necessary.
WE ALREADY USE SURVEYS AND FOCUS GROUPS—WHY RUN AN EXPERIMENT? Surveys and focus groups are an excellent way to learn about people’s intentions. However, what people say they would do is often very different from what they actually do. This is because people are affected by the context in which they make a decision, by the amount of attention they put in that decision, and by their emotions. Surveys may not capture these factors because people often do not know ex-ante what they will pay attention to or they do not consciously realize that context clues or emotions affect their decisions. Experiments allow us to observe how different interventions affect behavior in natural environments.
WHAT IS THE TIMELINE OF AN EXPERIMENT? Economic experiments are set up to take place organically within your business activity. How long the experiment is run and when it is implemented will be decided in its planning, with all partners involved.
WILL EXPERIMENTS "POISON" MY POOL OF CUSTOMERS? Firms and organizations are rightly concerned that some "bad" treatments may drive some customers away. While possible, experiments are set up precisely to avoid applying a "bad" intervention to all customers. Only a subset of your customers will be exposed to new treatments and once we find the most effective way to solve your problem, you will be able to use the solution with your entire pool. As the proverb goes, it is better to be safe than sorry.
WE ALREADY USE A/B TESTS, WHAT IS THE DIFFERENCE WITH FIELD EXPERIMENTS? Many companies, particularly young startups, routinely use A/B testing: from choosing the right user interface, to choosing the best day to send out newsletters, to finding the right wording for communications with clients. However, A/B testing alone can hardly be used for anything besides optimizing business practices at the margin. Without a motivating theory, A/B testing alone can merely reveal what works, but are silent about why things work the way they do. This limits the generalizability of A/B tests, as well as their predictive power. To truly unleash the power of randomization and use experiments to address core business issues, A/B tests need to be informed by theory. The combination of tested (and testable) theories about human behavior and randomized controlled experiments designed to falsify these theories is what allows the most successful firms in the world to use experiments to gain a deep and transformative understanding of their business. It is no coincidence that major firms and institutions all have dedicated teams of economists that closely work with their data scientists to understand the economics behind their businesses. As economists say: "Theory without data is empty. Data without theory is blind".
WHAT IS A FIELD EXPERIMENT? Field experiments apply the scientific method to examine the effect of an intervention in the real world. Economic experiments randomize subjects into treatment and control groups and compare outcomes between these groups. Randomization guarantees that if we observe a difference between a treatment and a control group, that difference must be caused by the treatment. Randomization also allows you to control for several potential explanatory factors such as age, gender, and income level. This enables you not only to understand what works, but also why. Field experiments allow you to discover the most efficient way to allocate your resources.
WILL THE NAME OF OUR ORGANIZATION BE ON THE ACADEMIC PAPER? It is your choice whether the name of your organization will or will not appear in an academic paper.
WILL YOU ACQUIRE PERSONAL IDENTIFIABLE INFORMATION? No. Like medical experiments, economic experiments need to be approved by an Institutional Review Board (or Ethical Review Board). The IRB process is aimed at guaranteeing the safety of participants. In the case of economic experiments, safety typically means privacy. To receive IRB approval, two general conditions need to hold: 1) the experiment does not involve deception of any kind; 2) no personally identifiable information is retained. We abide by what academics call data de-identification: while we retain individual-level data that is useful for research purposes (e.g. age, gender, income, level of education etc.), we try to design our experiments so that no personal identifiable information is collected (e.g. names, addresses, phone numbers, social security numbers). If the nature of the experiment requires acquiring some sensitive information, we will ensure that the data is de-identified before any dataset is built. IBE academic fellows routinely work with sensitive data and we have established practices aimed at guaranteeing that sensitive information always remains within the organization. IBE fellows regularly sign nondisclosure agreements and data usage agreements according to the specific needs of the organization. If your legal team is interested in more details, please reach out to us and we will provide you with more information.
IF WE RUN AN EXPERIMENT, WILL THE SCHOLARS OWN THE DATA? Absolutely not. IBE academic fellows routinely prepare with our partners a Data Usage Agreement— a contract that specifies that the organization remains the sole owner of all data generated. The partner University, however, will own the results and analysis performed on the data generated by the experiment. This is to guarantee the academic freedom of publishing novel results in scientific journals.
HOW MUCH DOES IT COST TO COLLABORATE WITH IFE? The Institute for Behavioral Economics does not charge its members. However, organizations should be prepared to cover some costs for the experiments, if jointly deemed necessary.
WE ALREADY USE SURVEYS AND FOCUS GROUPS—WHY RUN AN EXPERIMENT? Surveys and focus groups are an excellent way to learn about people’s intentions. However, what people say they would do is often very different from what they actually do. This is because people are affected by the context in which they make a decision, by the amount of attention they put in that decision, and by their emotions. Surveys may not capture these factors because people often do not know ex-ante what they will pay attention to or they do not consciously realize that context clues or emotions affect their decisions. Experiments allow us to observe how different interventions affect behavior in natural environments.
WHAT IS THE TIMELINE OF AN EXPERIMENT? Economic experiments are set up to take place organically within your business activity. How long the experiment is run and when it is implemented will be decided in its planning, with all partners involved.
WILL EXPERIMENTS "POISON" MY POOL OF CUSTOMERS? Firms and organizations are rightly concerned that some "bad" treatments may drive some customers away. While possible, experiments are set up precisely to avoid applying a "bad" intervention to all customers. Only a subset of your customers will be exposed to new treatments and once we find the most effective way to solve your problem, you will be able to use the solution with your entire pool. As the proverb goes, it is better to be safe than sorry.
WE ALREADY USE A/B TESTS, WHAT IS THE DIFFERENCE WITH FIELD EXPERIMENTS? Many companies, particularly young startups, routinely use A/B testing: from choosing the right user interface, to choosing the best day to send out newsletters, to finding the right wording for communications with clients. However, A/B testing alone can hardly be used for anything besides optimizing business practices at the margin. Without a motivating theory, A/B testing alone can merely reveal what works, but are silent about why things work the way they do. This limits the generalizability of A/B tests, as well as their predictive power. To truly unleash the power of randomization and use experiments to address core business issues, A/B tests need to be informed by theory. The combination of tested (and testable) theories about human behavior and randomized controlled experiments designed to falsify these theories is what allows the most successful firms in the world to use experiments to gain a deep and transformative understanding of their business. It is no coincidence that major firms and institutions all have dedicated teams of economists that closely work with their data scientists to understand the economics behind their businesses. As economists say: "Theory without data is empty. Data without theory is blind".