The ease with which companies can experiment depends on how easily they can observe outcomes. Direct-mail houses, catalog companies and online retailers can accurately target individuals with different actions and gauge the responses. But many companies engage in activities or reach customers through channels that make it impossible to obtain reliable feedback. The classic example is television advertising. Coke can only guess at how viewers responded to its advertising during the last Olympics, a limitation recognized by John Wanamaker's famous axiom, "Half the money I spend on advertising is wasted; the trouble is, I don't know which half." Without an effective feedback mechanism, the basis for decision making reverts to intuition.

In practice, most companies fall somewhere between these two extremes. Many are capable of conducting tests only at an aggregate level, and they're forced to compare non-equivalent treatment and control groups to evaluate the response. If Apple wants to experiment with the prices of a new iPhone, it may be limited to charging different prices in different countries and observing the response. In general, it's easier to experiment with pricing and product decisions than with channel management or advertising decisions. It's also easier to experiment in consumer settings than in business-to-business settings, because B2C markets typically have far more potential customers to serve as subjects.

Our feedback mechanism allows you to observe how customers respond to different treatments. We cover both the types of feedback metrics: behavioral and perceptual. Behavioral metrics measure actions — ideally, actual purchases.

Perceptual measures indicate how customers think they will respond to your actions. This speculative form of feedback is most often obtained via surveys, focus groups, conjoint studies and other traditional forms of market research. These measures are useful in diagnosing intermediate changes in customers' decision processes.

Given that the goal of most firms is to influence customers' behavior rather than just their perceptions, experiments that measure behavior provide a more direct link to profit, particularly when they measure purchasing behavior.