Controlled experiments are like the gold nuggets in the world of data-driven decision making. They provide a platform for organizations to test their ideas by assigning different variants to users and observing the impact. For instance, Bing, the search engine, saw a 12% increase in revenue, amounting to over $100 million per year, just by experimenting with minor tweaks in ad display.

For an experiment to be effective, organizations need certain elements. Firstly, they need experimental units, like users, who can be assigned to different variants without any interference. Secondly, they need a sufficient number of these units. Thirdly, they need to agree on the metrics to evaluate, especially an Overall Evaluation Criterion (OEC). Lastly, they need the ability to make changes with ease. The OEC should be a quantifiable measure of the experiment's objective that can be measured in the short-term but also drives long-term goals.

It's important for organizations to understand that they are not always the best at predicting the value of ideas. In many companies, only about 10-30% of the ideas tested actually improve key metrics. Sometimes, small changes can lead to big impacts. For example, Google tested 41 shades of blue and saw a significant increase in engagement. Experiments can help fine-tune strategy by testing tactics and uncovering new opportunities.

Controlled experiments go hand in hand with agile development. They promote continuous improvement through a series of small changes that collectively make a big difference. While most ideas may not work, the few that do can yield substantial rewards. Therefore, organizations should test as many ideas as possible. Experiments help reduce uncertainty and risk, enabling data-driven iteration towards better strategies.

However, it's crucial to ensure that experimentation is trustworthy for it to effectively guide data-driven decisions. Often, results that are surprising or interesting are indicative of errors, as suggested by Twyman's Law. Common pitfalls to avoid include lack of statistical power, misinterpretation of p-values, premature peeking at results, and multiple hypothesis testing. Validity threats include sample ratio mismatch, survivorship bias, Simpson's paradox, and primacy and novelty effects. To enhance trust, it's advisable to run sanity checks, encourage skepticism, and understand the context when interpreting results. The focus should be on identifying meaningful differences, not just statistical significance. Proper experiment design and investment in the unknown can help foster a culture of trustworthy experimentation.