https://smile.amazon.com/gp/product/B007V2VEQO/

Author runs a business analytics company and is involved in economic/political thinktanks.

These notes are really sparse! There is a lot of interesting detail in the book.

Structure of science

Much in common with The Beginning of Infinity. Induction doesn’t work, competition between explanations etc.

Paradigm shift. Without any background assumptions it’s hard to even determine what has been falsified. Paradigm dictates how to test theories and how to interpret observations.

Branching hierarchy of paradigms, from broad trunks like falsification to narrow twigs like FMRI clustering.

Science itself is a paradigm which has competitors - common sense, tradition etc. Each embodies implicit learned/evolved knowledge about the world and has different mechanisms that generate, change and select ideas.

Rise of randomized experiments

Forces against MRA and similar post-hoc analysis:

Meta-analysis by Ioannidis of 49 influential studies. 80% of non-randomized studies found weaker or no effect vs 10% of ‘large’ randomized studies.

Meritful criticisms of randomized trials in medicine:

Fall of randomized experiments

Causal density - density of direct causal relationships between variables. Crud factor = correlation. Causal density = causation. Can we distinguish them?

Holistic integration - everything interacts with everything else, so separate experiments cannot be performed. Eg in macro-economics we can’t run an experiment on one country and use another country as a control if those two countries interact heavily.

With high causal density but low holistic integration, we can still run experiments on some groups and use others as controls, but analysis is hard.

With high holistic integration but low causal density, we can get by with post-hoc statistical controls but we can’t create randomized controls and replication is hard.

Problems like social policy and macro-economics display high causal density and high holistic integration.

Internal validity - did the treatment work in the sample. External validity - will similar treatments work in similar samples ie to what extent can we generalize this result.

Problem of generalization: given high causal density, it’s very hard to determine the extent of external validity.

The experimental revolution is like a huge wave that has lost power as it moved uphill through topics of increasing complexity and holism. Physics was entirely transformed; therapeutic biology required statistical experimentation due to higher causal density but could often rely on the assumption of uniform biological response to reliably generalize findings from randomized trials; the yet-higher causal densities in social science make generalization from even properly randomized experiments hazardous, and integrated complexity of certain topics in social science appears to be fully impervious to experimentation.

Attack this on many fronts:

Lacking a powerful compact theory, we resort to many specialized micro-theories.

(Why does astronomy work? No ability to perform true experiments but: Low causal density. Low holistic integration. Universe is really big. Means that there are many good natural experiments. Also, much of the underlying laws are universal and can be separately tested in controlled experiments.)

Business

Describes the rise of experimental methods of business.

Jam experiment - hazards of over-generalization. Single experiment on single day in single store. Many, many failed replications that never reached the public awareness.

In practice, businesses grapple with high causal density by repeated iterative testing (similar to pharma funnels):

ie using experiments not to discover some global immutable truth, but as part of an optimization process.

Practical advice:

Authors rule of thumb: experimental methods > experienced expert > non-experimental methods > inexperienced ‘expert’

Politics

Compared to medicine and business, social policy has a dearth of experiments. Discusses what decent work exists.

Folk wisdom:

Liberty-as-goal vs liberty-as-means. Former wants liberty as a fundamental value. Latter sees liberty as a useful mechanism for designing systems that learn and improve over time eg markets, science, experiments within business all rely on freedom to try new ideas combined with a framework for weeding out bad ideas.

Liberty-as-means implies that there are valid reasons for restricting liberty eg to leverage economies of scale of collective action (eg taxation), to reduce transaction overheads (eg trade standards, fraud laws) or to prevent gaming of selection mechanism (eg anti-monopoly / anti-collusion laws).

Last chapter gives a series of concrete suggestions for US policy.