Tag: experiment

The problem of unintended consequences

Eine praktisch verwertbare sozialtechnologische Analyse muß die Beschaffenheit der Ausgangssituation für politisches Handeln, zum Beispiel für die ins Auge gefaßte Gesetzgebung berücksichtigen. Und sie muß dabei auch der Tatsache Rechnung tragen, daß eine solche Gesetzgebung nur bestimmte – meist sehr eng umschriebene – Bestandteile der in­stitutionellen Konstellation in einer Weise ändern kann, die voraussehbare Auswirkungen in der erwünschten Richtung hat. [27]

Piecemeal social experiments

Such arguments in favour of Utopian engineering exhibit a prejudice which is as widely held as it is untenable, namely, the prejudice that social experiments must be on a ‘large scale’, that they must involve the whole of society if they are to be carried out under realistic conditions. But piecemeal social experiments can be carried out under realistic con­ditions, in the midst of society, in spite of being on a ‘small scale’, that is to say, without revolutionizing the whole of society. In fact, we are making such experiments all the time. The introduction of a new kind of life-insurance, of a new kind of taxation, of a new penal reform, are all social experiments which have their repercussions through the whole of society without remodelling society as a whole. Even a man who opens a new shop, or who reserves a ticket for the theatre, is carrying out a kind of social experiment on a small scale; and all our knowledge of social conditions is based on experience gained by making experiments of this kind. … But the kind of experiment from which we can learn most is the alteration of one social institution at a time. For only in this way can we learn how to fit institutions into the framework of other institutions, and how to adjust them so that they work according to our intentions. And only in this way can we make mistakes, and learn from our mistakes, without risking repercussions of a gravity that must endanger the will to future reforms. … But the piecemeal method permits repeated experiments and continuous readjustments. In fact, it might lead to the happy situation where politicians begin to look out for their own mistakes instead of trying to explain them away and to prove that they have always been right. This—and not Utopian planning or historical prophecy—would mean the introduction of scientific method into politics, since the whole secret of scientific method is a readiness to learn from mistakes. [ch. 9, 176-7]

Fisher on significance tests

In considering the appropriateness of any proposed experimental design, it is always needful to forecast all possible results of the experiment, and to have decided without ambiguity what interpretation shall be placed upon each one of them. Further, we must know by what argument this interpretation is to be sustained. …

It is open to the experimenter to be more or less exacting in respect of the smallness of the probability he would require before he would be willing to admit that his observations have demonstrated a positive result. It is obvious that an experiment would be useless of which no possible result would satisfy him. Thus, if he wishes to ignore results having probabilities as high as 1 in 20—the probabilities being of course reckoned from the hypothesis that the phenomenon to be demonstrated is in fact absent … . It is usual and convenient for the experimenters to take 5 per cent. as a standard level of significance, in the sense that they are prepared to ignore all results which fail to reach this standard, and, by this means to eliminate from further discussion the greater part of the fluctuations which chance causes have intro­duced into their experimental results. No such selection can eliminate the whole of the possible effects of chance co­incidence, and if we accept this convenient convention, and agree that an event which would occur by chance only once in 70 trials is decidedly “significant”, in the statistical sense, we thereby admit that no isolated experiment, how­ever significant in itself, can suffice for the experimental demonstration of any natural phenomenon; for the “one chance in a million” will undoubtedly occur, with no less and no more than its appropriate frequency, however surprised we may be that it should occur to us. In order to assert that a natural phenomenon is experimentally demonstrable we need, not an isolated record, but a reliable method of procedure. In relation to the test of significance we may say that a pheno­menon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us a statistically significant result. [12-4]

“Data” aren’t simply “given”

There are no sensory “data”. Rather, there is an incoming challenge from the sensed world, which then puts the brain, or ourselves, to work on it, to try to interpret it. Thus, at first, there are no data: there is, rather, a challenge to do some­thing; namely to interpret. Then we try to match the so-called sense data. I say “so-called” because I don’t think there are sense “data”. What most people hold to be simple sense “datum” is in fact the outcome of a most elaborate process. Nothing is directly “given“ to us: perception is arrived at only as a result of many steps involving interaction between the stimuli which reach the senses, the interpreting apparatus of the senses, and the structure of the brain. So, while the term “sense datum” suggests primacy in the first step, I would suggest that, before I can realize what is a sense datum for me (before it is ever “given” to me), there are a hundred steps of give and take which result from the challenge pre­sented to our senses and our brain. [430]

Shadows of Baconian induction (2)

The Large Hadron Collider is the most complicated scientific experiment ever built. But it’s still just an experiment like any other. At its heart, there is repeatable process, as with Newton’s prism. There are teams of people dedicated to making detailed measurements, as Cavendish did with his flammable air. And the same rigorous logical thought processes used by Bill Tutte are of course applied here too. These are simple principles, yet they hold great power. [51:15]

Shadows of Baconian induction

There’s table after table of results. But he also describes precisely how he got those results. There’s a beautiful diagram of his apparatus, and this is there so that anyone else reading this paper, if they’re sceptical about the results or even if they just want to check them, can rebuild the apparatus and redo the experiment and check that Tyndall didn’t make any mistakes.

So these results are not a matter of opinion: they’re here, they can be checked by other scientists, they can be verified. So this is how scientific knowledge progresses. Publishing is the reason why science gets to our best view of the way that nature works. [43:28]

Newtonian induction

The common explanation for the appearance of the colours was that they were added by impurities in the prism to the pure white light. Newton thought that the colours were already present in the white sunlight. But what set Newton apart was that he devised and performed an experiment to test his hypothesis. …

Green light into the prism equals green light out. That implies that the colours themselves are pure. The prism is not subtracting or adding anything. That means that Newton’s hypothesis was shown to be correct. …

Newton was one of the first to interrogate Nature using the principles of what we now call the scientific method. In other words, he observed the world, came up with theories to explain what he saw, then tested them with experiments to see if he was right. The power of this approach is that it aims to remove preconceived ideas and, in doing so, deliver a more accurate description of the natural world.  [7:40]

The shorter scientific method

British scientists have made, and continue to make, some of the great scientific discoveries. But of equal importance, from a historical perspective, was the development of the means by which we do science. The idea that you build theories, you test them by experiment, and you publish the results: this is known as the scientific method. It is the bedrock of science. [3:38]

The experiment of democracy

The methods of science — with all its imperfections — can be used to improve social, political, and economic systems, and this is, I think, true no matter what criterion of improvement is adopted. How is this possible if science is based on experiment? Humans are not electrons or laboratory rats. But every act of Congress, every Supreme Court decision, every Presidential National Security Directive, every change in the Prime Rate is an experiment. Every shift in eco­nomic policy, every increase or decrease in funding of Head Start, every toughening of criminal sentences is an experiment. Exchanging needles, making condoms available, or decriminalizing marijuana are all experiments. Doing nothing to help Abyssinia against Italy, or to prevent Nazi Germany from invading the Rhineland, was an experiment. Communism in Eastern Europe, the Soviet Union, and China was an experiment. Privatizing mental health care or prisons is an experiment. Japan and West Germany investing a great deal in science and technology and next to nothing on defense — and finding that their economies boomed — was an experiment. Handguns are available for self-protection in Seattle, but not in nearby Vancouver, Canada; handgun killings are five times more common and the handgun suicide rate is ten times greater in Seattle. Guns make impulsive killing easy. This is also an experiment. In almost all of these cases, adequate control experiments are not performed, or variables are insufficiently separated. Nevertheless, to a certain and often useful degree, policy ideas can be tested. The great waste would be to ignore the results of social experiments because they seem to be ideologically unpalatable. [423]

Science doesn’t teach us anything

When someone says science teaches us such and such, he is using the word incorrectly. Science doesn’t teach it; experience teaches it. … And you have as much right as anyone else, upon hearing about the experiments (but we must listen to all the evidence), to judge whether a reusable conclusion has been arrived at. [187]