IQ tests share this property of undervaluing creativity and overvaluing conformity with academic work in general.
Witness my favorite renegade intellectual, Robert Pirsig, from his Wikipedia page:
'While doing laboratory work in biochemistry, Pirsig became greatly troubled by the existence of more than one workable hypothesis to explain a given phenomenon, and, indeed, that the number of hypotheses appeared unlimited. He could not find any way to reduce the number of hypotheses--he became perplexed by the role and source of hypothesis generation within scientific practice. This led him to an awareness of a (to him) previously unarticulated limitation of science, which was something of a revelation to him. The question distracted him to the extent that he lost interest in his studies and failed to maintain good grades; he was finally expelled from the university.'
I score pretty well on IQ tests, but only by suppressing a constant incipient rage at many of the questions, and a constant effort to ask myself "what would be the most boring and conformist interpretation of this?" That is usually the "right" answer.
Witness my favorite renegade intellectual, Robert Pirsig
I went to high school with a classmate who used to have the Pirsig family over for dinner fairly regularly. Robert Pirsig's father Maynard Pirsig was a law professor at the University of Minnesota, where the classmate's father (and, years later, I) studied for a law degree. I think, having read only Zen and the Art of Motorcycle Maintenance among Robert Pirsig's writings, and after limited personal acquaintance with Maynard Pirsig, that Robert Pirsig takes a lawyer's "that's debatable" attitude as he approaches scientific and philosophical questions, which bogs down his quest for certainty.
AFTER EDIT: I've seen an interesting pattern of drive-by voting on this comment, but I'd like to know more about what other people who are acquainted with the writings of Maynard Pirsig (father) and Robert Pirsig (son) think about the influence of the one writer on the other.
I'm not familiar with the Pirsigs, but I certainly do not consider the attitude in question to be lawyerlike. While lawyers do occasionally face philosophical conundrums, they also tend to have the desired answer known to them. This is considerably easier than to, say, set a prior for the domain of a model parameter, where your preferences are in a vague war between convenience and justice. The internal debate arises when working from first principles and finding too few.
Interesting. I first read ZAMM first when I was 15, and really enjoyed it, but I would strongly encourage everyone who has not read "Lila" to please read it, as Pirsig basically retracts most of what he said in ZAMM. In his own words, "Lila" is his "smart child", and the book he wants to be remembered by. It is a much better book than ZAMM.
If anyone is interested, I just discovered William James Sidis's "The Animate And The Inanimate" here:
Science has several rules of thumb, Occams Razor says that the simpliest is usually the more likely, and the assumption that the universe operates according to universal, consistant mathematical rules means that you should favour theories that match every other part of observable reality, the rule of thumb that extraordinary claims require extraordinary evidence is like occam's razor and means, again, choose the simpliest explaination that matches your evidence.
Hence, you can sort theories, and then try to devise tests to falsify each one to differenciate between them.
The problem with Occam's Razor is that it requires you to identify the simplest explanation. In fields like for example molecular biology deciding what is 'the simplest' often is difficult and debatable.
The author didn't name Occam's Razor, but she did touch on the idea that simplicity can be an unhelpful or ambiguous criteria to apply:
"Usually when you are asked to continue a pattern the assumption is that you are supposed to choose the simplest way. But sometimes it is difficult to decide what the testers think the simplest way is. Can you replace the question mark with a number in the following sequence: 31, ?, 31, 30, 31, 30, 31, … You might say that the answer is 30 as the numbers alternate; or, you might say that the answer is 28 as these are the days of the month."
This isn't to say that Occam's Razor isn't helpful -- it often is, if for no other reason than the fact that model/theory with fewer "moving parts" is simply easier to work with than another with more (and a similar level of explanatory power). Just that simplicity sometimes can't help or that it can be a value judgment.
I think in this case simplicity does reveal the answer. I would put 30 because the sequence becomes a repeating sequences of 31, 30. Answering 28 requires making the assumption that the sequence of numbers represent days of the month, which has a higher information entropy making the sequence less simple.
In a lot of modern biology however you will have tens or hundreds of thousands of equally simple competing hypotheses. Biologist have had to adapt to this by using multiple hypothesis testing methods, but Prisig was working well before this became standard, so perhaps this was the barrier he was running up against.
Pirsig actually came up with an entire metaphysical system in response to this. It's in his book "Lila". Basically it deals with what "good" (as in a good theory, or answer) is, and the different kinds of goodness or Quality, as he calls it, there are.
It's quite detailed, but his first distinction is between 'static' and 'dynamic' good. A test-writer values static good more highly than dynamic, and while this is justifiable in the circumstances, in the big picture, dynamic good is the real deal.
I think that Occam's razor says, that if we have several theories that describe reality with the same accuracy, we should chose the most succint (least redundant).
I don't think of it in terms of redundancy, but of introduced entities. That is, if two theories have equal explanatory power, but one theory requires ten new concepts, but the other one requires one new concept, prefer the latter. This interpretation is, I think, the spirit of what he intended: http://en.wikipedia.org/wiki/Occams_razor#Ockham
That's a good way to put it. If you want to be mathematically rigorous about it, you could phrase it as a statement about probabilities: for any A and B, P(A and B) <= P(A).
And if more than one theory is more-or-less equally probable, simple, and in accord with observed reality then you test all of them to see which you can eliminate. Repeat.
Yes. The context that has to be taken into account includes the social context, ie. 'I am sitting in a room taking a test, I cannot request clarification of the question, the person who wrote this test may be far less intelligent than me, etc'.
By making an effort to choose the possible answer that I perceive as the most boring and conformist, rather than one I might find brilliant, funny, novel, and suggestive of further even more interesting ideas, I am pursuing a strategy of "favouring theories that match every other part of observable reality", as you put it.
Is there any reason that in this case and in the case of the parent comment, there's a tendency to jump to conclusions about the intelligence level of the test/puzzle writer? Do truly intelligent people never write lousy questions?
When I was young, I was furious at "Skip the first letter, keep every other letter, and what does it spell?" Data was something like "ghfeklmlto", and my answer was "hfeklmlto", of course.
Bayesian uses a broader prior -- a distribution over possible prior distributions of facts, instead of assuming a single possible prior (Gaussian). In that sense you could sort of squint and say that that's the contrast.
Technically, you're allowed to use whatever prior you like. A beautiful thing about Bayesian inference is that it tells you how to update your prior, without making any assumptions about what it may be.
Frequentist approaches do not use any prior at all. Indeed some of the issues that frequentists wind up getting genuinely concerned about, such as repeated significance testing errors, are clear logical fallacies if you're doing any sort of Bayesian analysis.
Not sure what you mean by 'frequentist', but one thing is for sure: the more hypotheses that occur to you, the longer it takes to go through them testing each one not only for consistency, plausibility etc, but also for how likely it is that you think the test-writer would prefer the rationale for that answer. If only a few hypotheses occur to you, and they tend to be in the same ballpark as those that would occur to the average test-writer, you have an advantage.
My understanding is that the approach of coming up with a hypothesis and testing it is called "frequentist inference", in opposition to "Bayesian inference". However, I'm not terribly well versed in the subject and was hoping somebody could give a clear explanation of the difference between the two.
"Bayesian" and "frequentist" are descriptions of interpretations of probability more than of methods of inference.
Briefly and inexactly: A frequentist says that a probability is the answer to a question of the form "if we repeat this situation many times, what fraction of the time will this happen?"; a Bayesian says that many other things, such as (idealized) subjective degrees of belief, behave like probabilities -- i.e., they obey the same mathematical rules -- and that anything that obeys those rules deserves to be called a probability.
There is such a thing as Bayesian inference, but its opposite isn't "frequentist inference" but something like "classical hypothesis testing". If you take the frequentist view of probability then you will reject questions like "how likely is it that this treatment works?" because either it works or it doesn't -- there's nothing for a probability to be a long-run frequency of. But of course that really is the kind of thing you want to know, so you'll look for other similar questions that do make sense, such as "If the treatment doesn't work, how likely is it that we'd get results as impressive as these?". Asking that question leads to "Neyman-Pearson hypothesis testing": form a "null hypothesis" (the treatment doesn't work; the two groups of people are equally intelligent; the roulette table is not rigged by the casino; ...), do some measurements somehow, figure out how likely it is if the null hypothesis is right that you'd get results as unfavourable to the null hypothesis as you actually did, and if it's very unlikely then you say "aha, the null hypothesis can be rejected". Here, "very unlikely" might mean probability less than 5%, or less than 1%, or whatever.
If you've ever seen an academic paper in science or economics or whatever that says things like "eating more white bread is associated (p<0.01) with being a Mahayana Buddhist", that "p<0.01" thing is that same "probability of results as unfavourable to the null hypothesis"; in this (made-up, of course) case it would be something like "probability of the chi-squared statistic being as large as we found it, if bread consumption and Mahayana Buddhism were independent".
Bayesians, on the other hand, are perfectly happy talking more directly about the probability that Mahayana Buddhists eat more white bread. However, then another difficulty arises. Obviously the experimental results on their own don't tell you that probability. (Simpler example: you know that a coin was flipped five times and came up heads every time. How likely is it that it's a cheaty coin with two heads? You'll answer that quite differently if (a) you just pulled the coin out of your own pocket or (b) some dubious character approached you in a bar and invited you to bet on his coin-flipping.) The probability after your experimental results are in is determined by two things: those results, and what you believed beforehand.
On the other hand, here's a possible advantage of the Bayesian approach: Instead of just saying how confidently you reject (if you do) the null hypothesis, you can talk about the probabilities of various different extents to which it could be violated. (Imagine two medical treatments. One is more confidently known to have some effect than the other -- but the effect the second one might have is ten times bigger. You might prefer the second treatment even though the risk that it doesn't really work is bigger.)
So the Bayesian statistician might end up saying something like this: Here's a reasonable "prior distribution" -- i.e., a reasonable assignment of probabilities to the various possibilities we're interested in, before the experimental results. Now here's what our probabilities turn into if we start there and then take account of the experimental results.
Or they might just describe the impact of the experimental results, and leave it up to individual readers to combine that with their own prior probabilities.
Here's the recipe that's at the heart of Bayesian inference: Take the prior probability for each possibility. Multiply it by the probability of getting exactly the observed results, if that possibility is the case. The result is the final probability except that the resulting probabilities won't generally add up to 1, so you have to rescale them all by whatever factor it takes to make them add up to 1.
A similar thing occurred to me not too long ago, when I was learning about type theory: if you have the type of some function, then the set of possible implementations of that function is (usually) countably infinite, so the type doesn’t really tell you anything. But of course you probably care only about one of the simplest, so the type really does tell you an awful lot.
Anyway, I’m with you on the business of IQ tests. The problem is that as answers become increasingly open-ended, they are both more representative of a person’s actual intelligence, and more difficult to score objectively.
Witness my favorite renegade intellectual, Robert Pirsig, from his Wikipedia page:
'While doing laboratory work in biochemistry, Pirsig became greatly troubled by the existence of more than one workable hypothesis to explain a given phenomenon, and, indeed, that the number of hypotheses appeared unlimited. He could not find any way to reduce the number of hypotheses--he became perplexed by the role and source of hypothesis generation within scientific practice. This led him to an awareness of a (to him) previously unarticulated limitation of science, which was something of a revelation to him. The question distracted him to the extent that he lost interest in his studies and failed to maintain good grades; he was finally expelled from the university.'
https://en.wikipedia.org/wiki/Robert_Pirsig
I score pretty well on IQ tests, but only by suppressing a constant incipient rage at many of the questions, and a constant effort to ask myself "what would be the most boring and conformist interpretation of this?" That is usually the "right" answer.