Malcolm Gladwell and the statistical basis of Outliers

Malcolm Gladwell promotes his book on Colbert. Go watch it.

Isn't it just wonderful how comedy can get to the nub of things and
throw thoughtful, serious people off-balance?

So, Malcolm Gladwell has published a new book: Outliers. It is
Gladwell's summary of, and meditation on, the large volumes of
research on the topic of extraordinary achievement. As usual, the
reviews marvel at his clear prose and credit him with making a tough
topic easy to understand.

Being quite a success story himself, people are getting tougher on
Gladwell. Part of his talent as a great writer is that he makes it
seem easy; lots of stuff is folded neatly in his sentences and
narrative. It takes some getting used to before you can attempt to
unravel his logic. Possibly the strongest criticism of Gladwell's
body of work (NYT, New Yorker, and the three books) is that he
simplifies too much.

I have not read the book! But he sure picks'em - the topics. From
the optimistic message of "Tipping Point", via the equally
optimistic "Blink", he now tackles a topic that has more or less
created the Self-Help movement: extraordinary success.

His message, as reported by the reviews, is again optimistic:
extraordinary success (outlier success) is a product of
highly-specific circumstances, and therefore there is an
awfully-large amount of talent out there that we should not
overlook because it does not act, look, or sound like "successful"
people do.

I once gave a research tutorial on my PhD topic: Data Clustering,
and in one of the feedback forms, I had this guy saying nice things
about my talk, except that because the subject of his PhD is
"Outliers" - he found my talk lacking in that respect. He wrote that
he wanted a "more solid treatment of the topic of Outliers". So,
take what I write below as totally un-solid.

In the field of Pattern Recognition, where my scientific training
has been, Outliers is a special topic. Pattern Recognition - for
those who need a small introduction - is a subject usually
classified under Computer Science that combines statistics and
computer algorithms for the purposes of learning models from data.
For example, you want a computer to learn someone's voice from a
number of his speech recordings for the purpose of recognising that
voice in new, unheard-before recordings. That's a typical pattern
recognition problem.

The fundamental assumption employed in these situations is that a
person's voice can be typified. Often, however, it is not. It varies
according to his stress level, background noise, time of day, etc.
And sometimes the voice that the computer thinks is his, is not, and
vice versa. Typically, the decision rests on whether a given voice
is within the margins of expectation of the voice we are trying to
recognise, or outside of it. Is it an outlier or not?

Thus, the topic of "Outliers" - an established sub-field of
statistics by itself - got linked-in to pattern recognition. I am
not an expert in Outliers - otherwise I would not have got that
comment from the student - but I know enough to know that it is
almost a philosophical question.

Something could be an outlier in one 'representation space' (just
think of 'space' for now) but very typical in another representation
space. For example, in one space - a courtroom (say) - a man could
be exceptionally important - the judge, but in another room - a
hospital waiting room - he is just another patient. Representation
spaces transform outliers into typicals, and vice versa.

Also, let's fix the representation space for now, just how much, how
far, do you go before you say something is an outlier? Maybe it is
noise (a random, unwanted artefact) and not an outlier.
Distinguishing between noise and outliers is a huge pain in the neck
for people in the field.

Another angle: maybe with more data, or in more time, the outliers
cluster around each other, which would mean they are not outliers
but actually a distinct but tiny cluster. Once something is part of
a cluster it is not an outlier.

To recap, the statistical definition of outliers is that they are
not noise, and they do not congregate in tiny clusters. When you
factor in that they are tricky to hunt down because they change
status from outlier to typical when the representation space is
changed, you realise how tough this problem of identifying outliers
is.

Malcolm Gladwell says that Bill Gates and Mozart are outliers. Why
aren't we saying they are noise? (Bill Gates is a random, unwanted
artefact!) And in what representation space are we working? Are we
measuring money, acclaim, extraordinary musical talent, what?

If we had more data, would Mozart still be an outlier, or would he
coalesce with Timbaland, Prince, Beethoven, and others into a
distinct, tiny cluster of "exquisite musicmanship"?

Still, I agree with Gladwell's fundamental message: there is an
awful lot of luck involved in success, and an awful lot of wasted
talent on earth. If Warren Buffet was born in Egypt, would he have
been as rich and as famous? I hope Malcolm Gladwell's book manages
to create a dent in how people assess the "potentially successful".
Amen. Yes sir, please.