Black Swan Ideas (1)

There’s a good reason why The Black Swan is a best seller. It’s written in a very lively style with great narratives, literary images, and vivid terms and phrases. Nassim Nicholas Taleb (NNT) talks about scalability, non-scalability, Extremistan, Mediocristan, the fallacy of silent evidence, confirmation error or platonic confirmation, epistemic arrogance, future blindness, the lottery-ticket fallacy, the ludic fallacy, Mandelbrodtian randomess and Gray Swans, the narrative fallacy, Platonic folds, Platonicity, randomness as incomplete information, retrospective distortion, and the round-trip fallacy. In future Knowledge Management blogs, I’ll be commenting on many of these ideas. In this blog, I’ll begin with NNT’s distinction between Medocristan and Extremistan, and will also comment on the idea of randomness as incomplete information.

Both Mediocristan and Extremistan are Weberian Ideal Types. They are “utopian” places, “countries,” “provinces,” or “lands.” What distinguishes these, first of all, is the idea of “non-scalability” vs. “scalability.” Non-scalability exists when the consequences of an activity are dependent on the labor or effort invested. The remuneration of a Doctor, Lawyer, or industrial worker isn’t scalable because its based on the number of hours worked. The work of an author, however, may produce gains that are unrelated to its quantity. These gains are scalable in the sense that a relatively small amount of work can lead to a huge return. NNT has had a career as a trader. Traders can make a fortune in a single day if they can sense the direction of the market and ride a wave.

In Mediocristan, nothing is scalable, everything is constrained by boundary conditions, time, the limits of biological variation, the limits of hourly compensation, etc. Because of such constraints and the limits of our knowledge, random variation of attributes exists in Mediocristan, and can be usefully described by Gaussian probability models (the bell curve or other distributions having a family resemblance to it). In such “orderly” randomness models, probability distributions are such that no single instantiation of the value of an attribute can greatly affect the sum of all values in the distribution. Even the most extreme attribute values do not materially affect the mean value of a distribution, because the more extreme any value is, the more improbable it is that the extreme value will actually occur in nature. Games of chance are the paradigmatic generators of phenomena found in Mediocristan. The distributions they produce provide the best fit to Gaussian models. And individual Black Swan events almost never occur because their probability, according to Gaussian models, is so low. When they do occur, it is because someone thinks that events are generated deterministically, when they are actually generated by Gaussian random processes, and so they encounter the unexpected events that violate their deterministic expectations.

In Extremistan, variation within distributions, is far less constrained than in Mediocristan. It is the land of scalability. Generators of events produce distributions with very large or very small extreme values, relatively frequently. And those extreme values often affect the sum of attribute values in a sample distribution, and the mean value of such distributions. The probability of occurrence of extreme values varies greatly from Gaussian models. In fact, many attribute value distributions in Extremistan do not fit any known models well. Examples of them include sales distributions for books per author, wealth and income distributions for individuals and businesses. Since extreme occurrences can greatly affect statistical properties of distributions from Extremistan, it is hard, in contrast with data from Mediocristan, to make reliable inferences from sample data.

The values in Extremistan distributions represent events that are Black Swans in NNT’s sense. But there are also rare events in Extremistan that are not Black Swans. They are Gray Swans in the sense that they are somewhat tractable scientifically, and reflect what NNT calls “Mandelbrodtian randomness.” These distributions follow or approximate “scalable, scale-invariant, power laws, Pareto-Zipf laws, Yule’s law, Paretian-stable processes, Levy-stable, and fractal laws.” (p. 37)

Now, the importance of the distinction between Extremistan and Mediocristan is that for NNT, “while weight, height, and calorie consumption are from Mediocristan, wealth is not. Almost all social matters are from Extremistan.” (p. 33) That’s because, he says, social quantities are informational, and can take on any value without expending energy. A consequence of this is that attempts to predict social phenomena based on techniques using Gaussian models will render us “fooled by randomness” because they won’t either account for or lead us to expect Black Swans or Gray Swans. We may, at first, and for some time use these models to help us predict successfully. But they will lull us into a false sense of security, and ultimately we will invest or otherwise act on a prediction from the model that will cost us dearly. On the other hand, if we keep in mind that social events are from Extremistan, where Black and Gray Swans occur frequently, we will be alert to the possibility of their occurrence, and will expect them to sooner or later occur. This will affect our strategies for investment, or action and will allow us to build in hedges that allow us to cope much more successfully with the unexpected.

The distinction between Extremistan and Mediocristan is thought-provoking and perhaps even a bit compelling. But I wonder what it adds to the idea that social phenomena can’t be described or explained by models using Gaussian probability or related models, while they can sometimes be described and explained much more successfully using models from what NNT has called the category of “Mandelbrodtian” randomness. That is, I can’t see what the invocation of the provinces of Extremistan and Mediocristan really accomplishes by way of explanation of distributions that don’t fit either Gaussian or other probability models with a family resemblance to them.

Moreover, in pointing out that Black Swans can occur in Mediocristan due to errors in models generating erroneous expectations, NNT reminds us of the subjective aspect of his notion of “randomness”. He views it “as incomplete information: simply what I cannot guess is random because my knowledge about the causes is incomplete, not necessarily because the process has truly unpredictable properties.” (p. 308) Thus, for NNT, whenever a model about a process is contradicted by data and violates our expectations we have “randomness.” So random events are always Black Swans, or at least Gray ones whether they occur in Mediocristan or Extremistan, so long as they don’t fit the particular models one has been using to study events in the domain in which the unexpected occurs.

One of NNT’s favorite philosophers is Karl Popper, whom he praises effusively, for his skeptical empiricism and emphasis on conjectures and refutations. But he seems unaware of, and does not discuss, Popper’s lifelong opposition to the idea than randomness is about incomplete information, that the world is fundamentally deterministic, and that, if we have enough information, everything can be explained with reference to efficient causation. In The Open Universe (1982) and in an Appendix to recent editions of The Logic of Scientific Discovery, Popper gives his account of the propensity interpretation of probability which views both probability and randomness as occurring as a function of the generating conditions of processes.

Randomness, according to the propensity interpretation and Popper’s axiom system for probability, is not, in general, a function of our ignorance, but is an objective attribute of certain real world processes. If this account is right, and NNT doesn’t confront it in the Black Swan, there are not two kinds of randomness, one most characteristic of Mediocristan, and the other most characteristic of Extremistan, but only one kind of randomness, the kind that is generated by certain real world processes. Errors in both “stans” are caused by our ignorance, and not by randomness, and they are to be cured in both cases by creating better models, theories, conjectures and other formulations of knowledge claims that can survive our best tests, criticisms, and evaluations.