Romanian Witches Aren’t The Only Ones Failing to Predict the Future


Recently, my aunt was discussing this article from the online website Salon about how there are far more men than women getting their books published (in the traditional way).

I don’t know why but I read Salon every morning, even though the writing there (including the linked piece) tends to often be of substandard quality. If you’re looking to improve your English writing skills, avoid Salon! However what I want to address today is a very common inductive fallacy from the article:

The imbalance in books published [i.e. more male authors than female authors] is indeed a puzzle; book publishers, like any other business, want to make money.

Let’s assume the following facts are true:

  • Publishing companies want to make money.
  • Women and men are more or less inherently equally skilled at writing books (i.e. gender is not a factor).
  • Well-written books sell better than poorly-written books.
  • Therefore any publisher wanting to make money would publish (roughly) an equal number of books written by female authors as by male authors.

This certainly seems to make sense. And yet even if all of the above are true, there could easily be a bias in favor of male authors (as indeed seems to be the case).

Why? Because of the fallacy of game theory, the most famous proponent of which is John Nash, whom several of you may know from the film A Beautiful Mind.

Military and political strategists took (and often still DO take) game theory quite seriously. So do members of an equally important cohort – economists – with devastating results.

The fatal error here is what’s known as a “rational actor”. In the case of a publishing company, which certainly does want to make money, this would mean that they will always act “rationally”. If a good manuscript comes across their desk, they will eagerly sign a contract with the author and publish it regardless of the gender (or race, etc) of the author. After all, it seems a little silly to turn down money simply because of the genitalia (or skin color, etc) of the person who wrote it.

And yet it happens all the time. The most famous case is that of “JK” Rowling, the author of all those incredibly profitable Harry Potter books (and at least one of you knew her personally before she was famous!).

We all know her as “JK” Rowling instead of “Jo” or “Joanne” precisely because her publisher felt that people wouldn’t read her books if they knew she was a woman. Obviously that wasn’t the case as the woman is a billionaire and I’ve never once heard (even anecdotally) of anyone refusing to read the book because they knew it was written by a woman.

So the question we have before us is how could the publishers be so wrong? How could five or six of them reject her original manuscript if it was obviously so good (at least in terms of sales)? How could the one who finally decided to publish her be so wrong about people not reading her books if she used her obviously female name on the cover?

Again, this dovetails right into one of my hobbies – probability theory. Publishing companies certainly DO want to make money. And the way they do this is by looking over manuscripts and trying to predict which ones will be successful. If John Nash and economists and game theorists were right, the publishing companies would have some kind of rational algorithm that would allow them to more or less accurately (and impartially) assess manuscripts and predict which of them will sell well (and thus be profitable).

The problem is that there is no way to predict which books will sell well. Sometimes big-name authors with previously well-received books will write one that doesn’t sell well at all. Likewise, sometimes a book by an unknown person (such as Ms. Rowling) with very little expectation of success will write a blockbuster that makes billions of dollars.

The publishers really have no way of knowing what will happen. They try to cover it up with “experience in the field” and their “gut feelings” and “intuition” and “market research” and everything else but at the end of the day, it’s inherently unpredictable. It is a classic Black Swan problem.

What makes the situation even worse is that the (traditional) publishing world is not even close to being a “free market”. It’s dominated by a few huge companies that have their own institutionalized beliefs about “what sells” and how to market a book. They have incestuous ties to one another (and literary agents) and it is a relatively small industry. Just a few thousand people decide the fate of millions of books.

We’ll never know the masterpieces that were submitted and rejected by these publishers (written by female authors or male). We’ll never know of the great books that might be best sellers but were poorly promoted or marketed. All we see are the “winners” – the books that have sold well, and yes, most of them were written by men. Perhaps some of the books we’ve never read were rejected by these publishers due to sexism but it could equally just be because of the fact that while the publishers (and literary agents) act like they can recognize a good book and predict which ones will sell, in reality they can’t.

There is a solution to the particular case of publishing books and it’s called self-publishing, which I’ve written about extensively here.

If people who buy books (aka “readers”) could walk into a store (or browse online) and read samples of books for free and then buy them on demand, then the public would start winnowing out the true winners (aka talented writers) from the losers (untalented writers). Then a woman (or a person with a “foreign” name, etc) would have an equal opportunity to sell their books and perhaps even make a decent living to boot.

The method above would still be skewed in an absolute sense due to something called the power law (also known as the Fat Tail). If readers in my hypothetical “print on demand” bookstore could rate books then the earliest books to receive high ratings would then disproportionately continue to receive high ratings. But it would be a much fairer system, since there would be no intervening screen of publishers deciding FOR US what is and what is not quality writing (and thus worth publishing and marketing).

I’m sure that in the near future I will write an extended article about Romania’s economy because a lot of the same aspects come into play, albeit on a larger scale. To give you an example of what I’m referring to, let’s consider the following as true:

  • People in the Romanian government (and Central Bank, etc) want Romania’s economy to grow.
  • Choices on what money to borrow and where to invest it are based purely on what is good for the economy.
  • Everything from the laws that are written to business regulation to tax codes are designed to maximize the profitability of businesses and society as a whole.

Even if – even if all of the above were true, there would still be catastrophic failures in policy implementation (i.e. unprofitable policies would be implemented). In other words, even if every politician and member of the government were a saint and pure-hearted and never, ever corrupt, greedy or stupid, they would still make gigantic mistakes due to “non-rational” choices that people make and Black Swans.

Understanding system complexity is regrettably something very few people expend their mental energy on. One notable exception is John Robb, who mostly talks about terrorists and the like, but a lot of what he says is equally applicable to the world of publishing books and the government of Romania making economic policy decisions.

Long story short, the smaller the group of people making decisions, the greater their error rate is going to be. This is why I often applaud corruption in Romania, not because it’s fair or good for people (and it certainly sucks when it happens to you) but because it’s actually mitigating the Black Swan errors in the ability to predict the future by the “good” members of the government and their economic policy decisions. Sometimes corruption actually blocks decisions made by “good” people which would end up being quite harmful.

To put it into publishing terms: imagine if a few hackers could log into the computers of major publishing houses and insert books they wrote (or their friends wrote) into the system and get them published and onto bookstore shelves. By random processes alone, a few of those “pirate” books would actually be written by talented authors and get recognized by the buying public and turn into bestsellers purely on their own merit.

Luckily for authors (and people who read books), the world of self-publishing is expanding. There have already been a few best sellers that were self-published, some of them from manuscripts rejected by the world of traditional publishing.

Every copy I sell of The Complete Insider’s Guide to Romania potentially means that one less copy of a traditional guidebook on Romania is sold. If things continue the way they are, and my book continues to get recognition (and continues to sell), then ultimately that will influence the traditional publishers directly as they’ll tell their guidebook authors to write something “like Sam’s book”, whatever that means.

Predicting the future is not just the domain of witches (taxed or not :P). It’s absolutely vital in some fields, whether publishing books or taking massive billion euro loans from the IMF in order to run Romania’s economy. Tragically it is however, largely misunderstood and quite often disastrously misapplied, even by people who DO have good intentions.

4 Comments Add yours

  1. Anca says:

    The only bit of Harry Potter I’ve read was the first two pages of the first book. I thought it was crap, but given that it was written for kids, it wasn’t surprising. I’m not sure what this says about the adults who love it. I have seen most of the movies, which are good, so it seems the story is solid even if the original writing was geared to 10-yr-olds.

    I don’t see why JK Rowling would have been rejected for being a female writer given that her book straddled two genres (fantasy & young adult) that have many many (successful) female authors (if not the majority).

    Like

  2. Dacia Felix says:

    Sam,

    Your posting fits perfectly in the Long Tail theory about the mass market moving to the niche market.
    http://www.longtail.com/the_long_tail/about.html
    Your book fills a neded in the niche market as Romania is not yet on the travel map

    Like

  3. Valentin says:

    This reminds me of a fallacy I consider typical to the male mind. The Romanian Script dictates that a perfect woman does not exist, but if it did, it would have all of these three qualities:
    1. Beautiful (Frumoasa)
    2. Smart (Desteapta)
    3. Faithful (Devreme acasa)
    These three are all rational premises, even if they are subjective (beauty), incomplete (smarts) or hard to quantify (faithfulness). The conclusion is fallacious: if someone would meet all the conditions above, she would be “perfect”. The mind is therefore content with finding a partner that will best satisfy these conditions, ignoring other aspects. When a relationship degrades to the point of breakup, one usual complaint is: “she was crazy”. This of course would be just as valid when considering what women see as “the perfect man” and the fallacious conclusions derived from considering this value system as quasy-complete.
    But this where similarities to big business inertia and big government incompetence ends, as individuals are more likely to fail and learn from their mistakes. After a couple of “crazy” partners, people will start to consider adding another point to that list above.
    On the subject of publishers, I’m sure that even after JK Rowling’s success with Harry Potter, the publishers will not “add to the list”, as they will simply dismiss it as a “fluke”. Big companies run themselves by using statistics, so the larger the company, the larger the fluke.
    A free market with true competition would have meant that more publishing houses would have seen Rowling’s book and the chances of them all ignoring it would significantly decrease.
    As far as publishing goes, I’m glad competition is coming from the online sector. On the long run this will be highly beneficial for everybody. At the moment people still insist cutting trees to make books and filling half their living room with carcasses of dead books would actually mean something. Yeah, I consider that once you read a book (can be multiple times), it’s just a carcass and keeping it is like keeping the skeleton of the chicken you ate yesterday to remind yourself how tasty it was!

    Like

  4. Sorin says:

    Nice blog my dear Sam; read it all in a few days ;)
    Interesting article (in the link), and it might be true, but my guess is that men write more books in general covering all domains, than women.
    Personally i don’t read novels or any other belletristic works, only history and science books interest me ( i hated romanian language classes in middleschool/highschool – together with french/english/german/philosophy/debate classes/civic classes/even “dirigentie” classes were a f-ing drag for me – and i was glad to have rid myself of most of them in college).
    Keep the posts coming.
    Gday

    Like

Got something to say? Try to be nice!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.