Nearly 20 years ago I was desperately poor and unemployed when I got a temporary job doing manual labor. Over the space of three or four days I was paid minimum wage to pack and load heavy boxes onto a truck. And yet that demeaning job all those years ago changed my life.
If you’re Romanian the following story will seem a little bizarre but if you’re American it will probably be perfectly understandable. Attending university in America is extremely expensive and one small part of that expense is the textbooks that the students have to buy each year. Some of the textbooks used for certain classes could cost as much as $200. And at the end of the school year, the lone bookstore which sold these overpriced books would bring in an outside company that would buy back these books (at a drastically reduced price). A book which originally sold for $100 might be bought at the end of the year for $20.
Since the school year is over, a lot of students felt that they no longer had any need for their books and while $20 is far less than the $100 that they originally paid, it’s still better than nothing and so they’d come to this little event and sell off all of their books. My job with this company was to pack these books into boxes and then stack them onto a truck. But the company I was working with didn’t buy all of the books that the students brought in. They had a little list and sometimes a title was being replaced with a new edition or something else and so they knew that they couldn’t re-sell the book the next school year. And so students were coming in with books that A) they no longer wanted and B) the company wouldn’t buy back at any price.
Occasionally but very rarely, the students would take their unsold books back home with them but far more commonly would simply throw them away in a bin we had set up for that very purpose. My boss was a Nazi and treated the laborers like crap but he was more than happy to allow me to take as many of the discarded books as I wished. Why not? To both he and the students they were trash. And yet just a few months earlier these were expensive text books used to teach advanced classes. Free books, man! Free university level textbooks on almost every subject imaginable.
I think you can guess that I took quite a few of these books home (and read them). And a short while later I began my career with the government as an information systems specialist. I was not only immersed in and working on a daily basis with the most cutting edge information available (in several languages) on a variety of subjects but I was also given free access to the internet via a T3 backbone. Not only was I using the internet during its early days (way before Google) but it was at a tremendous, blazing speed with government access that allowed me to… well to go almost anywhere and do almost anything.
Both my work on information systems as well as my stash of university textbooks are instances of a completely new phenomenon in human history, which can aptly be summarized as information overload. Just 100 years ago, information from the wider world was infrequent and “low bandwidth”, perhaps just a telegram or short news article of a few hundred words. If we consider the flow of information as akin to water, even the most connected person in the heart of a large city like London in those days was at most dealing with a small garden hose. Today in 2012 it’s like standing underneath the largest waterfall on the planet – the sheer weight and volume of the cascade of information is far too powerful and overwhelming to absorb in its entirety.
And so we’re all faced with that strange new phenomenon – with so much information flowing from so many sources, how can we make sense of it all? I once worked in a library that had 6 million books. Even if I locked myself in a cell and read one book a day I’d never live long enough to read even 1% of them. Add to this the billions of web sites and all the content being created on a daily basis and it becomes quite clear that absorbing and understanding all of this information in toto is quite impossible. So, what to do?
Well if you’ve been reading this blog for a while you know that I often refer to the “Unsleeping Eye”. This is sort of my answer to that question. I first began developing this project nine years ago when I judged that the internet had sufficiently penetrated the formerly offline world to make it feasible. On a first order level, nearly every newspaper, news agency and television broadcast is online or has an online presence. Those smaller newspapers and other sources which aren’t online yet have a second order presence because they are referred to or cited or quoted by first order sources, which are.
So therefore it’s a rather simple thing to harness the power of computers to set up algorithms to sort this data. You might note that there is some basic use of ngrams in the right-hand sidebar of this very blog. Every time you do a search on Google, you are essentially operating your own “Unsleeping Eye” because a computer algorithm is sorting and collecting the relevant information for you.
My dear “friends” at certain alphabet soup government agencies in America are obsessed with these algorithms like certain trading firms are obsessed over their quants as they attempt to master stochastic fluctuations (about as futile as trying to herd cats). But for reasons I’m not going to go into now, I find this approach severely limiting. What convinced me to try my own project nine years ago was the penetration of the internet into the lives of people. And people can do things that no computer can do. A perfect example of this is language translation. Even though computer translation can be useful and help you out in a pinch, only human beings can properly translate between two different languages.
And now a significant number of people are online. And that is the real difference because no matter how good my algorithms are, no matter how many geniuses work at Google, at some point something is going to be missed. But it’s the people out there, which means you and me, who are out there watching and paying attention and Stumbling Upon little hidden things that sometimes end up being extremely significant.
All of this is an interesting treatise on information systems but let’s go ahead and put it into practice with a concrete example. Let’s say you live in Romania and can speak or read several languages and you are interested in “all things having to do with Romania” and want to know what information is out there. How do you go about doing this? Obviously going to Google and typing in “Romania” will get you a lot of information but it’s still too much information. You’re going to get everything from industrial data on the tonnage of wheat crops produced to some blogger writing about their bicycle trip through the country. The “flow” of information is still far too powerful and too overwhelming to get any kind of useful extraction.
Yesterday I posted a video that several people had sent me the link for. I certainly didn’t discover it by doing a Google search or some automated algorithm to comb through news feeds. If you look on its Youtube page you’ll see it only has a few thousand pages so it certainly isn’t popular enough to rise out of the static. But one person saw it and told their friend who told their friend and then it came to me, who has now since passed it on to you, who has perhaps passed it onto your friends. And thus you can see that the Eye relies just as much on people as it does on any kind of automatic sorting algorithms.
As I’ve mentioned before, I considered marketing to be an evil art. The Unsleeping Eye project has nothing to do with that, has no connection whatsoever with manipulating people on a subconscious emotional level so that certain messages will get transmitted often enough that large-scale crowd behavior is manufactured (for profit and glory, mostly profit). No. Certainly the temptation is there as it is any time you study the dark arts. The Unsleeping Eye was created solely for the purpose of observing both the deliberate and the serendipitous transmission of information in all its forms, primarily for my own personal curiosity.
I used to live in a village here in Romania and every day life there was quite an experience and required no internet whatsoever. My Romanian language skills largely improved as a result of this experience and I certainly could go through my daily life with plenty of information received the “old-fashioned” way, from my biological senses of hearing and vision, and this is more than enough. I could watch the television news (or not), read a (physical) newspaper (or not) and have a chat with my friends about what they think about things and I wouldn’t consider that a tragedy at all.
But I live in such a unique time in history when so much more is possible, and like my two cats, I am one mighty curious son of a bitch. And so the Eye was born, to see and learn not just what can be gleaned from my own senses and experiences but from the collective total of all the senses and experiences of millions of people. It’s quite a thrill, let me tell you.
Anyway, time for me to wrap this up as I’ve got lots of work to do today and my cat is now eating a rather hideous bug out on my balcony. For those of you who have asked me about the Unsleeping Eye before, hope this answers some of your questions!