In an Uncertain but Predictable World

Uncertainty! In a world surrounded by corruption, scams, recession, job loss etc., the world we live in has become more volatile and highly uncertain. Amidst the uncertainty we are always curious and intrigued to know what happens next, tomorrow and so on and so forth. The mind is always speculative, take the example of news these days – Intense Speculation! We are very much eager to know what would happen to life – that’s why Astrology! We are very much eager to predict the stocks – that’s why Stock Trading. We are very much eager to  know the outcome of a match – that’s why Betting (And, Fixing)!Our mind continuously debates to predict the outcome of something or the other. 

Prediction has been existing since the Roman Republic where the tracking of all adult male fit for military service were maintained in records and since then existed the census. The word ‘Census’ is derived from the Latin word ‘censere’ which means ‘to estimate’. In the year 1880 the census of the American population took over 8 years, the next census was estimated to take over 14 years. This gave impetus to sampling techniques. Sampling techniques are used in various places like to predict the outcome of an election – the field known as Psephology, to forecast the weather(predicting seasonal changes goes back to as old as 25000 years ago), to determine the behaviour of consumers by taking a sample survey and so on and so forth. How to decide a sample is itself a challenge and could give incorrect outcomes chosen a wrong sample. Various sampling techniques have evolved over a period of time. These have been applied to various available data forms. But with the advancement of technology, digitization of data, Plethora of challenges and opportunities exist to apply the prediction techniques to the emerging field known as Big Data. We have unleashed and solved myriad of problems that could have been never solved without technology and digitized data.

The emergence of new technologies viz. cheaper machines, hard disks, Hadoop, No SQL data bases, Artificial Intelligence - Artificial Intelligence – Machine Learning, Natural Language Processing and Linguistics have all aided to solve the problem of Big Data mainly to predict the outcome of a particular problem. Let us look at some of them to understand the impact of these on the life of people.

Recommendation:  With the advent of social networks, the recommendation engines are seen to be handy tool for marketers to promote the products. Take for example of 

Facebook: There are various recommended things viz. ‘You may know’ (person), ‘Suggested Group’, ‘Add to news feed’, ‘you may also like’, etc. All this is possible because by profiling the person to whom these are recommended. If you have page, it also give you an analytics about the demographics of the user and helps you identify your target customer profile. It also suggests the place that you currently live in based on other friends whom you are connected with. Because of recommendations, I am able to connect to friends whom I might have missed due to the distance gap.

Linkedin: ‘People you may know’, ‘recommended news’, ‘jobs you may be interested in’, ‘Groups you may like’, ‘Companies you may want to follow’ and so on and so forth. This helps you know where jobs are available, which are your relevant companies depending on your profile in real time. The idea of searching jobs or companies is kind of vanishing.

IMDB: International Movie Database – a recommendation engine for movies. It is built using an algorithm that rates the movies based on the votes of the users. It becomes easier to know what movies to watch and what can be left out!

Likewise, these recommendations are used on most e-commerce sites like amazon.com, flip kart and many other places to recommend you a particular product based on your usage history.

Markets

Stock Market Predictions - There have been models built using twitter to predict the stock market. Some of the references are ‘Twitter Mood Predicts the Stock Market’ , ‘Tweet Predictions’ . In fact, I got an opportunity to work on similar problem in my previous firm known as ‘Real Time Intelligence’. The idea was to predict the behaviour of the company so that the fund managers can decide on which company the investment should be made. Using the same problem, it was extended to health care, Enterprise Risk analysis and many other domains. We used new that was aggregated form disparate sources to predict the trend and also tweeter was used as well. Even there have been studies using Google Trends to predict the markets. References are here , here and here.

Algorithmic Trading – If tweeter and news can help to predict the outcome of the stock value of the company. Then algorithmic trading helps you invest in large number of companies based on the algorithms  you choose. Of course, there are guidelines and in some instances these algorithmic trading has halted the stock exchanges due to a bug in the system too. But, this helps to invest on large scale which was humanely impossible to invest.

Human Resource

Predicting Employee Exit – Several companies are using the tools to predict whether an employee is about to exit and these tools help to engage the companies to retain the talent. Training the employee is quite expensive for organization and retaining the employees is one of the agendas for the HR.

Predict Performance before Employee Joins – There are also tools that predict whether the employee is going to be successful in his job before he or she actually joins the company. Now, this might be incredible. Thanks to the data trails left by the employees on the web.

BMI Prediction – Usually, to calculate BMI you take the weight and height of the person and then calculate the BMI. How about looking at the face in a photograph and then calculating the BMI? Not possible then click here to believe it. 

Farecast  (Acquired by Bing) is used to predict when the best time to purchase a flight ticket is, Decide is used to predict when to buy electronic products and likewise the list of prediction applications is endless. One of the interesting predictions that I came across was about ‘Algorithms Calculate a Couple's Chances of Having a Baby via IVF’. And, the best ever has been by IBM Watson’s DeepQA project.  No industry is left without leveraging the Big Data technologies and Prediction algorithms. This has opened the flood gates of opportunities to the so called Data Scientist, Entrepreneurs and Academicians to solve challenging problems and to make an Uncertain World in to a Predictable One!
 

Zindagi

Zindagi ka aisa mode atha hai,

Na tum jaano na hum,

Manzil tho dikhati hai,

kabhi paas kabhi dhoor..

Chalte chalte, 

kuch log wahi rukh gaye, 

kuch log aage nikal gaye,

Jaana tho wahi hai sab ko, 

bas chalte raho,

 pehle tum baad mein hum !

If You Cannot Convince, then Distract!

50,000!!! That’s the number of thoughts per day that a human processes in its brain. Perhaps some may be repetitive and the average is 0.5 thoughts per second. Incredible!  This tiny brain that weighs around 3 pounds and has 75% water boasts of processing 50000 thoughts per day. I am not going to dissect the anatomy and the science behind the brain functioning here.  But, delve in to how information has overloaded the brain and distracts the mind. 

Roti, Kapada, Makaan (Food, Clothing , Shelter) has been the basic necessities of life. Adding to these necessities, there are many other things like TV, Web, apps, social networks, gadgets, mobile phones and many other things.  It is this addition of things to life that has made life sometimes feel miserable and sometimes feel connected.  Take the example of a news channel on TV, the news screen has at least 5 varieties of information and goes up to 10. The information includes Time, Channel Name, Breaking News, Just in News, Top Story, tickers that scrolls the news, important news schedule and stock prices and so on and so forth. The main news that the anchor is presenting is surrounded by these many information and your eyes and mind is always looking for additional information on the screen without your knowledge.

Let us look at the information on a social network site.  When you login to your page, you have recommended friends, the wall, trending topic, the news feed, sponsored ads, the groups, the pages etc. From an era of searching the information we have moved to an era of scrolling for information on these social networking sites. Amidst the scrolling, there is humongous amount of information that distracts you and deflects your purpose of visiting the site.

Let’s move to email. The relevant mail is not to be seen at all. You need to dig out the relevant mail amidst the marketing mails all around. New features are accommodated in mail boxes like social, updates, automated unsubscribe etc.  A recent article by muHive mentions the use of emails for marketing purpose is less useful.  But still people thrive on mass marketing whether it’s useful to the person or not. I just hate to see more ads or marketing mails in my mail box then real mails. 

Duplicity of information is another nuisance. The same information is posted on every other social network. Visit any site or blog there is an option, share on Facebook, twitter, pinterest, googleplus, and some pages have buttons from almost all social networks on the earth where you can share. And, it is quite annoying to read the same information since everyone is on every other social network. People share information that I have shared X information on Y network so please check. Irritating? The mantra is simple, push the information anywhere and everywhere because it’s easy and cost is almost zero!  It was interesting to see the purpose of social networks to connect people. Now, it has moved on to make people consume all sorts of information whether they like it or not. Another nuisance from these social networks is, automatically your profile gets created without even your knowledge! This is insane and furious!

The number of online accounts with separate credentials easily goes over 50 accounts for an individual. Managing this information is itself quite challenging. With the enormous amount of data that is generated on different mediums, the eyeballs that  screens through information easily surpasses over the number of thoughts per second which in turn means the brain has to process at a speed faster than the speed of thought. This is quite impossible and leads to deviate from your original thoughts.

Even Arjuna, hit the fish eye with his bow and arrow by looking at the reflection in water, would find it hard to navigate through the information. Earlier, we used to search for information, now information is being delivered to you whether you like it or not. The marketing personnel are going with the strategy of pushing the information with the motto, 'if you cannot convince, then distract' . For the user Call it information Overload, Redundant Information, people will be ignorant about Information and we are entering in to Information Ignorant World which can also be called as Attention Deficit Disorder :)