In an Uncertain but Predictable World

Uncertainty! In a world surrounded by corruption, scams, recession, job loss etc., the world we live in has become more volatile and highly uncertain. Amidst the uncertainty we are always curious and intrigued to know what happens next, tomorrow and so on and so forth. The mind is always speculative, take the example of news these days – Intense Speculation! We are very much eager to know what would happen to life – that’s why Astrology! We are very much eager to predict the stocks – that’s why Stock Trading. We are very much eager to  know the outcome of a match – that’s why Betting (And, Fixing)!Our mind continuously debates to predict the outcome of something or the other. 

Prediction has been existing since the Roman Republic where the tracking of all adult male fit for military service were maintained in records and since then existed the census. The word ‘Census’ is derived from the Latin word ‘censere’ which means ‘to estimate’. In the year 1880 the census of the American population took over 8 years, the next census was estimated to take over 14 years. This gave impetus to sampling techniques. Sampling techniques are used in various places like to predict the outcome of an election – the field known as Psephology, to forecast the weather(predicting seasonal changes goes back to as old as 25000 years ago), to determine the behaviour of consumers by taking a sample survey and so on and so forth. How to decide a sample is itself a challenge and could give incorrect outcomes chosen a wrong sample. Various sampling techniques have evolved over a period of time. These have been applied to various available data forms. But with the advancement of technology, digitization of data, Plethora of challenges and opportunities exist to apply the prediction techniques to the emerging field known as Big Data. We have unleashed and solved myriad of problems that could have been never solved without technology and digitized data.

The emergence of new technologies viz. cheaper machines, hard disks, Hadoop, No SQL data bases, Artificial Intelligence - Artificial Intelligence – Machine Learning, Natural Language Processing and Linguistics have all aided to solve the problem of Big Data mainly to predict the outcome of a particular problem. Let us look at some of them to understand the impact of these on the life of people.

Recommendation:  With the advent of social networks, the recommendation engines are seen to be handy tool for marketers to promote the products. Take for example of 

Facebook: There are various recommended things viz. ‘You may know’ (person), ‘Suggested Group’, ‘Add to news feed’, ‘you may also like’, etc. All this is possible because by profiling the person to whom these are recommended. If you have page, it also give you an analytics about the demographics of the user and helps you identify your target customer profile. It also suggests the place that you currently live in based on other friends whom you are connected with. Because of recommendations, I am able to connect to friends whom I might have missed due to the distance gap.

Linkedin: ‘People you may know’, ‘recommended news’, ‘jobs you may be interested in’, ‘Groups you may like’, ‘Companies you may want to follow’ and so on and so forth. This helps you know where jobs are available, which are your relevant companies depending on your profile in real time. The idea of searching jobs or companies is kind of vanishing.

IMDB: International Movie Database – a recommendation engine for movies. It is built using an algorithm that rates the movies based on the votes of the users. It becomes easier to know what movies to watch and what can be left out!

Likewise, these recommendations are used on most e-commerce sites like amazon.com, flip kart and many other places to recommend you a particular product based on your usage history.

Markets

Stock Market Predictions - There have been models built using twitter to predict the stock market. Some of the references are ‘Twitter Mood Predicts the Stock Market’ , ‘Tweet Predictions’ . In fact, I got an opportunity to work on similar problem in my previous firm known as ‘Real Time Intelligence’. The idea was to predict the behaviour of the company so that the fund managers can decide on which company the investment should be made. Using the same problem, it was extended to health care, Enterprise Risk analysis and many other domains. We used new that was aggregated form disparate sources to predict the trend and also tweeter was used as well. Even there have been studies using Google Trends to predict the markets. References are here , here and here.

Algorithmic Trading – If tweeter and news can help to predict the outcome of the stock value of the company. Then algorithmic trading helps you invest in large number of companies based on the algorithms  you choose. Of course, there are guidelines and in some instances these algorithmic trading has halted the stock exchanges due to a bug in the system too. But, this helps to invest on large scale which was humanely impossible to invest.

Human Resource

Predicting Employee Exit – Several companies are using the tools to predict whether an employee is about to exit and these tools help to engage the companies to retain the talent. Training the employee is quite expensive for organization and retaining the employees is one of the agendas for the HR.

Predict Performance before Employee Joins – There are also tools that predict whether the employee is going to be successful in his job before he or she actually joins the company. Now, this might be incredible. Thanks to the data trails left by the employees on the web.

BMI Prediction – Usually, to calculate BMI you take the weight and height of the person and then calculate the BMI. How about looking at the face in a photograph and then calculating the BMI? Not possible then click here to believe it. 

Farecast  (Acquired by Bing) is used to predict when the best time to purchase a flight ticket is, Decide is used to predict when to buy electronic products and likewise the list of prediction applications is endless. One of the interesting predictions that I came across was about ‘Algorithms Calculate a Couple's Chances of Having a Baby via IVF’. And, the best ever has been by IBM Watson’s DeepQA project.  No industry is left without leveraging the Big Data technologies and Prediction algorithms. This has opened the flood gates of opportunities to the so called Data Scientist, Entrepreneurs and Academicians to solve challenging problems and to make an Uncertain World in to a Predictable One!