Amara’s Law states we overestimate a technology’s effect in the short-run and underestimate it in the long run. This is happening with the proliferation of “big data” announcements in the capital markets space. Big Data’s quasi-definition currently relates to identifying patterns in unstructured data. The best example is that of Sentiment Analysis (twitter feeds, social media, etc..) being cited as a price movement indicator. This model of ‘big data’ is only a short-term arbitrage opportunity.
Sentiment Analysis is a small step in the right direction, but the big money to be made will come with the monitoring of embedded devices. This concept is called the “Internet of Things” (or IoT). Although I find it optimistic, some groups have predicted 30 billion devices will be monitored via the Internet in the next 7 years. As the number of inter-connected devices continues to skyrocket, the winners will be those that can quickly interpret the terabytes of unstructed data.
Why will this have a dramatic effect on capital markets? Take the Sentiment Analysis example and apply this to a world of connected devices. Let’s apply this to a market like gasoline futures. Instead of waiting for the US Energy Department to release the weekly inventory report, unstructured pump-level data is now sent across the internet from 10,000 gas stations and 50 refineries. The data is constantly collected and analyzed using deep visualization techniques. A data analyst and modeler look at real-time consumption trends for arbitrage opportunities and create a algorithm to predict future-state consumption patterns. This algorithm is now used utilized to trade futures and equities (oil refiners, agricultural producers, etc..) and fed into a larger algorithm for retail consumption. Trading and investment now becomes a command and control activity of monitoring activity and adjusting algorithms for non-quantitative factors.
Unstructured data is a key predictor of future economic activity. No longer will analysts need to rely on qualitative sentiment indicators like Manufacturing Purchasing Index and Consumer Confidence that claim to be forward indicator (but are really emotional glances at the rear-view mirror.) In an IoT world, analysts use aggregated data to identify real-world trends. ( Note: Like discussions with with my local Costco manager who noted that in late 2007 consumers had moved heavily into buying basis foodstuffs [eggs, fruit] and away from higher-end items such as furniture and electronics).
Big Data may also create mini-futures and micro-exchanges via commoditization of goods and services. If you have the infrastructure to reliable monitor and predict supply and demand of goods and services, then providing a mini-contract helps eliminate risk. Providing a method for hedge funds and/or traders to serve as risk brokers via buyers and sellers of a good or services helps overcome asymetric information. Previously, only inside buyers and sellers in a market had this information. Unstructured data would allow other risk takers to enter a market and help in price discovery and long term market stability.
Like the title mentions, we have seen very little of the potential of Big Data analysis in the capital markets….but it is coming and sooner than you think.
PTT Monitor 