Interest in Big data technologies has significantly grown in the last few years. Data growth has exploded in the last decade. Information systems are generating huge amounts of data, trading data is becoming much more granular and at higher frequency, unstructured data from Twitter feeds, blogs etc is becoming mainstream and technologies to produce, transmit, store and process these massive datasets is collectively enabling the Big data revolution. Data science is the one of the hottest areas for internet startups. Cloud computing is making access to unlimited hardware at your fingertips. Significant venture capital money is flowing to support newer and innovative technologies that leverage vast amounts of data to create platforms and applications that could enable decision making close to the proverbial “speed of light”. Technology industry stalwarts like Google, Microsoft, Amazon and Oracle are making significant investments in Big data technologies. The applications and opportunities seem endless and the promise of revolutionary applications in healthcare, finance, space research and pure science has led various industry and academic organizations to significantly ramp up efforts to develop technologies to help analyze massive data sets. Quants have been in the forefront in the financial industry in adopting revolutionary technologies and the Big-Data phenomenon has undoubtedly caught interest in the quant community. In every quant gathering, there is at least one reference to Big data and discussions on the potential and possibilities. Quants have started to frequent Big data conferences and events to know more about technologies and opportunities in this space. I have had multiple discussions with clients who want to start Big data projects but many still struggle to understand what it is about and how they can leverage these technologies to further their quantitative research ideas. The information overload on Big data and the umpteen numbers of sources of Big data, primarily vendor and media driven, is adding to the confusion leaving quants to ponder on whether Big data is another fad or is there a real opportunity they should try out to gain an edge. I started out to write an article to introduce Big data but rather than just rehash information that is widely available, I thought it may be useful to share the my perspective as a quant practitioner with experiences from the analytics and the quant worlds on pointers to help quants understand the realities before embarking on a Big data project. In this article, I will start out with a brief introduction to Big data and point you to sources where you can learn much more about Big data. I then discuss five key things quants should consider before embarking on your first Big data project. I will then conclude with pointers on how quants can keep themselves in tune with the rapid innovations happening in the Big data world.

Download the article at

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>