Wall Street has been dealing with big data since before it was a thing. But the rise of algorithmic and high frequency trading is making big data even bigger and has led many firms to adopt the same free open source tools that tech startups and scientific researchers are using, according to Dharmesh Malam, CEO and co-founder of the Y Combinator alum BigCalc.
Many firms are using mathematical programming environments like R and MatLab, but those tools don’t scale well enough Malam says. So firms are adopting Apache Hadoop, the open source big data crunching system modeled on technology pioneered at Google. Hadoop distributes data across clusters of inexpensive servers. The MapReduce framework, a component of Hadoop, makes it possible to harness the collective power of these servers to solve difficult mathematical problems. But writing MapReduce jobs is hard, even for experienced programmers. BigCalc developed a framework called QuantCalc for programming on Hadoop using R without MapReduce specifically for programmers at high frequency and algorithmic trading firms.
There are other ways of using R with Hadoop, including RHadoop from Revolution Analytics and RHIPE. But Malam says QuantCalc is different in that it translates normal R code into the C++ programming language, which is much faster than R. Also, existing R on Hadoop solutions still require programmers to write MapReduce jobs in R, but QuantCalc abstracts that away and lets you write normal R code.
Malam and fellow co-founders Rikin Shah and Chris Roebuck all met while studying computer science at Imperial College in London. After graduation they went on to work for different firms. Malam worked at Blackrock and later Fidelity London. Shah worked for UBS. Roebuck worked for Barclays Capital’s Fund Solutions and later Credit Suisse’ Algorithmic Trading desk. Eventually they realized that they all had similar problems with scalability of R and usability of Hadoop and decided to start a company that solved that problem.
For now the product is geared entirely towards the problem domain of algorithmic trading and high frequency trading. Malam says that going after a particular domain make it possible to translate normal R code into MapReduce C++ code effectively. And the team’s knowledge of the industry make them uniquely suited to designing such a vertical solution. And of course the team’s contacts and credentials in the industry will make it easier for them to landing meetings with customers.
But he says eventually they’d like to expand their product to the broader financial market and beyond.
Y Combinator is a venture fund which focuses on seed investments to startup companies. It offers financing as well as business consulting along with other opportunities to 2-4 person companies looking to take an idea to a product. Y Combinator looks for companies with “good” ideas over companies with experience and a business model. The company made its first investments in Summer 2005. Y Combinator selects companies to finance and consult with twice a year. They are located in...
Financial analytics should just work, no matter how big the data. Analysts and traders shouldn’t have to be expert programmers just to test a trade idea. The BigCalc team initially met at college, and then worked in quantitative finance and algorithmic trading at leading firms. Their first hand experience of ever increasing data sets led to the founding of BigCalc.
Seattle, WA
San Diego, CA
Menlo Park, CA
San Francisco, CA
Berlin, Germany
Boston, MA
