We also work with very large financial services firms, getting their computational analytics on the cloud as well. By the time you launch your product, it may even be possible to take advantage of such specialized cloud service offerings as “analytics on the cloud”, offered by major cloud service providers.

It is important to realize that even though EC2 is the biggest and most mature cloud provider out there – and today’s announcement proves once again their leadership – it is not the only game in town. For example, for real-time, on-demand
analytics node start-up is still an issue. Joyent can get you down to under 30 sec start-up on a new node – which sometimes is the difference between feasible/not feasible approach. Other cloud providers offer better SLAs or better data access solution.

Let us know if we can help you design a scalable, cloud-ready infrastructure. You’ll probably need to pick a good real-time scheduler too, and solve data transfer/access/management issues,
amongst other things, and we specialize in this space.

Very good tips and informative. It is good to know that your company consults in this area of high performance computing, because I may be looking for outside expertise to help in what I am doing at some stage (which depends on funding that I will get access to). I am assembling a team for a startup and our app is a real-time analytic that we must do it right from the beginning. Realtime analytic is a very challenging task in today’s computing environment. In the world of finance, computing the volatility of a single asset (ie, a single stock) using different algorithms that are based on linear algebra decomposition (ie, matrix factorisation) are very compute intensive. Eg, doing real-time GARCH is a memory intensive process. One can still do it non real-time (perhaps every 5 minutes or so) but this means that one has to accept losing a few important opportunities that have arised in the market without being exploited. There is always a trade that is registered in the market every seconds (or even less) or so, this means that the GARCH has to be recomputed everytime a single trade has been registered in the exchange, ie, every seconds. This is the problem with real-time analytic.

Let me comment of each. Large-scale part is addressable by EC2 today. Nearly a year ago we ran Monte-Carlo simulations on EC2 to test scalability of the underlying platform given the perfect scalability of math, and it worked perfectly linearly up to 512 nodes (we didn’t run it further). I know of multi-thousand node computations and basically believe that scalability is not an issue. Here is a (now rather old) blog with more details on that benchmark: http://blog.griddynamics.com/2008/07/gridgain-on-ec2.html

(b) Real-time can be the problem. First of all, there is a cost of starting up a cluster, which is several minutes. Once you have the computational cluster up and running, the scheduling can happen right away, depending on the speed of the scheduler. We have ran real-time schedules on EC2, doing things like annuity calculations. The other SIGNIFICANT issue related to real-timeness is QoS. Basically, you can’t control how loaded EC2 servers and network is at the time you ran your calculations. If it’s really oversubscribed, you’ll see significant performance degradation as compared to the best-case times seeing when the infrastructure is more available. Since there is no way to secure QoS, there is no way to guarantee resource availability or time completion. Hence, for some real-time applications, this is a significant problem.

(c) Analytics. The algorithms that are
1. Embarrasingly parallel
2. Do not depend on data location or transfer
3. Share nothing

should be runnable on EC2. Algorithms that either
1. Depend on data off-the-edge of the Cloud
2. Use message-passing between the compute nodes, or
3. Share some sort of state

are going to be trouble to run on EC2 in any kind of real-time scenario.

There is a large gray area in between where a lot is possible with a lot of level of effort I hope this helps.

Thanks for the tip. That’s what I thought that EC2 will be able to handle those high performance numerical linear algebra computations. Some of my codes are developed using the HpJava (a high performance development environment for Java), which is basically an extension of the standard Java.

“JUST A THOUGHT”

I don’t know which specific environment you wish to setup. However, Amazon has recently announced Elastic Mapreduce (hadoop on AWS infrastructure) service.

Many users are adopting Elatic Mapreduce for intensive computations.

And many developers have ported MPI to EC2. So its all very much possible.