Airbnb just announced that it has open-sourced Chronos, the company’s replacement for the standard time-based Cron schedule that is probably familiar to everybody who has ever used a Unix-based system. Chronos runs on top of Apache’s Mesos cluster manager and allows developers to execute shell scripts at set intervals and, unlike Cron, you can also trigger jobs by the completion of other jobs.
Chronos, the team writes, was born of Airbnb’s need to automate more of its data-processing pipeline. Until last December, Airbnb was apparently using a single hourly Cron job that would start its complex processing pipeline. The problem with this, the company’s engineers write, was that they “use a managed Hadoop product from Amazon, called Elastic Map/Reduce. High variance in network latency, virtualization and not having predictable I/O performance is an ongoing challenge in a cloud environment.” This increased the chance of something going wrong, so the team decided it needed a system that would allow for retries and would give its analysts the ability to see which jobs had failed.
To make keeping track of Chronos even easier, the team also developed a web-based user interface that makes it easier to manage and run jobs than the usual text file Cron users have to rely on (Chronos also features an API, so you can always build your own GUI, too).
Airbnb’s GitHub repository delves a little bit deeper into the details of how Chronos works and if you’ve got 45 minutes to burn and would love to hear more about Chronos, here is the tech talk the team did at Airbnb’s headquarters earlier this year:
Founded in August 2008 and based in San Francisco, California, Airbnb is a trusted community marketplace for people to list, discover, and book unique spaces around the world – online or from a mobile phone. Whether an apartment for a night, a castle for a week, or a villa for month, Airbnb connects people to unique travel experiences, at any price point, in more that 26,000 cities and 192 countries. And with world-class customer service and a growing community...
Austin, TX
Seattle, WA
San Diego, CA
Menlo Park, CA
Disrupt Europe: Berlin Hackathon
Berlin, Germany
San Francisco
