Major U.K. Real-Time Train Database Opens Up To More Developers

A long-standing pain-point for U.K. developers wanting to do cool things with real-time train data looks like it’s going to ease up, thanks to changes in the terms of access and usage being announced today.

From the user side, this will also (hopefully) mean more U.K. apps offering better and cheaper access to useful up-to-date train data.

From June 1, the U.K.’s National Rail Enquiries’ (NRE) Darwin database — which analyses raw data from numerous rail industry sources to predict the arrival times of trains — is switching to free for developers to access, waiving the fee that had been levied on some services up to now.

Most existing developers who are paying NRE for access to the data will no longer need to pay (although it should be noted that the licensed services that currently incur a charge are the minority: just 26 out of a total of 523).

Darwin access will also likely be free for all developers of new commercial apps wanting to use the API until they scale their user-base to a size where a fee becomes required.

And that’s really the key: devs who were being put off before, by the  cost and complexity of NRE’s restrictions to accessing the Darwin database, may now be encouraged to use it.

Developers with commercial apps whose services are used more than five million times in a four-week period will still be charged. As will private users of that same scale.

There won’t be any charges for public bodies using the data regardless of their size — unless they are using the data for commercial purposes, in which case the same rules apply. NRE said only one of its existing clients falls into the ‘will still be charged’ category.

The database is offered as a pull API, making it valuable for devs wanting to easily add things like train arrival push notifications to their apps.

Arduous licensing requirements

NRE has been strongly criticized for several years — as far back as 2010 — for a lack of transparency around access to the data, in terms of what can and can’t be done with it, and for slapping a fee on some users (even, in some instances, for non-commercial uses).

Critics argued the fee and arduous licensing requirements for using the API have made it much harder for (especially) smaller developers to incorporate useful real-time train data into their apps. Some free apps that had been using the data for free — prior to 2010 — were even forced to close as a result of NRE actively starting to enforce licensing in late 2010. At that time it started requiring a token to access what had previously been left open for non-commercial/public use.

On the licensing side, NRE has also been criticized for the sweeping conditions it imposed on developers wanting access to the data — one developer even likened parts of the license to a gagging clause.

Another criticism has been the length of time it can take NRE to grant access, even for those who were willing/able to pay for the licence. This timeframe could apparently stretch to months. Not exactly app-maker friendly.

NRE is owned by the Association of Train Operating Companies — so it’s a for-profit industry group, rather than a public body. But it’s worth pointing out that some train operating companies do pull in taxpayer cash in the U.K., so some of that public money has effectively been filtering down to sustain the Darwin database.

Arguably, therefore, NRE has been jealously guarding publicly funded data — even if it’s not actually profiting from it (it claims it doesn’t profit off Darwin, saying that revenues only go towards maintaining the system).

No surprise, then, that NRE’s proprietary attitude has stuck in the craw of developers and open data enthusiasts alike for years. Especially as that attitude has increasingly been cutting against the grain of U.K. open government data initiatives — which in recent times have been trying to widen access to valuable public data stores to get more utility from the info.

At root, though, the protectionist walls around Darwin have meant that the traveling, app-using British public has been punished by having to shell out over the odds to get their hands on apps containing the most useful kinds of real-time train data.

I know this, because I was one of them — paying for the excellent UK Train Times app more years ago than I care to remember (when it cost around £5 if memory serves). It’s a great app, but it wasn’t a cheap app and I did have a moment’s pause before tapping purchase just so I could figure out when a delayed train back to London would be limping into the station.

Photo 20-05-2014 18 08 13 (1)

Fast forward to today and it appears that NRE has seen the error of its ways. At long last. Or at very least realized that making it harder for developers to do cool stuff with its data ends up disadvantaging the traveling public who use — and pay for — its services. That in turn arguably encourages travelers to consider other, more convenient on-demand alternatives to trains. Ergo, it’s bad for their business too.

Yep, it really has taken some four+ years for NRE to figure out that being more open with its data might actually be a good thing for everyone involved. As one person I spoke to judiciously put it, “the railway moves at 1 / the speed of the news”.

Ch-ch-ch-changes

So, after June 1, instead of the fee for commercial usage and current arduous licensing process, smaller developers and private users will be able to apply for free access to Darwin online, with only a T&Cs page standing in the way of them and the data. Even if they’re building a commercial app.

I asked to see these new T&Cs but the NRE spokesman said they are “being worked on at the moment” — so it remains to be seen how developer-friendly they end up being.

He added that they will be “based on our review of conditions used by other public sector transport information and open data providers, such as TfL, Traveline and Network Rail, who use a version of the Open Government Licence (OGL). The new terms will be based on the old TfL terms and conditions (they have recently changed to an OGL) and the existing NRE service licence… Network Rail and Traveline use terms similar to an OGL, which has also influenced the terms we are putting together.”

All of which makes the process sound far more torturous than transparent. But at least he’s bandying the word ‘open’ around.

In a call with TechCrunch he did also say the aim would be to have a “prejudice towards saying yes” to a developer applying to use Darwin in future. ie, rather than steamrollering any use-cases that don’t conform to NRE’s prior sweeping conditions (not that the spokesman put it that way).

“All you will have to do is go on a website, probably somewhere on NRE, it will say ‘do you agree to these terms and conditions?’… Say yes and you’re away. That’s the idea. It’s quicker, easier, there’s a prejudice towards saying yes, rather than ‘we’ll check’,” he said. “You won’t have to have a licence anymore, you won’t have to go through NRE, you’ll just go to the website.”

“The reason that we’re keen that people use Darwin is because it’s the most reliable. We want to have one click, consistent source of information that’s available to passengers — and Darwin is it. So we’re keen that whether it’s through NRE or third party apps that’s getting out to as many people as possible,” he added.

The detail in the smallprint of the new T&Cs — and whether the process really lives up to that low-friction description — is going to determine how radical a shift this is for U.K. devs wanting to work with real-time train data.

Amy Worrell, a developer working for the company that makes the trusty UK Train Times app, which does license the Darwin system, reckons the shift when Darwin switches from fee to free could be significant — but only if what she dubbed the “onerous licensing process” gives way to a genuinely open API.

“If Darwin went free, but still had the approval process before you could make an app, it’d probably bring down the price of apps slightly, but that’s about it. However, if it went open (i.e. free and anyone can use it), that would change everything. There would be innovative apps doing things we can’t currently do with live data,” she told TechCrunch.

Open data drivers

One thing that’s worth noting is that in the years since the Darwin API ‘paywall’ went up, U.K. devs have not stood still but have taken action to route around the NRE database.

So there are alternatives to using that data — and they are clearly helping to railroad NRE towards being more open. (Disruptive weather is another driver for liberalising the data as train users have been critical about the lack of accurate, up-to-date info in times of acute service stress. Social media can also, therefore, take some credit — having provided an open customer criticism channel to pile more pressure on.)

The dev team behind Real Time Trains, for instance, devised a workaround based on using open data from another industry body, Network Rail, and “a lot of manual labour mapping signalling info”, as developer, Tom Cairns puts it.

NRE still hold a lot of data that is of use and value to everyone

“I’ve looked into [using the Darwin system] several times in the past and investigated the fees for what I wanted to do. The bar (and control over how I could use data) was sufficiently high that I ended up writing @realtimetrains,” he added.

Real Time Trains does charge for access to its API but its fees are evidently lower than Darwin’s (at least until June 1).

Despite an apparent brave new era of transparency for NRE, the spokesman would not provide details of the fees it currently charges for access to Darwin, claiming that’s commercially confidential data so would require client sign off to disclose.

He did say that collectively the switch from paid to mostly free will cost NRE around £250,000. And that there are approximately 169 customers who are licensed to use Darwin, 124 of whom are actively using it.

Another open data advocate who took up arms against NRE throwing up protectionist walls around its data, back in 2010, was Peter Hicks. His response was to replicate the Darwin data by approaching Network Rail to ask for access to their timetable and real-time data (access which they granted).

He then spent time decoding it and set up the Network Rail Data Feeds — which helps services such as Real Time Trains work around Darwin (although the Network Rail Data does not do predictions — that’s an add-on by Real Time Trains).

Hicks is currently employed for a company that’s helping make more of Network Rail’s data open and accessible, and runs the not-for-profit Open Train Times website.

Echoing the NRE spokesman, he also makes the point that Network Rail’s data is not perfect — “neither complete, nor 100% accurate” — so believes there is still a lot of value in the data held within Darwin, and therefore value in it being opened up to more users.

“NRE still hold a lot of data that is of use and value to everyone — they could really change the landscape by making their version of the truth, which is replicated on station boards and apps, universally accessible, liberally licensed and ‘free at the point of use’.  It would help them achieve their goal of a “single version of the truth”,” he said.

Come June 1, more U.K. devs will be able to start assessing that version of ‘train times truth’ for themselves.

Plus there are a few more tidbits: NRE said it will also be providing developers with “greater availability of information about service disruptions”; and providing them with more info about interchanges between national rail and other modes of transport, such as the DLR or the London Underground.

It has also committed to making it easier for passengers using NRE to find info about the routes on which their ticket is valid — another sizeable pain-point for train users thanks to fiendish ticketing complexities that appear intentional and profiteering in their scope.

And, according to the NRE spokesman, it’s “working on” getting Darwin data fed into all of the information screens at U.K. stations. “It’s in some of the stations at the moment, but getting it into all of the stations will again make sure there’s this one consistent source of what’s happening,” he added.

Commenting on the changes in a statement, (yet another) rail industry body, RDG, put up its lead on transparency, David Brown, who said: “The rail industry already publishes more information and data than many business sectors and leads the way among European railways, and we are committed to going further. Better access for developers to live train information will make it easier for even more passengers  to get the most up to date information about trains where and when they need it.”

So full steam ahead on the openness rhetoric at least.