Fixing cities

Local information is a mess, but tech and journalism are joining forces to help

E

xciting news for people who live, well, everywhere: Incremental changes happening across tech, local news and local government are finally making the world around you transparent.

This isn’t some “smart cities” concept fantasy, like those corporate marketing videos you see from time to time. This is about all of us finally getting answers to the real problems facing cities today that affect our personal lives.

The last big pieces are just now falling into place, with tech companies finally pushing major improvements in two areas:

Local distribution: Google, Facebook, Nextdoor and other companies with local-facing products are increasingly trying to surface the most relevant data points, articles and other information to their users.

Local data: Yelp, Foursquare, Trulia and many other local-category tech companies are opening up and analyzing their data for everyone, in addition to the public data sets and analytical tools that have been coming together in recent years. It’s part of the data “fuel for the future.”

With more relevant local information at your fingertips, you might finally know where to rent an apartment that’s perfect for your needs, open a brick-and-mortar business with a better chance of success or understand what motivates your neighbors.

City governments are getting a much deeper understanding of how policy changes on deep-seated issues like housing, transportation and local retail interact with growing populations, new technologies, and related issues like evictions and loss of local businesses.

Tech products will also be able to provide more good local information, contextualized to the end user based on local graphs of topics, interests and places down to the neighborhood level. End users get value that way, but then can return it through actions like leaving better reviews or frequenting more local businesses.

And, perhaps most surprisingly to many, local news stands to benefit. It could finally get strong local distribution channels and better tools and data to lead the way in really understanding cities. This could pave the way to much better revenues from local news subscriptions.

In this article, I will examine the new distribution channels for local information, the terabytes of new data sets and the potential that this emerging ecosystem now has.

Disclosure: I might be a little biased, because I left TechCrunch in 2014 to cofound Hoodline, a neighborhood news and tech company that is in the middle of all of these trends.

Images: Hoodline

The new ecosystem of local information

Why is “local” getting a bigger focus from tech these days? One factor was probably the 2016 election. It revealed, among other things, that many people in Silicon Valley are not that familiar with local issues in other parts of the country and that major distribution platforms didn’t provide that much helpful information about major civic issues. But this was just a newer development in the longstanding problems around local information.

You’ve heard about the worsening situation for local media — the years of ad revenue losses, the ongoing newsroom layoffs, the resulting decline in quality, the growth of news deserts, etc. Tech companies are in the last stages of destroying traditional media by providing better advertising and consumer products.

The consequence of this, for these companies, goes beyond civic duty or public perception. Their users actually want good local information more than ever, and right now this information is often bad or not available.

Local search and social

Google and Facebook are the main online sources of traffic for many local publications. But neither consumer platform has made local discovery a priority. This seems to have changed, both because these companies are perhaps more concerned by the future of journalism these days, and because they are fiercely competing with each other and many others for local users.

Still, Google search results for local topics tend to be stale. In fact, publications like mine see considerable search traffic about long-tail neighborhood topics since we’re often the only sources.

Facebook, meanwhile, is trying to promote more substantive content, especially given the “fake news” controversies last year. When every local media company is reblogging a New York Times scoop in order to ride the publication’s viral traffic wave, they’re not publishing local stories that a lot of Facebook users want.

Since both companies rely on user data to target ads, they ultimately need users to generate more data and engagement around local interests to provide the targeting that many advertisers want.

Google has experimented with local news distribution over the years, from when it first added a way for you to search Google News by city and ZIP code in 2008, to when it added a Local Source tag last year to highlight local news sources when their stories get picked up nationally.

The company has begun testing new ways of featuring local news articles about things like restaurants from local publications like the Washingtonian and national groups like Vox’s Curbed network. And through initiatives like its News Lab, it is helping journalists to get better at using technology. Meanwhile, its Google Maps product is providing better and better local information. I expect the company to start surfacing more local information directly to users in features like the OneBox, as it already does for many national and global stats and topics.

Facebook has been providing one-size-fits-all distribution tools for years, which have naturally rewarded larger publishers who provide broadly interesting content. Local publishers have benefited, as well, just not nearly as much. This is changing. As part of the company’s plan to focus more on promoting quality news and information, it has just launched a few new features.

One lets you designate yourself as a local via a badge icon, when you comment on articles by publications in your city of residence. Another feature points people to local groups, and a related third one points users in these community-oriented groups to local news. Facebook also recently bought Crowdtangle, a social media analytics tool that “publishers use to win,” and made it free expressly to help publications see what kinds of stories are of most interest to readers. We and many others are starting to use it to better hone our local coverage.

 

Recent examples of Facebook's new locally focused features.

Nextdoor, the private neighborhood social network that has been carving out an audience despite Facebook’s popular local groups features, has been testing out new tools for local publications to reach its users. Publications can create their own pages, then publish article links and target specific neighborhoods for relevant topics. Hoodline has been a part of these tests, and has noticed some quality new traffic as a result.

Other social media platforms like Twitter, Snapchat and Instagram have robust local activity, and each has been adding local content filters — most recently Instagram’s Location Stories feature.

Real estate

Online real estate platforms have become deeply embedded in local journalism and even local political discourse in a way that’s so natural one forgets that the data these companies provide didn’t really used to be available. In fact, the leading companies in this category have established themselves partly through bringing new transparency to the market for home buyers and sellers. Check out how easy it is for a local journalist these days to credibly write about, say, a neighborhood’s housing price increases with just a few clicks.

Trulia provides a wide range of data and content marketing, including neighborhood-level home price trends over time. It also offers more widely available, useful visualizations of city data than what most city governments themselves can achieve, on topics like crime, school grades, commute times and general demographics. Recurring blog posts like the Most Expensive Homes for Sale in America become easy stories for the media to write about. The company is also continuing to expand its local content operations, as you may have noticed with its new neighborhood-focused marketing campaign.

Trulia’s maps let you explore housing prices and a lot more.

Zillow, Trulia’s fairly recent new owner, has been a de facto resource, to the degree that it gets featured in articles about the factions in San Francisco’s housing policy wars. Here’s a nice example from The New York Times last year:

“Every few weeks, when a company like Zillow puts out a new price report, both sides hold up the numbers as an example of how San Francisco has failed the middle class. Zillow puts the city’s median home price at $1.1 million, neck and neck with Manhattan. The region’s rent, at $3,500 a month for an average apartment, is the highest in the nation.”

Redfin, the main online listings competitor to Zillow/Trulia, is also a standard reference point and trend data source. You may have read “Redfin Predicts the Hottest Neighborhoods of 2017,” for example, or one of the many follow-up articles about it from news sources. To make things easy for reporters, the company offers a Data Center page that provides a wide range of interactive visuals, including downloads (and prominent citation directions for press).

Other companies that offer online real estate products competing for the most interesting real estate market analysis include Zumper and Propertyshark.

Local reviews and events

Plenty of people still want to read news and professional criticism to navigate where to go and what to do, whether about a new restaurant, mechanic or musical act. But if you want to get specific information about, say, a doctor or a health spa — or learn what most people think of a restaurant, for that matter — you’ll look at a local review site. While this area is not as far along as real estate, there are some exciting recent developments.

Yelp, the oldest and largest local reviews site, has built a fascinating data set over the years, that it has been analyzing together with academics and media. For example, it collaborated with the Harvard Business School and Mathematica Policy Research to look at how minimum wage laws affect local businesses. It found that lower-ranked businesses are more likely to close if wages go up — to the great interest of those who don’t like minimum wage laws — but also that higher-ranked businesses are not affected, and the overall economy isn’t, either. Yelp has begun making more and more data available to third parties by API, including to Hoodline. We’ve used it to find neighborhood cuisine trendsnational donut rankings and as a source for neighborhood retail vacancy analysis.

Neighborhood cuisine analysis using Yelp data. (Source: Hoodline)

Foursquare has gone through a few iterations to emerge as a local data company, combining its own local reviewer app data with location data generated by a wide variety of tech partners, including Snapchat and Twitter. This means it isn’t just using its own check-ins and such: it can analyze location data generated from its partner apps.

The result is that it can get a sense for significant, even market-moving changes based on where people are going with their phones—like when it famously observed Chipotle’s foot traffic issues following food safety problems, before nearly anyone else. The company says it is on a tear in terms of revenue, slicing and dicing its location analysis for anyone from online advertisers who want to do targeting, to local businesses trying to understand foot traffic.

I expect it to become an increasingly important data source for journalists, especially as its data-analysis abilities become more widely understood.

Foursquare’s analysis of Chipotle’s foot-traffic problems from last year.

Eventbrite, one of the main online ticketing sites for local events, has been developing a much deeper content marketing strategy to help define itself as a discovery tool. In addition to producing its own local content, it has begun syndicating articles in city-based email newsletters—from my company, so far. It’s a new way for us to reach a large group of local users.

Travel and transportation

Airbnb has tested a range of local guide-type products over the years, most recently a product that turns its short-term rental hosts into local reviewers. As part of this bigger effort, it recently partnered with Hearst to produce a special travel print magazine, in which it uses internal data to figure out interesting new locations for Hearst writers to cover.

Lyft tested a sponsored data-driven, content-marketing product last year with major local publications including DNAinfo in Chicago and The New Tropic in the Miami area: a map of popular city destinations generated by data from its ride-hailing passengers in the area.

Uber has been experimenting with how to add more local context to its rides, with the addition of Foursquare place data last year, and experiments in showing relevant news articles in new locations (which we were a part of, and shared a little bit more about last year). The company is also promising to open up its own data through a project called Movement.

Visualization of street grids in Uber’s demo video for Movement.

Public data

Public information about cities has existed since the modern census was first developed. But historically, government officials, journalists and academics who wanted to analyze issues had to spend months or years, and lots of money, creating their own data sets out of surveys or manual record-taking.

As city information technology systems have gradually digitized, the available data has gotten far better. For example, you can now find up-to-date public maps of crime data on Crimemapping.com via a government software provider called TriTech Software Solutions. California’s Open Justice platform, developed by the Department of Justice, tracks and publishes criminal justice data.

And at least in some cities, there has been a strong drive to make this taxpayer-funded data — which includes information about transportation, property and sales tax rates, zoning and much more — available to the public. The US City Open Data Census provides a great overview of these efforts to date. San Francisco’s open data site is particularly well-done, from what I’ve experienced.

While making government data public is a complex task that involves navigating laws, politics and aging integrated software solutions, analyzing it in a useful way can be even harder. All sorts of companies have been integrating more public data for their own purposes, as it becomes available. Some, like Enigma, are specializing in the cleaning and structuring process to aid better analysis by anyone.

These data sets have so far been getting the most attention from local media — think of the big screen maps of county election results you see on local TV stations during election nights. This is possible because local governments are publishing detailed public election results for all to see.

This trend is exciting in its own right, but combined with company data and products, it can get very interesting. For example, when the city of San Francisco began making its health scores available by API a few years ago, Yelp incorporated the data so you can see the score of any restaurant you’re considering going to.

Enigma uses public data sets to explore all sorts of trends, including the booms and busts of North Dakota oil wells, above.

New data software tools

A sophisticated range of software products and techniques have evolved to help the world understand all the new data available online. Data science, artificial intelligence and machine learning are constant discussion topics in the tech industry, because they provide big new ways of understanding all the data being generated by modern technology.

These tools can be applied to local news in important ways. The Associated Press has been leading the way, in terms of guiding news organizations through research like this big report on augmented journalism, to actually using these tools. It has partnered with a machine learning company called Automated Insights to generate drafts of wire-type stories about local sports, crime and other topics. This is allowing journalists in newsrooms across the country to lightly edit these automated drafts, then have more time to work on higher-value stories.

The possibilities are just starting to emerge, with most of the focus still going toward national topics (like the AP’s recent look at Twitter sentiment about Trump). Speaking from our experience, hiring data experts has allowed us to quickly start producing a much deeper view of all sorts of data sets we’re getting access to. And with that, let’s take a closer look at local news.

What’s happening in local news

The examples of tech efforts, outlined above, could make huge differences, especially for distribution.

Local media has been appreciating the new love it’s been getting from tech companies, as this perceptive recent Digiday article noted — even if the efforts are new, and the results are not exactly life-saving right now.

But some of us have also been taking matters into our own hands when it comes to the new possibilities. And I don’t just mean setting up social media accounts or reblogging real estate listicles.

Newish groups of locally focused publications have reoriented themselves around data and visualizations, generally combining social media tips and third-party analysis by companies and governments to augment their own reporting.

Vox’s Curbed Network, for example, is today essentially a map-based exploration of real estate (Curbed), restaurants (Eater) and shopping (Racked). Curbed, for example, makes heavy use of reports from companies like Zillow and Redfin to cover real estate trends. But it also does original work, like this map of the big developments around mid-Market, a part of San Francisco that houses many low-income residents and nonprofits, that has also become its newest tech corridor.

Spirited Media, recently merged from sites like Billy Penn in Philadelphia and Denverite in Denver, regularly features city-based research to help it provide more depth for its target readers.

DNAinfo, a neighborhood news site for Chicago and New York City (and new owner of Gothamist), has gone further than most in using tech to tell local stories. Check out its clever crowdsourced map of New York neighborhood boundaries or its visualization of what planned developments would look like in the city skyline once they’re built.

Curbed tracks the changes on Mid-Market (Source: Curbed)
DNAinfo used public data about real estate development to visualize how neighborhood skylines are changing in New York. (Source: DNAinfo)

Patch, revived to profitability as an independent company after being spun out of Aol, has focused on distribution. For example, it’s syndicating local newspaper articles in addition to its own reporting about cities across the country, to help publications that struggle to get their work seen by online audiences.

My own company, has used public data to find local stories since its earliest days. Now, we are combining this work with tech-company data sets to which we’re getting access, in order to really dig into cities.

For example, we investigated major retail vacancy problems in San Francisco’s Castro neighborhood, because we were able to use a detailed appendix from a 2015 joint city-community analysis and combine it with Yelp categories and our own shoe-leather reporting. We found that, despite many efforts, the vacancy problem has gotten noticeably worse, with major losses concentrated among local retail shopping options.

Retail vacancies, Castro neighborhood, San Francisco - 2015. Source: Hoodline

Retail vacancies, Castro neighborhood, San Francisco - 2017. Source: Hoodline

Hoodline has also used city data to identify problems with city transit fare ticketing, historical Yellow Pages data to track the decline of laundromats over the decades and neighborhood-level violent crime increases, among other examples.

We’ve found that videos can often tell these data-driven stories in ways that text-based online articles can’t, by guiding people step-by-step through what the analysis means. For example, we’ve partnered with Disney’s ABC7 station in the Bay Area, with their news team doing on-camera interviewson-air visualizations and digital-first videos alongside our reporting. Like local TV weather reports, sports scores and election coverage, we think video will become the main way that many people will consume this data to understand the world around them.

Besides this editorial work, we’ve also focused on distribution solutions. We’ve partnered with more than 200 other local publications to create a content distribution platform that is beginning to feed stories into a variety of sites and apps. As an exhibit of the product, check out this map of the articles from ourselves and partners in San Francisco, tagged automatically via machine learning for location, topic categories, sentiment and shelf-life.

ABC7’s Jessica Castro using some of our data analysis for a look at neighborhood cuisine trends. (Source: ABC7)

Next, we’re using public and private data to create a more comprehensive analytical layer on the platform. Eventually, we’ll be able to predict things like what kind of business would do well in that vacant storefront down the street from your house.

We believe in building a system for understanding the topics where each additional story adds value to the whole. And we think other local publishers should do the same.

Unfortunately, most local publications still treat data analysis as a bespoke operation. For example, The Los Angeles Times made an excellent map of voting patterns by precinct across the state of California, revealing at a granular level which locations preferred Trump or Clinton.

Maybe that map product will be brought out for the election cycle, and maybe there’ll be some analysis of changes in voting patterns over time. But voting patterns reflect all the other issues that the LA Times is writing about in each of these places, day in and day out. How do the voting patterns correlate with issues that might make voters lean one way or another? This map could be connected to local industries, housing and transit infrastructure, types of businesses, and everything else. It could help provide a much deeper story.

Click to use a live demo of Hoodline’s content platform.

Good news for the business of local media?

All of this brings us to local media revenues. Many in tech and media think that local news subscriptions are the best known business model, and it might start to work a lot better with new distribution and better content. There are also some other interesting possibilities.

First, subscriptions, which are also starting to result in success stories. “Most local newspapers are simply not worth saving,” widely read tech analyst Ben Thompson wrote in a recent post, “not because local news isn’t valuable, but rather because everything else in your typical local newspaper is worthless (from a business perspective).”

Tech products and larger online publishers have replaced or diminished the value of traditional local media newspaper sections and television segments, from sports to shopping to stock prices. To build a strong subscription business, as Thompson outlines, you need to regularly deliver a high-value product to the subscriber.

But local subscriptions all have to work harder, because local conversion funnels are a lot tougher to operate than global ones. The portion of the world that has become inspired to pay for The New York Times after seeing enough of its Trump scoops on Facebook is far larger than the possible number of subscribers who care about San Francisco, who might want to pay the Chronicle to cover City Hall.

If tech companies build robust distribution channels for local media, potential subscribers will be far more likely to be exposed to the free versions of content on a regular basis, leading them to hit the paywall and cough up the money. Or, their contextualized local content could augment the user experience for tech companies enough that tech companies end up being the subscribers (on behalf of their end users). Data companies might even pay to have their data distributed. 

There is a lot left to figure out. What topics and news formats would most appeal to potential local subscribers? Which distribution channels would work best for the funnel? What technology expertise is needed in-house to create the needed content? Which partners are best suited to help?

Now that tech companies are adding their firepower to solving these sorts of questions, I think the answers are on the way. And the result will be a whole new view of how cities work and how we can make them better for everyone.

Top and bottom image: Getty Images / Michael H