With 17 million monthly uniques at its disposal, news aggregator Digg, which is attempting to overcome the user exodus caused by its unpopular Version 4 redesign in late August 2010, needs to find a way to keep its usership engaged and growing, if it ever wants to move beyond being a Silicon Valley cautionary tale.

It’s solution? The Topic Newsrooms beta, which separates news into top categories like Technology, Entertainment, World News and even topics as granular as Lady Gaga or Apple. Basically Digg is attempting to rebuild its user communities around interest in specific news. “There are a lot of signals out there,” Digg CEO Matt Williams, who describes what happened with Digg Version 4 as a “tragedy,” tells me, “It’s difficult to sort out what’s meaningful on a given topic versus what’s popular.”

Newsrooms will consist of two elements, a Front Page, reminiscent of the Digg “Popular” page, and a Newswire page, which is a realtime feed of all the stories submitted for a given topic. The Digg algorithm will take a look at signals like recency, Likes, LinkedIn shares and Tweets to determine whether a story will hit the Newsroom Frontpage.

In order to make the Front Page of a Newsroom, a story will also have to hit a critical mass of Diggs and Comments by Digg users, which signify a story’s “meaningfulness,” similar to how stories became popular on Digg in the past. The most “meaningful” stories from the Newsrooms will eventually hit Top News, or the overall Digg front page.

“With this new beta, Digg is also experimenting with showing user activity in realtime, and rewarding users with badges for participation, like the “Ace Reporter” badge, which users get for a story they submitted to Digg on the Newsroom Front Page.

Will it work? “Overall, I think the ‘newsrooms’ project is a gutsy attempt by Digg to bring in new users as well as some of the old ones,” says long time power user and VP of BizDev at Hasai Social Media John Boitnott, “However, they had better be ready to stick to their guns and make this new beta work over the long-term.”

The beta, which is invite only initially, is slowly rolling out to hundreds of users today and will eventually hit thousands, Williams says. I guess Digg has learned its lesson with regards to sudden changes.

Digg has raised a new round of financing, we’ve confirmed. It’s an inside round, meaning one or more existing investors put in the money.

We’ve heard the amount raised was single digit millions, probably around $5 million. CEO Matt Williams has confirmed the financing to us, but won’t give an exact number on the amount raised.

The real question is around the terms of the deal. Digg’s last round in 2008 valued the company at $150 million post money. But that was a long time ago, and Digg’s influence has waned. One source says the valuation is probably around $35 million. But Williams told me “this was not a down round.”

There are ways to structure a deal to give a better real valuation to investors while keeping the official valuation high. By increasing the liquidity preference, or using debt, or warrants. Williams wouldn’t comment further, though, on the round.

The company has $1.5 million outstanding debt with Silicon Valley Bank, and one source says they were pressuring the company to raise new money to improve the balance sheet. Without this round Digg had about 6 months of runway left before they ran out of cash, says one source. Now, they have more time.

 The strong interest [in Renren's IPO] stems from the fact that there is no major social media or social networking company open to public investment. With a Facebook IPO at least a year off, many investors are keen for a slice of “the Facebook of China”.

The offering is set to price on Tuesday in the US and begin trading on Wednesday. Highlighting the success of the pitch to market Renren as a Facebook proxy, the price range was raised on Friday from the initial $9-$11 to $12-$14. That could increase the deal size to as much as $743m, from the $584m planned originally.

The link in the excerpt above is from another Financial Times story that questions Renren’s valuation and user numbers, and is one of many signs that investors should take a cool and calm look at the company. But before looking at the doubts, let’s look at why the company is worth the excitement:

THE PROS

1. It’s probably the closest thing China has to Facebook

2. Highly competitive management, social network experience

Renren’s CEO is Joseph aka Joe Chen, a smart, tough investor and manager of Chinese Internet companies with a reputation as a formidable and experienced player and a savvy deal maker. Bill Bishop, one of the most astute observers of the Chinese Internet scene said “Joe Chen is a brilliant CEO with exquisite timing”.

Joe Chen and his team have experience building social networks long before the current craze: one of the company’s previous successes is Mop.com, a forum website that is something like a cross between Digg, Fark, 4chan and a Huffington-Post-for-bored-students. It was founded in 1997 and remains popular amongst the 18-32 age group (more than 20 million active users claimed). Mop.com will be spun off and will not be included in Renren’s IPO, but its survival and profitability are evidence of Chen’s team’s strength in monetizing social media and maintaining growth.

Another positive in terms of ownership and management : Japanese giant Softbank owns a stake of about 30% in Renren.

3. Advertiser support and early monetization

In the last few years, big brand advertisers in China have drunk the social media Kool-Aid. Renren was there, right from the beginning, offering brands a “Facebook for China” and the website is already very well known amongst media buying agencies and their clients.

Aside from advertising, Renren has also added two additional revenue streams: Nuomi.com, a Groupon-type service layered on top of the social network, and Renren Games which operates casual and MMORPGs on the Renren platform and elsewhere.

THE CONS

1. It’s not Facebook

Looking just at U.S. numbers, more than half of America’s 230 plus million Internet users are on Facebook. China has more than 400 million Internet users, but Renren only has only 20 to 30 million active monthly users and perhaps around 100 million registered users in total.

That’s a long way behind Facebook. There is no ubiquitous Facebook “Like” button on the Chinese Internet, and I have never heard of a young Renren user being horrified because their mom just signed up for the service.

Renren has simply not grown to occupy the place in Chinese life that Facebook has assumed in the U.S. and elsewhere.

Also worth scrutinizing all of the numbers very carefully. See for example: Renren Changes Key User Figure Before IPO, and the diplomatically titled Is Renren Seeing Explosive Active User Growth?

2. Brutal competitive landscape

In terms of the product, Renren’s closest competitor is Kaixin001, another Chinese Facebook clone that may seek an IPO this year. Kaixin is most famous in China for launching a Farmville type game long before Farmville was popular in the U.S.

But Renren is also up against dozens of other websites, including the following major players all of whom have very deep pockets:

Finally, a story breaking today on Reuters: Renren’s audit committee chair quits ahead of IPO

Further reading:

RenRen Inc. in 25 Slides  and “China’s Facebook” Renren to IPO Soon: Strengths and Challenges on Techrice
China Web 2.0: Joe Chen Wants it All, 2007 BusinessWeek profile of Joe Chen
The Cloner, Forbes profile of Wang Xing, orginal founder of Renren
The Rise and Stall of SNS in China, report by RedTech Advisors

Well by Silicon Valley standards, that certainly wasn’t a long “stealth” period. Just weeks after he quit Digg, Kevin Rose is announcing details of his new startup. Knowing Rose, I can tell why he’s so excited to get started. He’s constructed a company that plays perfectly to his strengths in the early days– but will test his weaknesses long term.

It’s called Milk, and it’s going to be a development lab in San Francisco’s hipster Mission District for mobile Web ideas. Along the Silicon Valley grapevine people have been calling it an “incubator,” but that term usually implies an organization that accepts entries from would-be entrepreneurs, funds them and helps groom them for the real world. That’s not Milk’s playbook at all.

Milk is not only counter to the typical incubator thinking– it’s counter to a lot of Valley trends right now. Rose and a team of just five other coders, designers and thinkers are going to set out to solve a handful of big old-industry problems using the mobile Internet. A year from now, he expects the company won’t have launched 20 small, cool ideas, but it will have developed four-to-six big, audacious ones.

Unlike a traditional incubator, Rose’s hope is these ideas are so out-there that several will fail, and one or two will become viable companies that have a big impact. And a year from now, he expects Milk will still be a small, elite team of less than ten people– a world apart from the talent landgrab going on in the Valley today. “We’ve been upfront with investors that the lab’s companies are going after big ideas, not launching continuous small projects,” he said in an interview yesterday. “There is so much opportunity to disrupt old media and old business.” So no Kevin Rose take on a mobile photo sharing app? “No,” he laughed. “Nothing like that.” Wow–that alone is counter-Valley thinking these days.

Rose has always had a flood of creative, disruptive ideas, but he has struggled to focus on developing the good ones. He says part of that was the architecture of Digg, which constrained how quickly he could build them, endlessly frustrating him. But this environment will allow his team to throw ideas out in the world quickly, see how they do, and kill the ones that don’t perform. The latter is something that doesn’t happen enough in the Valley, he argues. “People talk about pivoting all the time now, but if something isn’t working after four months, we’ll just shoot it in the head and start again,” he says.

Milk also reunites Rose with his designer from Digg’s glory days Daniel Burka. The two of them can lose days brainstorming on product ideas– one of which was the short-lived but pretty file sharing service Pownce. I once described Burka as the Robin to Rose’s Batman– an analogy Rose liked a good deal more than Burka did. “Robin is the worst superhero!” he moaned. Burka had been at Glitch, an online game built by Stewart Butterfield’s Tiny Speck. Burka says he loved working at Glitch, but couldn’t pass up the opportunity to be Robin again. (Sorry, Daniel.) He officially joins Milk as a co-founder today.

But reuniting of Batman and Robin aside, this isn’t a Digg do-over. This time around, Rose is the CEO and says he’ll remain the CEO– that’s a huge change from the early days of Digg when he wanted nothing to do with the business and only wanted to innovate on the product. “I’ve grown a lot in the last six and a half years,” he says. “I didn’t know what I was doing at Digg, and I gave away too much control. That’s part of the reason I got antsy at Digg. I know the mistakes we made at Digg, and I’m excited to start something new and ready to do this as a CEO.” Rose wouldn’t share details on the funding announcement, but we’ve heard it’s just a matter of finishing up the paperwork and that the roster includes a lot of Digg’s investors.

I’ve always seen a lot of similarities between Rose and Evan Williams. Williams, too, eschewed the idea of being the CEO after Odeo saying he never wanted to be “the guy” again. Then a year later, he took over as CEO of Twitter. Now, Williams has left Twitter and is getting back to his product roots too, with a new quiet company. If I had to guess, I’d say it’ll be reminiscent of Pyra Labs, which spawned Blogger, and Obvious, his product lab that spawned Twitter.

There are a lot of similarities between that idea and what Rose is building with Milk. And we hear Williams has been looking at office space in the Mission as well. It’ll be interesting to see what the two will come up with unencumbered from big companies and existing products.

Williams is extremely bullish on Digg. Traffic stabilized in January, he says, at about 20 million unique monthly visitors. The company has just 35 employees, he says, after 40% layoffs late last year. With the decreased burn rate the company is approaching cash flow positive, he says, and should get there this year. This is clearly a big priority for Williams, who is about to celebrate six months with the company.

But he also has big product plans for Digg, he says. “A year from now Digg will look very different. It will have a fantastic zeitgeist of the news product but it will also be many other things to news readers,” he told me.

That’s pretty vague and Williams isn’t saying much else about where Digg is going. But it’s clear that he’s planning yet another relaunch in the near future. Digg will remain a news aggregation site, he tells me, and notes that people still visit 4-6 sites per day for news. That’s an opportunity for Digg, he says. People today get a lot of news from people they follow on Twitter and Facebook, but that isn’t the whole solution.

He then spoke about the use of social networks and interest graphs to assist in finding news relevant to the user, presumably beyond just what your friends are digging or retweeting.

A customized news site has been a pipe dream for many failed startups. It’s just really hard to do. Partly because people consider news the stuff that everyone else considers news, so they know what to talk about around the water cooler. But also because it’s just really, really hard to figure out what people want to consume, and when.

If Digg gets this right, it’s a huge opportunity. If they don’t the show is probably over for them.

I also asked Williams about the difficulty in growing a startup when constrained by the competing goal of becoming cash flow positive. Former CEO Jay Adelson has told me that Digg became hyper focused on profitability during the recession a couple of years ago, and he probably cut too much fat and into the muscle. It was that period that Twitter grew dramatically and Digg lost status as an important news site.

Williams isn’t concerned, though, saying that cash constraints force good decisions. Once Digg figures out the winning product they can raise more cash to grow, if they want to.

I’m hoping to get Williams, Rose and Adelson in our studio next week to talk about the past and future of Digg at length. Would make a great show.

Update: Williams has a blog post up talking about the future of Digg as well:

The Digg Goes On

Got some love from a longtime Digg user at SXSW: “If I don’t start my day with Digg, it’s not a good day.” We’ve had some great comments like this from many of our users, and several questions at the same time.

We’ve been hearing a lot about how Kevin Rose is launching something new. Everyone knows Kevin is an entrepreneur at heart, and he’s had many projects in the works over the past several years. We’re excited to see what he comes up with next. Kevin continues to be committed to Digg’s success; his role as founder, Board member, and Diggnation host remains unchanged. When I took over as CEO last September, Kevin stepped back from the day-to-day decisions. I’m proud of the great team we’ve got at Digg, and they’re the ones to credit for the changes you’ve seen and the new direction we’re pursuing.

On behalf of the entire team, I’d like to highlight a few facts about Digg and then share some details about where we’re going.

When I joined Digg, we had just released a product that was not ready for prime time. It really upset our users. Over the first few months we dropped in the number of daily visitors and page views. But through this crisis, the lines of communication between Digg and our users opened to unprecedented levels. We received tens of thousands of comments and suggestions from the Digg community about how to restore the site they loved.

Since hitting a low point toward the end of 2010, our traffic has stabilized, and we’ve seen site engagement increase significantly: Diggs up 20%, time on site up 20%, and the total number of comments submitted per day is up nearly 50%. These strong numbers reflect both the passion of the Digg community and the tireless work of our engineering team. Longtime Digg users have been overwhelmingly positive about our ongoing improvements.

All told, we still have close to 20M monthly unique visitors worldwide, just about 1M unique visitors each day. We also have over 6M registered users, growing by hundreds of thousands each month. Depending on your favorite measurement service, Digg is ranked in the top 100-150 U.S. web sites, and our traffic puts us as one of the top news web sites in the world.

But even more interesting than where Digg is today is the role we will play in the years to come. Last week I presented at SXSW on the future of news. The amount of content published is growing by leaps and bounds over previous years – on a path to increase from hundreds of thousands to millions of major articles, blog posts, and media items published daily. A major factor is citizen journalism, set to explode with increased adoption of smartphones and content-sharing apps.

Meanwhile, it takes time and effort to find the news most relevant and interesting to you, given the overwhelming number of options. The average U.S. adult relies on multiple web sites for their news each day, and around 60% of Americans rely on both online and offline news sources. When we surveyed a sampling of our Digg users, we cataloged over 100 different web sites visited for the explicit purpose of reading the news.

In other words: massive opportunity. More news produced in a day than one can read in a year, yet no one has truly solved how to filter the news that each of us cares about individually. Digg’s community of users helps to curate what’s interesting – but even in our example, it’s just one crowdsourced view of the news. We must head toward a future where the crowdsourced view can be combined with news pertaining to your own interests and news shared within your own social circles. As a company that has been focused since day one on finding the most relevant and popular news online, we have a unique advantage in the industry.

So where is Digg going? Reinvention. Over the past 6 months, we’ve been rebuilding our team and restoring the web site to a stable, great experience for our community. We’ve made some recent changes in management, welcoming on board VP Engineering Ben Folk-Williams to replace our outgoing VP Product/Engineering Keval Desai. As importantly, we’ve hired some great engineers into Digg since the beginning of the year and our hiring continues. If you have a passion for news, join us!

Moving forward, we’ll continue to invest in making Digg.com better, and we’ll invest in innovative ways to discover news and share in the conversation beyond anything that exists today. Thanks to all of the active Digg users that have stood by as we continue to rebuild, experiment and innovate. We’re making a better Digg – and as you’ll see, we’re determined to deliver the news most relevant to you from the communities you trust. Stay tuned.

Startups in Silicon Valley are like old generals. They don’t die anymore, buoyed on life-rafts of lingering venture capital and modest revenues. They just fade away, eventually purchased for assets that are a shadow of their former promise. It’s pretty clear that Digg is on that path. The company isn’t dead, but it’s been fading away for a while, and its soul is all but gone. The company can spin it however it wants– the final nail in the coffin is news that founder Kevin Rose– long Digg’s greatest asset– is leaving.

I’m traveling in Indonesia, so the news will be old by the time you read this, but you’ll have to forgive another sentimental post. Digg has always represented the spirit of the early Web 2.0 movement to me. Facebook has never been the emblematic company of the Web’s mid-2000 resurgence, because it has always been such an outlier from the pack. But Digg– like Delicious, Six Apart, Flickr, YouTube and others– was one of those messy, risky companies founded at a time when no one was ready to believe in the Web again. The scars from the 2000 bust were too deep. These companies weren’t celebrated like Web startups today– they were mocked. People thought the founders were delusional.

The entrepreneurs were the exact opposite of the kids today seduced by the promises of Y Combinator, easy cash of super angels and lure of TechCrunch headlines. They were doing something that still stank of broken dreams and evaporated billions. And they were doing it for one simple reason: they couldn’t stop themselves.

And Digg was one of the first to prove you could take advantage of a decade of open source development to start a company for dirt cheap, one of the first to prove you didn’t have to be a technical genius to build a great product, and one of the first to prove a rabid community could make a site explode very, very quickly. Digg was never the biggest company of the movement, but it was bigger than many, and it stood for something. It was the everyman. This is why I put Kevin Rose on the cover of BusinessWeek in 2006. It was his first cover, my first cover, and one of the first national magazine covers about the Web 2.0 movement, period.

That cover– with provocative cover language cooked up by my wily New York editors to move copies– sparked a lot of hatred. It was my first brush with controversy, and one of BusinessWeek’s first big blog scandals as well.

But that cover also sparked inspiration, and the credit for that doesn’t go to BusinessWeek or me, it goes to Digg, Jay Adelson and Rose. It was the first time I saw young people reading BusinessWeek around San Francisco. On magazine racks it wasn’t put back with business publications, it was put back next to copies of FHM and Maxim. And recently BleacherReport founder Bryan Goldberg told me that when he read that cover back in 2006 he felt something he’d never felt reading a business magazine or even watching athletes and rockstars–sheer, consuming envy. If this kid– not a genius like Bill Gates, just a kid with an idea– could build Digg, why couldn’t he build what would later become BleacherReport? It was something that pushed him to quit his job and follow his own dream.

Fair disclosure: That cover probably helped me more than anyone. It landed me a book deal that changed my career. And I first met Michael Arrington right after it ran. He introduced himself to me just outside the Web 2.0 conference, and said he liked the story. That friendship changed my career too, and it was the first of many times he’d defend me against haters.

What Arrington got that others didn’t was that these companies and the Web 2.0 movement were only getting started. Among the article’s “outrageously overstated claims” was that YouTube could sell for $500 million. It sold for three times that a month or so later. The article argued Facebook could be worth more than MySpace. Again, that soon proved understated too. And Digg? Well we got Digg exactly right. We said it could sell for between $150 million and $200 million, and over the next few months and years there were several negotiations and at least one solid offer in that exact range. But Digg — unlike peers like Flickr and Delicious– said no, and its best days seemed ahead of it.

So what happened? In my view, Digg had a lot of things right. More than a million people loved its product– rabidly loved it. They loved it in a way we’d rarely seen until that point. Digg had top investors. And it had the vision part, too. Rose’s mission has played out. Digg helped transform how we consume media. While media properties balked at the idea in 2006, share buttons litter the Web today. We no longer rely on media gatekeepers for news. No one tells us what the front page should be– we create our own with the help of our friends.

Unfortunately Twitter is the one that’s pulled the bulk of his vision off, not Digg. It’s another example of what I’ve argued before– that it’s frequently not the company that comes up with something first that nails the execution. (And it might explain why Rose spends so much time on Twitter.)

The lesson from Digg is crucial as Silicon Valley’s ecosystem has made it easier and easier to start a company. It’s that a great product is necessary but not nearly enough. Building a real company is harder, and it takes execution and leadership. Things like a New York-based CEO and a sometimes-distracted co-founder took a toll on Digg in its most pivotal days. As I wrote in my book a year after that cover, startups reflect their founders’ personalities. Back then, Slide was characterized by silent intensity, Facebook was like a messy, pizza-stained dorm room, and Digg? Well, Digg’s offices were empty most evenings.

I have no doubt that Rose and Adelson are stronger after Digg than they were before. After all, few people remember that before Zynga, Mark Pincus’ Tribe didn’t live up to high expectations either. Like Pincus, I believe they both Rose and Adelson still have their biggest successes ahead of them. Adelson has already moved on with SimpleGeo, and Rose is moving on with a new mystery project.

There will be haters on this post. And that’s fine. But the people who write checks in the Valley have respect for what Digg built, whether the founders fell short or not. Smart people will always want to back these guys– as Mike Maples said on Ask a VC last week– and people like Arrington and me will root for them again.

That’s what makes the Valley such a unique place. (Photo by Thomas Hawk.)

Wow, when I wrote last night that Kevin Rose doesn’t really use Digg anymore, I had no idea how perfect the timing was. It turns out Rose really has tuned out. Because, say multiple sources, he’s already resigned from the company and is closing a $1+ million financing round for a new startup he’s founded.

Rose first launched Digg in December 2004. The service was an instant hit, and for a long while just all the big players thought about acquiring the company. Things never got so close as they did in mid-2008, when Google took Digg all the way to the altar before walking away at the last minute. Digg would have been sold for some $200 million. Every employee knew about the deal because Google had interviewed them all individually. Credit to then-CEO Jay Adelson for getting everyone back on track after the deal fell apart.

But those were the glory days for Digg. The site faded as newer services like Twitter and Facebook became ubiquitous. Rose and Adelson had a falling out, Rose stopped coming by the office much for months, and one of them had to go. It was Adelson. Rose took over as CEO until they hired Matt Williams last Fall.

Rose stayed on in vague executive role after Williams took over, but it’s been clear that Digg isn’t really top of mind for him anymore. He’s not using it much, as I showed last night. And he is very active with Revision3 and other projects.

So what’s the new startup? We’re still trying to figure that out, as well as who he’s hired and who’s invested. We’ll update as we hear more. Rose, for his part, isn’t responding to my emails, which isn’t surprising.

Update: Rose has tweeted that he’ll “continue advising Digg / on the board of directors, & taping Diggnation (as i have been since [Willaims] joined).”

How bad are things at the once-mighty Digg these days? Not so good. It’s been months since Digg relaunched in August in a quest for relevance. They had 18 million unique worldwide visitors that month according to Comscore. That dropped to just under 12 million in January, a 33% drop in just five months.

Everything official coming out of Digg says things are great and that a the company will find a way to success. But everyone knows how unlikely that is. Even, it seems, founder Kevin Rose.

He’s barely even using the service anymore. There was one 22 day period in December that he didn’t submit, comment on or even “digg” a single story.

Over the last 30 days, he’s only had seven actions on Digg, less than one action every 4 days. He hasn’t submitted a story in over a month, on February 13. You can see his account here. It’s not much better with CEO Matt Williams, although he manages to Digg, comment or submit a story about once a day on average.

In stark contrast, Rose is very active on Twitter, running up 181 tweets in the last month. He’s 26x more active on Twitter than the company he founded.

There isn’t much chance for Digg to ever turn things around if the senior team, particularly Rose, aren’t even using the service any more. A sad fate for a once mighty startup.

Folk-Williams was previously VP of Engineering at search and listings company Vast.com, and prior to that, led engineering at analytics startup Coremetrics, which was acquired by IBM last June.

Digg has a little bit of a shakeup at its executive level, losing its CFO and Chief Revenue Officer last Fall. And the company unfortunately succumbed to layoffs as well. It’s good to see the company adding more execs and talent.

We also talk about whether it has gotten too easy to start a company these days. Unlike much of the Valley, Maples still considers success to be a $1 billion outcome, and no matter how many companies get started, statistically only about 15 of them wind up being worth $500 million. There is a downside to those thousands that don’t– it wastes the time of talented would be executives, CTOs or even entrepreneurs who are working on a bad idea. While many VCs say they’ll back any great entrepreneur no matter the idea, Maples says a big reason he turns down deals is because he loves the entrepreneur but thinks the idea “isn’t worthy” of him or her.

Later, in the segment he tells us the three deals that got away, and what he learned from them. Oh, and we also give him shit for starting that whole PIVOT!!! thing last year. He explains the difference between a true pivot and a failed idea grasping at straws.

It’s a long video, but if you’re an aspiring entrepreneur you need to watch it. Maples is an entrepreneur’s VC. He admits to his failings, but has a non-lemming conviction for how he wants to invest.

Maples’ Ask a VC will air Friday and he candidly answers tough questions about two of his most noted investments Digg and Chegg. We’re taping another episode with NEA’s Peter Barris tomorrow, so send in questions for Barris to askavc(at)techcrunch(dot)com.

Editor’s note: This is a guest post submitted by Mahendra Palsule, who has worked as an Editor at Techmeme since 2009. Apart from curating tech news, he likes analyzing trends in startups and the social web. He is based in Pune, India, and you can follow him on Twitter.

What’s the Next Big Thing after social networking?

This has been a favorite topic of much speculation among tech enthusiasts for many years. I think we are already witnessing a paradigm shift – a move away from simple social sharing towards personalized, relevant content.

The key element of the next big thing is the increasing significance of the Interest Graph to complement the Social Graph. While Facebook, Twitter, and Google are already working on delivering relevant content, a slew of startups are focusing exclusively on it.

Relevance is the only solution to the problem of information overload.

The above matrix is a representation of how the process of online information discovery has evolved over time.

Phase I: The Search Dominated Web

This is how Google began its dominance over the web two decades ago, using PageRank to surface the most popular web pages as identified by other web pages that linked to them.

Phase II: Web 2.0 With Social Bookmarking

In the Web 2.0 era, social bookmarking services gained significant traction, surfacing popular content. Sites like Reddit and StumbleUpon are hugely popular even today, driving millions of page views.

Phase III: Personalized Recommendations

Services like Hunch, GetGlue, etc. have focused on building an Interest Graph for users, to deliver personalized recommendations using a ‘taste engine’.

Phase IV: Personalized Serendipity

The latest crop of startups is focusing on personalization using a combination of Interest and Social Graphs. Personalized Serendipity is what Jeff Jarvis calls ‘Unexpected Relevance’. Examples include Gravity, my6sense, Genieo, and TrapIt.

What Exactly Is Relevance?

The battle against information overload is sometimes presented as a choice between Relevance and Popularity, where ‘relevant’ is equated to ‘personalized’ as against popular.

However, Relevance does not always mean Personalized. Relevance is very dynamic – it depends on the needs of a person at a specific point in time. There are times when users want to know about the most popular stories, and other times when they seek personalized content.

There are multiple approaches to filtering information for Relevant Content. Google, Paper.li, and PostRank are examples of algorithmic filtering, while Reddit, Hacker News use a crowdsourcing approach. Klout can be used to filter Twitter streams by influence, while Facebook uses social affinity as a filter for its newsfeed and social signals for its new Comments Plugin. Location is another high-impact signal for delivering relevant content, gaining importance in a mobile world.

In other words, Relevance spans across all the quadrants of the Discovery Matrix above, and none of the above approaches to filtering for relevance is the ‘best approach’. There is no killer approach to Relevance. Henry Nothhaft, Jr., CMO of TrapIt, described it as “the myth of the sweet spot”. The competitive edge will be with services that support multiple discovery methods, multiple filtering approaches, have flexibility, and support multiple mobile platforms.

Quora: A Showcase Of The Interest Graph

Quora has pioneered the use of the Interest Graph as a dominant signal for its newsfeed. Quora asks new users to select Topics to follow, as part of its onboarding process, which is the first revelation that Topics are as important as Users to follow.

Quora’s newsfeed is an interesting showcase of what happens when you mix an Interest Graph with a Social Graph – and the result is the mysterious addictiveness so many have experienced, but found difficult to explain. An item pops up in your newsfeed not because you were following a user, but because you were following a related topic.

This often leads to Personalized Serendipity – or Unexpected Relevance – which is why Quora gets many people hooked.

The war over the Interest Graph began between Twitter and Facebook last year, as Erick described so eloquently. So how did Quora beat them to this game?

For starters, Quora is built from the ground-up with the Interest Graph being a backbone of the framework. Twitter’s ‘Browse Interests’ is too broad and primitive to be of use, even at present. And while Facebook has a mechanism for allowing publishers to push new items to your feed, most publishers have been unaware of this functionality.

This is also the reason why Facebook’s Like Button now publishes a full news feed story. The future clearly belongs to who best captures the Interest Graph as Max Levchin and Bill Gurley put it.

The implications of a Relevance-driven web are wide-ranging and broad in scope. Better utilization of the Interest Graph by services will lead to better ad targeting, and a potential decrease in reliance on CPM/CPC-based advertising. Monetization focus will be on higher yields through transactions and subscriptions as Dave McClure once described. Online media publishers will focus on Relevance Metrics revealing engagement and time-spent on site, than primitive metrics like page views and traffic.

Social media may lose its obsession with follower numbers and traffic, evolving to context-driven reputation systems and algorithms.

Interest Graphs will be used to build Better Social Graphs. Today’s monolithic Interest Graph will get further specialized into Taste Graphs, Financial Graphs, Local Network Graphs, etc., yielding higher relevance for different needs.

The Age of Relevance beckons!

The painful V4 redesign led to a user revolt and a drastic drop in traffic, with a corresponding increase in traffic over at competitor Reddit.

Cieri’s email says the RSS-submitted content is not performing well, which we can attest to judging by the single digit Diggs on stories and decline of Digg referral traffic on our site, and that the tool is being abused by spammers. The company will be reverting to manual submissions next week.

This is not the first time Digg has brought back V3 features as an attempt to bring back users after the initial V4 vision fell through. It brought back the controversial “Bury” button back in October as well as user profiles on submission pages and popular story statistics just last week.

The RSS submission tool was originally conceived to cater to publishers and expand possibilities for revenue. As this did not work out as planned Digg had no choice but to revert to the way things were says longtime user John Boitnott, “Digg bit off more than they could chew, and changed the way stories were promoted as well as the general conception of the site at the same time, so the massive number of people left.”

The consensus among former Digg users is that removing RSS submissions is a good move, and that V4 essentially turned the site into a “glorified RSS reader.” As the objective of a content aggregator is to filter the news and separate the signal from the noise, the “treating all publisher feeds as equal” model simply did not work.

Full email below:

Publisher Update From Digg

Publishers,

We hope this message finds you well. After a bumpy second half of 2010 at Digg, we are starting to see positive signs of improvement and are optimistic about the direction Digg is headed. In January 2011, we saw double digit growth of diggs and comments, as well as an increase in unique visitors and exit clicks out to publisher sites. We’ve taken a number of concrete steps to stay better connected with the Digg community, and we are taking action to improve Digg based on our community’s feedback. One important point of feedback we’ve heard is that RSS submitted stories are hurting Digg in a number of ways, and in the next week we are going to discontinue the ability to submit content via RSS. We’d like to share the reasoning behind the decision, and let you know what you can do to improve your performance on Digg.

Put very simply, most RSS submitted content is not performing well on Digg. For many of our users, RSS submissions take the fun out of finding and submitting great content. When users try to submit a story to Digg and find that the story has already been auto-submitted via RSS, they lose interest in helping spread the story on Digg by commenting and sharing with friends. Removing a user’s desire to champion a story results in less diggs, comments, exit clicks, and ultimately a much smaller chance of making the Top News section. Our analytics reflect this point – only 4.5% of all Top News content comes from RSS submitted content (95.5% is manually submitted).

At its core, Digg is a community of passionate users who take pride in the content they submit and engage with one another in discussion and promotion of viral content. There is a perception that some publishers don’t participate in the community, use RSS submit as an “auto-pilot” tool to submit content without discretion, and do little to promote submitted content or start discussions. This is one reason why many popular publishers, despite having tens of thousands of followers, are not seeing strong referral numbers for their submissions. Some publishers have cultivated a tight following on Digg by digging and commenting on content other than their own, adding Digg buttons prominently to articles on their site and limiting the content they submit to just their best content. These publishers are seeing much more value from Digg.

Finally, the RSS submission tool has been heavily abused by spammers and has been a constant drain on our technical resources to identify and fight off spam content. The simple act of forcing a manual submission helps to combat spam and ensures that quality content appears on Digg.

So in the next week, the feature will be disabled. We wanted to give advance notice of this change and encourage you to start submitting your best content manually to Digg. You can also enable your audience to help submit and spread your content on Digg by placing Digg buttons on each story item on your site. We are confident that removing RSS submissions will help increase exit clicks to your sites, and ultimately help you receive more value from Digg.

Please feel free to contact me with any questions or thoughts.

Thanks,
Mike

Editor’s note: Henry “Hank” Nothhaft, Jr. is the co-founder and CMO of Trapit, a virtual personal assistant for Web content still in private beta that was incubated out of SRI and the CALO project (as was Siri, the conversational search engine bought by Apple).

One of the most interesting concepts to emerge in media and tech lately is that of “serendipity”—showing people what they want even if they didn’t ask for it.

Despite its seemingly ubiquitous invocation, however, the concept of serendipity remains ill-defined and put forth as some vague panacea for a slew of emerging innovations hoping to attract new users in droves.  What is needed is a closer look at what we actually mean when we talk about serendipity.

From Search to Discovery

Eric Schmidt’s recent remarks about Google as a “Serendipity Engine” (and Facebook’s quick reply), emphasize an important shift in our daily interaction with the Web and how we use it.  Google-driven search provided us with an expectation of finding what we are looking for with increased precision.  But the rise of Facebook’s social relevance algorithms brought about more personalized content discovery based on the human graph—who we know and what they are reading, watching, or passing along.

In fact, I’d argue that we’re seeing the dominant portion of our interaction with Web content shift from search to discovery.

Jeff Jarvis has perhaps most succinctly defined the concept of serendipity, arguing that serendipity is simply “unexpected relevance.”  His explanation opens an entirely new can of worms, however, in the recognition that relevance is relative.

In seeking to achieve serendipity, the individual reader becomes both the target of content delivery mechanisms and the genesis of what that content may be. This is why serendipity is so closely associated with personalization—it requires a high-resolution understanding of the user.

Serendipity and personalization are in fact two sides to the same coin.  Personalization merely acknowledges intimacy, whereas serendipity pretends to have happened on it as if by accident.

Of course serendipity is not, in fact, at all random. In reality, it’s quite scientific. Good serendipity is a slight of hand—it requires deep and granular knowledge, and the fact of its seeming to happen by accident is an artifact of naivety, if anything.

Serendipity is really just an informed calculation based upon any number of our individually unique interests, habits, location, the time and date, and prior knowledge. This level of relevance is, of course, what the emerging personalized Web hopes to achieve for each user, whether for recommendations (GetGlue; Hunch), marketing and ads (Rapleaf; Facebook advertising), or news and content (my company, TrapIt).

Below I run through four different kinds of serendipity—each has its pros and cons. I end by talking about them all taken together, and “the myth of the sweet spot”.

Editorial Serendipity

Editorial Serendipity is the first and oldest form, the process of combining articles that we know we want to read (the day’s headlines) with unexpected stories (features, profiles, restaurant reviews). Yet the editorial voice and direction of a paper or aggregator is hardly serendipitous; it is a calculation of demographics and readership, whether you’re the New York Times, the Drudge Report, or TechCrunch.

On the plus side here, the human element of editorial serendipity (someone making decisions on what content to deliver) provides an effective flexibility of interest. The downside is that editorial serendipity is delivered by another’s interests, or at best their perception of their audience’s interests. Though the content’s relevance is targeted to a certain demographic of readers, it is a necessarily broad sweep of potential readers, and the level of interest is based on the editors’ perception of what is most in tune with those readers or what she thinks they should be interested in based on her own judgement.

Examples: Newspapers/Magazines, Curated Aggregators

Social Serendipity

Much of our content discovery now comes from the virtual watercooler of what our social circle is sharing directly online. The social aspect of staying informed with what our friends are discussing is valuable, not only for keeping “in the loop,” but also simply for the notion that what our friends like is parallel to our own interests.

The benefit of social serendipity is that our social groups have always been a primary indicator of how we choose to define ourselves and our interests. If something is important or relevant to our friends, there is a high likelihood that it is also relevant to ourselves, as well. The con is that social serendipity is therefore largely public by necessity, and thus a projection of ourselves we would present to others or like to be seen. The propensity to amplify the echo-chamber of like-mindedness is also exaggerated, whereas the goal of serendipity largely lies in the surprise and delight of unexpected content.

Examples: Facebook, Twitter

Crowdsourced Serendipity

Bridging the gap between editorial and social serendipity, the notion of crowdsourced relevance really only delivers a broad, lowest-common-denominator level of content discovery. While not without its usefulness to the degree that we want to be aware of what is most popular and most talked about, the trade-off is the lack of personalization.

The pro here is the viral component, which makes up a great deal of our online content-discovery routines. Crowdsourced serendipity provides a tier of distribution in touch with a larger zeitgeist, from trivial cat videos to important broad-based news. The downside is that the lowest common denominator lacks any precision and therefore has limited utility.

Examples: StumbleUpon, Reddit, Digg

Algorithmic Serendipity

Opposite editorial serendipity, the notion of algorithmic serendipity is the hardest to do well, but the most promising for future innovation. (Bias alert: this is the approach we are trying at TrapIt)

Based-upon any given set of data points, content is personalized to provide both the relevant, need-to-know information of news and content correlating to our interests, with varying degrees of flexibility through both active and passive inputs.

The best aspect of algorithmic serendipity is that it places the user back at the center of defining relevance. Content delivery emanates from the user, whether consciously or in the background based on habit. It also provides for a level of adjustability and fine-tuning based on individualized input and how narrowly or broadly a user may want the information delivered to him.

The con with algorithmic serendipity is that we need to be careful not to completely lose the human element of engagement, no matter how accurate the algorithm is. Of course, the biggest hindrance is that unlike the other forms of serendipity, a finely-tuned algorithmic Serendipity Engine has yet to be effectively realized. Still, it needs to only be the starting point rather than end point of achieving personalized serendipity.

Examples: Genieo, My6Sense, TrapIt

The Myth of the Sweet Spot

The challenge for any conception of serendipity, regardless of type, is the prevailing notion of a mythical “sweet spot” for users.

In all of the forms of content delivery outlined above, there is a notion that we can hone in on a user’s interests and find the right balance of relevance.  Presenting any such balance as stable or definitive is pure folly. We humans have no “sweet spot”—our interests are evolving and fluid in realtime.

To some extent, this recognition is obvious. Our interests change and evolve over time. Yet for the kind of precision that seeks to provide consistent serendipity in the ways we have been discussing, the indicators need to be equally sensitive.

The content that I want, and better yet, the content that I don’t even know that I want, is an ever-changing proposition based on any number of factors. To achieve that level of sophisticated customization requires a sensitive understanding of context for any proposed “serendipity engine”, both a context of the content and the user.

In the end, relevance is a goal based on context. The impossibility of fully understanding every intricacy of context at any given moment makes achieving the mythical, consistent sweet spot of serendipity impossible. Recognizing that serendipity is a constantly moving target of context, the best we can hope to achieve are fleeting moments relevance.

Photo credit: Flickr/Jennifer Konig

Jay Adelson, the CEO of Digg until April 2010, just landed in a new position. He’s taking over as CEO of location services startup SimpleGeo, and will join the company’s board of directors. Founding CEO Matt Galligan will become the company’s Chief Strategy Officer.

SimpleGeo, which has raised nearly $10 million in venture capital, allows companies to add location features to applications. Earlier this year we made light fun of the company for failing to clearly explain exactly what developers get out of their product.

Hopefully Adelson understands the company. And he certainly seems to. In a phone call this morning he said he’s excited to be working with former Digg employees Joe Stump and Jeffrey Kalmikoff, and said he’s looking forward working at a startup that is at the absolute center of things right now.

The company should make an announcement shortly, and we’ll update with anything they publish.

As of today you can find Breaking News in red widgets labeled “Breaking News” to the right of the Upcoming, My News, and Top News sections on the new Digg.  Breaking News stories are also designated by flame icons.

This is yet another large news aggregator experimenting with human editors,  Google News and Techmeme being the most visible news sites to make the jump into human curation thus far.  And similar to Google’s “Editors’ Picks” the widget will highlight stories deemed important but not replace the core news surfacing features of the site. Says Digg Product Lead Keval Desai, “There is no change to how the stories are submitted. Digging and burying are not done by Digg employees.”

So what criteria will the editors be using to deem a story Breaking News? Says Desai, “The widget will showcase stories that are submitted by Digg users that are currently hot, i.e. gaining in velocity on Digg as well as what’s going on across the Internet and what’s gaining in popularity in terms of news memes.”

It’s commonly acknowledged that aggregators like Reddit, Hacker News and even Digg have had at least some administrative supervision (like TOS enforcements). Digg itself faced accusations of manually intervening on behalf of certain publishers in early October. Desai says this is the first time any Digg staff will put a concerted effort into picking stories for editorial reasons.

This is a great idea, but it might be two years too late. Says Power Digger Andy Sorcini a.k.a Mr. Babyman on whether the feature will at least revive some interest in the new Digg, “I think it’s a smart move for Digg to meta-curate their already curated content. Especially in the wake of the drop of traffic from the switch to V4, they need all the help with content surfacing they can get.”

Not all attempts at human curated news have been successful however. Breaking News resembles what Jason Calacanis attempted with Digg clone Propeller, which ultimately failed despite its novel editorial oversight features in part because it confused people accustomed to the classic Netscape.com. While human editing does make a news engine more competitive with Twitter and younger upstarts like BuzzFeed and Spotery, it is not a panacea and comes with its own set of headaches.

Says Sorcini, “The only concern I would have is, It’d be nice to have some sort of transparency on their selection process, so users know there’s no slant (for example, favoring a advertising partner, such as BP).” And so it begins.

It’s been a difficult couple of months for Digg. The crowdsourced news site pushed out a major new design at the end of August which met with a lot of criticism and broken axles. There was literally a user revolt and things deteriorated so rapidly that earlier this week the company had to let go more than a third of its employees.

How bad did it get? Here’s one data point from comScore: Digg lost 30 percent of its audience in the month of September alone. Digg’s estimated unique visitors worldwide went from 18.4 million in August to 12.8 million in September. That is a drop of 5.6 million people in a single month. Remember, the new site went live for everyone on August 25, so September was the first full month of the new design. Compared to a year before, Digg’s worldwide audience shrank by 16 million visitors.

Other metrics show the same story or worse. Pageviews sank 70 percent from August to September (from 155 million pageviews to 46 million). Average visits and time spent per visitor were also down.

The numbers for October are not yet out, but I’d be surprised to see any immediate rebound. Can Digg stanch its losses and start to rebuild once again?