Update: Digg Recommendation Engine Confirmed For This Week


Digg has released some materials around their new Recommendation Engine, which we wrote about last night, and say that it will be released this week. Two overview videos are below, including an interview with Digg Lead Scientist Anton Kast. We’ve also included the text of a white paper on the Recommendation Engine.

Digg Recommendation Engine from Kevin Rose on Vimeo.

Anton Talks About The Digg Recommendation Engine from Kevin Rose on Vimeo.

The Digg Recommendation Engine
People love Digg because it’s a place to discover and share great content from around
the Web. The Digg homepage always has the most popular stories, but many Digg
users find their content in the Upcoming section, which gets over 15,000 new stories a
day. To help users filter this enormous amount of content, we have created a new
feature: The Digg Recommendation Engine.

When you Digg a story, you tell the Recommendation Engine two things: that you
recommend the story to other users and, less obviously, that the users who Dugg the
story before you are good at finding content. The Recommendation Engine keeps track
of users who Dugg particular stories before you did, and it recommends you the stories
they Dugg. The more content you Digg, the smarter the Recommendation Engine

Finding Diggers Like You
The Digg Recommendation Engine uses your Digg history over the last thirty days to
make Recommendations. (You can see the number of items you have Dugg over the
last month on the right-hand side of the Recommended view.) Every time you Digg a
story, the Engine matches you with other Diggers who Dugg the same story, and keeps
track of all your Diggs in common with them.

When it’s time to calculate your Recommendations, the Engine draws from this pool of
matched Diggers. For each matched Digger, it computes a correlation coefficient
between you and them. It then picks a cutoff for this correlation coefficient, and the
Diggers who make the cut are called “Diggers Like You.”

It’s easy to understand how the correlations are calculated. For each user with whom
you Dugg something in common, the Engine determines how many stories the two of
you Dugg in common, and divides that number by the total number of stories you or they
Dugg. The ratio is a correlation coefficient, a number between zero and one (zero if you
and the other user never agreed; one if you always did). Such a ratio is sometimes
called a “Jaccard coefficient.”

This scheme automatically accounts for the overall level of Digging activity. If another
user Diggs a lot, they have to agree with you on many stories to become a Digger Like
You. If another user Diggs rarely, then a small amount of agreement can suffice.
From Diggers Like You to Recommendations
Once the Engine has determined your Diggers Like You, your Recommendations consist
of stories that your Diggers Like You have already Dugg, minus the stories you already
Dugg or Buried. There are some extra steps, like the diversity rules and the
promotability constraint described below, but this is the basic idea.

Recommendations are always displayed together with your Diggers Like You and their
compatibility percentages. These percentages are just correlation coefficients. You may
notice that you are more compatible with a user that has fewer Recommendations than a
user with less compatibility but with more Recommendations. This is because although
you have Dugg more items in common with the more compatible user, that user has not
Dugg as much.

The Recommendations you get from any particular user will come from topics (such as
Technology or World News) where you have a shared Digging history. We figure that
two users may have similar interests in a subject like ‘playable web games’, but one
person might be into politics while the other follows celebrity gossip. So we actually
compute correlations, Diggers Like You, and compute Recommendations in several
collections of topics independently.

Promotable Stories
Since the Recommendation Engine works only with Upcoming stories, all the stories you
get from the Recommendation Engine are “promotable”, meaning that they are recent
enough to be eligible for the Digg homepage but haven’t appeared there yet. This
means that whenever you Digg one of your Recommendations, you are helping select
stories for the front page of Digg!

Just like stories on the homepage, we want your Recommendations to be diverse: a
balanced number of stories, not all on the same topic, and not all Dugg by the same

To make sure that your Recommendations are diverse, the Engine imposes limits that
keep things from getting too focused. It makes sure that no one Digger Like You
determines too many of your stories. It attempts to make your Recommendations reflect
the spectrum of topics that you’ve Dugg in the past, and it adjusts the compatibility cutoff
for Diggers Like You so you don’t get too many or too few stories.

The Engine also limits the influence of any single one of your Diggs. For instance, if you
are Digg number 1,000 on a popular story, you will have 999 similar users from that one
Digg alone, and those users are not necessarily more compatible with you than the two
or three who may have Dugg a less popular story you also liked. The Engine limits the
total pool of users you can get from a single Digg to balance things out.

We hope you enjoy using the Recommendation Engine and look forward to helping you
uncover even more great stories on Digg!
Digg on!
Anton Kast – Lead Scientist Digg

More TechCrunch


Zen Educate, an online marketplace that connects schools with teachers, has raised $37 million in a Series B round of funding. The raise comes amid a growing teacher shortage crisis…

Zen Educate raises $37M and acquires Aquinas Education as it tries to address the teacher shortage

“When I heard the released demo, I was shocked, angered and in disbelief that Mr. Altman would pursue a voice that sounded so eerily similar to mine.”

Scarlett Johansson says that OpenAI approached her to use her voice

A new self-driving truck — manufactured by Volvo and loaded with autonomous vehicle tech developed by Aurora Innovation — could be on public highways as early as this summer.  The…

Aurora and Volvo unveil self-driving truck designed for a driverless future

The European venture capital firm raised its fourth fund as fund as climate tech “comes of age.”

ETF Partners raises €284M for climate startups that will be effective quickly — not 20 years down the road

Copilot, Microsoft’s brand of generative AI, will soon be far more deeply integrated into the Windows 11 experience.

Microsoft wants to make Windows an AI operating system, launches Copilot+ PCs

Hello and welcome back to TechCrunch Space. For those who haven’t heard, the first crewed launch of Boeing’s Starliner capsule has been pushed back yet again to no earlier than…

TechCrunch Space: Star(side)liner

When I attended Automate in Chicago a few weeks back, multiple people thanked me for TechCrunch’s semi-regular robotics job report. It’s always edifying to get that feedback in person. While…

These 81 robotics companies are hiring

The top vehicle safety regulator in the U.S. has launched a formal probe into an April crash involving the all-electric VinFast VF8 SUV that claimed the lives of a family…

VinFast crash that killed family of four now under federal investigation

When putting a video portal in a public park in the middle of New York City, some inappropriate behavior will likely occur. The Portal, the vision of Lithuanian artist and…

NYC-Dublin real-time video portal reopens with some fixes to prevent inappropriate behavior

Longtime New York-based seed investor, Contour Venture Partners, is making progress on its latest flagship fund after lowering its target. The firm closed on $42 million, raised from 64 backers,…

Contour Venture Partners, an early investor in Datadog and Movable Ink, lowers the target for its fifth fund

Meta’s Oversight Board has now extended its scope to include the company’s newest platform, Instagram Threads, and has begun hearing cases from Threads.

Meta’s Oversight Board takes its first Threads case

The company says it’s refocusing and prioritizing fewer initiatives that will have the biggest impact on customers and add value to the business.

SeekOut, a recruiting startup last valued at $1.2 billion, lays off 30% of its workforce

The U.K.’s self-proclaimed “world-leading” regulations for self-driving cars are now official, after the Automated Vehicles (AV) Act received royal assent — the final rubber stamp any legislation must go through…

UK’s autonomous vehicle legislation becomes law, paving the way for first driverless cars by 2026

ChatGPT, OpenAI’s text-generating AI chatbot, has taken the world by storm. What started as a tool to hyper-charge productivity through writing essays and code with short text prompts has evolved…

ChatGPT: Everything you need to know about the AI-powered chatbot

SoLo Funds CEO Travis Holoway: “Regulators seem driven by press releases when they should be motivated by true consumer protection and empowering equitable solutions.”

Fintech lender SoLo Funds is being sued again by the government over its lending practices

Hard tech startups generate a lot of buzz, but there’s a growing cohort of companies building digital tools squarely focused on making hard tech development faster, more efficient and —…

Rollup wants to be the hardware engineer’s workhorse

TechCrunch Disrupt 2024 is not just about groundbreaking innovations, insightful panels, and visionary speakers — it’s also about listening to YOU, the audience, and what you feel is top of…

Disrupt Audience Choice vote closes Friday

Google says the new SDK would help Google expand on its core mission of connecting the right audience to the right content at the right time.

Google is launching a new Android feature to drive users back into their installed apps

Jolla has taken the official wraps off the first version of its personal server-based AI assistant in the making. The reborn startup is building a privacy-focused AI device — aka…

Jolla debuts privacy-focused AI hardware

The ChatGPT mobile app’s net revenue first jumped 22% on the day of the GPT-4o launch and continued to grow in the following days.

ChatGPT’s mobile app revenue saw its biggest spike yet following GPT-4o launch

Dating app maker Bumble has acquired Geneva, an online platform built around forming real-world groups and clubs. The company said that the deal is designed to help it expand its…

Bumble buys community building app Geneva to expand further into friendships

CyberArk — one of the army of larger security companies founded out of Israel — is acquiring Venafi, a specialist in machine identity, for $1.54 billion. 

CyberArk snaps up Venafi for $1.54B to ramp up in machine-to-machine security

Founder-market fit is one of the most crucial factors in a startup’s success, and operators (someone involved in the day-to-day operations of a startup) turned founders have an almost unfair advantage…

OpenseedVC, which backs operators in Africa and Europe starting their companies, reaches first close of $10M fund

A Singapore High Court has effectively approved Pine Labs’ request to shift its operations to India.

Pine Labs gets Singapore court approval to shift base to India

The AI Safety Institute, a U.K. body that aims to assess and address risks in AI platforms, has said it will open a second location in San Francisco. 

UK opens office in San Francisco to tackle AI risk

Companies are always looking for an edge, and searching for ways to encourage their employees to innovate. One way to do that is by running an internal hackathon around a…

Why companies are turning to internal hackathons

Featured Article

I’m rooting for Melinda French Gates to fix tech’s broken ‘brilliant jerk’ culture

Women in tech still face a shocking level of mistreatment at work. Melinda French Gates is one of the few working to change that.

2 days ago
I’m rooting for Melinda French Gates to fix tech’s  broken ‘brilliant jerk’ culture

Blue Origin has successfully completed its NS-25 mission, resuming crewed flights for the first time in nearly two years. The mission brought six tourist crew members to the edge of…

Blue Origin successfully launches its first crewed mission since 2022

Creative Artists Agency (CAA), one of the top entertainment and sports talent agencies, is hoping to be at the forefront of AI protection services for celebrities in Hollywood. With many…

Hollywood agency CAA aims to help stars manage their own AI likenesses

Expedia says Rathi Murthy and Sreenivas Rachamadugu, respectively its CTO and senior vice president of core services product & engineering, are no longer employed at the travel booking company. In…

Expedia says two execs dismissed after ‘violation of company policy’