We’re building a social+ world, but how will we moderate it?

@doshkim / 8:50 am PST • February 4, 2022

Image Credits: Artur Debat / Getty Images

John S. Kim Contributor

John S. Kim is the CEO of Sendbird, the leading API provider to embed chat, voice and video experiences inside of mobile apps.

Learning from Facebook

As the OG social pioneer, Facebook redefined how society functions. In doing so, it endured some very painful lessons. Notably, it must bear the burden of monitoring individual, group and organization posts from 1.93 billion daily active users– all while trying to cultivate a sense of community without censorship and driving platform adoption, engagement and profits. While social+ companies are not likely to see this kind of volume, at least in the near-term, they will still have to contend with the same issues – only they no longer have the excuse of not being able to foresee that such things could happen.

If Facebook and its army of developers, moderators and AI technology struggle, what kind of chance do you have if you don’t make moderation and community guidelines a priority from the start?

Let’s look at a few areas where Facebook stumbled on moderation:

Failing to account for bad user behavior amid rapid growth: In Facebook’s early days, platform moderation wasn’t deemed necessary in what was considered a free, user-driven space. The company was merely a conduit for connection. Facebook failed to recognize the potential for user harm until it was too late to manage effectively. Even with the most advanced software and a workforce in which 15,000 employees are dedicated solely to reviewing content across 70 languages, content moderation remains an enormous problem that has cost the company users, ad dollars and vast amounts of reputational capital.

Underestimating the language barrier: While we live in an increasingly global society, connected through online services and networks, documents released to Congress showed that 87% of Facebook’s global budget allocated for identifying misinformation was reserved for the United States. Just 13% goes to moderation practices for the rest of the world, even though North Americans represent only 10% of its daily users. Facebook tried to apply AI-based software for content moderation into markets where language is incredibly nuanced in an attempt to address the issue, which has not gone well. In Facebook’s largest market (India, with 350 million users) misinformation and calls for violence have proliferated because of a language deficit. It’s even worse with the varied dialects of North Africa and the Middle East. As a result, both human and automated content reviews have mistakenly allowed hate speech to run rampant while benign posts are removed for seemingly promoting terrorist activities.

Getting political: The most clear-cut language has become weaponized in the U.S. Deep fakes and disinformation campaigns have become normalized, yet posts that Facebook rightfully removes or flags according to its service terms draw the ire of users who feel that their rights of expression are being violated and their voices suppressed. This has caused significant public backlash, along with a smattering of new legal proceedings. As recently as December 1, a federal judge blocked a Texas law from going into effect that would allow state residents to sue Facebook for damages if their content was removed based on political beliefs. A similar law in Florida, which attempted to hold Facebook liable for censoring political candidates, news sites and users, was also struck down. These attempts, however, show just how incensed people have become about content moderation practices they don’t like or that they perceive as changing over time to work against them.

Determining what to do with banned content: There is also the issue of what happens to that content once it is removed and whether a company has an ethical responsibility to turn over objectionable content or alert authorities regarding potential illegal activity. For example, prosecutors are currently demanding that Facebook hand over data that will help them identify members of a group, the New Mexico Civil Guard, who were involved in a violent incident in which a protester was shot. Facebook claims it can’t help because it deleted records of the group, which had been banned. Tensions continue to flare between law enforcement and social companies in terms of who owns what, reasonable expectations of privacy, and whether companies can release content

All of these issues should be considered carefully by companies planning to incorporate a social component into their app or service.

The next generation of social apps

Social engagement is key to sales, adoption and much more, but we must not forget that humans are flawed. Trolling, spam, pornography, phishing, and money scams are as much a part of the internet as browsers and shopping carts. They can wipe out and destroy a community.

Consider: If Facebook and its army of developers, moderators and AI technology struggle, what kind of chance do you have if you don’t make moderation and community guidelines a priority from the start?

Companies must build moderation features – or partner with companies that provide robust solutions – that can scale with the company, especially as services go global. This cannot be overstated. It is fundamental to the long-term success and viability of a platform– and to the future of the social+ movement.

For moderation tools to do their part, however, companies must create clearly defined codes of conduct for communities, ones that minimize the gray areas and that are written clearly and concisely so that all users understand the expectations.

Transparency is vital. Companies should also have a structure in place for how they handle inappropriate conduct – what are the processes for removing posts or blocking users? How long will they be locked out of accounts? Can they appeal?

And then the big test – companies must enforce these rules from the beginning with consistency. Any time there is ambiguity or a comparison between instances, the company loses.

Organizations must also define their stance on their ethical responsibility when it comes to objectionable content. Companies have to decide for themselves how they will manage user privacy and content, particularly that which could be of interest to law enforcement. This is a messy problem, and the way for social companies to keep their hands clean is to clearly articulate the company’s privacy stance rather than hide from it, trotting it out only when a problem arises.

Social models are getting baked into every app from fintech to healthcare to food delivery to make our digital lives more engaging and fun. At the same time, mistakes are unavoidable as companies carve out an entirely new way of communicating with their users and customers.

What’s important now is for social+ companies to learn from pioneers like Facebook in order to create safer, more cooperative online worlds. It just requires some forethought and commitment.

We’re building a social+ world, but how will we moderate it?

Learning from Facebook

The next generation of social apps

Latest Stories

Midi is building a digital platform for an oft-overlooked area of women’s health

Razer hit with $1.1M FTC fine over glowing ‘N95’ mask COVID claims

With $175M in new funding, Island is putting the browser at the center of enterprise security

Screenshots suggest TikTok is circumventing Apple App Store commissions

Twitch’s TikTok-like discovery feed is rolling out to all users

Amazon CodeWhisperer is now called Q Developer and is expanding its functions