Platform Accountability and Online Harm

Platform accountability and online harm examines the legal, regulatory, and governance frameworks through which digital platform companies are held responsible for the criminal and harmful content and conduct that their services facilitate, amplify, and in some cases incentivize. Within Cyber Criminology, platform accountability represents a critical policy question because the major digital platforms — Meta (Facebook, Instagram), Alphabet (Google, YouTube), X (formerly Twitter), TikTok, Snapchat, and others — function as the primary environments in which many categories of cybercrime occur, from fraud and harassment through child exploitation and terrorist recruitment. The platforms’ design decisions — about content moderation, algorithmic recommendation, anonymity, and user safety features — collectively shape the opportunity structure for online criminal activity at a scale that no law enforcement intervention can match. Section 230 of the Communications Decency Act, which provides broad immunity to platforms for user-generated content, has been the focal point of debates about whether the current legal framework adequately incentivizes platforms to prevent online harm or instead shields them from accountability for harms they could mitigate (Kosseff, 2019; Citron & Wittes, 2017). This article examines the legal framework governing platform liability, the content moderation practices through which platforms manage harmful content, the regulatory proposals that seek to increase platform accountability, and the international approaches that provide alternative models within the broader field of Criminology.

Introduction

The concentration of online social interaction on a small number of corporate platforms has created a governance challenge without historical precedent. Social media platforms function as the primary venues for public discourse, personal communication, commercial activity, and information dissemination for billions of people worldwide — yet they are owned and operated by private companies whose primary obligations are to their shareholders rather than to the public interest. The content moderation decisions these companies make — what content to permit, what to remove, what to amplify through algorithmic recommendation, and what to suppress — affect the distribution of information, the quality of public discourse, and the prevalence of criminal activity in ways that rival the impact of governmental regulation (Gillespie, 2018; Klonick, 2018).

The criminal dimensions of platform governance are substantial. Platforms host and distribute content that constitutes or facilitates criminal activity: child sexual abuse material, terrorist propaganda, fraud schemes, harassment campaigns, drug sales, weapons trafficking, and the full range of digitally mediated criminal conduct that this encyclopedia documents. The platforms’ capacity to detect and remove this content through automated systems and human review exceeds the capacity of any law enforcement agency — Meta alone reported removing billions of pieces of violating content in 2023 — yet the volume of content generated on major platforms ensures that a significant proportion of criminal content evades detection and reaches its intended audience.

The Section 230 Framework

Origins and Core Provisions

Section 230 of the Communications Decency Act of 1996 provides the foundational legal framework governing platform liability in the United States. The statute’s core provision — Section 230(c)(1) — states that “[n]o provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider.” This provision has been interpreted by courts to provide broad immunity to platforms for third-party content, shielding them from civil liability for user-generated posts, messages, and uploads — including content that is defamatory, fraudulent, harassing, or otherwise harmful (Kosseff, 2019).

Section 230(c)(2) provides a complementary protection: immunity for “good faith” content moderation decisions, protecting platforms from liability when they remove or restrict access to content they consider “objectionable” regardless of whether the content is constitutionally protected. This provision was designed to encourage voluntary content moderation by ensuring that platforms would not be penalized for exercising editorial judgment — the concern being that without this protection, platforms would avoid moderating any content to prevent the inference that they had assumed responsibility for the content they chose not to remove (Citron & Wittes, 2017; Goldman, 2020).

The historical context of Section 230’s enactment illuminates its purpose. The statute was a legislative response to Stratton Oakmont, Inc. v. Prodigy Services Co. (1995), in which a New York court held that an online service provider that exercised editorial control over user-generated content could be treated as the publisher of that content and held liable for defamatory posts. Congress’s concern was that this ruling would create a perverse incentive: platforms that attempted to moderate content would face greater liability than platforms that did nothing, discouraging the very moderation that Congress wanted to encourage. Section 230 resolved this incentive problem by immunizing platforms regardless of whether they moderated content, ensuring that platforms could filter objectionable material without assuming publisher liability for the material they missed (Kosseff, 2019).

Judicial Interpretation and Expansion

The scope of Section 230 immunity has been expanded through judicial interpretation well beyond what many scholars argue the statute’s text and purpose support. Courts have applied Section 230 to dismiss claims involving platform algorithms that recommend harmful content, platform features that facilitate criminal contact between users, and platform design decisions that allegedly contribute to user harm. The breadth of judicial interpretation has produced what Citron and Wittes (2017) characterized as “near-complete immunity” that shields platforms from accountability not only for user-generated content but for their own design choices, algorithmic amplification decisions, and failures to implement available safety measures.

The Supreme Court’s engagement with Section 230 has been limited and inconclusive. In Gonzalez v. Google LLC (2023), the Court considered whether YouTube’s algorithmic recommendation of ISIS recruitment videos fell outside Section 230 immunity — a question that, if answered affirmatively, would have established that platform amplification of harmful content is distinct from mere hosting and could give rise to liability. The Court vacated the Ninth Circuit’s decision without reaching the Section 230 question, declining to provide the guidance that lower courts and commentators had anticipated. The companion case, Twitter, Inc. v. Taamneh (2023), held that the platforms’ failure to prevent ISIS from using their services did not constitute “aiding and abetting” terrorism under the Anti-Terrorism Act, a holding that was based on statutory interpretation of the ATA rather than on Section 230 (Citron, 2023).

Content Moderation Practices

Scale, Systems, and Limitations

The content moderation systems operated by major platforms represent the most extensive private governance apparatus in history. Meta’s content moderation operations employ a combination of AI-based automated detection systems, human content reviewers (numbering in the tens of thousands, predominantly employed through outsourcing firms), and user reporting mechanisms that collectively process billions of content items annually. Meta’s Community Standards Enforcement Report documents the volume of content removed or restricted across violation categories including violence, hate speech, terrorism, child exploitation, and fraud — figures that illustrate both the scale of the moderation effort and the scale of the harmful content that platforms must manage (Roberts, 2019; Gorwa, Binns, & Katzenbach, 2020).

Automated content moderation systems employ multiple detection methods. Hash-matching databases (including the GIFCT shared database for terrorist content and NCMEC’s hash-sharing program for CSAM) identify previously flagged content with high accuracy. Machine learning classifiers trained on labeled datasets detect new harmful content based on learned patterns in text, images, and video. Natural language processing systems analyze text for hate speech, threats, fraud indicators, and other prohibited content categories. Each detection method has characteristic strengths and limitations: hash-matching is precise but cannot detect novel content, ML classifiers can detect new content but produce false positives and false negatives, and NLP systems struggle with context-dependent meaning, coded language, and cross-cultural variation (Gorwa et al., 2020; Gillespie, 2018).

The limitations of content moderation are systemic rather than merely technical. The volume of content generated on major platforms — Meta processes over 100 billion pieces of content daily — ensures that even well-resourced moderation systems cannot review every item. Automated systems must balance sensitivity (catching harmful content) against specificity (avoiding false positives that remove legitimate content), and any threshold chosen will miss some harmful content while incorrectly flagging some legitimate content. The moderation of content in languages other than English, which receives less investment in training data and human review capacity, is substantially less effective — a disparity that concentrates the benefits of moderation on English-speaking users while leaving non-English-speaking communities more exposed to harmful content (Dwoskin, 2021).

The Human Cost of Content Moderation

The human dimension of content moderation — the psychological toll on workers who review the most disturbing content that platforms host — has been documented through investigative reporting and academic research. Content moderators, who are predominantly employed through outsourcing firms at wages far below those of direct platform employees, review streams of content that include graphic violence, child sexual abuse material, self-harm, and extreme hate speech. Roberts (2019) documented the psychological impact on moderators, including symptoms of post-traumatic stress, anxiety, depression, and emotional numbing comparable to those experienced by first responders and forensic investigators. Class action litigation against platform companies by former content moderators has resulted in settlements that include provisions for psychological counseling and improved working conditions, though critics argue that the structural conditions of outsourced moderation — high volume, low pay, limited support — remain fundamentally exploitative (Newton, 2019).

The ethical implications of content moderation labor connect the platform accountability discussion to broader questions about the distribution of harm in the digital economy. The platforms’ content moderation systems protect billions of users from exposure to the most disturbing content the internet produces, but the cost of that protection is borne disproportionately by a workforce that is predominantly located in developing countries, employed under precarious conditions, and provided with inadequate psychological support. The invisibility of moderation labor — most platform users are unaware of the human workforce that maintains their online environment — compounds the ethical concern by obscuring the human cost of the digital safety that content moderation provides.

Regulatory Proposals and Legislative Developments

Section 230 Reform Proposals

Section 230 reform has attracted bipartisan interest, though the reform proposals reflect divergent and often contradictory objectives. Republican-led proposals have focused on perceived liberal bias in content moderation, seeking to restrict platforms’ ability to remove or restrict conservative political speech. Democratic-led proposals have focused on platform failure to adequately address harmful content, seeking to condition immunity on reasonable content moderation practices or to create exceptions for specific categories of harm (Kosseff, 2019; Citron & Wittes, 2017).

The EARN IT Act (introduced in multiple sessions) would condition Section 230 immunity for child exploitation claims on platform compliance with best practices determined by a federal commission — effectively creating an incentive for platforms to adopt specific safety measures, potentially including content scanning in encrypted environments, that the commission identifies. The SAFE TECH Act would narrow Section 230 immunity by excluding paid content, creating exceptions for civil rights violations, and permitting claims against platforms for enabling criminal conduct — modifications that would significantly expand platform liability exposure without eliminating the core immunity for user-generated content. The KIDS Online Safety Act (KOSA), which passed the Senate in 2024, would impose a duty of care on platforms to prevent and mitigate harm to minors, creating a new regulatory framework specific to children’s online safety.

The difficulty of Section 230 reform lies in the unintended consequences that broad changes to platform immunity could produce. Expanding platform liability for user-generated content could incentivize aggressive content removal that suppresses legitimate speech, disproportionately affecting marginalized communities whose expression is already subject to disproportionate moderation. Restricting content moderation protections could discourage platforms from removing harmful content to avoid the inference that they have assumed editorial responsibility. The design of reform proposals that effectively increase platform accountability for demonstrably harmful content while preserving the protections that enable free expression and voluntary moderation remains a legislative challenge that Congress has not yet resolved (Goldman, 2020).

Children’s Online Safety

The protection of children from online harm has emerged as the most politically viable vector for platform accountability legislation, generating bipartisan support that other reform proposals have not achieved. KOSA’s “duty of care” framework would require platforms to act in the “best interests” of minor users, preventing and mitigating specific harms including promotion of suicide, eating disorders, substance abuse, sexual exploitation, and bullying. The legislation would empower the FTC to enforce these obligations and would create a state attorney general enforcement mechanism.

The children’s safety framing avoids many of the political obstacles that block broader Section 230 reform by focusing on a population (minors) and a set of harms (exploitation, self-harm promotion, predatory contact) that generate broad public concern and bipartisan legislative support. However, implementation challenges are significant: defining “best interests” of minors in operational terms that platforms can implement, avoiding age verification requirements that compromise adult users’ privacy, and establishing enforcement standards that are specific enough to guide platform behavior without micromanaging platform design decisions all represent policy design challenges that legislative text must resolve (Harding & Solon, 2024).

International Approaches

The European Union Model

The European Union’s regulatory approach to platform accountability provides the most developed alternative to the American Section 230 framework. The Digital Services Act (DSA), which became fully applicable in February 2024, establishes a tiered system of obligations for digital service providers, with the most extensive requirements falling on “very large online platforms” (VLOPs) — those with more than 45 million monthly active users in the EU. VLOPs must conduct annual systemic risk assessments evaluating the risks their services pose for the dissemination of illegal content, the exercise of fundamental rights, public discourse, and public security. They must implement risk mitigation measures proportionate to identified risks. And they must submit to independent auditing of their compliance with these obligations (Helberger, Pierson, & Poell, 2018).

The DSA’s approach differs from the American model in fundamental respects. While Section 230 immunizes platforms from liability and relies on voluntary content moderation, the DSA imposes affirmative obligations that platforms must fulfill to maintain their legal operating authority in the EU. The DSA does not eliminate platform immunity for user-generated content — its notice-and-action provisions maintain a framework in which platforms are not liable for content they have not been notified about — but it imposes systemic obligations (risk assessment, mitigation, auditing, transparency) that create regulatory accountability for platform design and governance decisions that Section 230 leaves entirely to platform discretion.

The United Kingdom’s Online Safety Act

The United Kingdom’s Online Safety Act, which received Royal Assent in October 2023, establishes duties of care requiring platforms to prevent users from encountering illegal content and, for platforms likely to be accessed by children, to prevent children from encountering content harmful to them. The Act empowers Ofcom (the UK communications regulator) to develop codes of practice specifying how platforms should fulfill their duties, with enforcement mechanisms including substantial fines and, for the most serious violations, the ability to block access to non-compliant services within the UK.

The Online Safety Act’s inclusion of provisions addressing encrypted messaging services — potentially requiring platforms to develop technology for detecting CSAM within encrypted communications — has generated particular controversy. The tension between child safety and encryption is the same as in the American context, but the UK’s legislative approach resolves it more aggressively in favor of detection, creating potential obligations that major platform companies (including Meta, which operates WhatsApp) have warned may require the withdrawal of encrypted services from the UK market. Whether this legislative threat produces compliance, technological innovation, or service withdrawal will be an important data point for American policymakers considering similar approaches.

Conclusion

Platform accountability and online harm occupy a central position in the governance of digital society, as the decisions that platform companies make about content moderation, algorithmic design, and user safety collectively shape the environment in which billions of people interact and in which a significant volume of criminal activity occurs. The American legal framework, centered on Section 230’s broad immunity, provides platforms with freedom to develop and implement content moderation practices without the threat of liability, but also shields them from accountability for design decisions that create or amplify harm. The international regulatory trend — exemplified by the EU’s Digital Services Act and the UK’s Online Safety Act — moves toward greater affirmative obligations for platforms, creating a regulatory divergence between the United States and its democratic allies that may eventually pressure American policy toward convergence.

The development of effective platform accountability frameworks requires balancing multiple competing values: the protection of free expression against censorship, the prevention of online harm against the costs of over-moderation, the accountability of private power against the risk of government overreach, and the safety of vulnerable users against the privacy of all users. The resolution of these tensions will determine the character of the digital public sphere and the extent to which platforms function as environments where crime is facilitated and harm is amplified or as environments where safety and accountability are structurally embedded in the systems through which billions of people communicate, transact, and associate.

References

Balkin, J. M. (2018). Free speech is a triangle. Columbia Law Review, 118(7), 2011–2056.
Citron, D. K. (2023). Section 230 at the Supreme Court. Harvard Law Review Forum, 136, 292–310.
Citron, D. K., & Wittes, B. (2017). The Internet will not break: Denying bad Samaritans Section 230 immunity. Fordham Law Review, 86(2), 401–423.
Dwoskin, E. (2021, March 22). Facebook’s content moderation advantage: Most of its workers are in the U.S. Washington Post.
Gillespie, T. (2018). Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press.
Goldman, E. (2020). An overview of the United States’ Section 230 Internet immunity. In G. Frosio (Ed.), The Oxford handbook of intermediary liability online (pp. 155–172). Oxford University Press.
Gonzalez v. Google LLC, 598 U.S. 617 (2023).
Gorwa, R., Binns, R., & Katzenbach, C. (2020). Algorithmic content moderation: Technical and political challenges. Big Data & Society, 7(1). https://doi.org/10.1177/2053951719897945
Harding, L., & Solon, O. (2024). Children’s online safety legislation: International developments and U.S. prospects. Brookings Institution TechStream.
Helberger, N., Pierson, J., & Poell, T. (2018). Governing online platforms: From contested to cooperative responsibility. The Information Society, 34(1), 1–14. https://doi.org/10.1080/01972243.2017.1391913
Klonick, K. (2018). The new governors: The people, rules, and processes governing online speech. Harvard Law Review, 131(6), 1598–1670.
Kosseff, J. (2019). The twenty-six words that created the Internet. Cornell University Press.
Meta. (2024). Community standards enforcement report. https://transparency.fb.com/data/community-standards-enforcement/
Newton, C. (2019, February 25). The trauma floor: The secret lives of Facebook moderators in America. The Verge.
Roberts, S. T. (2019). Behind the screen: Content moderation in the shadows of social media. Yale University Press.
Section 230, Communications Decency Act, 47 U.S.C. § 230.
Stratton Oakmont, Inc. v. Prodigy Services Co., 1995 N.Y. Misc. LEXIS 229 (N.Y. Sup. Ct. 1995).
Twitter, Inc. v. Taamneh, 598 U.S. 471 (2023).
UK Parliament. (2023). Online Safety Act 2023. https://www.legislation.gov.uk/ukpga/2023/50
European Parliament and Council. (2022). Regulation (EU) 2022/2065 on a Single Market for Digital Services (Digital Services Act).
Wall, D. S. (2007). Cybercrime: The transformation of crime in the information age. Polity Press.
Yar, M., & Steinmetz, K. F. (2019). Cybercrime and society (3rd ed.). SAGE Publications.

Related Articles