Hostile Social Manipulation
Hostile Social Manipulation
Hostile Social Manipulation
Manipulation
Present Realities and Emerging Trends
C O R P O R AT I O N
For more information on this publication, visit www.rand.org/t/RR2713
RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors.
Support RAND
Make a tax-deductible charitable contribution at
www.rand.org/giving/contribute
www.rand.org
Contents
Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
CHAPTER ONE
Introduction: Information and Democracy—
A Perilous Relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
The Growing Danger of Social Manipulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Study Design, Methodology, and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
The Broader Danger: A Corrupted Information Environment. . . . . . . . . . . . . . . 8
CHAPTER TWO
Understanding Social Manipulation: Definitions and Typologies. . . . . 11
Defining Key Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
The Goals of Social Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
The Range of Options: Examples of Hostile Social Manipulation . . . . . . . . 20
The Past: Understanding the Context for Social Manipulation . . . . . . . . . . . . 23
CHAPTER THREE
Hostile Social Manipulation: Russian Activities . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
History: Soviet and Russian Approaches to Social Manipulation. . . . . . . . . . . 33
Doctrine: Russia’s Conceptualization of Social Manipulation . . . . . . . . . . . . . . 49
Strategies: How Russia Conducts Social Manipulation Today . . . . . . . . . . . . 62
iii
iv Hostile Social Manipulation: Present Realities and Emerging Trends
CHAPTER FOUR
Hostile Social Manipulation: Chinese Activities . . . . . . . . . . . . . . . . . . . . . . . . 105
History: China’s Approach to Social Manipulation . . . . . . . . . . . . . . . . . . . . . . . . 107
Doctrine: China’s Goals for Foreign Policy and Information
Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
“Magic Weapons”: China’s Offensive Approach to Defense . . . . . . . . . . . . . . . 113
Strategies: Who Manages the Media and the Messages? . . . . . . . . . . . . . . . . . . . 121
Actions: China’s Information Operations Through Social Media . . . . . . . . 131
Effectiveness of China’s Efforts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Conclusions and Implications for U.S. Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
CHAPTER FIVE
Does Hostile Social Manipulation Work? Measures of Success in
Russian Activities in Europe and the United States . . . . . . . . . . . . . . . . 167
United States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
United Kingdom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
France. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Germany . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
The Baltic States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Poland . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
Summary of Outcome Evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
CHAPTER SIX
Hostile Social Manipulation: The Experience to Date—
Conclusions and Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Figures
v
vi Hostile Social Manipulation: Present Realities and Emerging Trends
vii
Summary
ix
x Hostile Social Manipulation: Present Realities and Emerging Trends
The authors would like to thank Ryan Bauer and Sarah Heintz for their
assistance with research on some of the future technologies referenced
in the scenarios in the report. We would like to thank Leah Hershey
for her expertise in preparing the document for publication, and Rosa
Maria Torres for her dedicated and timely assistance with the proj-
ect. We are deeply indebted to our RAND colleague John Godges for
his diligent work to enhance the tone and flow of the document. We
extend our sincere thanks to the sponsor, the Office of Net Assessment
in the Office of the Secretary of Defense, for support of the project.
xvii
Abbreviations
AI artificial intelligence
AIDS acquired immune deficiency syndrome
AP Associated Press
API Application Programming Interface
CAATSA Countering America’s Adversaries Through
Sanctions Act
CAC Cyber Administration of China
CCP Chinese Communist Party
CCTV China Central Television
CDU Christian Democrats
CGTN China Global Television Network
CIA Central Intelligence Agency
CNKI China National Knowledge Infrastructure
CPSU Communist Party of the Soviet Union
DNI Director of National Intelligence
EC European Commission
EU European Union
EW electronic warfare
FARA Foreign Agents Registration Act
FN National Front
FNC Framework Nation Concept
xix
xx Hostile Social Manipulation: Present Realities and Emerging Trends
In the 1997 James Bond film Tomorrow Never Dies, the villain is Elliot
Carver, head of a media conglomerate who has come to believe that
information is a more powerful weapon than military force. He black-
mails senior British leaders and ultimately tries to spark a war between
China and Britain to bring his ally to power in Beijing. His plot is a
combination of real-world actions—luring a British frigate into Chi-
nese waters and then sinking it—with a global media blitz to “demon-
strate” to the world that the action represents Chinese aggression and
stoke the flames of British nationalism.
At one point in the film, Carver stands underneath massive televi-
sion screens in the headquarters of his media empire, personally draft-
ing the headlines—many of them referring to fabricated events—that
will push the world toward war. “We’re both men of action,” he tells
Bond, “but your era . . . is passing. Words are the new weapons, satel-
lites the new artillery. . . . Caesar had his legions, Napoleon had his
armies. I have my divisions—TV, news, magazines.” Control of the
global narrative, Carver suggests, will give him more power than any
of those military leaders could ever wield.
The Bond plot offers one vision of a dystopian future: The idea
that mass media in the electronic age has become so powerful, prev-
alent, and capable of manipulation that—with some “real” events
thrown in for leavening—they could make whole populations believe
the opposite of the truth. Elliot Carver represents a turbocharged ver-
sion of the historical media figures who trafficked in rumor, innuendo,
1
2 Hostile Social Manipulation: Present Realities and Emerging Trends
1 Hannes Grassegger and Mikael Krogerus, “Weaken from Within,” New Republic,
November 2, 2017.
Introduction: Information and Democracy—A Perilous Relationship 3
2 Alexander Smith, “British PM Theresa May Says Russia Seeks to ‘Weaponize’ Informa-
tion,” NBC News, November 14, 2017.
3 Joe Biden and Michael Carpenter, “How to Stand Up to the Kremlin,” Foreign Affairs,
December 2017.
4 Scott Shane, “The Fake Americans Russia Created to Influence the Election,” New York
Times, September 7, 2017a.
4 Hostile Social Manipulation: Present Realities and Emerging Trends
5 Itxu Diaz, “Venezuela and Russia Teamed Up to Push Pro-Catalan Fake News,” Daily
Beast, November 28, 2017.
6 DFR Lab, “#FakeNews: Made in China,” Medium, November 25, 2017.
7 “German Intelligence Unmasks Alleged Covert Chinese Social Media Profiles,” Reuters,
December 10, 2017.
8 Josh Rogin, “China’s Foreign Influence Operations Are Causing Alarm in Washington,”
Washington Post, December 10, 2017. See also Ishaan Tahroor, “China’s ‘Long Arm’ of Influ-
ence Stretches Ever Further,” Washington Post, December 14, 2017.
Introduction: Information and Democracy—A Perilous Relationship 5
11 For a similar definition, see John Arquilla and David Ronfeldt, The Emergence of Noopoli-
tik: Toward An American Information Strategy, Santa Monica, Calif.: RAND Corporation,
MR-1033-OSD, 1999, pp. 11–12, 16–17. For a U.S. Defense Department definition of the
information environment, see U.S. Joint Chiefs of Staff, Department of Defense Dictionary
of Military and Associated Terms, Joint Publication 1-02, Washington, D.C., November
2010.
8 Hostile Social Manipulation: Present Realities and Emerging Trends
12 See, for example, Rand Waltzman, “The Weaponization of the Information Environ-
ment,” Defense Technology Program brief, American Foreign Policy Council, September
2015, pp. 4–6.
13 On the potential for this outcome, see Christopher Walker, “A New Era of Competition,”
Konrad Adenauer Stiftung International Reports, No. 2, 2017.
10 Hostile Social Manipulation: Present Realities and Emerging Trends
The first step on this analytical path is to be clear about the subject of
inquiry. This report is concerned with what we are calling hostile social
manipulation. It is at once a very specific notion and one that overlaps
with many other concepts. As a first step, therefore, we surveyed the
terms and concepts that are part of the current debate.
11
Table 2.1
12
Categorizing Informational Behavior
Misinformation Often none; • Inadvertent sharing of false or misleading Inaccurate news stories
disseminators believe information; “information that is initially that are corrected;
it to be true presented as true but later found to be spreading invalid facts
false”a about vaccines
“Fake news” Boost traffic, drive • Misleading or completely invented infor- Invented stories, hoaxes,
clicks and shares, mation generally spread online satire, parody, intentionally
garner revenue or • Often generated by web publish- deceptive angle or slant to
publicity ers to generate views, profit; can be news
state-directed
• Sometimes intentionally spread, can be
inadvertent
Marketing, public Affect beliefs, shape • Spin on factual information to support Consumer marketing
affairs, public narratives with case for product, issue, country campaigns; U.S.
diplomacy, “strategic “truth” (often a form • Provides context and slant but not inten- government public
communications” of propaganda) tionally deceptive diplomacy efforts
• Can be employed by corporations, coun-
tries, nongovernmental organizations
Propaganda Affect beliefs, shape • “The deliberate and systematic attempt Public service campaigns
narratives with any to shape perceptions, manipulate cogni- (antismoking); wartime
techniques available tions, and direct behavior to achieve a patriotic drives; autocratic
response that furthers the desired intent regimes’ efforts to control
of the propagandist”b thoughts of citizens
• Can use falsehoods but typically pack-
ages factual information in narratives to
achieve desired effect
Table 2.1—Continued
Computational Affect beliefs, • “The use of algorithms, automation, and Various types of social media
propaganda attitudes human curation to purposefully distribute bots, autonomous agents
misleading information over social media combined with cognitive
networks”c psychology insights
Political warfare Damage target • Efforts to coerce or weaken target state Propaganda to shape
society’s will through intervention in domestic affairs beliefs; actions to reinforce
• Discourage enemy troops from fighting narrative
“Active measures” Weaken Western • Category of Soviet techniques of disinfor- Soviet campaign to
societies; shape mation used during the Cold War inflame racial tensions by
narratives • Political warfare to affect global trend of suggesting HIV was a U.S.
correlation of forces government conspiracy
• Broader information channels; included against the African
support for proxy groups, sabotage, American community
13
violence
Table 2.1—Continued
14
Hostile Social Manipulation: Present Realities and Emerging Trends
Category Intent Definition Examples
NOTE: PSYOPS = psychological operations; HIV = human immunodeficiency virus. The definitions and concepts in the table are
drawn from a number of sources, most notably National Endowment for Democracy, “Issue Brief: Distinguishing Disinformation
from Propaganda, Misinformation, and ‘Fake News,’” October 17, 2017; Jennifer Kavanaugh and Michael Rich, Truth Decay: An
Initial Exploration of the Diminishing Role of Facts and Analysis in American Public Life, Santa Monica, Calif.: RAND Corporation,
RR-2314-RC, 2018; Claire Wardle, “Fake News: It’s Complicated,” First Draft, February 16, 2017; and D. J. Flynn, Brendan Nyhan, and
Jason Reifler, “The Nature and Origins of Misperceptions: Understanding False and Unsupported Beliefs About Politics,” Advances in
Political Psychology, Vol. 38, Supplement 1, 2017, pp. 128–129.
a John Cook, Ullrich Ecker, and Stephan Lewandowsky, “Misinformation and How to Correct It,” in Robert Scott and Stephan
Kosslyn, eds., Emerging Trends in the Social and Behavioral Sciences, New York: John Wiley and Sons, 2015.
b Garth Jowett and Victoria O’Donnell, Propaganda and Persuasion, Beverly Hills, Calif.: Sage Publications, 1986, p. 16. Edward
Bernays (in Propaganda, New York: IG Publishing, 2005, originally published 1928) offers on p. 52 a related definition: “A consistent,
enduring effort to create or shape events to influence the relations of the public to an enterprise, idea or group.”
c Samuel C. Woolley and Philip N. Howard, “Computational Propaganda Worldwide: Executive Summary,” in Samuel Woolley and
Philip N. Howard, eds., Working Paper 2017.11, Oxford, United Kingdom (UK): Project on Computational Propaganda, 2017, p. 3.
d U.S. Joint Chiefs of Staff, Information Operations, Joint Publication 3-13, Washington, D.C., November 27, 2012, incorporating
Change 1, November 20, 2014.
e Nathaniel Persily, “Can Democracy Survive the Internet?” Journal of Democracy, Vol. 28, No. 2, April 2017, p. 68.
Understanding Social Manipulation: Definitions and Typologies 15
1 Herbert Lin and Jackie Kerr, “On Cyber-Enabled Information/Influence Warfare and
Manipulation,” in Oxford Handbook of Cybersecurity, August 14, 2017, pp. 5–7.
Figure 2.1
16
The Boundaries of Social Manipulation
• Disinformation, propaganda
Hostile social • Social media campaigns
• Hacking to gain data for manipulation • Fabricated information, images
information operations • Targeted harassment
• Cyberbullying of targeted
opponents
• Disruption of automated trading • Suppression of civilian
• Disruption of internet of things broadcast systems
Cyberwar Electronic
warfare • EW jamming, disruption of
military systems
• Malware attacks on power grids • Attacks on civilian
• Distributed denial of service information systems
attacks on governments
Cyberattacks on military systems
Competitors see these realms as integrated and holistic, do not distinguish among components
Table 2.2
Goals and Objectives of Social Manipulation Campaigns
Objective Examples
lator. That is a possible goal but often not the predominant one. The
tools of social manipulation may be best suited to sow chaos and to
weaken a target state by influencing social, political, and economic
outcomes. If Russia’s social manipulation efforts were designed to ease
Western confrontation, for example, they have manifestly failed. There
is much contrary evidence, in fact, that these campaigns have been
counterproductive at a geopolitical level, prompting a much more vig-
orous Western response than would have been the case without them.
But the campaigns may have a more long-term purpose—to under-
mine social institutions, exacerbate social and political divisions, and
intimidate political or ideological opponents in target countries in ways
that weaken them as international actors over time.
3 A fine recent summary of such efforts is Samantha Bradshaw and Philip N. Howard,
“Troops, Trolls and Troublemakers: A Global Inventory of Organized Social Media Manip-
ulation,” Working Paper No. 2017.12, Oxford, UK: Project on Computational Propaganda,
2017.
Understanding Social Manipulation: Definitions and Typologies 21
4 Hunt Allcott and Matthew Gentzkow, “Social Media and Fake News in the 2016 Elec-
tion,” Journal of Economic Perspectives, Vol. 31, No. 2, Spring 2017; Levi Boxell, Matthew
Gentzkow, and Jesse M. Shapiro, “Is Media Driving Americans Apart?” New York Times,
December 6, 2017.
5 Zoey Chong, “Up to 48 Million Twitter Accounts Are Bots, Study Says,” CNET,
March 14, 2017.
6 Philip N. Howard and Bence Kollanyi, “Bots, #Strongerin, and #Brexit: Computational
Propaganda During the UK-EU Referendum,” Research Note 2016.1, Oxford, UK: Project
on Computational Propaganda, 2016, p. 4.
22 Hostile Social Manipulation: Present Realities and Emerging Trends
Table 2.3
Techniques and Mechanisms for Social Manipulation
Table 2.3—Continued
9 Jacques Ellul, Propaganda, New York: Vintage Books, 1965, p. 33; see also pp. 25–35.
10 U.S. House of Representatives Permanent Select Committee on Intelligence, Soviet
Active Measures, Washington, D.C.: Government Printing Office, 1982, p. 31.
11 Fred Barbash, “U.S. Ties ‘Klan’ Olympic Hate Mail to KGB,” Washington Post,
August 7, 1984.
26 Hostile Social Manipulation: Present Realities and Emerging Trends
15 “Are HIV/AIDS Conspiracy Beliefs a Barrier to HIV Prevention Among African Ameri-
cans?” Journal of Acquired Immune Deficiency Syndromes, Vol. 38, No. 2, February 1, 2005,
pp. 213–218.
16 Kavanaugh and Rich, 2018, pp. 41–78.
CHAPTER THREE
Much ink has been spilled over the Kremlin’s attempts to influence
foreign audiences in recent years. Although there is general agreement
that these efforts exist, there is less agreement on exactly what Moscow
seeks to accomplish with these efforts; how Russia exerts influence in
the informational realm; and who orders, designs, and executes the
efforts. Additionally, few recent studies address the effectiveness of
Russian efforts. This chapter aims to assess what is understood and
identify areas where additional inquiry is necessary.
One of the robust debates on this issue is a terminological one.
Russian efforts to use information to influence target audiences have
been called by many a name: hybrid warfare, psychological warfare, gray
zone activities, information operations, disinformation, active measures,
and so on. We use the term social manipulation. As defined earlier,
hostile social manipulation is the purposeful, systematic generation and
dissemination of information to produce harmful social, political, and
economic outcomes in a target country by affecting beliefs, attitudes,
and behavior.
It is important to acknowledge that social manipulation is not
a term Russia uses. Nor is there any Russian equivalent of the term
in publicly available Russian doctrine or relevant literature. Instead,
Russia uses the term information warfare (informatsionnaya voyna).1
1 As Keir Giles notes, Russian sources use the terms information warfare (informatsionnaya
voyna) and “information confrontation” (informatsionnoye protivoborstvo). The distinction
between these terms is debated in the literature (Keir Giles, Handbook of Russian Information
Warfare, Research Division, NATO Defense College, November 2016, p. 6).
29
30 Hostile Social Manipulation: Present Realities and Emerging Trends
Methodology
2 For a discussion of interview types, see Margaret C. Harrell and Melissa A. Bradley,
Data Collection Methods: Semi-Structured Interviews and Focus Groups, Santa Monica, Calif.:
RAND Corporation, TR-718-USG, 2009, pp. 27, 29–46.
32 Hostile Social Manipulation: Present Realities and Emerging Trends
3 In his classic work Art of War, Sun Tzu wrote “all warfare is based on deception. [. . .]
Thus one who is skillful at keeping the enemy on the move maintains deceitful appearances,
according to which the enemy will act” (Sun Tzu, Art of War, military treatise, circa fifth
century BCE).
4 Dennis Kux, “Soviet Active Measures and Disinformation: Overview and Assessment,”
Parameters, Vol. 15, No. 4, 1985, p. 20; also see Wojciech Karpinski, “Agents and Exiles,”
Survey, Vol. 27, Autumn–Winter, 1983.
5 CIA translation of S. Simoni, “Soviet Anti-Semitism and the Prague Trial,” Yedioth
Hayom, 1952; Randall L. Bytwerk, “Believing in ‘Inner Truth’: The Protocols of the Elders of
Zion in Nazi Propaganda, 1933-1945,” Holocaust and Genocide Studies, Vol. 29, No. 2, 2015.
34 Hostile Social Manipulation: Present Realities and Emerging Trends
6 Peter Kenez, The Birth of the Propaganda State: Soviet Methods of Mass Mobilization,
1917–1929, Cambridge, UK: Cambridge University Press, p. 49.
7 Kux, 1985, p. 19.
8 Max Holland, “The Propagation and Power of Communist Security Services Dezinfor-
matsiya,” International Journal of Intelligence and Counterintelligence, Vol. 19, No. 1, 2006,
p. 5.
9 The terms disinformation and active measures, which have been used to describe recent
Russian efforts to influence foreign audiences using information, descend from Soviet intel-
ligence terminology. The former is a transliteration of the Russian term dezinformatsiya, and
the latter a translation of the Russian phrase aktivnyye meropriyatiya. Though exact defini-
tions of these terms differ, disinformation is generally agreed to mean the deliberate use of
partially or wholly false information to mislead (Kux, 1985, p. 19).
Hostile Social Manipulation: Russian Activities 35
15 “CIA Study: Soviet Covert Action and Propaganda,” in U.S. House of Representatives,
Permanent Select Committee on Intelligence, Subcommittee on Oversight, 1980, p. 61;
Kux, 1985, p. 21.
16 Robert W. Pringle, “Andropov’s Counterintelligence State,” International Journal of Intel-
ligence and Counterintelligence, Vol. 13, No. 2, 2000, p. 199.
17 Oleg Kalugin, Vechernyaya Moskva, November 3, 1990, as referenced in Pringle, 2000,
p. 199; Giles, 2016, p. 36; in-person interview with former CIA analyst 024, November 20,
2017.
Hostile Social Manipulation: Russian Activities 37
Putin and his associates have been at Russia’s helm for nearly
two decades. This continuity in leadership indicates a continuity in
thought.19 Even in the interim years following the Soviet collapse, jour-
nalists accepted bribes in exchange for publishing false information
about government officials or other public figures, and the security
services collected compromising information, or kompromat, which
was revealed to the public at opportune times.20 Thus, the influence of
Soviet social manipulation practices has likely endured despite internal
institutional and external geopolitical changes.
A former CIA analyst responsible for covert action directed
against the Soviet Union in the 1970s and 1980s also noted the simi-
larities between Soviet-era social manipulation that he experienced
and Russian efforts today, in terms of their conception, principles, and
desired outcomes. The analyst said the only differences they observed
are the mechanisms used to disseminate Russian messages—mecha-
nisms brought about by advances in technology.21 Though Soviet intel-
ligence practices are likely an important influence on current Russian
social manipulation, these appear to be but one driver shaping the Rus-
18 Andrei Soldatov and Irina Borogan, The Red Web, New York: Public Affairs, 2015, p. 90.
19 Giles, 2016, p. 36.
20 Soldatov and Borogan, 2015, pp. 61–62.
21 In-person interview with former CIA analyst 024, November 20, 2017.
38 Hostile Social Manipulation: Present Realities and Emerging Trends
41 CIA, “TASS: Its Role, Structure, and Operations,” June 1959, pp. 3–4.
42 CIA, “The Soviet Foreign Propaganda Apparatus: A Research Paper,” 1986a, pp. iii.
43 CIA, Political Information: The Role of TASS in Soviet Propaganda Activities, Shanghai,
Information Report, December 13, 1948; CIA, 1959, p. 6.
44 CIA, 1959, pp. 17–18.
45 Kux, 1985, p. 23.
46 Congressional Record, 1987, p. E4717; Holland, 2006, p. 4.
47 Kux, 1985, p. 25.
Hostile Social Manipulation: Russian Activities 43
positive light.48 The hope was that this would affect the policies of the
newspaper editors who thought they should “keep in tune with their
readers.”49
Soviet-sponsored radio programming included overt and clandes-
tine transmissions. Moscow ran two overt stations: Radio Moscow’s
International Service, which was openly affiliated with the govern-
ment, and Radio Peace and Progress, which was considered an unof-
ficial station. Radio Moscow’s programming was broadcast in the local
language of the target audience. A CIA memorandum describes Radio
Moscow’s foreign-language broadcasts as follows:
53 CIA, “Moscow Drops Clandestine Radio, Sustains Criticism of Tehran,” FBIS Trends
newsletter, December 10, 1986b.
54 CIA, 1980.
55 CIA, 1986a, pp. 9–10.
56 U.S. House of Representatives, Permanent Select Committee on Intelligence, Subcom-
mittee on Oversight, 1980.
Hostile Social Manipulation: Russian Activities 45
amount of press a story received. Others have argued that while Soviet
social manipulation efforts did not change the tide of opinion against
the United States, they were a distraction.64 Still others have cited the
mediocre quality of Soviet propaganda, issues with its timeliness due to
internal bureaucratic and parochial hang-ups, and its lack of credibility
as potential detractors from Soviet success.65
But the difficulty of measuring effectiveness goes both ways. A
former CIA analyst responsible for U.S. covert action against the Soviet
Union in the 1970s and 1980s told us that, when asked during budget
discussions to justify their program’s expenditures on such efforts, they
were unable to provide a definitive answer. Whereas certain indicators
showed that some in their target audience were reading stories planted
by U.S. social manipulation efforts, there was ultimately no way to
prove that their efforts were achieving the desired impact.66
Ultimately, the tide of opinion did not appear to shift largely in
favor of the Soviet Union or its interests during the Cold War. Whether
this is due to a lack of success of Soviet social manipulation efforts or
to other factors is difficult to definitively assess.
69 A former political strategist with whom we had discussions confirmed this, noting that
the consultants recruited by Yeltsin’s team were former colleagues (phone interview with
former political consultant 028, December 5, 2017).
70 Michael Kramer, “Rescuing Boris,” TIME, July 15, 1996, pp. 29–37.
71 Kramer, 1996, p. 36.
72 Wilson, 2005, p. 50. Kompromat refers to information that is collected with the intent
that it may be used to blackmail or embarrass its subjects.
73 Wilson, 2005, pp. 54–55.
48 Hostile Social Manipulation: Present Realities and Emerging Trends
strate the virtues of the internet as a tool for manipulation and achiev-
ing political ends.74 As Wilson notes, Pavlovsky was one among many
such technologists operating in the post–Cold War era. Though he
appears to have left the field, many others remain.
Another expert we spoke with offered an alternative view of the
role of the American team aiding Yeltsin in 1996. The expert noted
that this effort may have shaped Russian perceptions of U.S. inten-
tions and methods; that while the American consultants likely viewed
their efforts as the implementation of a typical campaign strategy, the
view from Moscow was likely very different; and that the Russian
intelligence services likely saw this effort as secret U.S. interference
in a Russian election to engineer an outcome that was preferable to
Washington.75
Others told us that Russian technology experts in the early post–
Cold War years may have also influenced Russian thinking on social
manipulation. Russian tech personalities, such as Igor Ashmanov,
warned of the potential dangers posed by the internet, given its rise
in the West.76 Ashmanov appears to have had some influence with the
Russian government; in 2010, the Kremlin consulted with his firm,
Ashmanov and Partners, regarding the Kremlin’s plans to build a Rus-
sian national search engine.77
When asked about the relationship between search engines and
the state in 2010, Ashmanov noted, “U.S. authorities often say that
Google is advancing the causes of democracy in China. How should
the Chinese government view this? As an intervention in their affairs.
That’s exactly what they are doing. The U.S. government would be silly
not to use it for America’s own good.”78 In response to questions about
Google’s growing popularity in Russia, Ashmanov remarked that “a
search engine is a means of influencing public opinion, and second, it’s
74 Soldatov and Borogan, 2015, pp. 92–99; Wilson, 2005, pp. 54–55.
75 Phone interview with expert 030, January 15, 2018.
76 Phone interview with former Russian journalist and editor 026, November 28, 2017.
77 Evgeny Morozov, “Is Russia Google’s Next Weak spot?” Foreign Policy, March 26, 2010.
78 Morozov, 2010.
Hostile Social Manipulation: Russian Activities 49
a source of unique information about what people think and what kind
of information they want.”79
Lessons from the Second Chechen War also appear to have rein-
forced Russian suspicions of Western attempts to interfere in Russian
domestic affairs using information, information technologies, or other
mechanisms by which information is disseminated. Andrei Soldatov,
a Russian investigative journalist, spoke of these suspicions in a recent
interview:
Many of the tactics and techniques of the Soviet era have flowed
directly into Russian practices today. These include an embrace of the
general practice of propaganda and efforts to control narratives, manu-
facturing partly true material or outright falsehoods; efforts to reach
into other societies with targeted appeals; and conscious attempts to
shape political outcomes abroad. The way Russia is able to undertake
these campaigns, however, has evolved significantly under the influ-
ence of new technologies.
79 Morozov, 2010.
80 Michael Kirk, interview of Andrei Soldatov, “The Putin Files,” Frontline, PBS, July 25,
2017b; this point was reinforced in our discussion with a Russian expert 030, January 15,
2017.
50 Hostile Social Manipulation: Present Realities and Emerging Trends
81 Ministry of Foreign Affairs of the Russian Federation, “National Security Concept of the
Russian Federation,” January 10, 2000.
Hostile Social Manipulation: Russian Activities 51
and regional stability.”84 The document declares one of the main inter-
nal military risks to be “subversive information activities against the
population, especially young citizens of the State, aimed at undermin-
ing historical, spiritual and patriotic traditions related to the defense of
the Motherland.”85
In 2016, the “Doctrine of Information Security” continued to
stress the threat from the outside:
It goes on to say,
113Andrew Monaghan, The New Politics of Russia, Manchester, UK: Manchester University
Press, 2016.
114 Phone interview with expert 030, January 15, 2018; phone interview with former military
information activities specialist 022, November 2, 2017.
115 “An Ex St. Petersburg ‘Troll’ Speaks Out: Russian Independent TV Network Interviews
Former Troll at the Internet Research Agency,” Meduza, October 15, 2017.
60 Hostile Social Manipulation: Present Realities and Emerging Trends
116 U.S. District Court for the District of Columbia, “United States of America v. Internet
Research Agency LLC, [et. al],” filed February 16, 2018, pp. 4, 17l; Office of the Director of
National Intelligence, “Assessing Russian Activities and Intentions in Recent US Elections,”
Intelligence Community assessment, January 6, 2017, p. 6. The IRA has been registered
under many names since its inception (refer to p. 7 of the 2018 indictment for a full list).
This chapter will refer to the organization as the IRA, given that this is the moniker most
commonly used in public discourse.
117 “United States of America v. Internet Research Agency LLC, [et. al],” p. 23.
118“An Ex St. Petersburg ‘Troll’ Speaks Out: Russian Independent TV Network Interviews
Former Troll at the Internet Research Agency,” 2017.
119 In-person interview with European Defense Official 020, October 26, 2017.
Hostile Social Manipulation: Russian Activities 61
mate in the West, and by doing so, to strengthen the Soviet Union in
what was perceived as a zero-sum game on a global scale.”120 However,
because this objective is not known to be explicitly stated in doctrine,
it remains, at least for now, to be an inferred hypothesis of an objective.
Some analysts studying this issue argue that another of Moscow’s
ambitions is to drive society toward a “posttruth” environment—one
in which the distinction between fact and falsehood is immaterial,
objectivity is unattainable, and reality is malleable. Others question
whether this is an end in itself for the Kremlin, or whether it is instead
a potential means to other ends.
Journalist and Russia hand Peter Pomerantsev asserts that the diz-
zying array of explanations for the downing of MH-17 propagated by
Kremlin-linked actors was not intended to “convince viewers of any one
version of events, but rather to leave [audiences] confused, paranoid,
and passive—living in a Kremlin-controlled virtual reality that can no
longer be mediated or debated by any appeal to ‘truth.’”121 If, in this
case, Russia had intended to perplex and pacify audiences, the question
remains whether this was the ultimate objective of these efforts.
While Russia may not have tried to convince its target audience of
the legitimacy of any one narrative, as Pomerantsev argues, it is plau-
sible that the Kremlin sought to avert audiences from one specific nar-
rative: its involvement in the MH-17 incident and the Ukraine crisis
more broadly. Thus, its efforts to saturate public discourse with com-
peting and often contradictory explanations for the tragedy may have
been a means to more specific foreign policy ends. It may have sought
to obscure its connection to the incident to prevent further tarnishing
to its image in the eyes of eastern Ukrainian, regional, and global audi-
ences, and to preserve its influence in the region.
This is to say that it is possible that Russia seeks a postfactual real-
ity and uses social manipulation efforts in an effort to achieve this end.
It is also possible that Russian social manipulation efforts employ this
instrument to achieve other, specific objectives.
Most likely, the identities of those who plan and execute Russia’s social
manipulation efforts are deliberately concealed and are therefore hard
to reveal using entirely open-source material. Those who are visible, or
who have become visible, such as the troll farms, are not necessarily
the most important of the Russian state-sponsored entities engaged in
social manipulation. Some have argued that troll farms serve a second-
ary function as a distraction for Western journalists and analysts.122
The farms could divert attention from other entities and activities that
are harder to uncover.123 These organizations still matter. In the view
of one Moscow journalist we interviewed, the IRA is likely part of a
much larger system, much of which may be unknown in the unclassi-
fied setting.124
While this chapter examines Russian social manipulation efforts
overseas, the domestic Russian information landscape matters, as the
two are inextricably linked.125 This is also the case with the percep-
tion of domestic and foreign threats more broadly in the eyes of the
Kremlin leadership.126 Just as Russia conceives of information war-
fare holistically, according to one interviewee, disinformation and
social manipulation strategies outside and inside Russia cannot be sep-
arated.127 The Russian government seeks to protect itself and the sur-
vival of the regime through the domestic information space, as it has
122Phone interview with expert 030, January 15, 2018; Adrian Chen, “The Agency,” New
York Times Magazine, June 2, 2015.
123 Keir Giles, “Indicators and Warnings for Detecting Information Threats,” YouTube
video of remarks at the Riga StratCom Dialogue: Perception Matters, NATO Stratcom
Center of Excellence, Riga, Latvia, August 20–21, 2015; phone interview with expert 030,
January 15, 2018.
124 Phone interview with Moscow correspondent 014, October 4, 2017.
125 Nearly every one of our interview subjects reinforced the interconnected nature of this
system.
126 Andrew Radin and Clint Reach, Russian Views of the International Order, Santa Monica,
Calif.: RAND Corporation, RR-1826-OSD, 2017, p. 9.
127 Phone interview with Moscow correspondent 014, October 4, 2017.
Hostile Social Manipulation: Russian Activities 63
felt under threat from information from abroad.128 The Russian gov-
ernment describes this priority as “securing of the national information
space against breaches.”129 The Russian defense appears to consist of a
well-established network of state and private entities engaging in a wide
range of activities.
suggested that this hybrid system of public and private, and of directed
and entrepreneurial efforts, has evolved organically over time rather
than by design.132
The exact balance between centralized direction and freedom
of action is very difficult to determine, and neither the literature nor
interviewees agreed on the matter. According to one interviewee, Rus-
sian actors conducting influence efforts and information operations are
highly entrepreneurial and decentralized, while others suggested that a
greater degree of control and direction exists.133
Putin’s role is also debated. It is common in Western literature
to assign direction and control of Russian social manipulation strate-
gies to him personally. The 2017 U.S. intelligence community report
on Russian interference in the 2016 U.S. elections explicitly identi-
fies Putin as the deciding authority in this case: “We assess with high
confidence that Russian President Vladimir Putin ordered an influ-
ence campaign in 2016 aimed at the US presidential election.”134 We
assume the U.S. intelligence community has come to this conclusion
based on concrete evidence demonstrating Putin’s role—evidence that
is unavailable to the public. In other cases of alleged Russian social
manipulation, Putin’s role is largely a mystery.
Putin may decide on certain strategic actions, but excessive per-
sonalization can occlude the wider network of government officials,
decisionmakers, and entrepreneurs potentially operating in the social
manipulation space.135 It can also underestimate the amount of com-
petition for attention, funding, and perceived success among both state
agencies and private entities.
According to interviewees, the competition among state and pri-
vate entities is intense and leads to freelancing and entrepreneurship,
which has not been centrally directed.136 These actors may float a nar-
rative, in line with broader Kremlin messaging, and then ask their gov-
ernment contacts whether to promote this narrative further.137 In an
interview, Pavlovsky underscored the self-motivated nature of some
Russian social manipulation efforts.138 One Moscow correspondent we
spoke with noted that, given the disaggregated nature of the system, it
is possible that the Russian state may not have the ability to rein in cer-
tain elements.139 Thus, it is difficult to definitively characterize Putin’s
role in the Russian social manipulation system based on the publicly
available evidence.
Several of those we spoke with, particularly those who had worked
in Russian media and interacted with individuals working for parts of
the state apparatus like RT, said that many of those who pursue social
manipulation strategies are not “true believers” in the narratives they
produce and disseminate, while others are.140
137 In-person interview with Department of State official 010, October 4, 2017.
138 Michael Kirk, interview of Gleb Pavlovsky, former adviser to Vladimir Putin, “The Putin
Files,” Frontline, PBS, July 13, 2017a.
139 Phone interview with Moscow correspondent 014.
140In-person interview with Department of State official 010, October 4, 2017; phone inter-
view with Russian journalist 027, December 4, 2017.
141 The Russian government’s targets and target audiences may not be one and the same.
66 Hostile Social Manipulation: Present Realities and Emerging Trends
and officials are Moscow’s principal targets.142 These include the Euro-
pean Union (EU), North Atlantic Treaty Organization (NATO), West-
ern European states, the United States, and the individuals who repre-
sent these bodies. Ukraine has also been the target of Russian efforts;
the authorities in Kyiv who came to power following the 2014 Ukrai-
nian Revolution have been accused of supporting fascist elements, and
Ukrainian soldiers have been accused of barbaric crimes.143 However,
even in these instances, the West is often indirectly implicated.
Of the known actors believed to be responsible for Russian social
manipulation strategies, two are overtly affiliated with the Russian
government: RT (formerly Russia Today) and Sputnik News. Given
their acknowledged relationship with the government, analysis of their
target audiences is instructive. RT is a news organization compris-
ing eight television channels, digital platforms in six languages, and a
robust social media presence. Its television programming is available in
over 100 countries in English, Arabic, Spanish, and French. Its docu-
mentary channel is available in English and Russian, and its digital
platforms are available in German, French, English, Arabic, Spanish,
and Russian.144 Sputnik News operates a news agency, websites, and a
142 Martin Kragh and Sebastian Asberg, “Russia’s Strategy for Influence Through Public
Diplomacy and Active Measures: The Swedish Case,” Journal of Strategic Studies, Vol. 40,
No. 6, 2017, pp. 782–784; Stephen Hutchings and Joanna Szostek, “Dominant Narratives
in Russian Political and Media Discourse During the Ukraine Crisis,” in Agnieska Piku-
licksa-Wilczewska and Richard Sakwa, eds., Ukraine and Russia: People, Politics, Propaganda,
and Perspectives, Bristol, UK: E-International Relations Publishing, 2015, pp. 174–178;
Maria Hellman and Charlotte Wagnsson, “How Can European States Respond to Russian
Information Warfare? An Analytical Framework,” European Security, Vol. 26, No. 2, 2017,
pp. 156–157.
143 Mark Galeotti, “’Hybrid War’ and ‘Little Green Men’: How It Works, and How It
Doesn’t,” in Agnieska Pikulicksa-Wilczewska and Richard Sakwa, eds., Ukraine and Russia:
People, Politics, Propaganda, and Perspectives, Bristol, UK: E-International Relations Pub-
lishing, 2015, pp. 153–154; Anton Shekhovtsov, “Pro-Russian Network Behind the Anti-
Ukrainian Defamation Campaign,” blog post, Anton Skekhovtsov’s Blog, February 3, 2014;
Peter Pomerantsev and Michael Weiss, The Menace of Unreality: How the Kremlin Weapon-
izes Information, Culture and Money, New York: The Interpreter, Institute of Modern Russia,
2014, pp. 10–11; Alina Polyakova, “Russia Can’t Decide If Ukrainian Jews Are Victims or
Villains,” New Republic, April 28, 2014.
144 RT, About RT, webpage, undated(a).
Hostile Social Manipulation: Russian Activities 67
the ads selected specific audience criteria for each ad—criteria such as
age, gender, language(s), interests, and behaviors.149 This is to say that
the IRA targeted audiences with specific beliefs about issues relevant
to the campaign by crafting messaging that appealed to their existing
views, and by using the platform’s tools to direct messaging at niche
populations.
For instance, one of these ads, which was allegedly sponsored by
the organization called “Secured Borders,” prompts users to join the
group with an image of a yellow sign reading “no invaders allowed”
placed against the backdrop of a high fence.150 The text, “Every man
should stand for our borders! Join!” sits atop the image. The corre-
sponding metadata show that its sponsors selected ad criteria to target
users between the ages of 18 and 85, in the United States, who had
“Conservatism,” “Confederate States of America,” “Donald Trump,”
“Republican Party (United States), or “Dixie” listed as interests on
their Facebook accounts.151
Other ads targeted audiences in specific U.S. geographic loca-
tions.152 According to the 2018 U.S. indictment of Russian players in
these strategies, IRA employees traveled to specific U.S. states to col-
lect intelligence prior to the organization’s campaign. Advice provided
by an unwitting, legitimate, Texas-based grassroots organization sug-
149 The Facebook advertisements and corresponding metadata represent only a small sample
(30 ads) of the overall corpus of ads reportedly purchased by Russia (3,300 ads). It is unclear
whether the content of the ads and metadata of the sample that were released is representative
of the full corpus. The metadata can be found here: U.S. House of Representatives Perma-
nent Select Committee on Intelligence, “HSPCI Minority Open Hearing Exhibits,” web-
page, undated. Note that the ability to target specific audiences using demographic criteria
is an option available to all who advertise through Facebook.
150 “Secured Borders Ad_Cultural _Metadata 1,” U.S. House of Representatives Permanent
Select Committee on Intelligence, 2017; Taylor Hatmaker, “Here’s How Russia Targeted Its
Fake Facebook Ads and How Those Ads Performed,” Tech Crunch, November 1, 2017.
151 “Secured Borders Ad_Cultural _Metadata 1,” 2017.
152 “BM Not My President Rally,” Twitter post, U.S. House of Representatives Permanent
Select Committee on Intelligence, 2017; Hatmaker, 2017.
Hostile Social Manipulation: Russian Activities 69
gested that the IRA focus its efforts on “purple states like Colorado,
Virginia & [sic] Florida.”153
Likewise, an analysis of 36,000 tweets promulgated by the 2,752
Russian-affiliated Twitter accounts during the 2016 U.S. presidential
election found that the accounts appear to have focused on amplify-
ing stories published by local U.S. news outlets, such as Cleveland Live
and KSN Topeka.154 Some names of the Russian-affiliated accounts—
DailyNewsDenver, DallasTopNews, TodayMiami, StLouisOnline,
Seattle_Post—suggest they were intended to impersonate local news
outlets.155 The falsified news accounts also “showed a pattern of sys-
tematically re-broadcasting local news outlets’ stories,” reinforcing the
assertion that Russian efforts during the election campaign were likely
focused on geographic target audiences.156
RT’s programming on YouTube offers another example of target-
ing. RT draws viewers with human interest stories and offers content
on local issues by region. For instance, RT Arabic’s YouTube channel
153 “United States of America v. Internet Research Agency LLC, [et. al],” p. 13.
154Jonathan Albright, “Trolls on Twitter: How Mainstream and Local News Outlets Were
Used to Drive a Polarized News Agenda,” Berkman Klein Center for Internet and Society at
Harvard University, February 15, 2018. The U.S. Congress released the full list of Twitter
accounts linked to Russia (according to Twitter) that were reportedly actively targeting the
U.S. audience during the 2016 presidential election; it can be found on the U.S. House of
Representatives Permanent Select Committee on Intelligence–Democrats website.
155“Exhibit B,” U.S. House of Representatives Permanent Select Committee on Intelligence,
2017.
156 Albright, 2018.
157Elizabeth Nelson, Robert Orttung, and Anthony Livshen, “Measuring RT’s Impact on
YouTube,” Russian Analytical Digest, December 2016, pp. 5–6.
70 Hostile Social Manipulation: Present Realities and Emerging Trends
Messages
The narratives associated with alleged Russian social manipulation
strategies vary depending on their intended audiences and objectives.
Given the multitude of suspected Russian targets discussed above,
capturing the range of messages of the alleged Russian efforts would
be lengthy. That said, there are some overarching trends in narratives
across efforts.
When targeting audiences that share cultural, linguistic, or his-
torical ties with Russia, Moscow appears to underscore these common-
alities. The legacy of the Second World War (or the Great Patriotic
War, as it is often referred to in the region) is a common trope used
in social manipulation targeting former Soviet states. This important
shared experience affected most of the population in the region, tran-
scends national boundaries, and is associated with both painful memo-
ries and pride. Some in the region feel the West has not been suffi-
ciently grateful for their sacrifices. Consequently, Russian messaging
underscores both the joint sacrifice and the West’s lack of adequate
gratitude.162
Russia also appears to use pronationalist narratives and to incor-
porate historical memories, such as those of lost territory, into its mes-
saging in Central, Eastern, and Southern Europe.163 Russian messag-
ing in these states often also focuses on the alleged moral corruption
and opulence of the Western liberal order, the failed promises made by
Western institutions, the threats posed by Western institutions, and
Sources of Narratives
Given the apparent lack of structure among the loosely connected
group of actors described above, we were curious to understand where
they get their narratives. Several of those with whom we spoke offered
the same answer: Putin and his officials’ public remarks offer general
guidance on the Kremlin’s stance toward various policies and actors.171
A Moscow journalist noted the existence of invisible red lines and the
expectation that those in the system know what these are.172 Remarks
from Pavlovsky in a recent interview appear to substantiate this point,
even though he might have been referring more specifically to the
domestic political context:
168
Sheera Frenkel and Daisuke Wakabayashi, “After Florida School Shooting, Russian ‘Bot’
Army Pounced,” New York Times, February 19, 2018.
169
Christopher Paul and Miriam Matthews, The Russian ‘Firehose of Falsehood’ Propaganda
Model, Santa Monica, Calif.: RAND Corporation, PE-198-OSD, 2016, pp. 7–9.
170 Phone interview with expert 030, January 15, 2017.
171 In-person interview with Department of State official 010, October 4, 2017; in-person
interview with European defense official 020, October 26, 2017.
172 Phone interview with Russian journalist 027, October 31, 2017.
74 Hostile Social Manipulation: Present Realities and Emerging Trends
analysts interviewed, temniki are eight- to ten-page documents sent to television stations
and selected newspapers with directions on what and how to report during that week. These
appeared to be drafted and sent by the presidential administration of Leonid Kuchma,
though they were not signed or stamped to maintain plausible deniability. These directives
were initially sent to selected television stations, but their distribution was later expanded to
include all stations and some newspapers. Editors, journalists, and media analysts reported
feeling pressure to comply, concerned that failure to do so would result in negative reper-
cussions such as job loss, salary cuts, or decreased budgets for their outlet (“Negotiating the
News: Informal State Censorship of Ukrainian Television,” Human Rights Watch, Vol. 3,
No. 2, March 2003, pp. 13–24.
179 “Temnik—The Kremlin’s Route to Media Control,” EU vs Disinfo, March 29, 2017;
Dmitry Skorobutov, “Ispoved’ Propagandista. Chast’ I. Kak Delajut Novosti na Gosudarst-
vennom TV,” The Insider, June 9, 2017. The use of temniki was brought up in two of our
discussions, both with former journalists at Russian state news agencies. Both recalled hear-
ing about such meetings from their editors. Phone interview with Russian journalist former
desk chief (interview 27, October 31, 2017), and Moscow journalist (interview 26), Novem-
ber 28, 2017. A former RT journalist noted that in her time with the organization, she did
not receive direct orders from above on what to report on, but rather some stories she put
forth were declined by the Russian news director (“Russian Propaganda Broadcast into
Canadian Homes,” CBC News, The Weekly, January 21, 2018).
180 Office of the Director of National Intelligence, 2017, p. 9.
181 Matthew Bodner, Matthew Kupfer, and Bradley Jardine, “Welcome to the Machine:
Inside the Secretive World of RT,” Moscow Times, June 1, 2017.
182 Bodner, Kupfer, and Jardine, 2017.
76 Hostile Social Manipulation: Present Realities and Emerging Trends
183 The original interview can be found here in Russian: Evgenia Kotlyar, “U Nas Byla Cel’
. . . Vyzvat Besporjadki’: Interv’ju s Jeks-Sotrudnikom ‘Fabriki Trollej’ v Sankt-Peterburge,”
Dozhd, October 14, 2017. A summary of the interview in English can be found in “An Ex
St. Petersburg ‘Troll’ Speaks Out: Russian Independent TV Network Interviews Former
Troll at the Internet Research Agency,” 2017. The “foreign desk” is allegedly the department
within the IRA responsible for the 2016 U.S. election social manipulation efforts.
184 The 2018 U.S. indictment of the IRA filed by U.S. authorities accuses the organization
of “engag[ing] in operations to interfere with elections and political processes” and confirms
Prigozhin as its financier (“United States of America v. Internet Research Agency LLC, [et.
al].” pp. 2–3). However, it is important to note that while the indictment acknowledges that
Prigozhin’s company Concord has other Russian government contracts, nowhere in the doc-
ument does it explicitly link the 2016 U.S. campaign with the Kremlin. Additionally, docu-
ments leaked by the group Anonymous International demonstrate the IRA’s financial ties to
Concord and to other individuals working at or connected to the agency. The leaked docu-
ments were originally posted with “Chast’ pervaja. Zoloto trollej” [“Part One. Troll Gold”],
Anonymous International, May 26, 2014, but have since been removed. For a description
and analysis of the documents, see Max Seddon, “Documents Show How Russia’s Troll
Army Hit America,” BuzzFeed News, June 2, 2014. Other analyses detailing the personal
connections can be found in Russian; see Aleksandra Garmazhapova, “Gde Zhivut Trolli. I
Kto ih Kormit” [“Where Trolls Live. And Who Feeds Them”], Novaya Gazeta, September 9,
2013.
185 “An Ex St. Petersburg ‘Troll’ Speaks Out: Russian Independent TV Network Interviews
Former Troll at the Internet Research Agency,” 2017.
Hostile Social Manipulation: Russian Activities 77
188 Jill Dougherty, “Everyone Lies: The Ukraine Conflict and Russia’s Media Transforma-
tion,” Discussion Paper Series, Shorenstein Center on Media, Politics, and Public Policy,
Harvard Kennedy School, July 2014, pp. 3–7.
Hostile Social Manipulation: Russian Activities 79
189 Phillip Shishkin and James Marson, “Ukraine Accuses Kremlin Agents of Coordinating
Separatist Unrest,” Wall Street Journal, April 20, 2014.
190 Dougherty, 2014, pp. 4–5.
191 For a discussion of potential Russian interference in the Bulgarian media landscape,
see Dimitar Bechev, Russia’s Influence in Bulgaria, Brussels: New Direction: The Founda-
tion for European Reform, May 12, 2015, pp. 22–23; for a discussion of potential Russian
interference in the Serbian media landscape, see “Eyes Wide Shut: Strengthening of Russian
Soft Power in Serbia: Goals, Instruments, and Effects,” Center for Euro-Atlantic Studies,
May 2016, pp. 56–64; for a discussion of potential Russian interference in the Hungarian
media landscape, see Attila Juhász et al., “‘I Am Eurasian’: The Kremlin Connections of
the Hungarian Far-Right,” Political Capital and Social Development Institute, March 2015,
pp. 32–51; phone interview with Hungarian academic 012, October 6, 2017.
80 Hostile Social Manipulation: Present Realities and Emerging Trends
192 Eli Lake, “Putin’s Latest Dirty Trick: Leaking Private Phone Calls,” Daily Beast,
March 26, 2014.
193 Lake, 2014; Soldatov and Borogan, 2015, pp. 285–288.
194 Soldatov and Borogan, 2015, pp. 285–287.
195 Soldatov and Borogan, 2015, pp. 285–288.
196 Soldatov and Borogan, 2015, p. 287.
Hostile Social Manipulation: Russian Activities 81
197“‘F**k the EU’: Snr US State Dept. Official Caught in Alleged Phone Chat on Ukraine,”
RT, February 6, 2014.
198 “Anonymous Ukraine Klitschko E-mails and Nuland/Pyatt Dialogue Prove US-Backed
Coup,” Sputnik (then Voice of Russia), February 25, 2014.
199“Victoria ‘F*ck the EU’ Nuland Leaves Her Post at the US State Department,” Sputnik,
January 27, 2017.
200 Office of the Director of National Intelligence, 2017, pp. 2–3.
201 Office of the Director of National Intelligence, 2017, p. 3.
202 Andy Greenberg, “Hackers Hit Macron with Huge Email Leak Ahead of French Elec-
tion,” Wired, May 5, 2017a.
203 Andy Greenberg, “Don’t Pin the Macron Email Hack on Russia Just Yet,” Wired, May 8,
2017b.
82 Hostile Social Manipulation: Present Realities and Emerging Trends
204 “Russia’s Top Lies About Ukraine. Part 1,” blog post, StopFake, July 10, 2014a.
205 “Russia’s Top Lies About Ukraine. Part 2,” blog post, StopFake, July 10, 2014b. The video
and corresponding story were originally posted on Zvezda’s Russian-language site, but the
footage has since been removed (“Nacgvardija Obstreljala Semenovku Fosfornymi Bom-
bami,” Zvezda, June 6, 2014).
206 Kragh and Asberg, 2017, pp. 773–816.
Hostile Social Manipulation: Russian Activities 83
207 “Fake Swedish Letter in Russian Media,” blog post, StopFake, September 15, 2015; “Fake
‘Swedish’ Letter Spread in Russian Media,” Radio Sweden, September 13, 2015.
208 Kragh and Asberg, 2017, pp. 793–795.
209 Kragh and Asberg, 2017, p. 806. For another example of a suspected Russian forgery, see
John R. Haines, “Distinguishing the True from the False: Fakes and Forgeries in Russia’s
Information War Against Ukraine,” Foreign Policy Research Institute, September 28, 2016.
210 Soldatov and Borogan, 2015, pp. 284–285.
84 Hostile Social Manipulation: Present Realities and Emerging Trends
211
Max Seddon, “Russian TV Airs Clearly Fake Image to Claim Ukraine Shot Down
MH17,” BuzzFeed News, November 15, 2014b.
212Lucy Crossley, “The ‘Aggrieved Housewife’, the ‘Soldier’s Mother’ and the ‘Kiev Resi-
dent’: Did Russian Television ‘Use Actress to Portray FIVE Different Women’ As It Reported
Normal Ukrainians Backed Kremlin,” Daily Mail, March 5, 2014.
213Amos Chapple, “War of Words over Ukraine ‘Combat’ Photo,” Radio Free Europe Radio
Liberty, August 25, 2016.
Hostile Social Manipulation: Russian Activities 85
214 Vladimir Putin reportedly made this statement on a 2013 visit to RT’s Moscow head-
quarters (Jim Rutenberg, “RT, Sputnik and Russia’s New Theory of War,” New York Times,
September 13, 2017).
215“Products and Services,” webpage, Sputnik, undated; “Telling the Untold,” webpage,
Sputnik, undated.
216“Crimean ‘Nazi’ Billboard Highlights Propaganda Problem: U.S.,” CBS News,
March 10, 2014; Alan Yuhas, “Russian Propaganda over Crimea and the Ukraine: How
Does It Work?” The Guardian, March 17, 2014.
86 Hostile Social Manipulation: Present Realities and Emerging Trends
217 Though Garth S. Jowett and Victoria O’Donnell use the nomenclature “techniques to
maximize effect” to categorize the use of certain propaganda techniques in their book Pro-
paganda and Persuasion, the techniques described are different from those in this discussion
(Jowett and O’Donnell, 2012). For a brief discussion of some of these techniques employed
in a real-world campaign, see Chen, 2015.
218 It is important to acknowledge that there is little agreement on specific definitions for
the terms used to describe human- and computer-driven online accounts. In an effort to be
consistent, in this chapter we use terms as they are defined by Robert Gorwa and Doug-
las Guilbeault in their typology, based on a survey of the recent relevant literature. Robert
Gorwa and Douglas Guilbeault, “Understanding Bots for Policy and Research: Challenges,
Methods, and Solutions,” Prague: Conference of the International Communication Associa-
tion, May 2018.
219 Gorwa and Guilbeault, 2018, pp. 8–9.
220 Gorwa and Guilbeault, 2018, p. 8.
221 Sergey Sanovich, Computational Propaganda in Russia: The Origins of Digital Misinforma-
tion, Working Paper No. 2017.3, Computational Propaganda Research Project, University of
Oxford, Oxford, UK, 2017.
Hostile Social Manipulation: Russian Activities 87
Moscow has since used this tool to target external audiences, evidence
suggests.
Investigations jointly conducted by the U.S. Congress and tech-
nology firms have identified a bot campaign that operated during
the 2016 U.S. presidential campaign that evidence suggests may have
been sponsored by the Russian government. As of January 2018,
Twitter identified over 50,000 bots linked to Russia that were opera-
tional during the 2016 U.S. presidential campaign.222 Analyses of the
social bot traffic during the campaign found that the number of bot-
generated tweets classified as pro-Trump was significantly higher than
pro-Clinton tweets during the campaign, and that bots “strategically
colonized pro-Clinton hashtags, and then disabled activities after Elec-
tion Day.” However, this analysis includes bots sponsored by U.S.
political entities.223 Still, evidence suggests that pro-Trump bot traffic
may have been engineered by the Russian government.224 Russia has
also been accused of deploying bot campaigns against several other
target states.
Employing Sockpuppets (Trolls)
Politically driven sockpuppets, or what are now commonly referred
to as trolls in public discourse, are manually controlled (i.e., human-
controlled) “accounts that impersonate humans for political
purposes.”225 In other words, these are accounts that pose as legitimate
222
April Glaser, “Twitter Admits There Were More Than 50,000 Russian Bots Trying to
Confuse American Voters Before the Election Campaign,” Slate, January 19, 2018.
223 Bence Kollanyi, Phillip N. Howard, and Samuel C. Woolley, “Bots and Automation over
Twitter During the U.S. Election,” Comprop Data Memo 2016.4, Computational Propa-
ganda Research Project, University of Oxford, Oxford, UK, November 17, 2016.
224 Alice
Marwick and Rebecca Lewis, “Media Manipulation and Disinformation Online,”
Data and Society Research Institute, undated, p. 38.
225 Gorwa and Guilbeault, 2018, p. 10. As with bots, the meaning of the term troll has meta-
morphosed over time. Even now there is little consistency in the use of this term within
the computer science community, among policymakers, or in the media. Internet troll-
ing has existed since people have been able to interact online. For much of the 2000s, the
term trolls referred to humans who generated and/or communicated inflammatory material
online, with the intent to offend, irritate, or provoke. The ultimate objectives of these trolls
are varied: “They do this for many reasons, from boredom, to making people think, but
88 Hostile Social Manipulation: Present Realities and Emerging Trends
individuals or groups and are operated by humans, just not those the
accounts claim to be. Sockpuppets can post comments on articles,
share content, and “like” content while cloaked behind the anonymity
of the internet, engendering false impressions about public discourse
or sentiment. They can create the illusion that many (or few) “people”
support certain messages. This provides sponsors of these activities
with both credibility and mass.
The Russian government is suspected of employing sockpup-
pets to influence audiences and drive political outcomes. In 2012,
leaked emails from the leader of the Kremlin-funded youth group
Nashi offered evidence that the Russian government had paid blog-
gers and commenters to post pro-Putin content.226 Sockpuppet cam-
paigns have been identified in Ukraine, Poland, and Finland, but it is
unclear whether these were sponsored by Russia or were the work of
pro-Kremlin enthusiasts.227 More recently, evidence has come to light
that helps to further substantiate Russia’s use of sockpuppet campaigns
for political ends.
The IRA employs hundreds of individuals to generate content,
operate accounts, and use these to post content. Information made
public by the U.S. Congress and tech companies demonstrates the
IRA’s use of sockpuppets to generate and post content related to the
2016 U.S. presidential campaign.228 Over 2,500 Twitter handles were
operated by employees of the IRA during the campaign, and 3,300
most do it for the lulz. . . . Lulz is laughter at someone else’s expense” (Encyclopedia Dra-
matica, “Troll” and “Lulz,” referenced in E. Gabriella Coleman, “Phreaks, Hackers, and
Trolls: The Politics of Transgression and Spectacle,” in Michael Mandiberg, ed., The Social
Media Reader, New York: New York University Press, 2012, p. 111. The online image forum
4chan.org, founded in 2003, is often considered a birthplace of this early provocative trolling
behavior. Since then, the concept of trolling has evolved. Given the public’s recent focus on
Russian social manipulation efforts, the term troll is now often used in public discourse with
the assumption that the trolling always has political ends.
226 Miriam Elder, “Polishing Putin: Hacked Emails Suggest Dirty Tricks by Russian Youth
Groups,” The Guardian, February 7, 2012 .
227 NATO StratCom Center of Excellence, Internet Trolling as a Tool of Hybrid Warfare: The
Case of Latvia, undated.
228 David S.Cloud, “Facebook Tells Congress That 126 Million Americans May Have Seen
Russia-Linked Ads,” Los Angeles Times, October 31, 2017.
Hostile Social Manipulation: Russian Activities 89
229Ben Collins et al., “House Drops Motherlode of Russian Propaganda,” Daily Beast,
November 1, 2017.
230 RBC, “Rassledovanie RBK: Kak ‘Fabrika Trollej’ Porabotala Na Vyborah v SShA,” Vol.
11, No. 35, October 17, 2017; “An Ex St. Petersburg ‘Troll’ Speaks Out: Russian Indepen-
dent TV Network Interviews Former Troll at the Internet Research Agency,” 2017.
231 Gorwa and Guilbeault, 2018, pp. 9–10.
232Dipayan Ghosh and Ben Scott, “#DigitalDeceit: The Technologies Behind Precision
Propaganda on the Internet,” New America, Public Interest Technology Program, January
2018, p. 17.
233 Ghosh and Scott, 2018, p. 18.
90 Hostile Social Manipulation: Present Realities and Emerging Trends
sian spy Sergei Skripal.234 Some suspect that it is unlikely these results
were organically derived. The same was true for searches conducted
using the query “ODNI hacking report” in reference to the 2017 Intel-
ligence Community Assessment on Russia’s activities targeting the
U.S. election.235
Bots and sockpuppets can drive visibility for Kremlin messages by
affecting search engine algorithms.236 Search engine optimization tech-
niques are driven by bot and sockpuppet campaigns, as well as through
the use of clickbait. RT uses clickbait and viral videos, such as those
of natural disasters, to increase the time spent watching RT videos
and to generate more likes. Longer watch times and higher numbers
of likes result in more favorable placement in YouTube’s search results
and recommendations.237
Tangled Web of Techniques
None of the aforementioned techniques are typically used in a vacuum.
Rather, they are often employed as one part of broader efforts. The
various elements of Russia’s social manipulation efforts can be used
to mutually reinforce one another, creating a complex, tangled web.238
The Digital Research Forensics Lab’s deconstruction of a specific
campaign, one in which the IRA attempted to malign actor Morgan
Freeman’s critique of Russian disinformation efforts, demonstrates
this in practice.239 On September 18, 2017, a nonprofit organization,
the Committee to Investigate Russia, released a video featuring Free-
234Chris Meserole and Alina Polakova, “Disinformation Wars,” Foreign Policy, May 25,
2018.
235Kaveh Waddell, “Kremlin-Sponsored News Does Really Well on Google,” The Atlantic,
January 25, 2017.
236 In-person interview with Department of State official.
237
Daisuke Wakabayashi and Nicholas Confessore, “Russia’s Favored Outlet Is an Online
News Giant. YouTube Helped,” New York Times, October 23, 2017.
238Digital Forensics Research Lab, “Russia’s Full Spectrum Propaganda: A Case Study
in How Russia’s Propaganda Machine Works,” The Atlantic Council, January 23, 2018;
Defense Intelligence Agency, “Russia Military Power: Building a Military to Support Great
Power Aspirations,” 2017.
239 Digital Forensics Research Lab, 2018.
Hostile Social Manipulation: Russian Activities 91
man. In it, the actor addresses Russia’s efforts to interfere in the 2016
presidential election and cautions, “we have been attacked. We are at
war.”240 Two days later, an online group that has since been revealed as
an IRA account by the U.S. Congress, AgitPolk, initiated a campaign
in response to Freeman’s video under the slogan “#StopMorganLie.”241
The group accused Freeman of “manipulat[ing] the facts of modern
Russian history and openly [slandering their] country,” and announced
the launch of a campaign in response to his video.242
The Digital Research Forensics Lab’s analysis reveals how the
campaign employed a labyrinth of actors and platforms to propagate
the desired narratives and to potentially suppress alternative narratives.
First, bot and sockpuppet accounts were used to amplify the hashtag
on social media platforms Twitter and VKontakte (VK). Only hours
later, official Russian government Twitter accounts such as that of the
consulate general weighed in, posting memes that attempted to deni-
grate Freeman’s credibility. This activity was followed by an RT arti-
cle, which featured the same hashtag as the social media posts. The
RT article was picked up by several other niche outlets.243 While this
analysis examines a single campaign believed to be sponsored by Russia,
it nevertheless illustrates how various actors masquerading as indepen-
dent from one another and from the Kremlin are used in concert.244
240 Committee to Investigate Russia, “Morgan Freeman Warns Russia Is Waging War on the
U.S.,” September 18, 2017.
241 Digital Forensics Research Lab, 2018.
242 Digital Forensics Research Lab, “Putin’s Online Cheerleaders: The ‘Patriots’ Behind Pro-
Kremlin, Anti–Morgan Freeman Memes,” The Atlantic Council, October 17, 2017.
243 Digital Forensics Research Lab, 2018. For the RT article referenced, see RT,
“#StopMorganLie: Twitterati Disappointed in Freeman After His ‘War with Russia’ Video,”
September 20, 2017.
244 Digital Forensics Research Lab, 2018.
92 Hostile Social Manipulation: Present Realities and Emerging Trends
251 In-person interview with Department of State official 010, October 4, 2017.
252 In-person interview with Department of State official 010, October 4, 2017.
253 Phone interview with Russian journalist 27, October 31, 2017.
254 Phone interview with Russian journalist 27, October 31, 2017.
255 Phone interview with Russian journalist 27, October 31, 2017.
Hostile Social Manipulation: Russian Activities 95
256 Josh Halliday, “BBC World Service Fears Losing Information War As Russia Today
Ramps Up Pressure,” The Guardian, December 21, 2014.
257 Glaser, 2018.
258Colin Lecher, “Here Are the Russia-Linked Facebook Ads Released by Congress,” The
Verge, November 1, 2017.
259For the first two budget figures, see Simon Shuster, “Inside Putin’s On-Air Machine,”
TIME, March 5, 2015. For the third figure, see Steven Erlanger, “What is RT?” New York
Times, March 8, 2017.
260 Seddon, 2014a. This figure is based on the assumption that spending continued at the
rate of spending between December 2013 and April 2014.
261 “United States of America v. Internet Research Agency LLC [et al.],” pp. 5–6.
96 Hostile Social Manipulation: Present Realities and Emerging Trends
Audience Exposure
Some discourse on Russian efforts cites the number of people that have
been exposed to social manipulation as evidence of its effectiveness.
RT alleges that a survey conducted in 2015 found that its television
network was viewed by 70 million people every week.264 Yet, according
to 2015 ratings by Nielsen Media Research, RT was not in the top 94
channels in the United States. Likewise, Britain’s Broadcast Audience
Research Group found that RT captured only 0.04 percent of British
viewers in December 2016.265
262 Colin Stretch, General Counsel, Facebook, “Hearing Before the United States Senate
Select Committee on Intelligence,” testimony, November 1, 2017, p. 4.
263 U.S. Congress, “Report on Russian Active Measures,” House Permanent Select Commit-
tee on Intelligence, March 22, 2018, p. 32.
264RT, “RT Watched by 70mn Viewers Weekly, Half of Them Daily—Ipsos Survey,”
March 10, 2016.
265 “RT’s Propaganda Is Far Less Influential Than Westerners Fear,” The Economist,
January 19, 2017.
Hostile Social Manipulation: Russian Activities 97
270 Nancy K. Baym, “Data Not Seen: The Uses and Shortcomings of Social Media Metrics,”
First Monday, Vol. 18, No. 10, October 7, 2013.
271 Glaser, 2018.
272 Ben Popken, “Russian Trolls Went on Attack During Key Election Moments,” NBC
News, February 13, 2018.
273 Leslie K. John, Oliver Emrich, Sunil Gupta, and Michael I. Norton, “Does ‘Liking’ Lead
to Loving: The Impact of Joining a Brand’s Social Network on Marketing Outcomes,” Jour-
nal of Marketing Research, Vol. 54, No. 1, 2017a, p. 4.
274 Leslie K. John, Daniel Mochon, Oliver Emrich, and Janet Schwartz, “What’s the Value
of a Like?” Harvard Business Review, March–April, 2017b.
Hostile Social Manipulation: Russian Activities 99
280 In-person interview with former CIA analyst 024, November 20, 2017.
281 In-person interview with former CIA analyst 024, November 20, 2017.
282Keir Giles, Russia’s ‘New’ Tools for Confronting the West: Continuity and Innovation in
Moscow’s Exercise of Power, London: Chatham House, March 2016a, p. 31.
283 Giles, 2016a, p. 31, quoting John Besemeres, “Russian Disinformation and Western Mis-
conceptions,” in Besemeres, A Difficult Neighbourhood: Essays on Russia and East-Central
Europe Since World War II, Canberra: Australian National University Press, 2016.
Hostile Social Manipulation: Russian Activities 101
Policy Changes
Russian social manipulation efforts, or the perceived threat of such
efforts, have spurred policy changes across Europe and in the United
States in attempts to shield or inoculate populations from Moscow’s
influence. In 2014, the Ukrainian government banned 14 Russian tele-
vision channels from its cable networks, citing their “broadcasting pro-
paganda of war and violence” as the motivation for its action.284 Three
years later, Kyiv announced that it would block Russian-owned inter-
net sites and social media platforms, including VK, Odnoklassniki,
Mail.ru, and Yandex.285 Finland sponsors classes educating “border
guards, child protection agencies, educators, and civil servants how to
respond to propaganda.”286 The U.S. government has pressured RT
America to register as a foreign agent under the Foreign Agents Regis-
tration Act (FARA).287 Individuals believed to be involved in Russian
social manipulation efforts, such as Dmitry Kiselev, have been placed
on the European Union’s individual sanctions list.288
These describe only some of the offensive and defensive policy
actions taken by the West in response to Russian social manipulation
efforts. Although Moscow can claim to have motivated such policy
actions, it is questionable whether these are positive developments for
Russia.
284 “Ukraine Bans Russian TV Channels for Airing War ‘Propaganda,’” Reuters, August 19,
2014.
285 “RT’s Propaganda Is Far Less Influential Than Westerners Fear,” 2017.
286 Linda Kinstler, “How to Survive a Russian Hack: Lessons from Eastern Europe and the
Baltics,” The Atlantic, February 2, 2017.
287 Aron Mate, “RT Was Forced to Register as a Foreign Agent,” The Nation, November 16,
2017.
288
Natalka Pisnia, “Why Has RT Registered As a Foreign Agent in the US?” BBC News,
November 15, 2017.
102 Hostile Social Manipulation: Present Realities and Emerging Trends
289 Dalibor Rohac, “Cranks, Trolls and Useful Idiots,” Foreign Policy, March 12, 2015.
290 Kragh and Asberg, 2017, p. 807.
Hostile Social Manipulation: Russian Activities 103
more, social science scholarship has debated whether attitudes are reli-
able indicators of behavior.291
Given the information available, we cannot determine whether
Russian social manipulation efforts have been effective in influencing
attitude, opinion, belief, or behavior change in their target audiences.
Humans are inclined toward attitude preservation, particularly with
attitudes that are deeply held, like those in the political realm. If Russia
has been successful in such efforts, it is because it navigated our own
internal gatekeepers.
Conclusions
291 Phone interview with former military information activities expert 022, November 2,
2017; Alice H. Eagly and Shelly Chaiken, “Attitude Structure and Function,” in D. T. Gil-
bert, S. T. Fiske, and G. Lindzey, eds., The Handbook of Social Psychology (4th ed.), New York:
Oxford University Press, 1998, pp. 269–322; Richard T. LaPiere, “Attitudes vs. Actions,”
Social Forces, Vol. 13, No. 2, 1934, pp. 230–237.
104 Hostile Social Manipulation: Present Realities and Emerging Trends
actors take their cues from the Kremlin’s publicly stated positions on
various issues and operate based on these signals.
Many mysteries remain. Our work has demonstrated how little
the policymaking, scholarly, and analytical fields know about Russian
social manipulation. More may be known in the classified setting. Yet
with the centrality of this issue in today’s public discourse, it is impor-
tant that those being targeted by such efforts have access to evidence-
based insights about the efforts.
Perhaps the most important missing piece of information pertains
to Russia’s effectiveness. Understanding whether and how Russian
target audiences have been influenced is as important as identifying the
perpetrators and their means. Our questioning of success should not be
interpreted as a determination that Russian attempts have been ineffec-
tive in influencing opinions, attitudes, beliefs, or behavior. Instead, we
believe that this issue merits further inquiry.
If Russia does, in fact, use reaction as a measure of its success, as
some suggested to us, it could reasonably interpret its recent efforts in
the West as victories. Western reactions could incentivize future Rus-
sian social manipulation efforts, further underscoring the need for
additional examinations of the effectiveness of its efforts.
CHAPTER FOUR
1 For academic studies of Chinese ideology and propaganda, see, for example, Anne-
Marie Brady, Marketing Dictatorship: Propaganda and Thought Work in Contemporary China,
Lanham, Md.: Rowman and Littlefield Publishers, 2008; Anne-Marie Brady, ed., China’s
Thought Management, New York: Routledge, 2012; Daniel C. Lynch, After the Propaganda
State: Media, Politics, and ‘Thought Work’ in Reformed China, Stanford, Calif.: Stanford Uni-
versity Press, 1999. In conducting this study, RAND leveraged open-source primary and
secondary source information in Chinese and English and conducted a small number of
face-to-face interviews with subject-matter experts knowledgeable about China’s propaganda
and social media operations.
This chapter also draws on Nathan Beauchamp-Mustafaga and Michael Chase, The Chi-
nese Military and Social Media: A New Tool for Peacetime and Wartime Propaganda at Home
and Abroad, Washington, D.C.: John Hopkins SAIS, forthcoming.
105
106 Hostile Social Manipulation: Present Realities and Emerging Trends
2 National Security Strategy of the United States of America, Washington, D.C.: The White
House, December 2017, pp. 34–35.
3 Mark Landler, “Trump Accuses China of Interfering in Midterm Elections,” New York
Times, September 26, 2018.
4 David Nakamura and Ellen Nakashima, “Without Offering Evidence, Trump Accuses
China of Interfering in U.S. Midterm Elections,” Washington Post, September 26, 2018.
Hostile Social Manipulation: Chinese Activities 107
5 Murong Xuecun, “The New Face of Chinese Propaganda,” New York Times,
December 20, 2013.
6 Kristin Shi-Kupfer, “Governance Through Information Control,” China Monitor, No.
26, Mercator Institute for China Studies, January 19, 2016. Moreover, as one analysis has
pointed out, “The Party [has recognized] that social media, carefully managed, can help
spread its messages effectively in the country with the world’s largest number of internet
users”; Anil Azad Pandey, “How the Chinese Communist Party Is Using Social Media to
Win Friends and Influence People,” OZY, October 25, 2017.
7 For an overview of recent work on Chinese influence operations targeting New Zealand,
Australia, Germany, and the European Union, respectively, see Anne-Marie Brady, Magic
108 Hostile Social Manipulation: Present Realities and Emerging Trends
Weapons: China’s Political Influence Activities Under Xi Jinping, Washington, D.C.: Wilson
Center, 2017; Clive Hamilton, Silent Invasion: China’s Influence in Australia, Richmond,
Australia: Hardie Grant, 2018; “China’s Operation Australia,” Sydney Morning Herald,
2017; Didi Kirsten Tatlow, “China Reaches Into the Heart of Europe,” New York Times,
January 25, 2018; Thorsten Benner, Jan Gaspers, Mareike Ohlberg, Lucrezia Poggetti,
and Kristin Shi-Kupfer, “Authoritarian Advance: Responding to China’s Growing Political
Influence in Europe,” Global Public Policy Institute and Mercator Institute for China Stud-
ies, February 2018.
8 Shi Anbin and Peiyan Wang, “Stealth Propaganda: Concepts Evolution, Strategy”
[“隐性宣传:概念·演进·策略”], International Communications, January 2016.
9 For an overview of China’s lessons learned from the collapse of the Soviet Union, see
David Shambaugh, China’s Communist Party: Atrophy and Adaptation, Washington, D.C.:
Woodrow Wilson Center Press, 2009; William Wan, “In China, Soviet Union’s Failure
Drives Decisions on Reform,” Washington Post, March 23, 2013; James Palmer, “What
China Didn’t Learn from the Collapse of the Soviet Union,” Foreign Policy, December 24,
2016. For an authoritative article analyzing the danger of Western social media (Twitter,
Facebook, and Blackberry phones) to social stability and their role as a tool for the U.S.
government to interfere in other countries based on the Arab Spring, see Wu Zaiping, “The
Social Challenges and Responses to New Media” [“新媒体的社会挑战与应对”], Journal of
the Party School of the Central Committee of the C.P.C, October 2012.
Hostile Social Manipulation: Chinese Activities 109
logical and cultural fields are the focal areas of their long-term infiltra-
tion”; Hu’s prescription, in part, was to improve Chinese soft power.10
In April 2013, shortly after taking power, President Xi asserted
more work was to be done, saying in a secret document that “West-
ern forces hostile to China and dissidents within the country are still
constantly infiltrating the ideological sphere” and specifically assert-
ing that regime opponents “have stirred up trouble about disclosing
officials’ assets, using the Internet to fight corruption, media controls
and other sensitive topics, to provoke discontent with the party and
government.”11 Xi called for the CCP to “conscientiously strengthen
management of the ideological battlefield,” including “strengthen
guidance of public opinion on the Internet [and] purify the envi-
ronment of public opinion on the Internet.”12 As China deepens its
engagement with the world, this struggle for controlling information
about the CCP has extended to global public opinion, and the internet
is only the latest battlespace.
Thus, while some observers hoped that the widespread adoption
of personalized networked information technology, most commonly in
the form of internet-enabled cellular phones with access to social media
websites, might undercut the Party’s high degree of control over infor-
mation, Chinese authorities not only managed to rise to the challenge
10 Hu Jintao, “Resolutely Follow the Cultural Development Path of Socialism with Chinese
Characteristics, Work to Build a Socialist Strong Culture Country” [“坚定不移走中国特
色社会主义文化发展道路努力建设社会主义文化强国”], Seeking Truth, January 1, 2012;
Edward Wong, “China’s President Lashes out at Western Culture,” New York Times, Janu-
ary 3, 2012; Evan Osnos, “China’s Culture Wars,” New Yorker, January 5, 2012; Damien
Ma, “Beijing’s ‘Culture War’ Isn’t About the U.S.—It’s About China’s Future,” Atlantic,
January 5, 2012; “China’s New Cultural Revolution,” Wall Street Journal, January 4, 2012.
11 Chris Buckley, “China Takes Aim at Western Ideas,” New York Times, August 19, 2013.
For translation, see “Document 9: A ChinaFile Translation,” ChinaFile, November 8, 2013.
12 Xi mirrored this tone in his speech to the 19th Party Congress in October 2017: “We
will [. . .] strengthen the penetration, guidance, influence, and credibility of the media. We
will provide more and better online content and put in place a system for integrated internet
management to ensure a clean cyberspace. [. . .] We must oppose and resist various erroneous
views with a clear stand” (Xi Jinping, “Secure a Decisive Victory in Building a Moderately
Prosperous Society in All Respects and Strive for the Great Success of Socialism with Chi-
nese Characteristics for a New Era,” speech to the 19th National Congress of the Commu-
nist Party of China, via Xinhua, October 18, 2017).
110 Hostile Social Manipulation: Present Realities and Emerging Trends
but also found that they could use such new technologies to expand
their influence. In retaining substantial control over information flows
among the Chinese populace, China has built up one of the world’s
most sophisticated capacities for human- and machine-enabled key-
word blocking and censorship and has also used such new technologies
and platforms in innovative ways to shape domestic and foreign infor-
mation flows related to China. Reflecting this, in November 2017 the
U.S. nongovernmental organization Freedom House noted that, on the
basis of Beijing’s widespread content blocking, content removal, and
content fabrication regimes, “for the third consecutive year, China was
the world’s worst abuser of Internet freedom.”13
The next section describes China’s foreign policy doctrine, includ-
ing its goals for information operations. The following section lays out
China’s strategies for who manages social media and online messages
to support the regime. The chapter then examines actual instances of
Chinese information operations through social media. The chapter
continues with an analysis of how effective China has been and con-
cludes with an assessment of the implications.
Chinese foreign policy has, in recent years, adopted two key postulates,
one being the importance of defending China’s core interests (核心利
益), and the other and more recent formulation centered on achiev-
ing the great rejuvenation of the Chinese nation (中华民族伟大复兴),
often referred to more colloquially as realizing the China dream (中国
13 Freedom on the Net 2017: Manipulating Social Media to Undermine Democracy, Washing-
ton, D.C.: Freedom House, 2017. As one interviewee we spoke with noted, “The [Chinese]
state’s ability to collect, analyze, target and deploy data [is] now far greater than that of
[Chinese] society” (RAND Interviewee #3). Another of the subject-matter experts we spoke
with for this study pointed out that, “In its ideal world, China wants everything that is said
about China internationally to follow what is said about China inside its borders,” giving the
regime total information control (RAND Interviewee #1).
Hostile Social Manipulation: Chinese Activities 111
梦).14 These two organizing precepts parallel fairly closely the notions
of defensive and offensive realism in international relations theory,
though with a considerable degree of fuzziness in delineating between
the more defensive approach to security policy (core interests) and the
more aggressive, ambitious, or offensive security strategy (the China
dream).15
The adoption by China of the core interests organiz-
ing framework in the late 2000s centered on three basic goals:
preserving China’s basic state system and national security
(维护基本制度和国家安全); protecting China’s sovereignty and terri-
torial integrity (主权和领土完整); and continuing the stable develop-
ment of China’s economy and society (经济社会的持续稳定发展).16
In practice, the first core interest is largely consonant with the preserva-
tion of the ruling status of the CCP, while the second and third inter-
ests serve as means to this end through the retention of control over
Xinjiang and Tibet; the defense of China’s claims in the South and
East China seas; and the prevention of Taiwan independence, leading
to the island’s ultimate absorption.
By contrast, the China dream, while necessarily entailing the
retention and/or integration of territories that Chinese leaders regard
as theirs, looks further afield to a more ambitious set of goals. These
include domestic economic goals such as achieving the two 100s,17
which are linked to the centenaries of the founding of the CCP, in
2021, and of the PRC itself, in 2049; reducing social inequality; clean-
ing up the environment; developing national morals; and achieving the
14 “Xi Calls for Persistently Pursuing Chinese Dream of National Rejuvenation,” China
Daily, September 26, 2017.
15 Chinese military theorists further complicate such matters by talking in terms of active
defense (积极防御) and preemptive counterattack (先发制人的反击), concepts that blur the
lines between cause and effect so as to justify China’s own efforts to take the initiative in
shaping its environment or engaging its adversaries during peacetime or on the battlefield.
16 Michael D. Swaine, “China’s Assertive Behavior, Part One: On ‘Core Interests,’” China
Leadership Monitor, No. 34, September 2010.
17 The two 100s refers to the economic goals of becoming a moderately well-off society (小康
社会) by 2021 (the centenary of the founding of the CCP) and of completing the dream of
national rejuvenation by 2049 (the 100th anniversary of the founding of the PRC).
112 Hostile Social Manipulation: Present Realities and Emerging Trends
18 Robert Lawrence Kuhn, “Xi Jinping’s China Dream,” New York Times, June 4, 2013.
19 Bonnie Glaser and Matthew P. Funaiole, “Xi Jinping’s 19th Party Congress Speech Her-
alds Greater Assertiveness in Chinese Foreign Policy,” Center for Strategic and International
Studies, October 26, 2017.
20 For analyses of the goals of Chinese foreign policy over the past decade, see Susan Shirk,
China: Fragile Superpower, Oxford, UK: Oxford University Press, 2007; Andrew J. Nathan
and Andrew Scobell, China’s Search for Security, New York: Columbia University Press,
2012; David Shambaugh, China Goes Global: The Partial Power, Oxford, UK: Oxford Uni-
versity Press, 2013; Jae Ho Chung, “China’s Evolving Views of the Korean-American Alli-
ance, 1953–2012,” Journal of Contemporary China, Vol. 23, No. 87, 2014, pp. 425–442;
Yan Xuetong, “From ‘Keeping a Low Profile’ to ‘Striving for Achievement,’” Chinese Journal
of International Politics, Vol. 7, No. 2, 2014, pp. 153–184; Oriana Skylar Mastro, “Why
Chinese Assertiveness Is Here to Stay,” Washington Quarterly, Vol. 37, No. 4, Winter 2015,
pp. 151–170; Camilla T. N. Sørensen, “The Significance of Xi Jinping’s ‘China Dream’
for Chinese Foreign Policy: From ‘Tao Guang Yang Hui’ to ‘Fen Fa You Wei,’” Journal of
Chinese International Relations, Vol. 3, No. 1, 2015, pp. 53–73; Robert S. Ross and Jo Inge
Bekkevold, eds., China in the Era of Xi Jinping, Washington, D.C.: Georgetown University
Hostile Social Manipulation: Chinese Activities 113
Press, 2016; Robert Blackwill and Kurt Campbell, Xi Jinping on the Global Stage: Chinese
Foreign Policy Under a Powerful but Exposed Leader, Council Special Report No. 74, New
York: Council on Foreign Relations, February 25, 2016; and Adam Liff, “China and the U.S.
Alliance System,” China Quarterly, April 2017.
21 “Xi’s Secret Economic Weapon: Overseas Chinese,” Nikkei Asian Review, April 3,
2017; “Inside China’s Secret ‘Magic Weapon’ for Worldwide Influence,” Financial Times,
October 26, 2017.
22 Sarah Cook, “Resisting Beijing’s Global Media Influence,” The Diplomat, December 10,
2015; Sarah Cook, “Chinese Government Influence on the U.S. Media Landscape,” testi-
mony before the U.S–China Economic and Security Review Commission, May 4, 2017.
114 Hostile Social Manipulation: Present Realities and Emerging Trends
23 Gerry Groot, “The Long Reach of China’s United Front Work,” Lowy Interpreter, Novem-
ber 6, 2017.
24 Brady, 2017; “Inside China’s Secret ‘Magic Weapon’ for Worldwide Influence,” 2017.
25 For an authoritative analysis of Chinese lessons learned from Iraq, see Dean Cheng,
“Chinese Lessons from the Gulf Wars,” in Andrew Scobell, David Lai, and Roy Kamphau-
sen, eds., Chinese Lessons from Other Peoples’ Wars, Carlisle, Pa.: Strategic Studies Institute,
2011, pp. 153–200.
For a specific Chinese analysis of U.S. public opinion warfare, see Cai Huifu, Wang Lin,
Sheng Peilin, Yu Qi, Liu Xuemei, and Zheng Yu, “Research into News and Public Opinion
Warfare During the Iraq War,” China Military Science, August 2003, pp. 28–34.
Hostile Social Manipulation: Chinese Activities 115
more recent articles, Chinese military analysts have also drawn lessons
from Russia’s actions in Ukraine and Syria, highlighting the impor-
tance of starting information operations before political or military
actions, as well as the benefits of going on the offensive to counter
Western narratives, though these articles do not touch specifically on
social media.26 Other, broader articles have covered ISIS’ use of social
media for recruitment and the role of social media in the United States’
war on ISIS, Japan’s foreign propaganda work under Prime Minister
Shinzo Abe, and Germany’s use of social media.27
We found no authoritative articles drawing substantive lessons
for future application from Russia’s interference in the 2016 U.S. elec-
tion, in contrast with extensive analysis and lessons learned from Rus-
sia’s activities in Ukraine and Syria.28 Most articles that did address
allegations of Russian interference presented neutral or negative views,
focusing on the security risks to computer systems, though such tech-
nical analysis of how “micro-propaganda” and misinformation spread
through U.S. social networks would be applicable to potential Chinese
influence operations in the future.29 Early Chinese military analysis
26 Zhu Ningning, “An Analysis of Russia’s Unfolding of Media Warfare Tactics amid the
Turbulent Political Situation in Ukraine,” Military Correspondent, May 2014; Li Qiaoming,
“Analysis of Modern Warfare Development Based on Russia’s Two Conflicts,” PLA Daily,
August 16, 2016; Wang Jichang, “Main Experience of Russia’s Military Operations in Syria,”
China Military Science, March 2016, pp. 119–126.
27 Huang Dahui, “Analyzing the Abe Government’s Foreign Propaganda Strategy” [“试
析安倍政府的对外宣传战略”], Contemporary International Relations, June 2017; Zhou
Yang, “Examination of Social Media Actions in U.S. Strikes on ISIS” [“美军打击ISIS的
社交媒体行动探索”], Military Correspondent, July 2017; Chen Zheng, “Analysis of Infor-
mation on Social Media for German Audience” [“德国受众社交媒体获取信息情况分析”],
International Communications, August 2013; Bao Yu, “The Political Network Marketing
Strategies of Islamic State Towards Western Countries,” Journal of Jiangnan Social Univer-
sity, June 2016, pp. 17–21.
28 Instead, most articles on the topic covered the events and even the negative consequences
of stricter U.S. counterpropaganda rules on Chinese propaganda work. See Xu Shaomin,
“Insights into the Impact of the Proliferation of Fake News in the United States and Europe
on China’s Public Diplomacy” [“欧美假新闻泛滥对我国开展公共外交的启”], Public
Diplomacy, Vol. 2, 2017.
29 For example, see Chen Hui-hui, “Commentary Analysis on ‘Artificial Intelligence Tech-
nology Manipulating The US Election’ Research Report,” Information Security and Commu-
116 Hostile Social Manipulation: Present Realities and Emerging Trends
Figure 4.1
Articles Referencing Russia Today in Chinese-Language Publications Since
2016
12
10
8
Percentage
0
2010 2011 2012 2013 2014 2015 2016 2017
SOURCE: China National Knowledge Infrastructure (CNKI) 2017 data through August
2017.
33 Shi Anbing and Liu Ying, “Three Steps to Advance the Construction of Interna-
tional Communications Power” [“三步走”推进国际传播力建设”], People’s Daily Online,
May 19, 2014; Xu Lei, “What Can We Learn from Russia Today?” [“我们向“今日俄罗斯”
学什么?”], People’s Daily Overseas Edition, September 19, 2014; Gao Han, “Russia Today:
Russia’s External Propaganda Aircraft Carrier” [“今日俄罗斯”:俄罗斯的“外宣航母”],
Modern Audiovisual, May 2016.
34 Li Yiqing, “两家俄官媒推特账号广告功能遭关闭,曾被美指责“干预大选””], The
Paper, October 27, 2017.
118 Hostile Social Manipulation: Present Realities and Emerging Trends
38 Ben Bland, “China Censorship Drive Splits Leading Academic Publishers,” Financial
Times, November 4, 2017.
39 Michael Cieply, “Deal Expands Chinese Influence on Hollywood,” New York Times,
May 20, 2012; Ryan Faughnder, “China-Owned AMC Seals Deal to Buy Carmike Cinemas,
Making It the Largest Theater Chain in U.S.,” Los Angeles Times, November 15, 2016; Clare
Baldwin and Kristina Cooke, “How Sony Sanitized the New Adam Sandler Movie to Please
Chinese Censors,” Reuters, July 24, 2015; Richard Berman, “China’s Rising Threat to Hol-
lywood,” Politico, October 4, 2016; Ben Fritz and John Horn, “Reel China: Hollywood Tries
to Stay on China’s Good Side,” Los Angeles Times, March 16, 2011; Frank Langfitt, “How
China’s Censors Influence Hollywood,” NPR, May 18, 2015.
40 “China Is Spending Billions to Make the World Love It,” The Economist, March 23, 2017.
41 Mike Isaac, “Facebook Said to Create Censorship Tool to Get Back into China,” New
York Times, November 22, 2016; Alexandra Stevenson, “Facebook Blocks Chinese Billion-
aire Who Tells Tales of Corruption,” New York Times, October 1, 2017.
120 Hostile Social Manipulation: Present Realities and Emerging Trends
42Koh Gui Qing and John Shiffman, “Beijing’s Covert Radio Network Airs China-Friendly
News Across Washington, and the World,” Reuters, November 2, 2015.
43 Koh and Shiffman, 2015.
44Steven Jiang, “Beijing Has a New Propaganda Weapon: Voice of China,” CNN Business,
March 21, 2018.
45 Gary King, Jennifer Pan, and Margaret E. Roberts, “How the Chinese Government
Fabricates Social Media Posts for Strategic Distraction, Not Engaged Argument,” American
Political Science Review, Vol. 111, No. 3, 2017, pp. 484–501.
Hostile Social Manipulation: Chinese Activities 121
46 Kenneth Lieberthal, Governing China: From Revolution Through Reform, New York:
W.W. Norton & Co., 1995. Although the PLA is the armed wing of the CCP, and hence not
technically a third organ of governance that can be separated from the Party, in practice the
military operates in a space largely ungoverned by civilian authorities, including either state
or Party officials. For another list of Chinese government organizations undertaking online
propaganda, see Bradshaw and Howard, 2017, p. 17.
47 In China, province-level administrative units include provincial governments; the gov-
ernments of “autonomous regions,” where ethnic minority populations are heavily clustered,
including Guangxi, Inner Mongolia, Ningxia, Tibet, and Xinjiang; and the four province-
level municipalities (Beijing, Chongqing, Shanghai, and Tianjin).
48 King, Pan, and Roberts, 2017, pp. 484–501.
122 Hostile Social Manipulation: Present Realities and Emerging Trends
49 The former head of the Propaganda Department, Liu Yunshan, was quoted in one 2008
speech as saying, “Through the pertinent struggle of international public opinion, we have
established a good international image and promoted the establishment of an international
public opinion in favor of China. By comprehensively implementing the cultural strategy of
going out, holding theme activities on Chinese culture, setting up overseas institutions of
Chinese culture and vigorously establishing mainstream media in foreign countries, we have
constantly expanded the international influence of Chinese culture” (Liu Yunsan, “Review
and Outlook—This Article Is Abridged from the Speech Made by Comrade Liu Yunshan
at the Meeting for Leading Comrades in Central Propaganda and Cultural Units on 25
December, 2008,” Seeking Truth, January 2009.
50 For recent research on the United Front, see Brady, 2017; Yimou Lee and Faith Hung,
“How China’s Shadowy Agency Is Working to Absorb Taiwan,” Reuters, November 26,
2014; James Kynge, Lucy Hornby, and Jamil Anderlini, “Inside China’s Secret ‘Magic
Weapon’ for Worldwide Influence,” Financial Times, October 26, 2017; Jamil Anderlini
and Jamie Smyth, “West Grows Wary of China’s Influence Game,” Financial Times, Decem-
ber 19, 2017; Gerry Groot, “United Front Work after the 19th Party Congress,” China Brief,
December 22, 2017b; June Teufel Dreyer, “A Weapon Without War: China’s United Front
Strategy,” Foreign Policy Research Institute, February 6, 2018; John Dotson, “The United
Front Work Department in Action Abroad: A Profile of the Council for the Promotion of the
Peaceful Reunification of China,” China Brief, February 13, 2018.
Hostile Social Manipulation: Chinese Activities 123
Figure 4.2
Articles Referencing Social Media in Party Construction
45
40 Social media
Weibo
35 WeChat
Facebook
30 Twitter
Percentage
25
20
15
10
0
2010 2011 2012 2013 2014 2015 2016 2017
SOURCE: CNKI database, 2017 data through October.
51 The United Front Department’s official journal, China United Front, does not have any
articles publicly available on the database CNKI after 2014, so we chose the Chongqing
journal as a better source of data. China United Front had a similar focus on domestic social
media platforms (Weibo from 2011 to 2014 and WeChat in 2014).
For articles on the need for the United Front Department to keep pace with evolving
trends (i.e., social media) and the value of WeChat for domestic propaganda work, see Bei-
jing Municipal Committee United Front Department, “The Historical Status and Practical
Role of the United Front” [“统一战线的历史地位和现实作用”], China United Front, Octo-
ber 2012; Song Suxia, “Small WeChat and Big Family: Hebi City Builds WeChat Platform
for United Front Work” [“ ‘小’ 微信 ‘大’ 家庭——鹤壁市委统战部建立统一战线微信平
台”], China United Front, May 2013.
124 Hostile Social Manipulation: Present Realities and Emerging Trends
Figure 4.3
Articles Referencing Social Media in United Front Science
16
Social media
14 Weibo
WeChat
Facebook
12 Twitter
10
Percentage
0
2010 2011 2012 2013 2014 2015 2016 2017
Figure 4.4
Articles Referencing Social Media in International Communications
35
Social media
30 Weibo
WeChat
25 Facebook
Twitter
Percentage
20
15
10
0
2009 2010 2011 2012 2013 2014 2015 2016 2017
SOURCE: CNKI 2017 data through August.
52 RAND Interview #3. A 2010 profile of the SCIO director provides some insight into the
SCIO’s overseas work and quotes the director as saying, “It is necessary for us to effectively
carry out a campaign to win world opinion and to safeguard national security and social sta-
bility.” See: Liu Jun, “Wang Chen, Guard of China’s Image: A Review of His Statements and
Actions in the Past Year Shows That the Question-and-Answer Papers Handed in by This
Ministerial-Level Official, Who Used to Be a Journalist, Are Outstanding,” Guoji Xianqu
Daobao, March 12, 2010.
126 Hostile Social Manipulation: Present Realities and Emerging Trends
53 Zhou Xiang and Han Weizheng, “Using Image Social Media to Improve China’s Inter-
national Communication Power” [利用图像社交媒体提升中国国际传播力研究”], Aca-
demic Journal of Zhongzhou, March 2017.
54 For one overview, see Ai Weiwei, “China’s Paid Trolls: Meet the 50-Cent Party,” New
Statesman, October 17, 2012.
55 King, Pan, and Roberts, 2017.
Hostile Social Manipulation: Chinese Activities 127
56 Meng Jing and Celia Chen, “China Fines Tencent, Baidu, Weibo over Banned Contents
in On-Going Crackdown,” South China Morning Post, September 26, 2017.
57 Beina Xu and Eleanor Albert, “Media Censorship in China,” Council on Foreign Rela-
tions, February 17, 2017; Yaqiu Wang, “The Business of Censorship: Documents Show
How Weibo Filters Sensitive News in China,” blog post, Committee to Protect Journalists,
March 3, 2016; Cate Cadell, “China Investigates Top Local Social Media Sites in Push to
Control Content,” Reuters, August 10, 2017.
58 For an overview of the PLA’s approach to political warfare, see Mark Stokes and Russell
Hsiao, “The People’s Liberation Army General Political Department: Political Warfare with
Chinese Characteristics,” Project 2049 Institute, October 14, 2013.
59 Chinese Academy of Military Science Military Strategy Department, ed., Science of Mili-
tary Strategy [战略学], 3rd edition, Beijing: Academy of Military Science Press, 2013, p. 129.
128 Hostile Social Manipulation: Present Realities and Emerging Trends
60 For an overview of the three warfares, see Stephan Halper, China: The Three Warfares,
Washington, D.C.: Office of Net Assessment, 2013; Dean Cheng, “Winning Without Fight-
ing: Chinese Public Opinion Warfare and the Need for a Robust American Response,” Heri-
tage Foundation, Backgrounder No. 2745, November 21, 2012; Dean Cheng, “Winning
Without Fighting: The Chinese Psychological Warfare Challenge,” Heritage Foundation,
Backgrounder No. 2821, July 11, 2013; Elsa Kania, “The PLA’s Latest Strategic Thinking on
the ‘Three Warfares,’” Jamestown Foundation China Brief, Vol. 16, No. 13, August 22, 2016.
61 Nathan Beauchamp-Mustafaga, Cristina Garafola, Astrid Cevallos, and Arthur Chan,
“China Signals Resolve with Bomber Flights over the South China Sea,” War on the Rocks,
August 2, 2016.
62 Nathan Beauchamp-Mustafaga, Derek Grossman, and Logan Ma, “Chinese Bomber
Flights Around Taiwan: For What Purpose?” War on the Rocks, September 13, 2017; PLAAF
Weibo status, December 12, 2017. For articles about PLA propaganda targeting Taiwan, see
Ma Yi, “Strengthening Agenda-Setting for Military News Coverage Targeting Taiwan,” Mil-
itary Correspondent, September 2010; Lu Wenxing, “Innovative Developments in Military
Broadcasts to Taiwan in the New Communication Age,” Military Correspondent, December
2010; Zhong Zhigang, “New Explorations on Military Propaganda Toward Taiwan Under
the Goal of Building a Strong Military,” Military Correspondent, November 2013.
Hostile Social Manipulation: Chinese Activities 129
Figure 4.5
Articles Referencing Social Media in Military Correspondent
70
Social media
60 Weibo
WeChat
50 Facebook
Twitter
Percentage
40
30
20
10
0
2009 2010 2011 2012 2013 2014 2015 2016 2017
SOURCE: CNKI 2017 data through August.
63 Sheng Peilin, and Li Xue, “On ‘Media Decapitation,’” Journal of the PLA Nanjing Insti-
tute of Politics, May 2006, pp. 114–117; Wu Rui, “Be on Guard Against Other Kinds of Soft
Warfare,” Military Correspondent, November 2013, pp. 53–54; Zhu Yuping, “Factors and
Inspiration for Public Opinion Warfare Under Informationized Conditions,” Military Art
Journal, October 2003, pp. 29–30.
130 Hostile Social Manipulation: Present Realities and Emerging Trends
Provincial Actors
Finally, provincial-level and local officials engage in efforts to censor,
swamp, and distribute information online, including through social
media. For example, circumstantial evidence suggests that the govern-
ments of both the Tibet Autonomous Region (TAR) and the Xinji-
ang Uyghur Autonomous Region (XUAR) have sought to shape global
opinions about these two regions, where substantial human rights vio-
lations are ongoing, and have coordinated these operations with cen-
tral authorities and Chinese tech companies.66 In the case of Tibet,
Cultural System Conscientiously Study the Spirit of the Sixth Plenary Session of the 17th
CPC Central Committee,” Tibet Daily, October 24, 2011; Chen Lin, “Effectively Strengthen
Internet Propaganda and Management Work to Create Sound Internet Public Opinion and
Cultural Environment for Society,” Tibet Daily, May 26, 2012; “Further Present a Real Tibet
to the World—Sixth Discussion on Earnestly Studying and Implementing Spirit of Com-
rade Li Changchun’s Important Speech,” Tibet Daily, August 5, 2012; Tang Dashan, “Tibet
Should Build a Major External Propaganda Structure,” Tibet Daily, September 14, 2013,
p. 3; Shi Lei and Xiao Tao, “Chen Quanguo, Lobsang Gyaincain Meet Media Delegation
‘Beijing Internet Media Red Land—Tibet’; Wu Yingjie Present at Meeting,” Tibet Daily,
August 20, 2014, pp. 1–2.
For articles explaining the Inner Mongolia and Shenzhen governments’ foreign propa-
ganda, see: Bi Lifu, “Innovating Foreign Propaganda in Ports and Improving Inner Mongo-
lia’s Image,” Theory Construction, 2009, pp. 10–12; Wang Pan, “Casting a New ‘Window to
China’—Explorations and Thoughts on Shenzhen’s Foreign Propaganda Work in the New
Era” [“铸造新的“中国窗口”—新时期深圳特区外宣工作探索与思考”], International Com-
munications, February 2012.
67 Jonathan Kaiman, “Free Tibet Exposes Fake Twitter Accounts by China Propagandists,”
The Guardian, July 22, 2014; Chen, 2012.
68 Anne Henchowitz, “Thousands of Local Internet Propaganda Emails Leaked,” China
Digital Times, December 3, 2014.
69 Other uses of such platforms to advance national security goals clearly exist but lie out-
side the scope of this research effort. For example, Chinese state and military intelligence
organs have reportedly used social media platforms to engage in espionage and recruit-
ment, scraping foreign users’ social media postings to create a personal dossier on targets
of influence attempts. While not directly related to the effort to push propaganda on social
media, this approach has recently received substantial attention, and so we include it here as
132 Hostile Social Manipulation: Present Realities and Emerging Trends
Table 4.1
Taxonomy of Chinese Influence Operations via Social Media
Type of Approach
offensive actions in the order listed in the table. Many of these activities
have fluid categorization and a degree of overlap, so readers can create
their own breakdown of the examples provided. Thereafter, we discuss
a few of China’s self-restraints in using social media.
a footnote. See, for example, “German Intelligence Unmasks Alleged Covert Chinese Social
Media Profiles,” 2017; Javier C. Hernandez and Melissa Eddy, “China Denies Using Linke-
dIn to Recruit German Informants,” New York Times, December 11, 2017.
Hostile Social Manipulation: Chinese Activities 133
70 “Successfully Do Foreign Propaganda Work for the 90th Anniversary of the Party’s
Founding: Fully Show Our Party’s Positive Image” [“做好建党90周年对外宣传工作 充
分展示我党良好形象”], International Communications, January 2011; Xiao Lili, “Chal-
lenges and Public Opinion Responses for China’s National Image in Africa” [“中国国家
形象在非洲面临的挑战及舆论应对”], International Communications, August 2011; Liu
Chen, “Audience Strategy for Foreign Communications on the Image of China’s Economy”
[“中国经济形象对外传播的受众策略”], International Communications, November 2011;
Sun Ming, “International Public Opinion on This Year’s ‘Two Congresses’” [“今年“两会”的
国际舆论关切”], International Communications, March 2013; Lian Xiaotong, “Analysis of
Leaders’ Public Diplomacy Strategy from a Cross-Cultural Perspective—Xi Jinping’s 2012
Visit to the United States as Example” [“跨文化视野下领导人公共外交策略分析—以2012
年习近平访美为例”], International Communications, September 2013; Jia Min, “Creator’s
Plight: The Good and Bad of Shaping Obama’s Image” [“创新者的窘境:奥巴马形象塑造
中的得与失”], International Communications, April 2014; Yao Yao, “The West’s View of
China, or the World’s View of China? New Thinking on Building China’s Global Image”
[“西方的中国观,还是世界的中国观?—中国建构国际形象的新思路”], International Com-
munications, July 2014; Xu Hua, “How Did Putin Create the Image of a Leader” [“普京
如何塑造领袖形象”], International Communications, March 2015; Wang Chen and Zhou
Ting, “Three Problems for Building and Communicating National Leader’s Public Image”
[“国家领导人公共形象的构建与传播三问”], International Communications, June 2015;
Jiang Yunai, “Shaping National Leaders’ Image Through Foreign Communication via New
Media—2015 Twitter Reporting by Xinhua, People’s Daily and CCTV as Examples” [“新
媒体对外传播中的国家领导人形象塑造—以2015年新华社、 《人民日报》、央视的推特
报道为例”], International Communications, April 2016; Zhao Mingwu, “Messaging One
Belt One Road Strategy: Problems and Responses” [“’一带一路’的政策传播:问题与应对”],
International Communications, April 2016; Hu Yu and Lu Jun, “Experiences and Thoughts
on Construction of Central-Level State-Owned Enterprises Image Abroad” [“央企海外形
象建设的经验与思考”], International Communications, October 2016; “An Examination of
International Public Opinion on One Belt One Road” [“一带一路”议题的国际舆情分析”],
International Communications, May 2017.
134 Hostile Social Manipulation: Present Realities and Emerging Trends
71 For one example of overseas propaganda targeting Chinese abroad, see Ji Deqiang,
“Global Communications for China’s Anti-Corruption Campaign: Problems and Solu-
tions—Based on Real Research of Chinese Students Studying Abroad” [“中国反腐的国际
传播 : 困境与出路 一一基于对在华外国留学生的实证研究”], International Communica-
tions, December 2016.
For examples of Chinese research on global discussion and opinion about China, see
Xiang Debao and Zhang Renwen, “Characteristics of Public Opinions About China on
International Social Media in 2012” [“2012国际自媒体涉华舆情特征”], Journal of Intelli-
gence, Vol. 32, No. 8, 2013, pp. 31–34; Xiang Debao, “Rules, Characteristics and Guidance
Strategy for Public Debate over Tibet in International Social Media and the Public Opinion
Struggle” [“国际自媒体涉藏舆情及舆论斗争的规律、特征及引导策略”], Journal of Intel-
ligence, Vol. 35, No. 5, 2016, pp. 20–26.
72 Kaiman, 2014. According to Kaiman, the human rights nongovernmental organization
Free Tibet “found that the fake accounts had overlapping qualities. Most of their names were
comprised of two Western-sounding first names strung together. About 90 of them were also
closely intertwined—they followed one another and frequently retweeted each other’s posts,
often identical statements and links.”
For more recent research, see Gillian Bolsover, “Computational Propaganda in China:
An Alternative Model of a Widespread Practice,” Computational Propaganda Research Proj-
ect, University of Oxford, Oxford, UK, April 2017.
73Nicholas J. Monaco, “Computational Propaganda in Taiwan: Where Digital Democracy
Meets Automated Autocracy,” Computational Propaganda Research Project, University of
Oxford, Oxford, UK, June 2017.
Hostile Social Manipulation: Chinese Activities 135
78 “Hotels Turn Away South Koreans, Chinese Smash Goods as Missile Row Widens,”
Radio Free Asia, March 13, 2017.
79 Timothy Heath, “Beijing’s Influence Operations Target Chinese Diaspora,” War on the
Rocks, March 1, 2018.
80 Bethany Allen-Ebrahimian, “Chinese Police Are Demanding Personal Information from
Uighurs in France,” Foreign Policy, March 2, 2018a.
Hostile Social Manipulation: Chinese Activities 137
14
Social media
12 Weibo
WeChat
10 Facebook
Twitter
Percentage
0
2009 2010 2011 2012 2013 2014 2015 2016 2017
SOURCE: CNKI 2017 data through October.
81 Yu Mingsong, “Research on United Front Work for Hong Kong Middle Class Profession-
als” [“香港中产专业人士统战工作研究”], United Front Science, March 2017.
82 On the value of Chinese students abroad, see Zhao Liangying and Xu Xiaolin, “Actively
Build China’s National Strategic Communication System” [“积极构建中国国家战略传播
体系”], Media Outpost, September 2016; Han Song and Ping Chuan, “Grasp Important
Points, Explain Difficult Points, Decipher Confusion Points: How to Explain 3rd Plenum
Meeting of 18th Party Congress to Foreigners” [“抓重点 解难点 释疑点—如何做好十八届
三中全会的对外解读”], International Communications, January 2014; Ma Han, “Research
and Opinion on Current Problems in Building China’s International Voice” [“当前中国
国际话语权构建问题研究谫论”], Journal of Yunnan Provincial Committee School of CCP,
December 2016; Bi, 2009; Hou Dongsheng, “Comparison and Analysis of Foreign Propa-
138 Hostile Social Manipulation: Present Realities and Emerging Trends
ganda Related to Tibet Between Chinese Government and Dalai Lama Clique” [“中国政府
与达赖集团在涉藏外宣上的比较和分析”], Journal of Chongqing Institute of Socialism, June
2012.
Foreign students in China are also targets for propaganda indoctrination (Bi, 2009; Yang
Yunsheng, “Research on Foreign Propaganda for the China Dream” [“中国梦海外宣讲研
究”], New Orient, June 2016).
For criticism of overseas Chinese students as poor conduits for influence, see Tan Feng,
“Why China Became the ‘Sacrificial Lamb’ of U.S. Elections” [“中国为何成为美国大选的‘
替罪羊’”], International Communications, September 2016.
83 Cai Yintong, “Study Abroad Students: An Important Force for People-to-People Exter-
nal Propaganda” [“留学生: 民间外宣的重要力量”], International Communications, March
2009; Hou, 2012.
84 Song Shunan, “Gather the Abroad Students to Strengthen the Dream of National
Revival—A Report of the Speech of General Secretary Xi Jinping Learning the 100th Anni-
versary” [“凝聚留学人员力量共筑民族复兴梦想—欧美同学会学习习近平总书记在百年
庆典上的讲话纪实”], China United Front, February 2014.
85 Bethany Allen-Ebrahimian, “China’s Long Arm Reaches into American Campuses,” For-
eign Policy, March 7, 2018b.
Hostile Social Manipulation: Chinese Activities 139
At the same time, the Chinese propaganda system clearly treats the
overseas students as a vector for influencing foreign public opinion.86
Notably, the article touts “Soft Power, Discourse Power, Cultural Iden-
tity, and Ethnic Awareness” as key terms for its content.
86 A 2016 article extols the “irreplaceable special role” of overseas Chinese students in
spreading the Chinese narrative on Tibet, especially their ability to use social media for pro-
paganda against the Dalai Lama, and suggests the government needs to shape their opinions
and guide their propaganda efforts; Qin Yongzhang, “Utilizing Overseas Chinese Students’
Role for Foreign Propaganda Related to Tibet” [“发挥海外中国留学生群体在涉藏外宣工
作中的作用”], International Communications, May 2016. The article further notes that “this
non-governmental propaganda is more ‘flexible’ and ‘lively’ compared to the stereotypical
image of our ‘rigid’ and ‘formulaic’ official propaganda. Their ‘external propaganda’ is more
easily accepted by the majority of foreign citizens. It’s easy to get twice the result with half
the effort.” One example of this is cited in another 2016 article about the role of overseas
Chinese for Chinese soft power that recounts how Chinese alumni and the Chinese student
association, among others, at Cornell University used social media to organize toward lobby-
ing the administration to alter the wording of its congratulatory statement on Tsai Ing-wen’s
victory as Taiwan president to bring it in line with China’s party line. RAND was unable
to verify that the announcement’s wording had actually changed. See Blaine Friedlander,
“Taiwan Elects Its Second Cornell Alumnus as President,” Cornell Chronicle, January 29,
2016; Yi Changjun, “Research on New Overseas Chinese Associations and the Construction
of ‘Soft Power’” [“海外新华侨华人社团与国家 ‘软实力’ 建设研究”], Journal of Huaqiao
University, May 2016.
140 Hostile Social Manipulation: Present Realities and Emerging Trends
flood of online complaints against the schools where the faculty were
employed.87
When officials discern that information disadvantageous to the
party or the government is circulating, the regime swings into action
on social media by blocking or censoring the unauthorized informa-
tion if it lives on a platform over which China has control. Then, once
approved messages have been created and authorized, propaganda
organs flood a variety of social media platforms with messages aimed
at distracting, swamping, or drowning out anti-government arguments
that could provoke social mobilization against the regime.88 Such efforts
are mainly targeted at countering domestic popular action, but insofar
as no real barriers exist that would stop information in China’s online
space from flowing out to the outside world (in contrast with the Great
Firewall that blocks information from the outside world from getting
into China), such actions can have effects on the global discussions of
China that are carried on in Chinese-language media anywhere world-
wide where users access PRC social media platforms.
This effort has, in recent months, extended to foreign compa-
nies as part of a larger crackdown on perceived corporate sympathies
for disputed territorial claims.89 The campaign began in January 2018
when Marriott International, Delta Airlines, and Zara, among others,
were forced to apologize for listing Tibet and Taiwan as separate coun-
tries. Then, a Marriott employee accidently “liked” a Twitter post by a
Tibetan independence group that supported Marriott for listing Tibet
as a separate country, leading to further Chinese criticism and the
employee’s eventual firing.90 The next month, Mercedes-Benz posted a
photo on its Instagram account with a quote from the Dalai Lama, and
87 Josh Horwitz, “Australian Professors and Universities Are Being Shamed into Apologiz-
ing for Offending Chinese Students,” Quartz, August 29, 2017.
88 King, Pan, and Roberts, 2017; RAND Interview #5.
89Richard Bernstein, “The Brands That Kowtow to China,” New York Review of Books,
March 2, 2018.
90 Teddy Ng, “Marriott Sacks Employee Who ‘Liked’ Twitter Post from Tibet Indepen-
dence Group,” South China Morning Post, January 13, 2018; Wayne Ma, “Marriott Employee
Roy Jones Hit ‘Like.’ Then China Got Mad,” Wall Street Journal, March 3, 2018.
Hostile Social Manipulation: Chinese Activities 141
91 Amy B. Wang, “Bowing to Pressure from China, Mercedes-Benz Apologizes for Quoting
the Dalai Lama in Ad,” Washington Post, February 6, 2018.
92 Pei Li and Adam Jourdan, “Mercedes-Benz Apologizes to Chinese for Quoting Dalai
Lama,” Reuters, February 6, 2018; Bernstein, 2018.
93 Paul Mozur, “China Presses Its Internet Censorship Efforts Across the Globe,” New York
Times, March 2, 2018.
94 Separately, the Chinese government also conducts cyberattacks against perceived regime
opponents abroad, such as pro-Tibet groups (Nithin Coca, “The High-Tech War on Tibetan
Communication,” Engadget, June 27, 2015; John Markoff, “Vast Spy System Loots Comput-
ers in 103 Countries,” New York Times, March 28, 2009; “Tracking Ghostnet: Investigat-
ing a Cyber Espionage Network,” Citizen Lab, March 28, 2009; Katie Kleemola, Masashi
Crete-Nishihata, and John Scott-Railton, “Targeted Attacks Against Tibetan and Hong
Kong Groups Exploiting CVE-2014-4114,” Citizen Lab, June 15, 2015.
142 Hostile Social Manipulation: Present Realities and Emerging Trends
95 Michael Forsythe, “Billionaire Who Accused Top Chinese Officials of Corruption Asks
U.S. for Asylum,” New York Times, September 7, 2017b.
96 RAND Interview #3; Paul Mozur, “Facebook Briefly Suspends Account of Outspo-
ken Chinese Billionaire,” New York Times, April 21, 2017; Michael Forsythe, “He Tweeted
About Chinese Government Corruption. Twitter Suspended His Account,” New York Times,
April 26, 2017a; Choi Chi-yuk, “Voice of America Fires Three Staff over Explosive Guo
Wengui Interview,” South China Morning Post, November 15, 2017.
97 Interview #3. The giveaway was apparently that the Russian Twitter bots, while post-
ing slight varieties on a core anti-Guo-themed tweet in Chinese, nonetheless retained their
pseudo-Russian names. Some of the bots reportedly did nothing but spam Guo’s account
with garbled messages that made no sense but that may have been intended merely to over-
whelm the system or make it impossible for real users to get their own messages through.
98 Mike Ives, “Chinese Student in Maryland Is Criticized at Home for Praising U.S.,” New
York Times, May 23, 2017.
Hostile Social Manipulation: Chinese Activities 143
99 Josh Horwitz, “A Chinese Student’s Commencement Speech Praising “Fresh Air” and
Democracy Is Riling China’s Internet,” Quartz, May 23, 2017a; “Student Heckled by Chi-
nese Netizens After Praising US Fresh Air and Free Speech,” Study International, May 24,
2017.
100
Tom Phillips, “Chinese Student Abused for Praising ‘Fresh Air of Free Speech’ in US,”
Guardian, May 23, 2017.
101 For a report on China’s judicial punishment of social media–related dissent, see For-
bidden Feeds: Government Controls on Social Media in China, New York: PEN America,
March 13, 2018.
102 This paragraph draws heavily from Jojje Olsson, “Beware of Chinese Social Media,”
Taiwan Sentinel, September 23, 2017.
144 Hostile Social Manipulation: Present Realities and Emerging Trends
experts suspect his phone was hacked and not that his WhatsApp’s
encryption was broken.103 China’s goal with this approach is very likely
to deter others from making similar social media posts.
107Bethany Allen-Ebrahimian, “Did China Just Ban Maroon 5?” Foreign Policy, July 16,
2015.
108 Zachary Keck, “Justin Bieber Visits Japan’s Yasukuni War Shrine,” The Diplomat,
April 24, 2014.
109 Grace Tsoi, “Why Katy Perry and Gigi Hadid Were Missing from Shanghai’s Victoria’s
Secret,” BBC News, November 20, 2017.
110 Mara Hvistendal, “Inside China’s Vast New Experiment in Social Ranking,” Wired,
December 14, 2017; Josh Chin and Gillian Wong, “China’s New Tool for Social Control: A
Credit Rating for Everything,” Wall Street Journal, November 28, 2016; Rachel Botsman,
“Big Data Meets Big Brother As China Moves to Rate Its Citizens,” Wired, October 21,
146 Hostile Social Manipulation: Present Realities and Emerging Trends
2017; Jiayang Fan, “How China Wants to Rate Its Citizens,” New Yorker, November 3, 2015;
“China Invents the Digital Totalitarian State,” Economist, December 17, 2016.
111Zheping Huang, “China Wants to Build a Credit Score That Dings Online Chat Group
Users for Their Political Views,” Quartz, September 8, 2017.
Hostile Social Manipulation: Chinese Activities 147
tion of Tibet as a separate country, but the authors were unable to cor-
roborate this claim.125
This, however, does not mean bots are not actively posting about
China on Twitter; the Oxford report found that anti-China dissident
groups, including prodemocracy and pro-Tibet activists, have likely
created their own armies of bots to spam Chinese language speakers.126
Moreover, as another Oxford report concludes, “these facts do not pre-
clude usage of malicious political bots in future Chinese propaganda
efforts, but they lead to the conclusion that bots do not currently play
a central role in China’s official propaganda apparatus.”127
125Josh Rogin, “How China Forces American Companies to Do Its Political Bidding,”
Washington Post, January 21, 2018. For the Marriott Twitter post in question, see Marriott
Corporation, tweet, Twitter, January 10, 2018.
126 Bolsover, 2017.
127 Monaco, 2017.
128 “China Is Spending Billions to Make the World Love It,” 2017.
152 Hostile Social Manipulation: Present Realities and Emerging Trends
Figure 4.7
Global Opinion of China
60
50
Percentage
40
30
20
10
0
2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
SOURCE: Pew Research Center, undated.
Figure 4.8
Chinese Journal Articles on Foreign Public Opinion of China
30
25
20
Percentage
15
10
0
2003 2005 2007 2009 2011 2013 2015 2017
SOURCE: CNKI, 2017 data through August. Search term for all Chinese journals was
129 Paul Mozur, “China Spreads Propaganda to U.S. on Facebook, a Platform It Bans at
Home,” New York Times, November 8, 2017b.
130 “Chinese Embassy in US now on Facebook,” China Daily, February 13, 2018.
154 Hostile Social Manipulation: Present Realities and Emerging Trends
Figure 4.9
International Communications Articles on Shaping Foreign Public Opinion
of China
70
60
50
Percentage
40
30
20
10
0
1995 1997 1999 2001 2003 2005 2007 2009 2011 2013 2015 2017
SOURCE: CNKI, 2017 data through August. Search term was “(�����������������������������)
or
and .“
131 “Table of Chinese Foreign Propaganda Accounts for News on Twitter” [“Twitter中国外
宣帐号列表之新闻类”] Medium, November 12, 2017; “Table for Chinese Foreign Propa-
ganda Pages for News on Facebook” [“Facebook中国外宣专页列表之新闻类”], Medium,
September 17, 2017; “Table of Chinese Foreign Propaganda Accounts on Instagram” [“Ins-
tagram中国外宣帐号列表”], Medium, December 19, 2017.
132For a brief discussion of CGTN’s social media strategy, see Yu Xiaoqing, “The Growth
of China’s Outreach Flagship Media: 20 Years of Transformation of an English Anchor-
woman” [“中国外宣旗舰媒体成长记:一位英文女主播的20年蜕变”], The Paper, April 1,
2017.
Hostile Social Manipulation: Chinese Activities 155
Table 4.2
Select Chinese Government and Media Accounts on Western Social Media
Platforms
Twitter Facebook
Account Twitter Account Created Facebook
Created (Year) Followers (Year) Followers
noted that while these accounts currently support China’s broader pro-
paganda efforts, they could easily be used for more malign purposes in
the future, especially a conflict scenario, with a vast audience already
harnessed through peacetime activities.
It is difficult to gauge how much the Chinese government is
spending on propaganda in foreign countries, but one U.S. scholar has
156 Hostile Social Manipulation: Present Realities and Emerging Trends
133 David Shambaugh, “China’s Soft-Power Push,” Foreign Affairs, July 2015.
For various attempts to catalog Chinese propaganda spending, see “China Is Spending
Billions to Make the World Love It,” 2017; Jamie Smyth, “China’s $10bn Propaganda Push
Spreads Down Under,” Financial Times, June 9, 2016; Anne-Marie Brady, “China’s Foreign
Propaganda Machine,” Journal of Democracy, Vol. 26, No. 4, October 2015.
134 Chris Buckley and Jane Perlez, “By Buying Hong Kong Paper, Alibaba Seeks to Polish
China’s Image,” New York Times, December 13, 2015.
135 Kristina Cooke, “China News Agency Leases Plum Times Square Ad Space,” Reuters,
July 26, 2011; Angela Doland, “Watch the Chinese Propaganda Ad Playing 120 Times a Day
in Times Square,” AdAge, July 27, 2016.
136 Mozur, 2017b.
137Nicholas Confessore, Gabriel J. X. Dance, Richard Harris, and Mark Hansen, “The Fol-
lower Factory,” New York Times, January 27, 2018.
Hostile Social Manipulation: Chinese Activities 157
138 Tom Grundy, “Did China’s State-Run News Agency Purchase Twitter Followers?” Hong
Kong Free Press, April 14, 2015; Alexa Olesen, “Where Did Chinese State Media Get All
Those Facebook Followers?,” Foreign Policy, July 7, 2015.
139 Former White House Chief of Staff Reince Priebus in July 2017 suggested that China
and North Korea also interfered with the 2016 election, but this was later clarified to refer
to broader hacking activities not specifically targeted at the election (“Reince Priebus Breaks
Down Trump’s Trip to the G-20 Summit,” Fox News, July 9, 2017; Jason Silverstein, “North
Korea and China Also Interfered in U.S. Election, Reince Priebus Says,” New York Daily
News, July 9, 2017).
140 This was the conclusion too of a New York Times report in early November 2017. See
Mozur, 2017b.
141 Relevant articles include Zhai Huixia, Xie Lianghong, and Yu Yunquan, “New Perspec-
tive on Western Research on ‘China Model’ Since Global Financial Crisis” [“国际金融危
机以来西方对 ‘中国模式’ 研究的新视角”], International Communications, January 2012;
Zhou Xinyu and Feng Bo, “Foreign Communication of Chinese Values Under the Waves
of Populism in the West” [“西方民粹主义浪潮下的中国价值观对外传播”], International
158 Hostile Social Manipulation: Present Realities and Emerging Trends
144 For an overview of Chinese-Americans in the 2016 election, see “Chinese-Americans Are
Becoming Politically Active,” Economist, January 19, 2017.
For an overview of WeChat’s entrance into the United States, see Emily Parker, “Can
WeChat Thrive in the United States?” MIT Technology Review, August 11, 2017.
145 Louise Lucas, “Questions over Pace of Growth As Wechat Nears 1bn Users,” Financial
Times, August 30, 2017; Mengzi Gao, “Chinese Trump Supporters Thank WeChat,” Voices
of New York, November 11, 2016.
146 Esther Wang, “Conservative Chinese Americans Are Mobilizing, Politically and Digi-
tally,” Pacific Standard Magazine, October 11, 2017; Liu Zhen, “How One Chinese Ameri-
can Became Politically Aware . . . and Joined the Ranks of Trump Supporters,” South China
Morning Post, November 2, 2016; Kate Linthicum, “Meet the Chinese American Immi-
grants Who Are Supporting Donald Trump,” Los Angeles Times, May 27, 2016; Andi Wang,
“Meet Some of the Chinese Americans Voting for Trump,” PBS, August 20, 2016; Jessica
Stone, “Chinese-Americans Voters Mobilize Ahead of US Election,” CGTV, November 1,
2016; “独家:用中国社交网 在美华裔组建特朗普支持团,” Sina, May 11, 2016; Stephanie
Zu, “揭秘特朗普最大华裔助选团 组织集资全靠微信,” Sohu, November 6, 2016.
147 Eileen Guo, “How Wechat Spreads Rumors, Reaffirms Bias, and Helped Elect Trump,”
Wired, April 20, 2017.
160 Hostile Social Manipulation: Present Realities and Emerging Trends
more powerful than any other promotion tools,” and the discourse on
WeChat apparently had an impact on the Chinese-language media in
California.148 There are no indications there was any Chinese govern-
ment support or involvement in these activities, but the reliance of U.S.
political activities on Chinese-hosted platforms raises questions about
potential future censorship or manipulation.
Moreover, there is evidence the Chinese government is interested
in activating Chinese-Americans to play a more “positive” role in set-
ting the course of U.S.-China policy, as one article criticized Chinese-
Americans for being a “silent group” in the electoral process and allow-
ing China to become a “scapegoat” for U.S. politics, implicitly arguing
that making them more proud to be Chinese would “improve how
Americans see China.”149 Chinese researchers have also published arti-
cles on factors affecting Chinese-American political participation in
both voting and running for office.150
It is not beyond the imagination to project a future election, at
any level of government and for any country, where the Chinese gov-
ernment orders Chinese-owned social media platforms to censor views
critical of China and/or views critical of China’s preferred candidate.
This censorship would qualify under the report’s definition of hostile
social manipulation because it would seek to have a malign and harm-
ful impact on social discourse about U.S. domestic politics and shape
the election in China’s favor. This would probably be predominantly,
if not exclusively, targeted at Chinese-Americans, due to some of the
population’s consumption of primarily Chinese-language informa-
tion. From a Chinese perspective, such operations might be considered
148Gao, 2016; Grace Wyler, “What Do Chinese-Americans Think of Trump’s Tough China
Talk? We Asked Them,” Los Angeles Daily News, January 4, 2017.
149 Tan, 2016, pp. 13–14.
150 Ye Xiaoli and Gu Haoyu, “The Factors on Election Campaign of Modern Chinese-
American: Based on the Analyses of Sustainability” [“当代美国华人竞选影响因素: 基于可
持续性的分析”], Overseas Chinese Journal of Bagui, September 2017, pp. 31–38; Ye Xiaoli
and Gu Haoyu, “The Analysis for Influence Factors of Chinese-American Political Par-
ticipation: Take the Protesting Action to the Insulting Chinese for an Example” [“当代美
国华人政治参与影响因素分析: 以抗议ABC辱华行动为例”], Overseas Chinese Journal of
Bagui, June 2015, pp. 13–20.
Hostile Social Manipulation: Chinese Activities 161
151 The Canadian internet freedom nongovernmental organization Citizen Lab has found
that WeChat can censor commentary without alerting the sender that his or her message
was not received by the intended recipient. This is currently only for China-based accounts,
and censorship is focused mostly on group chats, but this censorship could very likely be
extended to international accounts (Lotus Ruan, Jeffrey Knockel, Jason Q. Ng, and Masashi
Crete-Nishihata, “One App, Two Systems: How WeChat Uses One Censorship Policy in
China and Another Internationally,” Citizen Lab, November 30, 2016).
152According to the 2010 Census, nearly 3 million people in the United States speak Chi-
nese at home while roughly 850,000 speak Russian. See: “Number of Russian Speakers in
U.S. Quadruples in 30 Years, Census Report Says,” Moscow Times, August 8, 2013.
153 Amy Qin, Twitter, February 1, 2018.
154 Yu Baozhu, “The ‘Chinese Times’ [Huaxia Shibao] Builds a Bridge of China–US Cul-
tural Exchange,” Military Correspondent, January 2012, p. 54.
162 Hostile Social Manipulation: Present Realities and Emerging Trends
159Sarah Cook, “China’s Party Congress Hints at Media Strategy for a ‘New Era,’” The Dip-
lomat, November 4, 2017b.
Hostile Social Manipulation: Chinese Activities 165
160 Wu Feng and Li Yaofei, “The Latest Status and Operation Model of Overseas Anti-
China Media and Countermeasures” [“境外反华媒体的最新态势,及应对策略”], Journal
of Intelligence, March 2017, pp. 36–42.
166 Hostile Social Manipulation: Present Realities and Emerging Trends
1 Laura Rosenberger, “Shredding the Putin Playbook,” Democracy Journal, No. 47, Winter
2018.
2 On this issue, see Carina Storrs, “How Effective Are Misinformation Campaigns to
Manipulate Public Opinion?” Scientific American, September 29, 2017.
167
168 Hostile Social Manipulation: Present Realities and Emerging Trends
3 Braden R. Allenby, “The Age of Weaponized Narrative,” Issues in Science and Technology,
Summer 2017, p. 66, admits that “experts disagree on whether these techniques were decisive
in the Brexit vote or the US election” but then suggests that “that is beside the point.”
Does Hostile Social Manipulation Work? Measures of Success 169
pushed a great deal of content into the public debate and generated
measurable outputs in social media activity, such as views and shares.
In a very few cases, specific outcomes can be identified—such as an
anti-Trump political protest sparked by a Russian Facebook post. But
apart from such anecdotal cases, for the time being we have no authen-
tic way of knowing their larger effect on attitudes or behavior. Most of
what is known so far is in terms of abstract statistics of production and
viewership (what might be termed output measures)—how many posts
were made by Russian-controlled sites, how many people may have
clicked on or “liked” them. This information tells us very little, how-
ever, about beliefs or attitudes: What did people think before they saw
the posts? Did they change their thinking or likely behavior? Even the
basic statistic of retweeting a post does not indicate whether the person
was retweeting it to support or condemn the message.
To gain a better sense of the possible effectiveness of such cam-
paigns, we reviewed available evidence about the effects of known
campaigns. Existing campaigns with measurable data at this point are
almost entirely of Russian origin and focused on the United States and
Europe. Because it is so difficult to disaggregate the effects of social
media or disinformation campaigns from other variables—and even
within those, to identify the specific effects of Russian activities—we
looked at outcome effects that the Russian campaigns might be seeking.
If these campaigns are succeeding, we ought to see movement in the
directions Russia desires in several indicators, including4
4 Another outcome that Russia appears to seek is the strengthening of right-ring or popu-
list parties throughout Europe with some sympathy for Moscow. We investigated this issue
in some depth but concluded that the variables at work in the waxing and waning of those
parties are so complex that the factor does not serve as even a good indirect indicator of the
outcome effects of Russian efforts. Broadly speaking, there is some evidence that the pro-
Russian right-wing parties reached something of a plateau of influence by 2017, but there are
worrisome hints of further growth, especially in places like Germany. That growth, however,
appears to have little direct connection to Russian support or sympathy for Russia in key EU
countries. We therefore have not used the status of right-wing parties as an outcome measure
for Russian social manipulation efforts.
170 Hostile Social Manipulation: Present Realities and Emerging Trends
United States
Multiple public reports suggest that the United States has, especially
since about 2015, been one of Russia’s leading targets for hostile social
manipulation. We evaluated evidence for possible effects of these pro-
grams. The United States and other Western societies have been beset
by several long-term political, social, and economic ills—including
economic insecurity and inequality, gridlocked governance, and rising
partisanship. Polling evidence shows strong signs of these trends across
many issues. The major question is whether we see significant addi-
tional movement since 2014.
Figure 5.1
Favorable Ratings for Russia Among American Public, 2013–2017
80
Percentage favorable 72
70 Percentage unfavorable
67
63
60
50
Percentage
43
40 37
30 29
"% 22
20 19
'"
10
0
2013 2014 2015 2017
5 Unfortunately, Pew began asking this question in only 2017. Jacob Poushter and Doro-
thy Manevich, “Globally, People Point to ISIS and Climate Change as Leading Security
Threats,” Pew Research Center, August 1, 2017. These graphs include results from similarly
phrased questions in consistent polls, which, in some cases, are not comprehensive across all
years but which give a clear sense of trends over time.
Does Hostile Social Manipulation Work? Measures of Success 173
8 Gallup, undated.
9 Kristen Bialik, “Putin Remains Overwhelmingly Unfavorable in the United States,” Pew
Research Center, March 26, 2018.
Does Hostile Social Manipulation Work? Measures of Success 175
10 Robert D. Blackwill and Philip H. Gordon, “Containing Russia, Again,” Foreign Affairs,
January 18, 2018.
11National Security Strategy of the United States of America, Washington, D.C.: The White
House, December 2017, p. 14.
12 Gallup, “Confidence in Institutions,” survey results, 2017.
13 Frank Newport, “Americans’ Confidence in Institutions Edges Up,” press release, Gallup,
June 26, 2017.
176 Hostile Social Manipulation: Present Realities and Emerging Trends
14 Art Swift, “Americans’ Trust in Mass Media Sinks to New Low,” press release, Gallup,
September 14, 2016.
15 Art Swift, “Democrats’ Confidence in Mass Media Rises Sharply from 2016,” press
release, Gallup, September 21, 2017.
16 Pew Research Center, “Partisanship and Political Animosity in 2016,” June 22, 2016.
Does Hostile Social Manipulation Work? Measures of Success 177
National Orientation
Between 2014 and 2017, U.S. national policies, especially in the secu-
rity realm, demonstrated a significant tilt toward greater confrontation
with Russia.
The United States continues to be a leader in NATO, deploying
troops and holding joint drills and exercises intended to send a message
of strength and cohesion to Russia. In 2015, the United States autho-
rized the European Reassurance Initiative (later termed the European
Deterrence Initiative) to provide funding to enhance deterrence and
defense and improve the readiness of forces in Europe. The amount
of American money dedicated to the security of Eastern Europe has
tripled under President Trump, and the number of deployed troops
has also increased.17 The United States deployed 300 troops to Esto-
nia and increased the amount of military equipment provided to the
Estonian government in 2017.18 The United States is leading a mul-
tinational battlegroup in Poland under NATO’s Enhanced Forward
Posture19 and has troops throughout Central and Eastern Europe as
part of Operation Atlantic Resolve.20 Partly in support of such activi-
ties, U.S. defense spending broke a recent trend and began to increase
in the fiscal year 2018 budget proposal.
U.S. policies toward Russia have become consistently more hostile
since 2014. That year, the United States imposed sanctions on Russia
due to Russia’s annexation of Crimea and incursion in Ukraine. Two
successive U.S. National Security Strategies have condemned Russian
aggression and pointed to Russia as a major national security threat;
the 2017 version argued that Russia “challenge[s] American power,
17 Tomáš Valášek, “Trump’s Relationship with NATO, One Year into His Presidency,”
Carnegie Europe, December 28, 2017.
18 Natasha Turak, “Estonia Has No Doubts on Trump’s Commitment to NATO, Says
Prime Minister Juri Ratas,” CNBC, January 26, 2018.
19“NATO’s Enhanced Forward Presence Factsheet,” NATO Public Diplomacy Division,
May 2017.
20 Atlantic Resolve is funded and enabled by the European Reassurance Initiative (America’s
Continued Commitment to European Security: Operation Atlantic Resolve, U.S. Department of
Defense Special Reports, undated).
178 Hostile Social Manipulation: Present Realities and Emerging Trends
25 Grassegger and Krogerus, 2017. This analysis was completed before the release of the
Mueller Report, which offered even more detailed evidence to confirm Russia’s efforts to
influence the 2016 elections. That report clarifies the extent of Russian efforts, but it does
not provide new evidence on their actual effects.
180 Hostile Social Manipulation: Present Realities and Emerging Trends
documents and their release through Wikileaks clearly had the effect
of distracting the leadership of the Clinton campaign in the final days
of the election.26 Campaign officials were scrambling to respond to the
controversies generated by leaked emails when they could have been
taking actions to affect the election results. A more quantifiable effect
of the release can be found in shifting poll numbers after the document
release, which has caused some observers to conclude that this action
alone may have had a significant effect on the election’s outcome.27
Our analysis neither validates nor refutes that hypothesis. Such
numbers and implications are more than reason enough, however, for
the United States to engage in determined efforts to ensure that Russia
or other outside actors cannot manipulate U.S. electoral outcomes. As
we will argue in the section of this report on future scenarios, more-
over, hostile social manipulators are only scratching the surface today
of what might be possible in a decade, and the reasons for concern are
many. At the same time, it is important to understand that the more
directly influence-seeking components of social manipulation cam-
paigns are not magic wands—they have significant limitations, at least
as of today, that constrain the actual effect on attitudes and behavior.
One limitation is the role of other variables in influencing elec-
tion outcomes. Economic insecurity in the United States, for example,
turned out to have been more profound than many understood before
the election, creating a much more viable basis for an insurgent candi-
date than some polling showed going into 2016.28 Social manipulation
efforts can take advantage of such conditions, but they cannot create
them. As one analysis of Russia’s activities concludes, they have “suc-
ceeded in stirring confusion only because there were so many weak-
nesses for them to exploit in the first place.”29
26 Ben Nimmo, “Election Watch: Beyond Russian Impact,” Atlantic Council Digital
Forensic Research Lab, February 27, 2018.
27 This case is made by Kathleen Hall Jamieson, Cyberwar: How Russian Hackers and Trolls
Helped Elect a President, New York: Oxford University Press, 2018.
28 Nate Silver, “It Wasn’t Clinton’s Election to Lose,” 538.com, January 23, 2017.
29 Henry Farrell, “American Democracy Is an Easy Target,” Foreign Policy, January 17,
2018.
Does Hostile Social Manipulation Work? Measures of Success 181
30 Based on voting rates and other baseline assumptions, researchers compiled a rough esti-
mate suggesting that these stories might have affected voting shares by tiny amounts—some-
thing like 0.001–0.005 percent (Allcott and Gentzkow, 2017).
31 See Pew Research Center, “Many Americans Believe Fake News Is Sowing Confusion,”
December 15, 2016b.
32 Shane, 2017a.
182 Hostile Social Manipulation: Present Realities and Emerging Trends
33 Robert Faris, Hal Roberts, Bruce Etling, Nikki Bourassa, Ethan Zuckerman, and Yochai
Benkler, “Partisanship, Propaganda, and Disinformation: Online Media and the 2016 U.S.
Presidential Election,” research paper, Berkman Klein Center for Internet and Society, Har-
vard University, August 2017, p. 5.
34 These data are derived from the testimony of Facebook, Twitter, and Google executives
before the Senate Committee on the Judiciary, October 31, 2017 (U.S. Senate Committee on
the Judiciary, “Extremist Content and Russian Disinformation Online: Working with Tech
to Find Solutions,” subcommittee hearing video, Washington, D.C., October 31, 2017).
Does Hostile Social Manipulation Work? Measures of Success 183
Part of the issue, obviously, is that the overall social media eco-
system is simply vast: 328 million Twitter users, 2 billion Facebook
members, 3.5 billion Google searches per day. Between 2015 and
2017, Facebook sent over 33 trillion stories to peoples’ News Feeds;
each person gets an average of 220 stories per day. In just the six-
week period from September 1, 2016, to November 15, 2016, there
were 16 billion tweets, 189 million of which were identified as being
election-related. The fact that Russian bots and human operators gen-
erated tens of thousands of Facebook posts and tweets sounds impres-
35Aaron Smith and Monica Anderson, “Social Media Use in 2018,” Pew Research Center,
March 1, 2018.
184 Hostile Social Manipulation: Present Realities and Emerging Trends
sive—until one realizes that between 2015 and 2017, Americans were
exposed to 33 trillion Facebook posts alone.36
A more relevant statistic may be the proportion of Russian-
generated messages received by smaller target audiences. Russian bot–
produced tweets or Facebook posts may have been a fraction of the
overall political messages in that period—but were they a much larger
proportion of the tweets and posts viewed by specific potential voters
of a particular political persuasion in specific states? As of this writ-
ing, we simply do not know. There is reason—from both the gen-
eral proportions of Russia-related content and the studies of partisan
content cited above—to doubt that Russian sources would have been
dramatically more influential even with such target audiences. Some
studies, as noted above, also show that some of the most significant
voting swings in 2016 took place among populations with the least
social media exposure. Nonetheless, more research is clearly required
on the specific reach and effect of targeted messaging.
In sum, the available evidence surveyed for this analysis does not
support a definitive judgment of the degree of effect achieved by Rus-
sian social manipulation efforts before the 2016 U.S. election. It does,
however, demonstrate a serious potential threat to the integrity of cur-
rent and future elections if such activities continue and become more
sophisticated. The available evidence also suggests that Russia appears
to have achieved more-direct effects through the theft and release of
documents—a form of political warfare sometimes known as “dox-
fare”—than with social media messages that aimed to shift attitudes
or behavior.
United Kingdom
Outside the United States—and indeed for a longer period and with a
wider range of social manipulation programs—Russia has been target-
ing social stability and democratic processes throughout Eastern and
37 Arguably the most extensive U.S. government statement of the issue is U.S. Senate, Com-
mittee on Foreign Relations, Putin’s Asymmetric Assault on Democracy in Russia and Europe:
Implications for U.S. National Security, Minority Staff Report, January 10, 2018.
38 Unfortunately, Pew began asking this question in only 2017. Margaret Vice, “Publics
Worldwide Unfavorable Toward Putin, Russia,” Pew Research Center, August 16, 2017.
186 Hostile Social Manipulation: Present Realities and Emerging Trends
Figure 5.2
Favorability Ratings of Russia Among British Public, 2013–2017
70
66
Percentage favorable 63
60 Percentage unfavorable 59
Percentage don’t know
50
Percentage
40 38 39
30
25 26
23
20 18
16 15
12
10
0
2013 2014 2015 2017
Russia as a serious threat. Unlike France and Germany, the British are
slightly more confident that Trump would do the right thing in world
affairs (22 percent) than they are that Putin would (19 percent). As in
the United States, then, there has been some variation since 2014 but
not a dramatic recovery from the precipitous drop in favorable atti-
tudes toward Russia beginning in 2014. And as in the United States,
in the wake of renewed Russian provocations—and particularly, in the
British case, of the alleged poisoning of former Russian citizens in the
United Kingdom—these attitudes worsened in 2018, with over 60 per-
cent of Britons saying Russia was a threat to world peace. And again,
these are all far lower than attitudes as recently as 2011, when 50 per-
cent of Britons held a favorable view of Russia.39
In the meantime, public support for NATO has remained strik-
ingly stable over the past several years, with about 60 percent of the
British public indicating a favorable attitude and only 20 percent
saying they had an unfavorable view (with the remainder saying they
40 Stokes, 2017.
41 “EU Referendum Results,” BBC News, 2016.
42 “The Vote to Leave the EU,” British Social Attitudes, Vol. 34, National Centre for Social
Research, undated, p. 2.
43 “The Vote to Leave the EU,” undated, p. 16.
188 Hostile Social Manipulation: Present Realities and Emerging Trends
advantage over the other, and that there was a significant segment of
the electorate that was undecided and could potentially be influenced.
A poll conducted in early June showed Remain at 44 percent, Leave
at 42 percent, and those who did not know how they would vote at
13 percent.50
Second, while some research suggests that “relatively few” Brit-
ish felt strongly committed to a European identity to begin with,51
Figure 5.3 shows a significant increase in the percentage of the public
wishing to leave the European Union between 2015 and 2016. This
appears to indicate some late changes in attitude, when a social manip-
ulation campaign might have been underway.
Third, a considerable proportion of voters were “fence-sitters,” not
strongly committed to a side until days before, or even the day of, the
Figure 5.3
British Attitude Toward Relationship with European Union
30
Percentage
26
25 24
22
20 19 18
19
16
15
10
10 8
6
5 3
4 3 4
2
0
2013 2014 2015 2016
50 Toby Helm, “Third of EU Referendum Voters Won’t Make Up Their Minds Until Week
Before Poll,” The Guardian, June 11, 2016.
51 “The Vote to Leave the EU,” undated, p. 20.
190 Hostile Social Manipulation: Present Realities and Emerging Trends
52 Michael Bruter and Sarah Harrison, The Impact of Brexit on Consumer Behavior, Lansons,
London School of Economics, and Opinium, June 9, 2016. The report also says that argu-
ments put forward by the Leave camp are met by voters with more skepticism than those
advanced by those in Remain, even among those who say they back Brexit.
53 Panagiotis T. Metaxas and Eni Mustafaraj, “Social Media and the Elections,” Science,
Vol. 338, No. 6106, October 2012.
54 Bruter et al., 2016.
55 Office for National Statistics, “Internet Access—Households and Individuals: 2017,”
August 3, 2017a, chapter 7.
56 Office for National Statistics, “Social Media Usage in the United Kingdom,” Statista
Dossier, August 2017b, p. 7.
Does Hostile Social Manipulation Work? Measures of Success 191
National Orientation
Under the governing Conservative Party (in power since 2010), Great
Britain’s basic national orientation has remained staunchly pro-NATO
and pro-West, with repeated reaffirmations of the alliance with the
United States and a growing hostility toward Russian aggression. Like
the United States, Great Britain has become increasingly confronta-
tional toward Russia since 2014.
In terms of defense spending, the British defense budget has
remained steady since 2014, at between 2.19 and 2.14 percent of gross
domestic product (GDP). In 2015, the British government committed
to continuing to meet NATO’s member defense spending target of
63 “Russian Twitter Trolls Meddled in the Brexit Vote. Did They Swing It?” The Economist,
November 23, 2017. The Senate Foreign Relations Committee report, Putin’s Asymmetric
Assault on Democracy (U.S. Senate, Committee on Foreign Relations, 2018), discusses the
UK case (pp. 116–119) but offers no meaningful evidence of a significant Russian campaign
or any effect on voting.
64 Based on information from “Number of Twitter Users in the United Kingdom (UK)
from 2012 to 2018 (in Million Users),” Statista, 2017; and Stuart Dredge, “More Than One-
Fifth of Britons Will Use Twitter This Year, Claims Report,” The Guardian, February 20,
2014.
65 “Market Share Held by the Leading Social Networks in the United Kingdom (UK) as of
July 2017,” Statista, 2017, p. 12.
Does Hostile Social Manipulation Work? Measures of Success 193
66 “Defence Budget Increases for the First Time in Six Years,” press release, United King-
dom Ministry of Defence, April 1, 2016.
67 “British Troops Land in Estonia for Nato Mission to Deter Russia,” The Guardian,
March 18, 2017.
68 “Defence Secretary Steps up UK Commitments to NATO,” press release, United King-
dom Ministry of Defence, June 29, 2017.
69 Arthur Beesley, “EU Sets Timetable for Tighter Military Coordination,” Financial Times,
June 22, 2017.
70“UK and France Commit to New Defence Cooperation,” press release, United Kingdom
Ministry of Defence, January 18, 2018.
194 Hostile Social Manipulation: Present Realities and Emerging Trends
France
Figure 5.4
Favorability Ratings of Russia Among French Republic, 2013–2017
80 Percentage favorable
Percentage unfavorable
73
70
70
64
62
60
50
Percentage
40
36 36
30
30
26
20
10
0
2013 2014 2015 2017
National Elections
Evidence from the French election suggests that the #MacronLeaks
disinformation campaign was ineffective because it did not reach the
only high-value community (if the goal was influencing the election):
French citizens of voting age. In the days before the runoff, online
alt-right communities collaborated to manufacture and allegedly steal
72 Vice, 2017.
73 Unfortunately, Pew began asking this question in only 2017.
196 Hostile Social Manipulation: Present Realities and Emerging Trends
Table 5.1
Comparing Metrics of Social Manipulation Efforts and Effects in Rounds
One and Two of French Election
Round One Round Two
Percentage of election-related Twitter 7.2% 16%
traffic driven by bots
Number of bots driving traffic about 100 500
each candidate
Ratio of links to professional news to 2:1 1:1
links to nonprofessional news
Percentage of election-related traffic 4% 6%
classified as “junk news”
81 OII did not analyze the content or valance of specific tweets, so it is not possible to deter-
mine whether these automated accounts were pushing positive or negative information about
the candidates or whether they were likely being run by the campaign itself or a saboteur.
82 Desigaud et al., 2017, p. 3.
83 Note that while “junk” or “fake” news is included in this category, this category also
includes civil society content and personal blogs. Much of this category is composed of
thoughtful work produced by civil society and individuals discussing political issues. About
21 percent of this category was judged to be junk news (Desigaud et al., 2017).
Does Hostile Social Manipulation Work? Measures of Success 199
Figure 5.5
Political Content Shared by Twitter Users in Rounds One and Two of the
2017 French Election
50
35
30
Percentage
25
20
15
10
0
Round One Round Two
by users in this sample led to either what OII classifies as “junk news”
based on misinformation84 or to content produced by known Rus-
sian sources of political information. However, there was a noticeable
shift in the second sample; users shared a lower proportion of credible
sources and a slightly higher proportion of fake news.
Overall, Twitter users discussing French politics proved less sus-
ceptible to spreading misinformation and fake news than users dis-
cussing American, German, or British politics.85 Figure 5.6 compares
the prevalence of several types of political content shared on Twitter
shortly before elections in France (round two), Germany (September
84 “This content includes various forms of propaganda and ideologically extreme, hyper-
partisan, or conspiratorial political news and information. Much of this content is deliber-
ately produced false reporting” (Desigaud et al., 2017, p. 3).
85 Monica Kaminska, John D. Gallacher, Bence Kollanyi, Taha Yasseri, and Philip N.
Howard, “Social Media and News Sources During the 2017 UK General Election,” Data
Memo 2017.6, Oxford, UK: Project on Computational Propaganda, Oxford Internet Insti-
tute, University of Oxford, June 5, 2017, p. 6.
200 Hostile Social Manipulation: Present Realities and Emerging Trends
Figure 5.6
Political Content Shared by Twitter Users in Several Elections (in
percentages)
60
Professional news content
Professional political content
50 Junk news
40
Percentage
30
20
10
0
2017 2017 Michigan 2017 UK
French German voters 2016 general
presidential parliamentary U.S. election
election elections presidential
round two election
86 Lisa-Maria Neudert, Bence Kollanyi, and Philip N. Howard, “Junk News and Bots
During the German Parliamentary Election: What Are German Voters Sharing over Twit-
ter?” Data Memo 2017.7, Oxford, UK: Project on Computational Propaganda, Oxford Inter-
net Institute, University of Oxford, September 19, 2017.
87 Howard et al., 2017.
88 Kaminska et al., 2017.
Does Hostile Social Manipulation Work? Measures of Success 201
Evaluating Impact
If we assume, as many observers do, that France resisted social manipu-
lation efforts effectively, we can draw out some interesting thoughts for
future research. With limited metrics and information, some potential
reasons for French resistance stood out: the relatively lower penetra-
tion of social media in France compared with other countries recently
plagued by social manipulation efforts and the improved awareness of
social manipulation and the capacity to combat it among social media
companies and traditional media (which France enjoyed because other
countries, such as the United Kingdom and the United States, were hit
by alleged social manipulation first). Finally, the runoff vote was not
close, with Macron garnering about double the votes of Le Pen. Even
if disinformation campaigns did succeed in persuading some voters to
support Le Pen, it was clearly not enough to change the outcome of the
election.
One reason may be that social media use in France is significantly
lower than in the United States or United Kingdom.91 Only 40 to 45
percent of the French population used social media in 2016.92 In com-
parison, 81 percent of Americans93 and 64 to 66 percent of the British
population has a social media account.94 Given that so much of social
manipulation is believed to occur on social media networks, this diver-
gence is notable and worthy of further examination. Susan Banducci,
a social scientist at the University of Exeter, does make the important
point that misinformation spread on social media may have “second-
order influence” beyond the immediate audience. Journalists could
perceive “bot-boosted” messages as a shift in the public mood, or bots
could push unsubstantiated rumors into the credible media, thus influ-
encing the wider public.95 However, this risk is likely decreased the
lower the proportion of the population that actively uses social media,
at least in part because the media understand this fact and know Twit-
ter cannot give the pulse of the entire French public.
A second reason for the limited effects in French elections may
be that social media companies such as Facebook were more aware of
the dangers of fake news and social manipulation by spring 2017 and
took steps to combat them. Better policing and proactive responses
to fake accounts or news stories by Facebook may have helped France
91 All of the data on social media use are sourced from Statista. Statista provides access to
statistics and studies gathered by market researchers, trade organizations, scientific publica-
tions, and government sources.
92 French use of social media was more difficult to pinpoint than that of other countries,
though most estimates fell within the 40 to 45 percent range. This range is provided by the
following sources: “Share of Individuals in France Participating in Social Networks from
2011 to 2016,” Statista, 2017; “Number of Social Network Users in France from 2014 to
2018 (in Millions),” Statista, 2017; “Social Network Usage in France,” Statista Dossier, 2017.
93 “Percentage of U.S. Population with a Social Media Profile from 2008 to 2017,” Statista,
2019.
94 “Social Media Usage in the United Kingdom,” Statista Dossier, August 2017, p. 7.
95 Chris Baraniuk, “Beware the Brexit Bots: The Twitter Spam out to Swing Your Vote,”
New Scientist, June 22, 2016.
Does Hostile Social Manipulation Work? Measures of Success 203
96 Tom Regan, “Facebook Helped Blunt Russian Meddling in French Elections,” Engadget,
July 27, 2017.
97 Shabnam Shaik, “Improvements in Protecting the Integrity of Activity on Facebook,”
Facebook, April 12, 2017.
98 Eric Auchard and Joseph Menn, “Facebook Cracks Down on 30,000 Fake Accounts in
France,” Reuters, April 13, 2017.
99 Joseph Menn, “Exclusive: Russia Used Facebook to Try to Spy on Macron Campaign—
Sources,” Reuters, July 27, 2017.
100 Jessica Davies, “Le Monde Identifies 600 Unreliable Websites in Fake-News Crackdown,”
Digiday, January 25, 2017.
204 Hostile Social Manipulation: Present Realities and Emerging Trends
right voters (partly because the FN party has been around for decades),
indicated that this would not be a close race.101 Polling suggested that
Macron would win by a decent margin; he received about two-thirds of
the vote to Le Pen’s one-third. Even if a considerable number of people
were influenced enough by misinformation campaigns to shift their
vote (which is entirely possible), it would likely not have been enough
to change the outcome.
101
Emily Schultheis, “What Went Right with the French Campaign Polls?” The Atlantic,
May 13, 2017.
102 Pierre Tran, “France Adds $2B to Defense Budget, Moving Closer to NATO Spending
Target,” Defense News, September 27, 2017.
Does Hostile Social Manipulation Work? Measures of Success 205
103Alice Pannier, “Between Autonomy and Cooperation: The Role of Allies in France’s New
Defense Strategy,” War on the Rocks, November 2, 2017.
104 Stokes, 2017.
105 “Sorbonne Speech of Emmanuel Macron—Full Text/English Version,” 2017.
106“President Macron’s Initiative for Europe: A Sovereign, United, Democratic Europe,”
French Ministry for Europe and Foreign Affairs, September 26, 2017.
107 “UK and France Commit to New Defence Cooperation,” 2018.
206 Hostile Social Manipulation: Present Realities and Emerging Trends
Germany
108 Michael Peel, “EU States Poised to Agree Joint Defence Pact,” Financial Times, Novem-
ber 7, 2017.
109 “Twenty-Five EU States Sign PESCO Defense Pact,” Deutsche Welle, December 11, 2017.
Figure 5.7
Favorability Ratings for Russia Among German Public, 2013–2017
80
70 79
70
60 67
60
Percentage Percentage
50
favorable unfavorable
Percentage
40
30
32
20 27 27
19
10
0
2013 2014 2015 2017
112 Unfortunately, Pew began asking this question in only 2017. German firm Bertelsmann
Stiftung found similar results in a 2016 poll: About 38 percent of Germans perceived Russia
to be a threat. Gabriele Scholer, “Russia—A Threat to European Security? A View from Ger-
many,” Bertelsmann Stiftung, October 1, 2016.
208 Hostile Social Manipulation: Present Realities and Emerging Trends
able views.113 A plurality (39 percent) of the group that indicated favor-
able views of Russia was between 18 and 29 years old (this is also typi-
cally the age group with the highest use of social media, which is an
interesting correlation). This age group is also the most likely to believe
that Russia respects the personal freedoms of its people.
Public support for NATO has increased in Germany over the past
few years, as noted in Figure 5.8.118 However, only a minority of Ger-
mans (40 percent) would support their country using force to defend
a NATO ally if it were to become engaged in a “serious military con-
flict” with Russia, making Germany less supportive than countries like
France (53 percent would support) and the United States (62 percent
would support).
In 2015, the German government pledged to increase defense
spending and overhaul its security strategy in the coming years in
response to Russian attempts to use “power politics and military force”
to assert its interests.119 Germany has recently led efforts to increase EU
defense cooperation, which culminated in the signing of the PESCO
Figure 5.8
Favorability Ratings for NATO Among German Public, 2013–2017
80
Percentage Percentage Percentage
70 favorable unfavorable don’t know
60
50
Percentage
40
30
20
10
0
2013 2015 2016 2017
118Bruce Stokes, “NATO’s Image Improves on Both Sides of Atlantic,” Pew Research
Center, May 23, 2017.
119“Germany Says New Security Strategy Will Respond to Russia,” Reuters, February 17,
2015.
210 Hostile Social Manipulation: Present Realities and Emerging Trends
issue: 14.5 percent of those polled said “life in Latvia is very bad,” and
almost 30 percent agreed that the society discriminates against those
who do not know Latvian. Some polls that distinguished households
that specifically spoke Russian at home showed higher numbers—
28 percent saying they felt a belonging to Russia, 41 percent saying
Russian intervention was required to protect Russian speakers in
Latvia, and over half believing Russia’s claim that the Latvian govern-
ment was pursuing a “restoration of fascism.” Evidence suggests that
economic variables, such as unemployment, play a major role in deter-
mining Russian speakers’ views of the society.126
Table 5.2
Recent Security Initiatives in the Baltics
Estonia • 2014: Estonian President Toomas Hendrick Ilves stated that “The current
security architecture in Europe, which relied on both the Helsinki Final
Act and the Paris Charter, has now collapsed, following Russia’s aggres-
sion in Ukraine.”a
• 2015: Estonian Air Force announced planned expansion of the Amari air
base to allow for additional NATO aircraft.
• 2016: Commander of Estonian defense forces, Lt General Riho Terras,
stated that Patriot missile defense systems are needed in the Baltic
states to deter a Russian invasion.
• 2017: Estonian national security concept stated that Estonia will con-
tinue to work closely with NATO and the European Union in the face of
Russia’s unpredictable, aggressive and provocative activity.
• 2017: Baltics signed a joint plan to simplify bureaucratic barriers and
facilitate the movement of NATO forces in the region.
Latvia • 2014: Riga (Latvia’s capital) mayor Nils Usakovs claimed that Putin has
brought stability to the region and is the best Russian president Latvia
can have at the moment.
• 2015: Latvia’s national security concept stated that Russia’s actions
have created long-term negative effects on the national security of the
Republic of Latvia.
• 2016: Latvia increased its defense budget by 42%, though still below
the 2% of GDP target set for NATO members.
• 2016: Latvia’s national defense concept stated that in recent years,
Russia had employed a number of methods to erode the security of
Latvia. 2012 iteration noted cooperation with Russia is a security and
stability strengthening aspect of the Baltic Sea region.
• 2016: Latvia promised to increase defense spending to 2% of GDP by 2018.
• 2017: Baltics signed a joint plan to simplify bureaucratic barriers and
facilitate the movement of NATO forces in the region.
Lithuania • 2014: Since 2014, Lithuania’s defense spending has steadily increased
each year.
• 2014: Lithuania established a rapid reaction force to address a potential
hybrid warfare scenario by Russia.
• 2014: Lithuania’s national security threat assessment stated that one of
the primary areas of threats emanates from Russian foreign policies.
• 2016: Lithuania increased its defense budget by 34%, though still below
the 2% of GDP target set for NATO members.
• 2016: Lithuania permanently reinstated a 9-month conscription service
to fully equip military units and prepare sufficient reserves.
Lithuania • 2016: Lithuanian national security threat assessment cited Russia’s impe-
rialistic ambitions and aggressive foreign policy as one of the greatest
national security threats to Lithuania.
• 2017: Baltics signed a joint plan to simplify bureaucratic barriers and
facilitate the movement of NATO forces in the region.
• 2017: Lithuania’s national security strategy stated that, “The main
threat for the security of the Republic of Lithuania is posed by aggres-
sive actions of the Russian Federation.”b
• 2017: Ahead of the 2017 Russian Zapad military exercise, Lithuania
constructed a high wire fence along its border with Kaliningrad to help
prevent provocations.
a Jeremy Bender, “Estonian President: Europe’s Security Architecture ‘Has Collapsed,’”
Business Insider, September 19, 2014.
b “National Security Strategy,” Republic of Lithuania, January 17, 2017.
214 Hostile Social Manipulation: Present Realities and Emerging Trends
In the past few years, Latvia has been involved in several initiatives
to enhance cooperation between various EU member states. Latvia is
among the 25 EU member states that signed the PESCO defense agree-
ment in December 2017. In 2017, Latvia and Norway signed memo-
randa of understanding to cooperate on various economic and climate
issues.135 Both Latvia and Estonia are participants in the Interreg Baltic
Sea Region Programme 2014–2020, an EU-funded program to encour-
age greater cooperation and innovation among countries in the region.136
In 2016, Latvia hosted the fifth summit of heads of government of Cen-
tral and Eastern European Countries and China (16+1).137
Poland
135 “Closer Cooperation with Latvia,” Mission of Norway to the European Union, Decem-
ber 14, 2017.
136 The program is an agreement between EU member states Denmark, Estonia, Finland,
Latvia, Lithuania, Poland, Sweden, and the northern parts of Germany, as well as partner
countries Norway, Belarus, and the northwest regions of Russia (“About the Programme,”
Interreg Baltic Sea Region, fact sheet, Rostock, Germany, undated).
137 The 16+1 is an initiative by the People’s Republic of China to expand cooperation with
11 EU member states and 5 Balkan countries (Albania, Bosnia and Herzegovina, Bulgaria,
Croatia, the Czech Republic, Estonia, Hungary, Latvia, Lithuania, Macedonia Montenegro,
Poland, Romania, Serbia, Slovakia, and Slovenia). Ministry of Foreign Affairs of the Repub-
lic of Latvia, “16+1 Summit Has Concluded,” press release, February 22, 2017.
216 Hostile Social Manipulation: Present Realities and Emerging Trends
Figure 5.9
Favorability Ratings of Russia Among Polish Public, 2013–2017
90
Percentage favorable
Percentage unfavorable 81 80
80
69
70
60
54
Percentage
50
40
36
30
21
20
15
12
10
0
2013 2014 2015 2017
Figure 5.10
Favorability Ratings for NATO Among Polish Public, 2013–2017
90
80
70
60
Percentage
50
Percentage Percentage Percentage
40 favorable unfavorable don’t know
30
20
10
0
2013 2015 2016 2017
the European Commission (EC) took Poland (as well as Hungary and
the Czech Republic) to the EU Court of Justice over the countries’
refusal to accept the EU refugee resettlement plan. A small majority
(51 percent) of Poles recently said that Poland should continue to refuse
Muslim refugees even if it resulted in losing EU membership.148
The PiS-led government’s recent democratic rollbacks have
prompted threatened reprisals of an unprecedented severity from the EC
(the EU executive arm), which has put serious strain on Poland’s rela-
tionship with the European Union. The EC began an investigation into
rule of law violations in Poland in January 2016. The Polish government
rejected the EC’s recommendations to improve rule of law in Poland as
interference in sovereign Polish affairs. In December 2017, the EC pro-
posed invoking Article 7 of the Treaty of the European Union (intended
to ensure that EU member states respect human dignity and democracy)
against Poland, which would be an unprecedented disciplinary step. If
this proposal were to escalate and receive unanimous support from other
EU members (which is unlikely), Poland could lose its voting rights.149
If the goal of the recent Russian campaign has been to affect strategic
positioning of states and populations and boost support for policies
Russia desires, that effort appears to be largely failing. One scholar
who is extremely concerned about Russian disinformation efforts
nonetheless argued that “so far, the impact of Russian active mea-
sures in Europe appears to have been somewhat hit-and-miss—with
an emphasis on ‘miss.’ Certainly, none of the past year’s elections
have yielded outcomes favorable to the Kremlin; in fact, European
voters . . . have been mostly hewing to the main.” Some egregious
examples of Russian fake news, such as the “Our Lisa” story about a
148Results from a July 2017 poll (Polish language) summarized in Remi Adekoya, “Why
Poland’s Law and Justice Party Remains So Popular,” Foreign Affairs, November 3, 2017.
149 Marek Strzelecki and Ewa Krukowska, “EU Sanctions Risk for Poland Rises on Demo-
cratic Backsliding,” Bloomberg, December 20, 2017.
220 Hostile Social Manipulation: Present Realities and Emerging Trends
Figure 5.11
Percentage of Various Publics with Favorable View of European Union
80
70
60
50
Percentage
40
30
20 France Russia
Germany U.S.
10 UK
0
2010 2012 2013 2014 2015 2016 2017
150 Quotations in this paragraph are from Constanze Stelzenmüller, “The Impact of Russian
Interference on Germany’s 2017 Elections,” testimony before the U.S. Senate Select Com-
mittee on Intelligence, June 28, 2017.
Does Hostile Social Manipulation Work? Measures of Success 221
Figure 5.12
Percentage of Various Publics with Favorable View of NATO
80
70
60
50
Percentage
40 France Russia
30 Germany U.S.
UK
20
10
0
2010 2012 2013 2014 2015 2016 2017
151This evidence is drawn from EC, “Special Eurobarometer 461: Designing Europe’s
Future,” April 2017.
222 Hostile Social Manipulation: Present Realities and Emerging Trends
152 Lucan Ahmad Way and Adam Casey, “Russia Has Been Meddling in Foreign Elections
for Decades. Has It Made a Difference?” Washington Post, January 8, 2018. The compre-
hensive study is Way and Casey, “Is Russia a Threat to Western Democracy? Russian Inter-
vention in Foreign Elections, 1991–2017,” draft memo prepared for conference on Global
Populisms as a Threat to Democracy? Stanford University, November 3–4, 2017.
153
Leonid Peisakhin and Arturas Rozenas, “When Does Russian Propaganda Work—and
When Does It Backfire? Here’s What We Found.” Washington Post, April 3, 2018.
154 Stelzenmüller, 2017.
Does Hostile Social Manipulation Work? Measures of Success 223
Table 5.3
NATO Collective Initiatives Since 2014
2014 • NATO agreed to establish a Very High Readiness Joint Task Force to
have the capability to better respond to a Ukraine scenario.
• NATO suspended all practical civilian and military cooperation with
Russia within the NATO-Russia Council.
• NATO’s Strategic Communications Centre of Excellence was devel-
oped and received its first task: to study Russia’s information cam-
paign against Ukraine.
• NATO stated that the developments in and around Ukraine are seen
to constitute a threat to neighboring allied countries, with serious
implications for security and stability.
Our research into the character and evolving nature of hostile social
manipulation supports several broad conclusions.
First, the United States needs an updated framework for organiz-
ing its thinking about the complex issues involved with manipulation of
infospheres by foreign powers determined to gain competitive advantage.
Chapter Two offers such a revised framework, in an effort to put social
manipulation into a broader context of information competition. One
challenge is that many concepts and terms overlap with one another
in confusing ways; cybertools, for example, can be employed as part
of social manipulation efforts or in diverse ways—directing attacks on
infrastructure targets, for example. Traditional military concepts such
as information operations, psychological operations, and military sup-
port to information operations do not begin to capture the full scope of
hostile social manipulation. A coherent framework for organizing the
various components of the challenge is the first step toward improved
policy.
Second, it is now clear that leading autocratic states have begun to
employ information channels for competitive advantage; these plans remain
in their initial stages and could unfold in several ways. States such as
Russia and China appear to view such techniques as a source of lever-
age relative to open societies. They believe themselves to be engaged
in an information war with the West—one begun by the United
States and its friends and allies—and have begun to invest significant
resources in such tools. They see many forms of information competi-
tion as parts of an overarching, holistic competitive space and pay less
225
226 Hostile Social Manipulation: Present Realities and Emerging Trends
231
232 Hostile Social Manipulation: Present Realities and Emerging Trends
Allenby, Braden R., “The Age of Weaponized Narrative,” Issues in Science and
Technology, Summer 2017.
Allen-Ebrahimian, Bethany, “Did China Just Ban Maroon 5?” Foreign Policy,
July 16, 2015.
———, “Chinese Police Are Demanding Personal Information from Uighurs in
France,” Foreign Policy, March 2, 2018a. As of April 30, 2019:
http://foreignpolicy.com/2018/03/02/chinese-police-are-secretly-demanding-
personal-information-from-french-citizens-uighurs-xinjiang/
———, “China’s Long Arm Reaches into American Campuses,” Foreign Policy,
March 7, 2018b. As of May 5, 2019:
http://foreignpolicy.com/2018/03/07/chinas-long-arm-reaches-into-american-
campuses-chinese-students-scholars-association-university-communist-party/
Alliance for Securing Democracy, Securing Democracy Dispatch, December 18,
2017.
America’s Continued Commitment to European Security: Operation Atlantic Resolve,
U.S. Department of Defense Special Reports, undated. As of May 7, 2019:
https://dod.defense.gov/News/Special-Reports/0218_Atlantic-Resolve
Anbin, Shi, and Peiyan Wang, “Stealth Propaganda: Concepts Evolution,
Strategy” [“隐性宣传:概念·演进·策略”], International Communications, January
2016.
Anderlini, Jamil, and Jamie Smyth, “West Grows Wary of China’s Influence
Game,” Financial Times, December 19, 2017. As of April 29, 2019:
https://www.ft.com/content/d3ac306a-e188-11e7-8f9f-de1c2175f5ce
“An Ex St. Petersburg ‘Troll’ Speaks Out: Russian Independent TV Network
Interviews Former Troll at the Internet Research Agency,” Meduza, October 15,
2017. As of April 17, 2019:
https://meduza.io/en/feature/2017/10/15/an-ex-st-petersburg-troll-speaks-out
“An Examination of International Public Opinion on One Belt One Road” [“一带
一路”议题的国际舆情分析”], International Communications, May 2017.
“Anonymous Ukraine Klitschko E-mails and Nuland/Pyatt Dialogue Prove
US-Backed Coup,” Sputnik (then Voice of Russia), February 25, 2014.
Arabie, P., and Y. Wind, “Marketing and Social Networks,” in Advances in
Social Networks Analysis, S. Wasserman and J. Galackiewicz, eds., London: Sage
Publications, 1994, pp. 254–273.
“Are HIV/AIDS Conspiracy Beliefs a Barrier to HIV Prevention Among African
Americans?” Journal of Acquired Immune Deficiency Syndromes, Vol. 38, No. 2,
February 1, 2005, pp. 213–218.
Bibliography 233
Bernstein, Richard, “The Brands That Kowtow to China,” New York Review of
Books, March 2, 2018. As of May 5, 2019:
http://www.nybooks.com/daily/2018/03/02/the-brands-that-kowtow-to-china/
Besemeres, John, A Difficult Neighbourhood: Essays on Russia and East-Central
Europe Since World War II, Canberra: Australian National University Press, 2016.
Bi Lifu, “Innovating Foreign Propaganda in Ports and Improving Inner Mongolia’s
Image,” Theory Construction, 2009.
Bialik, Kristen, “Putin Remains Overwhelmingly Unfavorable in the United
States,” Pew Research Center, March 26, 2018. As of October 12, 2018:
http://www.pewresearch.org/fact-tank/2018/03/26/
putin-remains-overwhelmingly-unpopular-in-the-united-states/
Biden, Joe, and Michael Carpenter, “How to Stand Up to the Kremlin,” Foreign
Affairs, December 2017.
Bittman, Ladislav, The KGB and Soviet Disinformation: An Insider’s View,
Washington, D.C.: Pergamon-Brassey’s, 1985.
Blackwill, Robert, and Kurt Campbell, Xi Jinping on the Global Stage: Chinese
Foreign Policy Under a Powerful but Exposed Leader, Council Special Report
No. 74, New York: Council on Foreign Relations, February 25, 2016.
Blackwill, Robert D., and Philip H. Gordon, “Containing Russia, Again,” Foreign
Affairs, January 18, 2018. As of May 7, 2019:
https://www.foreignaffairs.com/articles/russian-federation/2018-01-18/
containing-russia-again
Bland, Ben, “China Censorship Drive Splits Leading Academic Publishers,”
Financial Times, November 4, 2017.
“BM Not My President Rally,” Twitter post, U.S. House of Representatives
Permanent Select Committee on Intelligence, 2017. As of April 19, 2019:
https://democrats-intelligence.house.gov/uploadedfiles/6056284937087.pdf
Bodner, Matthew, Matthew Kupfer, and Bradley Jardine, “Welcome to the
Machine: Inside the Secretive World of RT,” Moscow Times, June 1, 2017. As of
April 24, 2019:
https://www.themoscowtimes.com/2017/06/01/
welcome-to-the-machine-inside-the-secretive-world-of-rt-a58132
Boghardt, Thomas, “Soviet Bloc Intelligence and Its AIDS Disinformation
Campaign,” Studies in Intelligence, Vol. 53, No. 4, December 2009. As of April 19,
2019:
https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/
csi-studies/studies/vol53no4/pdf/U-%20Boghardt-AIDS-Made%20in%20the%20
USA-17Dec.pdf
236 Hostile Social Manipulation: Present Realities and Emerging Trends
Buckley, Chris, “China Takes Aim at Western Ideas,” New York Times, August 19,
2013. As of April 28, 2019:
http://www.nytimes.com/2013/08/20/world/asia/chinas-new-leadership-takes-
hard-line-in-secret-memo.html
Buckley, Chris, and Jane Perlez, “By Buying Hong Kong Paper, Alibaba Seeks to
Polish China’s Image,” New York Times, December 13, 2015. As of May 6, 2019:
https://www.nytimes.com/2015/12/14/world/asia/alibaba-south-china-morning-
post-hong-kong.html
Bumiller, Elizabeth, and Michael Wines, “Test of Stealth Fighter Clouds Gates
Visit to China,” New York Times, January 11, 2011.
Burgess, Matt, “Here’s the First Evidence Russia Used Twitter to Influence Brexit,”
Wired, November 10, 2017.
Bytwerk, Randall L., “Believing in ‘Inner Truth’: The Protocols of the Elders of Zion
in Nazi Propaganda, 1933–1945,” Holocaust and Genocide Studies, Vol. 29, No. 2,
2015.
Cadell, Cate, “China Investigates Top Local Social Media Sites in Push to Control
Content,” Reuters, August 10, 2017. As of April 29, 2019:
https://www.reuters.com/article/us-china-cyber/china-investigates-top-local-social-
media-sites-in-push-to-control-content-idUSKBN1AR07K
Cai Huifu, Wang Lin, Sheng Peilin, Yu Qi, Liu Xuemei, and Zheng Yu, “Research
into News and Public Opinion Warfare During the Iraq War,” China Military
Science, August 2003, pp. 28–34.
Cai Yintong, “Study Abroad Students: An Important Force for People-to-
People External Propaganda” [“留学生: 民间外宣的重要力量”], International
Communications, March 2009.
Capon, Felicity, “Chinese Propaganda Cartoon Promotes Five Year Plan,”
Newsweek, October 27, 2015. As of April 30, 2019:
http://www.newsweek.com/
chinachinese-propaganda-videochina039s-five-year-plancommunist-598015
Central Intelligence Agency, Political Information: The Role of TASS in Soviet
Propaganda Activities, Shanghai, Information Report, December 13, 1948. As of
April 17, 2019:
https://www.cia.gov/library/readingroom/docs/CIA-RDP82-
00457R002100650004-7.pdf
———, translation of: S. Simoni, “Soviet Anti-Semitism and the Prague Trial,”
Yedioth Hayom, 1952. As of April 17, 2019:
https://www.cia.gov/library/readingroom/docs/CIA-RDP65-
00756R000500130006-7.pdf
———, “Tass: Its Role, Structure, and Operations,” June 1959.
238 Hostile Social Manipulation: Present Realities and Emerging Trends
———, “Winning Without Fighting: Chinese Public Opinion Warfare and the
Need for a Robust American Response,” Heritage Foundation, Backgrounder
No. 2745, November 21, 2012.
———, “Winning Without Fighting: The Chinese Psychological Warfare
Challenge,” Heritage Foundation, Backgrounder No. 2821, July 11, 2013.
Chessen, Matt, “Understanding the Psychology Behind Computational
Propaganda,” in Shawn Powers and Markos Kounalakis, eds., Can Public
Diplomacy Survive the Internet?: Bots, Echo Chambers, and Disinformation,
Advisory Commission on Public Diplomacy, May 2017.
Chin, Josh, and Gillian Wong, “China’s New Tool for Social Control: A Credit
Rating for Everything,” Wall Street Journal, November 28, 2016. As of May 5,
2019:
https://www.wsj.com/articles/
chinas-new-tool-for-social-control-a-credit-rating-for-everything-1480351590
“China Invents the Digital Totalitarian State,” The Economist, December 17, 2016.
As of May 5, 2019:
https://www.economist.com/news/briefing/21711902-worrying-implications-its-
social-credit-project-china-invents-digital-totalitarian
“China Is Spending Billions to Make the World Love It,” The Economist,
March 23, 2017.
“China’s Biggest Search Engine Baidu Looks Into 3 Billion Fake News Claims a
Year,” Bloomberg, October 10, 2017.
“China’s Five-Year Plan Now Has Its Own Psychedelic Music Video,” Wall Street
Journal, October 27, 2015.
“China’s New Cultural Revolution,” Wall Street Journal, January 4, 2012. As of
April 28, 2019:
https://www.wsj.com/articles/SB100014240529702035503045771382701917888
32
“China’s Operation Australia,” Sydney Morning Herald, 2017. As of April 28, 2019:
http://www.smh.com.au/interactive/2017/chinas-operation-australia/
“China’s Propaganda Department Not Good Enough at Propaganda—Gov’t,”
Hong Kong Free Press, June 9, 2016.
Chinese Academy of Military Science Military Strategy Department, ed., Science
of Military Strategy [战略学], 3rd edition, Beijing: Academy of Military Science
Press, 2013.
“Chinese-Americans Are Becoming Politically Active,” The Economist, January 19,
2017. As of May 6, 2019:
https://www.economist.com/news/united-states/21715066-long-slumbering-voter-
block-awakes-chinese-americans-are-becoming-politically-active
240 Hostile Social Manipulation: Present Realities and Emerging Trends
“Fake ‘Swedish’ Letter Spread in Russian Media,” Radio Sweden, September 13,
2015. As of April 24, 2019:
http://sverigesradio.se/sida/artikel.aspx?programid=2054&artikel=6254260
Fan, Jiayang, “How China Wants to Rate Its Citizens,” New Yorker, November 3,
2015. As of May 5, 2019:
https://www.newyorker.com/news/daily-comment/
how-china-wants-to-rate-its-citizens
Faris, Robert M., Hal Roberts, Bruce Etling, Nikki Bourassa, Ethan Zuckerman,
and Yochai Benkler, “Partisanship, Propaganda, and Disinformation: Online
Media and the 2016 U.S. Presidential Election,” research paper, Berkman Klein
Center for Internet and Society, Harvard University, August 2017. As of May 8,
2019:
https://dash.harvard.edu/bitstream/handle/1/33759251/2017-08_
electionReport_0.pdf?sequence=9
Farrell, Henry, “American Democracy Is an Easy Target,” Foreign Policy,
January 17, 2018. As of May 8, 2019:
http://foreignpolicy.com/2018/01/17/american-democracy-was-asking-for-it/
Farrely, M. C., K. C. Davis, M. L. Haviland, P. Messeri, and C. G. Healton,
“Evidence of a Dose-Response Relationship Between ‘Truth’ Antismoking Ads
and Youth Smoking Prevalence,” American Journal of Public Health, Vol. 95, 2005,
pp. 425–431.
Faughnder, Ryan, “China-Owned AMC Seals Deal to Buy Carmike Cinemas,
Making It the Largest Theater Chain in U.S.,” Los Angeles Times, November 15,
2016.
Fazio, Lisa K., Nadia M. Brasier, B. Keith Payne, and Elizabeth J. Marsh,
“Knowledge Does Not Protect Against Illusory Truth,” Journal of Experimental
Psychology, Vol. 144, No. 5, 2015, pp. 993–1002.
Fazio, Russell H., and Richard E. Petty, eds., Attitudes: Their Structure, Function,
and Consequences, New York: Psychology Press/Taylor & Francis, 2008.
Ferrara, Emilio, “Disinformation and Social Bot Operations in the Run up to the
2017 French Presidential Election,” First Monday, Vol. 22, No. 8, August 2017. As
of May 8, 2019:
http://firstmonday.org/ojs/index.php/fm/article/view/8005/6516
Flynn, D. J., Brendan Nyhan, and Jason Reifler, “The Nature and Origins of
Misperceptions: Understanding False and Unsupported Beliefs About Politics,”
Advances in Political Psychology, Vol. 38, Supplement 1, 2017, pp. 128–129.
Forbidden Feeds: Government Controls on Social Media in China, New York: PEN
America, March 13, 2018. As of May 5, 2019:
https://pen.org/wp-content/uploads/2018/06/PEN-America_Forbidden-Feeds-
report-6.6.18.pdf
246 Hostile Social Manipulation: Present Realities and Emerging Trends
Gao Han, “Russia Today: Russia’s External Propaganda Aircraft Carrier” [“今日
俄罗斯”:俄罗斯的“外宣航母”], Modern Audiovisual, May 2016. As of April 29,
2019:
http://www.globalview.cn/html/global/info_10863.html
Garcia, D., “Social Media Mavens Wield ‘Influence,’ and Rake in Big Dollars,”
CNBC Tech, August 12, 2017. As of August 24, 2017:
https://www.cnbc.com/2017/08/11/social-media-influencers-rake-in-cash-become-
a-billion-dollar-market.html
Garmazhapova, Aleksandra, “Gde Zhivut Trolli. I Kto ih Kormi,t,” Novaya
Gazeta, September 9, 2013. As of April 24, 2019:
https://www.novayagazeta.ru/
articles/2013/09/07/56253-gde-zhivut-trolli-i-kto-ih-kormit
Geng, Olivia, “‘Very Big Muscles’: Chinese Propaganda Video Lavishes Praise on
Putin,” blog post, Wall Street Journal, May 8, 2015. As of April 30, 2019:
https://blogs.wsj.com/chinarealtime/2015/05/08/
very-big-muscles-chinese-propaganda-video-lavishes-praise-on-putin
“German Intelligence Unmasks Alleged Covert Chinese Social Media Profiles,”
Reuters, December 10, 2017.
“Germany Says New Security Strategy Will Respond to Russia,” Reuters,
February 17, 2015. As of May 10, 2019:
https://www.reuters.com/article/us-germany-security/germany-says-new-security-
strategy-will-respond-to-russia-idUSKBN0LL1BK20150217
Ghosh, Dipayan, and Ben Scott, “#DigitalDeceit: The Technologies Behind
Precision Propaganda on the Internet,” New America, Public Interest Technology
Program, January 2018.
Giles, Keir, “Indicators and Warnings for Detecting Information Threats,”
YouTube video of remarks at the Riga StratCom Dialogue: Perception Matters,
NATO StratCom Center of Excellence, Riga, Latvia, August 20–21, 2015. As of
April 17, 2016:
https://www.youtube.com/watch?v=RW3DHoQYayM
———, Russia’s ‘New’ Tools for Confronting the West: Continuity and Innovation in
Moscow’s Exercise of Power, London: Chatham House, March 2016a.
———, Handbook of Russian Information Warfare, Research Division, NATO
Defense College, November 2016b.
Glaser, April, “Twitter Admits There Were More Than 50,000 Russian Bots
Trying to Confuse American Voters Before the Election Campaign,” Slate,
January 19, 2018. As of April 24, 2019:
https://slate.com/technology/2018/01/twitter-admits-there-were-more-than-50-
000-russian-bots-confusing-u-s-voters-in-2016.html
248 Hostile Social Manipulation: Present Realities and Emerging Trends
Glaser, Bonnie, and Matthew P. Funaiole, “Xi Jinping’s 19th Party Congress
Speech Heralds Greater Assertiveness in Chinese Foreign Policy,” Center for
Strategic and International Studies, October 26, 2017. As of April 28, 2019:
https://www.csis.org/analysis/xi-jinpings-19th-party-congress-speech-heralds-
greater-assertiveness-chinese-foreign-policy
Glatz, Rainer L., and Martin Zapfe, “Ambitious Framework Nation: Germany in
NATO,” German Institute for International and Security Affairs, September 2017.
As of May 10, 2019:
https://www.swp-berlin.org/fileadmin/contents/products/comments/2017C35_
glt_zapfe.pdf
Gorman, Jack, and Sara Gorman, Denying to the Grave: Why We Ignore the Facts
That Will Save Us, 2016.
Gorton, William A., “Manipulating Citizens: How Political Campaigns’ Use of
Behavioral Social Science Harms Democracy,” New Political Science, Vol. 38, No.
1, 2016.
Gorwa, Robert, and Douglas Guilbeault, “Understanding Bots for Policy and
Research: Challenges, Methods, and Solutions,” Prague: Conference of the
International Communication Association, May 2018. As of April 24, 2019:
https://arxiv.org/pdf/1801.06863.pdf
Grassegger, Hannes, and Mikael Krogerus, “Weaken from Within,” New Republic,
November 2, 2017.
Green, D. P., and A. S. Gerber, “Voter Mobilization, Experimentation, and
Translational Social Science,” Perspectives in Politics, Vol. 14, 2016, pp. 738–749.
Greenberg, Andy, “Hackers Hit Macron with Huge Email Leak Ahead of French
Election,” Wired, May 5, 2017. As of April 24, 2019:
https://www.wired.com/2017/05/macron-email-hack-french-election/
Greenberg, Andy, “Don’t Pin the Macron Email Hack on Russia Just Yet,” Wired,
May 8, 2017b. As of April 24, 2019:
https://www.wired.com/2017/05/dont-pin-macron-email-hack-russia-just-yet/
Groot, Gerry, “The Long Reach of China’s United Front Work,” Lowy Interpreter,
November 6, 2017a.
———, “United Front Work After the 19th Party Congress,” China Brief,
December 22, 2017b. As of April 29, 2019:
https://jamestown.org/program/united-front-work-19th-party-congress
Grundy, Tom, “Did China’s State-Run News Agency Purchase Twitter Followers?”
Hong Kong Free Press, April 14, 2015. As of May 6, 2019:
https://www.hongkongfp.com/2015/04/14/
did-chinas-state-run-news-agency-purchase-twitter-followers/
Bibliography 249
Guo, Eileen, “How WeChat Spreads Rumors, Reaffirms Bias, and Helped Elect
Trump,” Wired, April 20, 2017. As of May 6, 2019:
https://www.wired.com/2017/04/
how-wechat-spreads-rumors-reaffirms-bias-and-helped-elect-trump/
Győri, Lóránt, Peter Krekó, Jakub Janda, and Bernhard Weidinger, Does Russia
Interfere in Czech, Austrian, and Hungarian Elections? Political Capital, European
Values, in cooperation with Dokumentations Archiv des österreichischen
Widerstandes, 2017.
Haidt, Jonathan, The Righteous Mind: Why Good People Are Divided by Politics and
Religion, New York: Vintage Books, 2013.
Haines, John R., “Distinguishing the True from the False: Fakes and Forgeries in
Russia’s Information War Against Ukraine,” Foreign Policy Research Institute,
September 28, 2016. As of April 24, 2019:
https://www.fpri.org/article/2016/09/
distinguishing-true-false-fakes-forgeries-russias-information-war-ukraine/#_ftn49
Halliday, Josh, “BBC World Service Fears Losing Information War As Russia
Today Ramps Up Pressure,” The Guardian, December 21, 2014.
Halper, Stephan, China: The Three Warfares, Washington, D.C.: Office of Net
Assessment, 2013.
Hamilton, Clive, Silent Invasion: China’s Influence in Australia, Richmond,
Australia: Hardie Grant, 2018.
Han Song and Ping Chuan, “Grasp Important Points, Explain Difficult Points,
Decipher Confusion Points: How to Explain 3rd Plenum Meeting of 18th Party
Congress to Foreigners” [“抓重点 解难点 释疑点——如何做好十八届三中全会
的对外解读”], International Communications, January 2014.
Harrell, Margaret C., and Melissa A. Bradley, Data Collection Methods:
Semi-Structured Interviews and Focus Groups, Santa Monica, Calif.: RAND
Corporation, TR-718-USG, 2009. As of April 17, 2019:
https://www.rand.org/pubs/technical_reports/TR718.html
Harrington, Kent, “Will China Weaponize Social Media?” Project Syndicate,
February 5, 2018. As of July 10, 2019:
https://www.project-syndicate.org/commentary/
xi-jinping-china-foreign-influence-campaigns-by-kent-harrington-2018-02
Hatmaker, Taylor, “Here’s How Russia Targeted Its Fake Facebook Ads and How
Those Ads Performed,” Tech Crunch, November 1, 2017. As of April 17, 2019:
https://techcrunch.com/2017/11/01/list-russian-ads-facebook-instagram/
Heath, Timothy, “Beijing’s Influence Operations Target Chinese Diaspora,” War
on the Rocks, March 1, 2018. As of April 30, 2019:
https://warontherocks.com/2018/03/
beijings-influence-operations-target-chinese-diaspora/
250 Hostile Social Manipulation: Present Realities and Emerging Trends
Hellman, Maria, and Charlotte Wagnsson, “How Can European States Respond
to Russian Information Warfare? An Analytical Framework,” European Security,
Vol. 26, No. 2, 2017, pp. 156–157.
Helm, Toby, “Third of EU Referendum Voters Won’t Make up Their Minds Until
Week Before Poll,” The Guardian, June 11, 2016. As of May 8, 2019:
https://www.theguardian.com/politics/2016/jun/11/
brexit-eu-referendum-vote-last-minute-decision-decide-lse
Helmus, Todd, Christopher Paul, and Russell W. Glenn, Enlisting Madison
Avenue: The Marketing Approach to Earning Popular Support in Theaters of
Operation, Santa Monica, Calif.: RAND Corporation, MG-607-JFCOM, 2007.
As of April 17, 2019:
https://www.rand.org/pubs/monographs/MG607.html
Henchowitz, Anne, “Thousands of Local Internet Propaganda Emails Leaked,”
China Digital Times, December 3, 2014.
Hernandez, Javier C., and Melissa Eddy, “China Denies Using LinkedIn to
Recruit German Informants,” New York Times, December 11, 2017.
Hirss, Martins, “The Extent of Russia’s Influence in Latvia,” National Defense
Academy of Latvia, Working Paper No. 03/16, November 2016.
Holland, Max, “The Propagation and Power of Communist Security Services
Dezinformatsiya,” International Journal of Intelligence and Counterintelligence,
Vol. 19, No. 1, 2006.
Horvitz, Josh, “China’s Military Has Released a Rap Video in Order to Lure More
Recruits,” Quartz, May 3, 2016.
Horwitz, Josh, “A Chinese Student’s Commencement Speech Praising “Fresh Air”
and Democracy Is Riling China’s Internet,” Quartz, May 23, 2017a. As of May 5,
2019:
https://qz.com/989454/a-chinese-students-commencement-speech-at-the-
university-of-maryland-praising-fresh-air-and-democracy-is-riling-chinas-internet/
———, “Australian Professors and Universities Are Being Shamed into
Apologizing for Offending Chinese Students,” Quartz, August 29, 2017b.
“Hotels Turn Away South Koreans, Chinese Smash Goods as Missile Row
Widens,” Radio Free Asia, March 13, 2017.
Hou Dongsheng, “Comparison and Analysis of Foreign Propaganda Related to
Tibet Between Chinese Government and Dalai Lama Clique” [“中国政府与达赖
集团在涉藏外宣上的比较和分析”], Journal Of Chongqing Institute of Socialism,
June 2012.
Bibliography 251
Howard, Philip N., Gillian Bolsover, Bence Kollanyi, Samantha Bradshaw, and
Lisa-Maria Neudert, “Junk News and Bots During the U.S. Election: What Were
Michigan Voters Sharing over Twitter?” Data Memo 2017.1, Oxford, UK: Project
on Computational Propaganda, Oxford Internet Institute, University of Oxford,
March 26, 2017.
Howard, Philip N., and Bence Kollanyi, “Bots, #Strongerin, and #Brexit:
Computational Propaganda During the UK-EU Referendum,” Research Note
2016.1, Oxford, UK: Project on Computational Propaganda, 2016. As of April 19,
2019:
http://comprop.oii.ox.ac.uk/wp-content/uploads/sites/89/2016/06/
COMPROP-2016-1.pdf
Hu Jintao, “Resolutely Follow the Cultural Development Path of Socialism with
Chinese Characteristics, Work to Build a Socialist Strong Culture Country”
[“坚定不移走中国特色社会主义文化发展道路努力建设社会主义文化强国”],
Seeking Truth, January 1, 2012.
Hu Xiaojian, “Decoding the U.S. ‘Murphy-Portman Counter-Propaganda Bill,’”
International Study Reference, July 2017, pp. 26–29.
Hu Yu and Lu Jun, “Experiences and Thoughts on Construction of Central-Level
State-Owned Enterprises Image Abroad” [“央企海外形象建设的经验与思考”],
International Communications, October 2016.
Huang Chien and Hsieh Chun-lin, “Prosecutors: China Paid Wang for
Propaganda,” Taipei Times, January 3, 2018. As of May 6, 2019:
http://www.taipeitimes.com/News/front/archives/2018/01/03/2003685092/1
Huang Dahui, “Analyzing the Abe Government’s Foreign Propaganda Strategy”
[“试析安倍政府的对外宣传战略”], Contemporary International Relations, June
2017.
Huang, Zheping, “China’s Craziest English-Language Propaganda Videos Are
Made by One Mysterious Studio,” Quartz, October 27, 2015. As of April 30, 2019:
https://qz.com/533850/chinas-craziest-english-language-propaganda-videos-are-
made-by-one-mysterious-studio/
———, “China Wants to Build a Credit Score That Dings Online Chat Group
Users for Their Political Views,” Quartz, September 8, 2017. As of May 5, 2019:
https://qz.com/1072660/china-wants-to-build-a-credit-score-that-dings-online-
chat-group-users-for-their-political-views/
Hutchings, Stephen, and Joanna Szostek, “Dominant Narratives in Russian
Political and Media Discourse During the Ukraine Crisis,” in Agnieska Pikulicksa-
Wilczewska and Richard Sakwa, eds., Ukraine and Russia: People, Politics,
Propaganda, and Perspectives, Bristol, UK: E-International Relations Publishing,
2015.
252 Hostile Social Manipulation: Present Realities and Emerging Trends
Langfitt, Frank, “How China’s Censors Influence Hollywood,” NPR, May 18,
2015. As of April 29, 2019:
https://www.npr.org/sections/parallels/2015/05/18/407619652/
how-chinas-censors-influence-hollywood
LaPiere, Richard T., “Attitudes vs. Actions,” Social Forces, Vol. 13, No. 2, 1934,
pp. 230–237.
Lecher, Colin, “Here Are the Russia-Linked Facebook Ads Released by Congress,”
The Verge, November 1, 2017.
Lee, Georgina, “Here’s What We Know About the Alleged Russian Involvement in
Brexit,” 4News, November 16, 2017. As of May 8, 2019:
https://www.channel4.com/news/factcheck/
heres-what-we-know-about-alleged-russian-involvement-in-brexit
Lee, Yimou, and Faith Hung, “How China’s Shadowy Agency Is Working to
Absorb Taiwan,” Reuters, November 26, 2014. As of April 29, 2019:
https://www.reuters.com/article/us-taiwan-china-special-report/special-report-how-
chinas-shadowy-agency-is-working-to-absorb-taiwan-idUSKCN0JB01T20141127
Li, Pei, and Adam Jourdan, “Mercedes-Benz Apologizes to Chinese for Quoting
Dalai Lama,” Reuters, February 6, 2018. As of May 5, 2019:
https://www.reuters.com/article/us-mercedes-benz-china-gaffe/
mercedes-benz-apologizes-to-chinese-for-quoting-dalai-lama-idUSKBN1FQ1FJ
Li Qiaoming, “Analysis of Modern Warfare Development Based on Russia’s Two
Conflicts,” PLA Daily, August 16, 2016.
Li Yiqing, “两家俄官媒推特账号广告功能遭关闭,曾被美指责“干预大选””],
The Paper, October 27, 2017. As of April 29, 2019:
http://www.thepaper.cn/newsDetail_forward_1839554
Lian Xiaotong, “Analysis of Leaders’ Public Diplomacy Strategy from a Cross-
Cultural Perspective—Xi Jinping’s 2012 Visit to the United States as Example”
[“跨文化视野下领导人公共外交策略分析——以2012年习近平访美为例”],
International Communications, September 2013.
Lieberthal, Kenneth, Governing China: From Revolution Through Reform, New
York: W.W. Norton & Co., 1995.
Liff, Adam, “China and the U.S. Alliance System,” China Quarterly, April 2017.
Lin, Herbert, and Jackie Kerr, “On Cyber-Enabled Information/Influence Warfare
and Manipulation,” in Oxford Handbook of Cybersecurity, August 14, 2017. As of
April 19, 2019:
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3015680
Linthicum, Kate, “Meet the Chinese American Immigrants Who Are Supporting
Donald Trump,” Los Angeles Times, May 27, 2016. As of May 6, 2019:
http://www.latimes.com/politics/la-na-pol-asian-voters-20160527-snap-story.html
Bibliography 257
“Market Share Held by the Leading Social Networks in the United Kingdom
(UK) as of July 2017,” Statista, 2017.
Markoff, John, “Vast Spy System Loots Computers in 103 Countries,” New York
Times, March 28, 2009. As of May 5, 2019:
http://www.nytimes.com/2009/03/29/technology/29spy.html
Marriott Corporation, tweet, Twitter, January 10, 2018. As of May 6, 2019:
https://twitter.com/marriottrewards/status/951344031405084673?lang=en
Marwick, Alice, and Rebecca Lewis, “Media Manipulation and Disinformation
Online,” Data and Society Research Institute, undated. As of April 24, 2019:
http://www.chinhnghia.com/DataAndSociety_
MediaManipulationAndDisinformationOnline.pdf
Mastro, Oriana Skylar, “Why Chinese Assertiveness Is Here to Stay,” Washington
Quarterly, Vol. 37, No. 4, Winter 2015, pp. 151–170.
Mate, Aron, “RT Was Forced to Register as a Foreign Agent,” The Nation,
November 16, 2017.
Mattis, Peter, “Contrasting China’s and Russia’s Influence Operations,” War on the
Rocks, January 16, 2018. As of April 29, 2019:
https://warontherocks.com/2018/01/
contrasting-chinas-russias-influence-operations/
McKinsey and Company, “Targeted Online Marketing Programs Boost Customer
Conversion Rates,” webpage, undated. As of November 7, 2017:
https://www.mckinsey.com/business-functions/marketing-and-sales/
how-we-help-clients/clm-online-retailer
Meng Jing and Celia Chen, “China Fines Tencent, Baidu, Weibo over Banned
Contents in On-Going Crackdown,” South China Morning Post, September 26,
2017.
Menn, Joseph, “Exclusive: Russia Used Facebook to Try to Spy on Macron
Campaign—Sources,” Reuters, July 27, 2017. As of May 9, 2019:
https://www.reuters.com/article/us-cyber-france-facebook-spies-exclusive/
exclusive-russia-used-facebook-to-try-to-spy-on-macron-campaign-sources-
idUSKBN1AC0EI
Meserole, Chris, and Alina Polakova, “Disinformation Wars,” Foreign Policy,
May 25, 2018.
Metaxas, Panagiotis T., and Eni Mustafaraj, “Social Media and the Elections,”
Science, Vol. 338, No. 6106, October 2012. As of May 8, 2019:
http://dx.doi.org/10.1126/science.1230456
“Military Expenditure by Country As Percentage of Gross Domestic Product,”
Stockholm International Peace Research Institute, 2017. As of May 10, 2019:
https://www.sipri.org/sites/default/files/Milex-share-of-GDP.pdf
260 Hostile Social Manipulation: Present Realities and Emerging Trends
Parker, Emily, “Can WeChat Thrive in the United States?” MIT Technology
Review, August 11, 2017. As of May 6, 2019:
https://www.technologyreview.com/s/608578/
can-wechat-thrive-in-the-united-states
Paul, Christopher, and Miriam Matthews, The Russian ‘Firehose of Falsehood’
Propaganda Model: Why It Might Work and Options to Counter It, Santa Monica,
Calif.: RAND Corporation, PE-198-OSD, 2016. As of April 23, 2019:
https://www.rand.org/pubs/perspectives/PE198.html
Peel, Michael, “EU States Poised to Agree Joint Defence Pact,” Financial Times,
November 7, 2017. As of May 9, 2019:
https://www.ft.com/content/29f6fe76-c2eb-11e7-a1d2-6786f39ef675
Peisakhin, Leonid, and Arturas Rozenas, “When Does Russian Propaganda
Work—and When Does It Backfire? Here’s What We Found.” Washington Post,
April 3, 2018. As of May 10, 2019:
https://www.washingtonpost.com/news/monkey-cage/wp/2018/04/03/when-does-
russian-propaganda-work-and-when-does-it-backfire-heres-what-we-found/?utm_
term=.b05b1faf362b&wpisrc=nl_cage&wpmm=1
Pellerin, Cheryl, “Poland, a Valued NATO Member, Leads by Example, Mattis
Says,” press release, U.S. Department of Defense, September 22, 2017. As of
May 10, 2019:
https://www.defense.gov/News/Article/Article/1321306/
poland-a-valued-nato-member-leads-by-example-mattis-says/
People’s Liberation Army Air Force, Weibo microblog status, December 12, 2017.
As of April 29, 2019:
https://m.weibo.cn/status/4184081916589933
“Percentage of U.S. Population with a Social Media Profile from 2008 to 2017,”
Statista, 2019. As of May 8, 2019:
https://www.statista.com/statistics/273476/
percentage-of-us-population-with-a-social-network-profile/
Persily, Nathaniel, “Can Democracy Survive the Internet?” Journal of Democracy,
Vol. 28, No. 2, April 2017,
Pew Research Center, “Global Indicators Database,” database, undated. As of
October 12, 2018:
http://www.pewglobal.org/database/indicator/27/country/231/
———, “Partisanship and Political Animosity in 2016,” June 22, 2016a. As of
May 7, 2019:
http://www.people-press.org/2016/06/22/
partisanship-and-political-animosity-in-2016/
———, “Many Americans Believe Fake News Is Sowing Confusion,”
December 15, 2016b.
Bibliography 265
Phillips, Tom, “Chinese Student Abused for Praising ‘Fresh Air of Free Speech’ in
US,” The Guardian, May 23, 2017. As of May 5, 2019:
https://www.theguardian.com/world/2017/may/23/
china-yang-shuping-free-speech-university-of-maryland-us-student
Pisnia, Natalka, “Why Has RT Registered As a Foreign Agent in the US?” BBC
News, November 15, 2017.
PLAAF—See People’s Liberation Army Air Force.
“Poland, NATO Launch Defensive Exercises amid Security Concerns,” CBS
News, September 21, 2017. As of May 10, 2019:
https://www.cbsnews.com/news/
poland-nato-launch-defensive-drills-security-concerns/
“Polish President Signs Defence Spending Boost into Law,” Radio Poland,
October 23, 2017. As of May 10, 2019:
http://thenews.pl/1/9/Artykul/331690,Polish-president-signs-defence-spending-
boost-into-law
Polyakova, Alina, “Russia Can’t Decide If Ukrainian Jews Are Victims or
Villains,” New Republic, April 28, 2014. As of April 18, 2019:
https://newrepublic.com/article/117556/
putins-russia-using-ukrainian-jews-propaganda-tools
Pomerantsev, Peter, “Russia and the Menace of Unreality,” The Atlantic,
September 9, 2014.
Pomerantsev, Peter, and Michael Weiss, The Menace of Unreality: How the Kremlin
Weaponizes Information, Culture and Money, New York: The Interpreter, Institute
of Modern Russia, 2014.
Popken, Ben, “Russian Trolls Went on Attack During Key Election Moments,”
NBC News, February 13, 2018. As of April 26, 2019:
https://www.nbcnews.com/tech/social-media/
russian-trolls-went-attack-during-key-election-moments-n827176
Poushter, Jacob, and Dorothy Manevich, “Globally, People Point to ISIS and
Climate Change as Leading Security Threats,” Pew Research Center, August 1,
2017.
“President Macron’s Initiative for Europe: A Sovereign, United, Democratic
Europe,” French Ministry for Europe and Foreign Affairs, September 26, 2017. As
of May 9, 2019:
https://www.diplomatie.gouv.fr/en/french-
foreign-policy/european-union/events/article/
president-macron-s-initiative-for-europe-a-sovereign-united-democratic-europe
266 Hostile Social Manipulation: Present Realities and Emerging Trends
Schultheis, Emily, “What Went Right with the French Campaign Polls?” The
Atlantic, May 13, 2017. As of May 9, 2019:
https://www.theatlantic.com/international/archive/2017/05/
what-went-right-with-the-french-election-polls/526326/
“Secretary General Marks Deployment of NATO Battlegroups During Visit to
Latvia,” press release, NATO, June 19, 2017. As of May 10, 2019:
https://www.nato.int/cps/en/natohq/news_144993.htm
“Secured Borders Ad_Cultural _Metadata 1,” U.S. House of Representatives
Permanent Select Committee on Intelligence, 2017. As of April 19, 2019:
https://democrats-intelligence.house.gov/uploadedfiles/secured_borders_cultural_
metadata_1.pdf
Seddon, Max, “Documents Show How Russia’s Troll Army Hit America,”
BuzzFeed News, June 2, 2014a. As of April 24, 2019:
https://www.buzzfeed.com/maxseddon/documents-show-how-russias-troll-army-
hit-america?utm_term=.vcnbmwvY5o#.hh2b3EKayD
———, “Russian TV Airs Clearly Fake Image to Claim Ukraine Shot Down
MH17,” BuzzFeed News, November 15, 2014b. As of April 24, 2019:
https://www.buzzfeed.com/maxseddon/russian-tv-airs-clearly-fake-image-to-claim-
ukraine-shot-dow?utm_term=.rvNp9DABRE#.myO7zJpEaG
Semichastny, V., and Lubomir Strougal, “Record of a Conversation About
the Results of Cooperation and Further Coordination of Intelligence and
Counterintelligence Activities Between the MVD of the Czechoslovak Socialist
Republic and the KGB Under the USSR Council of Ministers,” transcript, Wilson
Center Digital Archive, June 12, 1962. As of April 19, 2019:
http://digitalarchive.wilsoncenter.org/document/175926
Sethuraman, Raj, Gerard J. Tellis, and Richard A. Briesch, “How Well Does
Advertising Work? Generalizations from Meta-Analysis of Brand Advertising
Elasticities,” Journal of Marketing Research, Vol. 48, No. 3, June 2011.
Shaik, Shabnam, “Improvements in Protecting the Integrity of Activity on
Facebook,” Facebook, April 12, 2017. As of May 9, 2019:
https://www.facebook.com/notes/facebook-security/improvements-in-protecting-
the-integrity-of-activity-on-facebook/10154323366590766/
Shambaugh, David, China’s Communist Party: Atrophy and Adaptation,
Washington, D.C.: Woodrow Wilson Center Press, 2009.
———, China Goes Global: The Partial Power, Oxford, UK: Oxford University
Press, 2013.
———, “China’s Soft-Power Push,” Foreign Affairs, July 2015. As of May 6, 2019:
https://www.foreignaffairs.com/articles/china/2015-06-16/china-s-soft-power-push
Shane, Scott, “The Fake Americans Russia Created to Influence the Election,”
New York Times, September 7, 2017a.
270 Hostile Social Manipulation: Present Realities and Emerging Trends
———, “These Are the Ads Russia Bought on Facebook in 2016,” New York
Times, November 1, 2017b.
“Share of Individuals in France Participating in Social Networks From 2011 to
2016,” Statista, 2017.
Shekhovtsov, Anton, “Pro-Russian Network Behind the Anti-Ukrainian
Defamation Campaign,” blog post, Anton Skekhovtsov’s Blog, February 3, 2014.
Sheng Peilin and Li Xue, “On ‘Media Decapitation,’” Journal of the PLA Nanjing
Institute of Politics, May 2006, pp. 114–117.
Shi Anbing and Liu Ying, “Three Steps to Advance the Construction of
International Communications Power” [“三步走”推进国际传播力建设”], People’s
Daily Online, May 19, 2014. As of April 29, 2019:
http://opinion.people.com.cn/n/2014/0519/c1003-25034295.html
Shi-Kupfer, Kristin, “Governance Through Information Control,” China Monitor,
No. 26, Mercator Institute for China Studies, January 19, 2016.
Shi Lei and Xiao Tao, “Chen Quanguo, Lobsang Gyaincain Meet Media
Delegation ‘Beijing Internet Media Red Land—Tibet’; Wu Yingjie Present at
Meeting,” Tibet Daily, August 20, 2014.
Shirk, Susan, China: Fragile Superpower, Oxford, UK: Oxford University Press,
2007.
Shishkin, Phillip, and James Marson, “Ukraine Accuses Kremlin Agents of
Coordinating Separatist Unrest,” Wall Street Journal, April 20, 2014.
Shuster, Simon, “Inside Putin’s On-Air Machine,” TIME, March 5, 2015
Sides, John, and Jack Citrin, “How Large the Huddled Masses? The Causes and
Consequences of Public Misperceptions About Immigrant Populations,” presented
at 2007 Annual Meeting of the Midwest Political Science Association, Chicago,
Ill., 2007.
Silver, Nate, “It Wasn’t Clinton’s Election to Lose,” 538.com, January 23, 2017. As
of May 8, 2019:
http://fivethirtyeight.com/features/it-wasnt-clintons-election-to-lose/
Silverstein, Jason, “North Korea and China Also Interfered in U.S. Election,
Reince Priebus Says,” New York Daily News, July 9, 2017. As of May 6, 2019:
http://www.nydailynews.com/news/politics/
north-korea-china-interfered-u-s-election-priebus-article-1.3312905
Sinovets, Polina, and Bettina Renz, “Russia’s 2014 Military Doctrine and Beyond,
Threat Perceptions, Capabilities and Ambitions,” Research Paper No. 117,
Research Division, NATO Defense College, July 2015.
Skorobutov, Dmitry, “Ispoved’ Propagandista. Chast’ I. Kak Delajut Novosti na
Gosudarstvennom TV,” The Insider, June 9, 2017. As of April 24, 2019:
https://theins.ru/confession/59757/2
Bibliography 271
Skurnik, Ian, Carolyn Yoon, Denise C. Park, and Norbert Schwarz, “How
Warnings About False Claims Become Recommendations,” Journal of Consumer
Research, Vol. 31, 2005, pp. 713–724.
Smith, Aaron, and Monica Anderson, “Social Media Use in 2018,” Pew Research
Center, March 1, 2018. As of May 8, 2019:
http://www.pewinternet.org/2018/03/01/social-media-use-in-2018/
Smith, Alexander, “British PM Theresa May Says Russia Seeks to ‘Weaponize’
Information,” NBC News, November 14, 2017.
Smyth, Jamie, “China’s $10bn Propaganda Push Spreads Down Under,” Financial
Times, June 9, 2016. As of May 6, 2019:
https://www.ft.com/content/324d82c4-2d60-11e6-a18d-a96ab29e3c95
“Social Media Usage in the United Kingdom,” Statista Dossier, August 2017.
“Social Network Usage in France,” Statista Dossier, 2017.
Soldatov, Andrei, and Irina Borogan, The Red Web, New York: Public Affairs,
2015.
Song Shunan, “Gather the Abroad Students to Strengthen the Dream of National
Revival—A Report of the Speech of General Secretary Xi Jinping Learning the
100th Anniversary” [“凝聚留学人员力量共筑民族复兴梦想 - 欧美同学会学习习
近平总书记在百年庆典上的讲话纪实”], China United Front, February 2014.
Song Suxia, “Small WeChat and Big Family: Hebi City Builds WeChat Platform
for United Front Work” [“ ‘小’ 微信 ‘大’ 家庭——鹤壁市委统战部建立统一战
线微信平台”], China United Front, May 2013.
“Sorbonne Speech of Emmanuel Macron—Full Text/English Version,” blog post,
Ouest France, September 26, 2017. As of May 8, 2019:
http://international.blogs.ouest-france.fr/archive/2017/09/29/macron-sorbonne-
verbatim-europe-18583.html
Sørensen, Camilla T. N., “The Significance of Xi Jinping’s ‘China Dream’ for
Chinese Foreign Policy: From ‘Tao Guang Yang Hui’ to ‘Fen Fa You Wei,’”
Journal of Chinese International Relations, Vol. 3, No. 1, 2015, pp. 53–73
“Soviet Use of Declassified Documents,” memorandum from A. Denis Clift to
Secretary of State Henry Kissinger, National Security Council Files, President
Gerald R. Ford Library, October 8, 1975.
Spencer, David, “Is the Incense Ban Furor More Than Just Simple Fake News?”
Taiwan News, July 27, 2017.
Sputnik, About Us, webpage, undated. As of February 5, 2018:
https://sputniknews.com/docs/about/index.html
Stelzenmüller, Constanze, “The Impact of Russian Interference on Germany’s
2017 Elections,” testimony before the U.S. Senate Select Committee on
Intelligence, June 28, 2017.
272 Hostile Social Manipulation: Present Realities and Emerging Trends
“Taiwan’s Taoists Protest Against Curbs on Incense and Firecrackers,” BBC News,
July 23, 2017.
“Taking on Taiwan’s Ruinous and Partisan Pension System,” The Economist,
May 18, 2017.
Tan, Chenhao, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian
Lee, “Winning Arguments: Interaction Dynamics and Persuasion Strategies in
Good-Faith Online Discussions,” International World Wide Web Conference
Committee, presented at WWW2016, Montreal, Canada, April 11–15, 2016. As
of April 16, 2019:
https://arxiv.org/pdf/1602.01103v1.pdf
Tan Feng, “Why China Became the ‘Sacrificial Lamb’ of U.S. Elections” [“中国为
何成为美国大选的‘替罪羊’”], International Communications, September 2016.
Tang Dashan, “Tibet Should Build a Major External Propaganda Structure,” Tibet
Daily, September 14, 2013.
Tatlow, Didi Kirsten, “China Reaches into the Heart of Europe,” New York Times,
January 25, 2018. As of April 28, 2019:
https://www.nytimes.com/2018/01/25/opinion/china-germany-tech-
manufacturing.html
“Telling the Untold,” webpage, Sputnik, undated. As of April 24, 2019:
https://sputniknews.com/files/newswires_sputnik.pdf
“Temnik—the Kremlin’s Route to Media Control,” EU vs Disinfo, March 29,
2017. As of April 24, 2019:
https://euvsdisinfo.eu/temnik-the-kremlins-route-to-media-control/
“The Vote to Leave the EU,” British Social Attitudes, Vol. 34, National Centre for
Social Research, undated. As of May 8, 2019:
http://www.bsa.natcen.ac.uk/media/39149/bsa34_brexit_final.pdf
Thomas, Timothy L., “Russia’s Reflexive Control Theory and the Military,”
Journal of Slavic Military Studies, Vol. 17, 2004, pp. 237–256.
Thorson, Emily, “Belief Echoes: The Persistent Effects of Corrected
Misinformation,” Political Communication, Vol. 33, No. 3, 2016, pp. 460–480.
“Tracking Ghostnet: Investigating a Cyber Espionage Network,” Citizen Lab,
March 28, 2009. As of May 5, 2019:
https://citizenlab.ca/2009/03/
tracking-ghostnet-investigating-a-cyber-espionage-network/
Tran, Pierre, “France Adds $2B to Defense Budget, Moving Closer to NATO
Spending Target,” Defense News, September 27, 2017. As of May 9, 2019:
https://www.defensenews.com/global/europe/2017/09/27/
france-adds-2b-to-2018-defense-budget/
Bibliography 275
Tsoi, Grace, “Why Katy Perry and Gigi Hadid Were Missing from Shanghai’s
Victoria’s Secret,” BBC News, November 20, 2017.
Turak, Natasha, “Estonia Has No Doubts on Trump’s Commitment to NATO,
Says Prime Minister Juri Ratas,” CNBC, January 26, 2018. As of May 7, 2019:
https://www.cnbc.com/2018/01/26/estonia-has-no-doubts-on-trumps-nato-
commitment-prime-minister.html
“Twenty-Five EU States Sign PESCO Defense Pact,” Deutsche Welle,
December 11, 2017. As of May 9, 2019:
http://www.dw.com/en/twenty-five-eu-states-sign-pesco-defense-pact/a-41741828
“UK and France Commit to New Defence Cooperation,” press release, United
Kingdom Ministry of Defence, January 18, 2018. As of May 8, 2019:
https://www.gov.uk/government/news/
uk-and-france-commit-to-new-defence-cooperation
“Ukraine Bans Russian TV Channels for Airing War ‘Propaganda,’” Reuters,
August 19, 2014.
U.S. Director of National Intelligence, “Assessing Russian Activities and
Intentions in Recent U.S. Elections,” unclassified assessment, January 6, 2017. As
of May 7, 2019:
https://www.dni.gov/files/documents/ICA_2017_01.pdf
U.S. District Court for the District of Columbia, “United States of America v.
Internet Research Agency LLC, [et. al],” filed February 16, 2018.
U.S. DNI—See U.S. Director of National Intelligence.
U.S. House of Representatives Permanent Select Committee on Intelligence,
“HSPCI Minority Open Hearing Exhibits,” webpage, undated. As of April 19,
2019:
https://intelligence.house.gov/hpsci-11-1/hpsci-minority-open-hearing-exhibits.
htm
U.S. House of Representatives Permanent Select Committee on Intelligence, Soviet
Active Measures, Washington, D.C.: Government Printing Office, 1982.
U.S. House of Representatives Permanent Select Committee on Intelligence,
Report on Russian Active Measures, March 22, 2018.
U.S. House of Representatives Permanent Select Committee on Intelligence,
Subcommittee on Oversight, Soviet Covert Action (The Forgery Offensive),
Washington, D.C.: U.S. Government Printing Office, 1980.
U.S. Joint Chiefs of Staff, Department of Defense Dictionary of Military and
Associated Terms, Joint Publication 1-02, Washington, D.C., November 2010. As
of April 19, 2019:
http://www.dtic.mil/doctrine/new_pubs/jp1_02.pdf
276 Hostile Social Manipulation: Present Realities and Emerging Trends
Wardle, Claire, “Fake News: It’s Complicated,” First Draft, February 16, 2017.
Watson, Kathryn, “How Did WikiLeaks Become Associated with Russia?” CBS
News, November 15, 2017.
Way, Lucan Ahmad, and Adam Casey, “Is Russia a Threat to Western Democracy?
Russian Intervention in Foreign Elections, 1991–2017,” draft memo prepared for
conference on Global Populisms as a Threat to Democracy? Stanford University,
November 3–4, 2017. As of May 28, 2019:
https://fsi-live.s3.us-west-1.amazonaws.com/s3fs-public/is_russia_a_threat_to_
western_democracy_russian_intervention_in_foreign_elections_1991-2017_.pdf
———, “Russia Has Been Meddling in Foreign Elections for Decades. Has It
Made a Difference?” Washington Post, January 8, 2018.
Weaver, Kimberlee, “Inferring the Popularity of an Opinion from Its Familiarity:
A Repetitive Voice Can Sound Like a Chorus,” Journal of Personality and Social
Psychology, Vol. 92, No. 5, 2007, pp. 821–833.
Webster, C. M., and P. D. Morrison, “Network Analysis in Marketing,” Australian
Marketing Journal, Vol. 12, pp. 8–18, 2004.
Werkhäuser, Nina, “German Military Spending Gets Political,” Deutsche Welle,
August 8, 2017. As of May 10, 2019:
http://www.dw.com/en/german-military-spending-gets-political/a-40016299
Whitson, Jennifer A., and Adam D. Galinsky, “Lacking Control Increases Illusory
Pattern Perception,” Science, Vol. 322, 2008.
Wikipedia, “Sputnik (news agency),” article, 2018. As of February 5, 2018:
https://en.wikipedia.org/wiki/Sputnik_(news_agency)
Wilson, Andrew, Virtual Politics: Faking Democracy in the Post-Soviet World, New
Haven, Conn.: Yale University Press, 2005.
Wong, Chun Han, “The Foreigner Advising Beijing on Propaganda,” blog post,
Wall Street Journal, May 13, 2016. As of April 30, 2019:
https://blogs.wsj.com/chinarealtime/2016/05/13/
the-foreigner-advising-beijing-on-propaganda/
Wong, Edward, “Chinese Warship May Be Nearly Ready,” New York Times,
April 7, 2011.
———, “China’s President Lashes out at Western Culture,” New York Times,
January 3, 2012. As of April 28, 2019:
http://www.nytimes.com/2012/01/04/world/asia/chinas-president-pushes-back-
against-western-culture.html
Bibliography 279
Xiao Fei and Caichan Minbao, “Analysis of Online Public Opinion Dangers
and Online Public Opinion Guidance Countermeasures for Overseas Border
Conflicts” [“涉外边境冲突的网络舆情风险与舆论引导对策探析”], International
Communications, August 2016.
Xiao Lili, “Challenges and Public Opinion Responses for China’s National Image
in Africa” [“中国国家形象在非洲面临的挑战及舆论应对”], International
Communications, August 2011.
Xiong Yuhua, “To Seek Advantages and Avoid Disadvantages and Make Good Use
of the ‘Double-Edged Sword’ of Internet—Cadres, Staff Members, and Workers
of Our Region’s Propaganda and Cultural System Conscientiously Study the Spirit
of the Sixth Plenary Session of the 17th CPC Central Committee,” Tibet Daily,
October 24, 2011.
“Xi’s Secret Economic Weapon: Overseas Chinese,” Nikkei Asian Review, April 3,
2017.
Xu Lei, “What Can We Learn from Russia Today?” [“我们向“今日俄罗斯”学什
么?”], People’s Daily Overseas Edition, September 19, 2014. As of April 29, 2019:
http://paper.people.com.cn/rmrbhwb/html/2014-09/19/content_1479579.htm
Xu, Beina, and Eleanor Albert, “Media Censorship in China,” Council on Foreign
Relations, February 17, 2017 As of April 29, 2019:
https://www.cfr.org/backgrounder/media-censorship-china
Xu Hua, “How Did Putin Create the Image of a Leader” [“普京如何塑造领袖形
象”], International Communications, March 2015.
Xu Shaomin, “Insights into the Impact of the Proliferation of Fake News in the
United States and Europe on China’s Public Diplomacy” [“欧美假新闻泛滥对我
国开展公共外交的启”], Public Diplomacy, Vol. 2, 2017.
Xu Xiujun, “Foreign Communication of Chinese Global Governance Ideas Under
Counter-Globalization Trend” [“逆全球化思潮下中国全球治理观的对外传播”],
International Communications, March 2017.
Xuecun, Murong, “The New Face of Chinese Propaganda,” New York Times,
December 20, 2013.
Yan Liang, “Global Changes and Foreign Communication Responses After
Trump’s Taking Office” [“特朗普上任后的世界变局与对外传播应对”],
International Communications, February 2017.
Yan Xuetong, “From ‘Keeping a Low Profile’ to ‘Striving for Achievement,’”
Chinese Journal of International Politics, Vol. 7, No. 2, 2014, pp. 153–184.
Yang Yunsheng, “Research on Foreign Propaganda for the China Dream” [“中国
梦海外宣讲研究”], New Orient, June 2016.
Bibliography 281
Yao Yao, “The West’s View of China, or the World’s View of China? New
Thinking on Building China’s Global Image” [“西方的中国观,还是世界的中国
观?——中国建构国际形象的新思路”], International Communications, July 2014.
Ye Xiaoli and Gu Haoyu, “The Analysis for Influence Factors of Chinese-
American Political Participation: Take the Protesting Action to the Insulting
Chinese for an Example” [“当代美国华人政治参与影响因素分析: 以抗议ABC辱
华行动为例”], Overseas Chinese Journal of Bagui, June 2015, pp. 13–20.
———, “The Factors on Election Campaign of Modern Chinese-American:
Based on the Analyses of Sustainability” [“当代美国华人竞选影响因素: 基于可
持续性的分析”], Overseas Chinese Journal of Bagui, September 2017, pp. 31–38.
Yi Changjun, “Research on New Overseas Chinese Associations and the
Construction of ‘Soft Power’” [“海外新华侨华人社团与国家 ‘软实力’ 建设研
究”], Journal of Huaqiao University, May 2016.
Ying Yu Lin, “China’s Hybrid Warfare and Taiwan,” The Diplomat, January 13,
2018.
Young, C. W. Bill, “Soviet Active Measures in the United States—An Updated
Report by the FBI,” U.S. Congressional Record, December 9, 1987.
Yu Mingsong, “Research on United Front Work for Hong Kong Middle Class
Professionals” [“香港中产专业人士统战工作研究”], United Front Science, March
2017.
Yu Xiaoqing, “The Growth of China’s Outreach Flagship Media: 20 Years of
Transformation of an English Anchorwoman” [“中国外宣旗舰媒体成长记:一位
英文女主播的20年蜕变”], The Paper, April 1, 2017. As of May 6, 2019:
http://www.thepaper.cn/newsDetail_forward_1652275
Yuhas, Alan, “Russian Propaganda over Crimea and the Ukraine: How Does It
Work?” The Guardian, March 17, 2014.
Zhai Huixia, Xie Lianghong, and Yu Yunquan, “New Perspective on Western
Research on ‘China Model’ Since Global Financial Crisis” [“国际金融危机以来西
方对 ‘中国模式’ 研究的新视角”], International Communications, January 2012.
Zhang Leilei, “Actively Use Overseas Social Media for Military Foreign
Propaganda” [“积极利用海外社交媒体参与军事外宣”], Military Correspondent,
August 2016, pp. 59–60.
Zhao Qinghai, “New Western Reflections on Globalization” [“西方对全球化的新
反思”], International Communications, January 2008.
Zhao Liangying and Xu Xiaolin, “Actively Build China’s National Strategic
Communication System” [“积极构建中国国家战略传播体系”], Media Outpost,
September 2016.
282 Hostile Social Manipulation: Present Realities and Emerging Trends
www.rand.org $45.00
ISBN-10 1-9774-0260-7
ISBN-13 978-1-9774-0260-8
54500