What percentage of edits to the Wikipedia main namespace are reverted because of the presence of peacock behavior?

This question will help us understand how frequently users are reverted due to peacock behavior vs other policy violations.

Proposed approach:

• Collect a recent sample of edits made in the last 90 days to a Wikipedia main namespace. This sample should include the revision id, revision content, wiki, user experience level platform, and if the edit was reverted.
• Use the new content edit tag (editcheck-newcontent) to limit to revisions that added new content to an article. Rationale: We would want to focus on edit type that add content (vs minor edits or edits that remove content) and use VE as those are the types of edits edit check will be presented within.
• Use a string-matching approach, looking for common peacock keywords in the sample revision text. Consult @diego to align with approach being used to calculate the baseline in T368274
• Calculate the percentage of edits that are reverted due to peacock behavior. Include breakdowns by wiki, user experience, and platform.

Per what @MNeisler and I talked about offline today, we're going to move forward with the approach she described in T371158#10134639 above.

Update: I reached out to @diego who clarified that he does not currently have a list of keywords but a TF-IDF approach, that uses words to classify articles. While the model itself doesn't output the list, additional steps could be completed to output a list of keywords related to peacock behavior and promotional tone. Diego indicated he might be able to find time to do this during the next week or so depending on task urgency and priorities.

Hi @MNeisler ! I've just shared on Slack the list and code to create the peacock related words. Just for the records a put the links to notebook and list here.

Related: https://en.wikipedia.org/wiki/Category:Articles_with_wikipuffery

@ppelberg Below is a summary of the initial results for review. This is based on all new content edits published between May and June 2024 on an English Wikipedia main namespace by newcomers, Junior Contributors, or unregistered users.

Methodology Overview:

• Identified new content edits from mediawiki_history using the edit tag (editcheck-newcontent).
• Queried mediawiki_wikitext_history to obtain the revision text for each revision id in the sample
• Used a string-matching approach to look for the set of identified common peacock-related words in the revision text.
• The list of peacock-related words is based on the list provided in T371158#10202597. Note: I manually filtered and removed some words that appeared due to data issues to a final list of 61 peacock-related words.
• Reviewed if edits were reverted within 48 hours.

Results

What percentage of edits published by newcomers and Junior Contributors to the Wikipedia main namespace at en.wiki are reverted because of the presence of peacock behavior?

• A little over half (56%) of all new content edits by newcomers and Junior contributors included one of the identified peacock-related words.
• Overall on English Wikipedia, 22% of all new content edits by newcomers and Junior Contributors that contain at least one of the identified peacock words is reverted. This represents a 46.7% increase (+7 percentage points) in the likelihood of being reverted compared to an edit without a peacock-related word.

Revert rate of peacock-related words on English Wikipedia

• New content edits with peacock words are equally likely to occur on both mobile web and desktop platforms. 57% of new content edits on mobile web and 56% of new content edits on desktop include peacock-related words.
• We observed a similar increase in the likelihood of an edit with a peacock-related word being reverted on a per-platform basis, as shown below.

• Newcomers are more likely to have their new content edits reverted than Junior Contributors. If their new content edit includes a peacock-related word, the likelihood of the edit being reverted increases by 30% (30% to 39%).

• As a reference point, when establishing a baseline for reference check in T332848#9077218, we found that about 21% of all new content edits without a reference by Junior Contributors and 40% of all new content edits without a reference by newcomers were reverted on English Wikipedia. This indicates that the revert rate for edits with peacock-related words is very similar to the revert rate for edits without a reference.

Some caveats:

• This is based on a sample list of peacock words and is not an exhaustive list of all possible peacock terms. It's possible some new content edits included a peacock word not identified on the list and were not counted in the analysis.
• Also, this does not consider the context of how the word was used. There may be some cases where the use was acceptable (i.e. part of a proper noun).

Code repo