PageRank on Wikipedia: Towards General Importance Scores for Entities

Entities are omnipresent in the landscape of modern information extraction and retrieval. Application areas range from natural language processing over recommender systems to question answering. For many of these application areas it is essential to build on objective importance scores of entities. One of the most successful amongst different methods is the PageRank algorithm [4]. It has been proven to provide objective relevance scores for hyperlinked documents, for example in Wikipedia [6, 8, 11]. Wikipedia serves as a rich source for entities and their descriptions. Its content is currently used by major Web search engine providers as a source for short textual summaries that are presented in knowledge graph panels. In addition, the link structure of Wikipedia has been shown to exhibit the potential to compute meaningful PageRank scores: connected with semantic background information (such as DBpedia [1]) the PageRank scores computed on the Wikipedia link graph enable rankings of entities of specific types, for example for scientists (see Listing 1.1). Although the provided PageRank scores [11] exhibit reasonable output in many cases, they are not always easily explicable. For example, as of DBpedia version 2015-04, “Carl Linnaeus” (512) has a much higher PageRank score than “Charles Darwin” (206) and “Albert Einstein” (184) together in the result of the query in Listing 1.1. The reason is easily identified by examining the articles that link to the article of “Carl Linnaeus”:^{Footnote 1} Most articles use the template Taxobox^{Footnote 2} that defines the field binomial_authority. It becomes evident that the page of “Carl Linnaeus” is linked very often because Linnaeus classified species and gave them a binomial name (see [9]). In general, entities that help to structure the geographic and biological domains have distinctively higher PageRank scores than most entities from other domains. While, given the high inter-linkage of these domains, this is expected to some degree, according to our computations articles such as “Bakhsh” (1914), “Powiat” (1408), “Chordate” (1527), and “Lepidoptera” (1778) are occurring in the top-50 list of all things in Wikipedia (see Table 5, column “DBP 2015-04”). These observations led us to the question whether these rankings can be improved. Unfortunately, this is not a straight forward task as a gold standard is missing and rankings are often subjective.

In this work we investigate on different link extraction^{Footnote 3} methods that aim to address the root causes of the observed effects. We focus on the question whether some links—based on their context/position in the article text—can be deemed more important than others. In our variants, we change the probabilistic impact of links in accordance to their context/position on the page and measure the effects on the output of the PageRank algorithm. We compare these variants and the rankings of existing systems with page-view-based rankings and provide statistics on the pairwise computed Spearman and Kendall rank correlations.

In this section we provide additional background on the used PageRank variants, link extraction from Wikipedia, and redirects in Wikipedia.

2.1 PageRank Variants

The PageRank algorithm follows the idea of a user that browses Web sites by following links in a random fashion (random surfer). For computing PageRank, we use the original PageRank formula [4] and a weighted version [2] that accounts for the position of a link within an article.

• Original PageRank [4] – On the set of Wikipedia articles W, we use individual directed links \(link(w_1, w_2)\) with \(w_1, w_2 \in W\), in particular the set of pages that link to a page l(w) = \(\{w_1 | link(w_1, w)\}\) and the count of out-going links \(c(w) = |\{w_1| link(w,w_1)\}|\). The PageRank of a page \(w_0 \in W\) is computed as follows:

  $$\begin{aligned} pr(w_0) = (1 - d) + d * \sum _{w_n \in l(w_0)}{\frac{pr(w_n)}{c(w_n)}} \end{aligned}$$

  (1)

• Weighted Links Rank (WLRank) [2] – In order to account for the relative position of a link within an article, we adapt Formula (1) and introduce link weights. The idea is that the random surfer is likely not to follow every link on the page with the same probability but may prefer those that are at the top of a page. The WLRank of a page \(w_0 \in W\) is computed as follows:

  $$\begin{aligned} wlr(w_0) = (1 - d) + d * \sum _{w_n \in l(w_0)}{\frac{pr(w_n)*lw(link(w_n, w_0))}{\sum _{w_m}{lw(link(w_n, w_m))}}} \end{aligned}$$

  (2)

  The link weight function lw is defined as follows:

  $$\begin{aligned} lw(link(w_1, w_2)) = 1 - \frac{first\_occurrence(link(w_1,w_2),w_1)}{|tokens(w_1)|} \end{aligned}$$

  (3)

  In order to form a correct probability model, the individual link weight is normalized in accordance to the link weights of all outgoing links of a page in Formula 2. If we set the link weight of every incoming link to the same value (e.g., 1) we obtain the original PageRank formula (see Formula 1). The used helper functions of Formula 3 can be described as follows:

  • \(first\_occurrence(link(w_1, w_2), w_1)\) – the token number of the first occurrence of a \(link(w_1, w_2)\) at the respective Wikipedia page \(w_1\). The token numbering starts at 1 (i.e., the first word/link in the wikitext).

  • \(tokens(w_1)\) – the total number of tokens of the Wikipedia page \(w_1\). Tokenization is performed as follows: we split the article text in accordance to white spaces but do not split up links (e.g., [[brown bear|bears]] is treated as one token).

Both Formulas (1) and (2) are iteratively applied until the scores converge. The variable d is called “damping factor”: in the random surfer model, it accounts for the possibility of accessing a page via the browser’s address bar instead of accessing it via a link from another page.

For reasons of presentation, we use the non-normalized version of PageRank in both cases. In contrast to the normalized version, the sum of all computed PageRank scores is the number of articles (instead of 1) and, as such, does not reflect a statistical probability distribution. However, normalization of the PageRank scores does not influence the final ranking (i.e., the resulting ordering relation between the Wikipedia articles does not change).

2.2 Wikipedia Link Extraction

In order to create a Wikipedia link graph we need to clarify which types of links are considered. The input for the rankings of [11] is a link graph that is constructed by the DBpedia Extraction Framework^{Footnote 4} (DEF). The DBpedia extraction is based on Wikipedia database backup dumps^{Footnote 5} that contain the non-rendered wikitexts of the Wikipedia articles and templates. From these sources, DEF builds a link graph by extracting links of the form [[article|anchor text]]. We distinguish between two types of links with respect to templates:^{Footnote 6}

1. 1.

   Links that are defined in the Wikipedia text but do not occur within a template, for example “[[brown bear|bears]]” outside {{and}}.

2. 2.

   Links that and provided as (a part of) a parameter to the template, for example “[[brown bear|bears]]” inside {{and}}.

DEF considers only these two types of links and not any additional ones that result from the rendering of an article. It also has to be noted that DEF does not consider links from category pages. This mostly affects links to parent categories as the other links (i.e., links to all articles of that category) are presented only in the rendered version of the category page (i.e., they do not occur in the wikitext). As an effect, the accumulated PageRank of a category page would be transferred almost 1:1 to its parent category. This would lead to a top-100 ranking of things with mostly category pages only. In addition, DEF does not consider links in references (denoted via \({<}\)ref\({>}\) tags).

In this work, we describe how we performed more general link extraction from Wikipedia. Unfortunately, in this respect, DEF exhibited certain inflexibilities as it processes Wikipedia articles line by line. This made it difficult to regard links in the context of an article as a whole (e.g., in order to determine the relative position of a link). In consequence, we reverse-engineered the link extraction parts of DEF and created the SiteLinkExtractor^{Footnote 7} tool. The tool enables to execute multiple extraction methods in a single pass over all articles and can also be extended by additional extraction approaches.

2.3 Redirected vs. Unredirected Wikipedia Links

DBpedia offers two types of page link datasets:^{Footnote 8} one in which the redirects are resolved and one in which they are contained. In principle, also redirect chains of more than one hop are possible but, in Wikipedia, the MediaWiki software is configured not to follow such redirect chains (that are called “double redirect” in Wikipedia)^{Footnote 9} automatically and various bots are in place to remove them. Therefore, we assume that only single-hop redirects are in place. However, as performed by DBpedia, also single-hop redirects can be resolved (see Fig. 1). Alternatively, for various applications (especially in NLP) it can make sense to keep redirect pages as they also have a high number of inlinks in various cases (e.g., “Countries of the world”)^{Footnote 10}. In that case, with reference to Fig. 1 and assuming that redirect pages only link to the redirect target, B passes most of its own PageRank score on to C (note that the damping factor is in place). Thus, we assume that the PageRank score of pages of type C is not heavily influenced by resolving/not resolving redirects.

Transitive resolution of a redirect in Wikipedia. A and C are full articles and B is called a “redirect page”, PL are page links, and \(PL^R\) are page links marked as a redirect (e.g., #REDIRECT [[United Kingdom]]). The two page links from A to B and from B to C are replaced by a direct link from A to C.

Full size image

We implemented five Wikipedia link extraction methods that enable to create different input graphs for the PageRank algorithm. In general we follow the example of DEF and consider type 1 and 2 links for extraction (which form a subset of those that occur in a rendered version of an article). The following extraction methods were implemented:

• All Links (ALL) This extractor produces all type 1 and 2 links. This is the reverse-engineered DEF method. It serves as a reference.

• Article Text Links (ATL) This measure omits links that occur in text that is provided to Wikipedia templates (includes type 1 links, omits type 2 links). The relation to ALL is as follows: \(ATL \subseteq ALL\).

• Article Text Links with Relative Position (ATL-RP) This measure extracts all links from the Wikipedia text (type 1 links) and produces a score for the relative position of each link (see Formula 3). In fact, the link graph ATL-RP is the same as ATL but uses edge weights based on each link’s position.

• Abstract Links (ABL) This measure extracts only the links from Wikipedia abstracts. We chose the definition of DBpedia which defines an abstract as the first complete sentences that accumulate to less than 500 characters.^{Footnote 11} This link set is a subset of all type 1 links (in particular: \(ABL \subseteq ATL\)).

• Template Links (TEL) This measure is complementary to ATL and extracts only links from templates (omits type 1 links, includes type 2 links). The relation to ALL and ATL is as follows: \(TEL = ALL \setminus ATL\).

Redirects are not resolved in any of the above methods. We executed the introduced extraction mechanisms on a dump of the English Wikipedia of February 5, 2015. This date is in line with the input of DEF with respect to DBpedia version 2015-04.^{Footnote 12} Table 1 provides an overview of the number of extracted links per link graph.

In our experiments, we first computed PageRank on the introduced link graphs. We then measured the pairwise rank correlations (Spearman’s \(\rho \) and Kendall’s \(\tau \))^{Footnote 13} between these rankings and the reference datasets (of which three are also based on PageRank and two are based on page-view data of Wikipedia). With the resulting correlation scores, we investigated on the following hypotheses:

H1 :

Links in templates are created in a “please fill out this form” manner and rather negatively influence the general estimate of salience that PageRank scores should represent.

H2 :

Links that are mentioned at the beginning of articles are more often clicked and, therefore, the ATL-RP and ABL rankings correlate stronger with the page-view-based rankings.

H3 :

The practice of resolving redirects does not strongly impact the final ranking (in accordance to PageRank scores) as redirect pages pass most of their score on to the respective target page.

4.1 PageRank Configuration

We computed PageRank with the following parameters on the introduced link graphs ALL, ATL, ATL-RP, ABL, and TEL: non-normalized, 40 iterations, damping factor 0.85, start value 0.1.

4.2 Reference Datasets

We use the following rankings as reference datasets:  

DBpedia PageRank (DBP) :

The scores of DBpedia PageRank [11] are based on the “DBpedia PageLinks” dataset (i.e., Wikipedia PageLinks as extracted by DEF, redirected). The computation was performed with the same configuration as described in Sect. 4.1. The scores are regularly published as TSV and Turtle files. The Turtle version uses the vRank vocabulary [10]. Since DBpedia version 2015-04, the DBP scores are included in the official DBpedia SPARQL endpoint (see Listing 1.1 for an example query). In this work, we use the following versions of DBP scores based on English Wikipedia: DBpedia 3.8, 3.9, 2014, and 2015-04.

DBpedia PageRank Unredirected (DBP-U) :

This dataset is computed in the same way as DBP but uses the “DBpedia PageLinks Unredirected” dataset.^{Footnote 14} As the name suggests, Wikipedia redirects are not resolved in this dataset (see Sect. 2.3 for more background on redirects in Wikipedia). We use the 2015-04 version of DBP-U.

SubjectiveEye3D (SUB) :

Paul Houle aggregated the Wikipedia page views of the years 2008 to 2013 with different normalization factors (particularly considering the dimensions articles, language, and time).^{Footnote 15} As such, SubjectiveEye3D reflects the aggregated chance for a page view of a specific article in the interval years 2008 to 2013. However, similar to unnormalized PageRank, the scores need to be interpreted in relation to each other (i.e., the scores do not reflect a proper probability distribution as they do not add up to one).

The Open Wikipedia Ranking (TOWR) :

The TOWR project is maintained by the Laboratory for Web Algorithmics of the Università degli Studi di Milano. It provides Wikipedia rankings in accordance to different ranking methods in a Web interface^{Footnote 16} for direct comparison. They provide the following measures:^{Footnote 17}  

TOWR-PR :

PageRank computed on the Wikipedia link graph with the parallel Gauß-Seidel method [7] of the LAW^{Footnote 18} library.

TOWR-H :

Harmonic centrality as introduced in [3] computed on the Wikipedia link graph.

TOWR-I :

Indegree, ranks Wikipedia pages in accordance to their number of incoming links.

TOWR-PV :

Page views, ranks Wikipedia pages in accordance to “the number of page views in the last year”^{Footnote 19}.

The two page-views-based rankings (i.e., SUB and TOWR-PV) serve as a reference in order to compare the different graph-based rankings. We show the mutual overlap of entities covered by the individual rankings in Table 2.

4.3 Results

We used MATLAB for computing the pairwise Spearman’s \(\rho \) and Kendall’s \(\tau \) correlation scores. The Kendall’s \(\tau \) rank correlation measure has \(\mathcal {O}(n^2)\) complexity and takes a significant amount of time for large matrices. In order to speed this up, we sampled the data matrix by a random selection of 1M rows for Kendall’s \(\tau \). The pairwise correlation scores of \(\rho \) and \(\tau \) are reported in Tables 3 and 4 respectively. The results are generally as expected: For example, the page-view-based rankings correlate strongest with each other. The DDP rankings correlate strongest with the respective neighboring DBP versions. Also DBP-U 2015-04 and ALL have a very strong correlation (these rankings should be equal).

H1 seems to be supported by the data as the TEL PageRank scores correlate worst with any other ranking. However, ATL does not correlate better with SUB and TOWR-PV than ALL. This indicates that the reason for the bad correlation might not be due to the “bad semantics of links in the infobox”. With random samples on ATL—which produced similar results—we found that the computed PageRank values of TEL are mostly affected by the low total link count (see Table 1). With respect to the initial example, the PageRank score of “Carl Linnaeus” is reduced to 217 in ATL. However, this subjective perception of improvement can not be generalized (with the used measures).

On a side note: we assume that computing PageRank on DBpedia’s RDF data would produce similar scores as TEL because DBpedia extracts its semantic relations mostly from Wikipedia’s infoboxes.

Indicators for H2 are the scores of ABL and ATL-RP. However, similar to TEL, ABL does not produce enough links for a strong ranking. ATL-RP, in contrast, produces the strongest correlation with SUB. The improvement of ATL-RP comparred to ATL is clearly visible. This is an indication that—indeed—articles that are linked at the beginning of a page are more often clicked. This is supported by related findings of Dimitrov et al. [5] where actual HTTP referrer data was analyzed.

With respect to H3, we expected DBP-U 2015-04 and DBP 2015-04 to correlate much stronger than the results suggest. As a reason, we found that DEF does not implement the full workflow of Fig. 1: although it introduces a link \(A \rightarrow C\) and removes the link \(A \rightarrow B\), it does not remove the link \(B \rightarrow C\). As such, the article B occurs in the final entity set with the lowest PageRank score of 0.15 (as it has no incoming links). In contrast, in DBP-U 2015-04, these pages often accumulate PageRank scores of 1000 and above. If B would not occur in the final ranking of DBP 2015-04, it would not be considered by the rank correlation measures. This explains the comparatively weak correlation between the redirected and unredirected datasets.

Further Observations. Another surprising result is the rather weak correlation of TOWR-PR with all the other PageRank-based rankings. As the Wikipedia dump date of DBpedia 2015-04 (that we also used for our measures, see Sect. 3) is only two months apart from the dump date used by TOWR, we expected much stronger correlations here. This is amplified by the observation that TOWR-PR correlates stronger with older DBP versions. However, Table 2 already suggests a clear difference with respect to the number of covered entities. Therefore, we assume that the preprocessing of the link graph performed by TOWR induces this bias. This is also supported by the strong correlations between the link-graph-based TOWR measures (i.e., TOWR-PR, TOWR-H, and TOWR-I) visible in Tables 3 and 4.

In addition to ATL-RP, also the link-graph-based TOWR measures exhibit a stronger correlation with SUB than the other PageRank-based measures. However, with respect to Table 2 it becomes clear that their overlap with SUB is 949 603 entities less than the one of ATL-RP (or \(-19\)% relative to the overlap of ATL-RP and SUB). With this difference, the correlation scores are not directly comparable.

4.4 Conclusions

Whether links from templates are excluded or included in the input link graph does not impact strongly on the quality of rankings produced by PageRank. WLRank on articles produces best results with respect to the correlation to page-view-based rankings. In general, although there is a strong correlation, we assume that link and page-view-based rankings are complementary. This is supported by Table 5 which contains the top-50 scores of SUB, DBP 2015-04, and ATL-RP: The PageRank-based measures are strongly influenced by articles that relate to locations (e.g., countries, languages, etc.) as they are highly interlinked and referenced by a very high fraction of Wikipedia articles. In contrast, the page-view-based ranking of SubjectiveEye3D covers topics that are frequently accessed and mostly relate to pop culture and important historical figures or events. We assume that a strong and more objective ranking of entities is most likely achieved by combining link-structure and page-view-based rankings on Wikipedia. For applications that deal with NLP, we recommend to use the unredirected version of DBpedia PageRank.

There are two common types of Wikipedia rankings: one is based on measures on the link graph, the other is based on consumption (e.g., page views). In the following, we briefly introduce the state of the art in both Wikipedia ranking methods.

Measures on the Wikipedia link graph: The work of Eom et al. [6] investigates on the difference between 24 language editions of Wikipedia with PageRank, 2DRank, and CheiRank rankings. The analysis focuses on the rankings of the top-100 persons in each language edition. We consider this analysis as seminal work for investigation on mining cultural differences with Wikipedia rankings. This is an interesting topic as different cultures often use the same language edition of Wikipedia (e.g., United Kingdom and the United States use English). Similarly, the work of Lages et al. provide rankings of universities of the world in [8]. Again, 24 language editions were analyzed with PageRank, 2DRank, and CheiRank. PageRank is shown to be efficient in producing similar rankings like the “Academic Ranking of World Universities (ARWU)” (that is provided yearly by the Shanghai Jiao Tong University). The Open Wikipedia Ranking (TOWR) also applies different graph measures on the Wikipedia link graph (see Sect. 4.2).

The above approaches vary the applied graph measures (PageRank, 2DRank, CheiRank, indegree, harmonic centrality) but do not vary the link extraction methods. In this paper, we experiment with both, different input graphs and a combination of a new weighted input graph and WLRank.

Wikipedia consumption patterns: The official page view statistics of various Wikipedia projects are publicly available as dumps^{Footnote 20} or as a Web API^{Footnote 21}. Our work on this paper was mainly influenced and motivated by an initial experiment that was performed by Paul Houle: in the Github project documentation of SubjectiveEye3D (see Sect. 4.2 for more details on SubjectiveEye3D), he reports about Spearman and Kendall rank correlations between SubjectiveEye3D and our published PageRank computations [11].^{Footnote 22} His results are similar to our computations. In a recent work, Dimitrov et al. introduce a study on the link traversal behavior of users within Wikipedia with respect to the positions of the followed links [5]. The authors conclude that a great fraction of clicked links can be found in the top part of the articles.

Comparing ranks on Wikipedia is an important topic and with our contribution we want to emphasize the need for considering the features “link graph” and “page views” in combination.

In this work, we compared different input graphs for the PageRank algorithm, the impact on the scores, and the correlation to page-view-based rankings. The main findings can be summarized as follows:

1. 1.

   Removing template links has no general influence on the PageRank scores.

2. 2.

   The results of WLRank with respect to the relative position of a link indicate a better correlation to page-view-based rankings than other PageRank methods.

3. 3.

   If redirects are resolved, it should be done in a complete manner as, otherwise, entities get assigned artificially low scores. We recommend using an unredirected dataset for applications in the NLP context.

Currently, we use the link datasets and the PageRank scores in our work on entity summarization [12, 13]. However, there are many applications that can make use of objective rankings of entities. Therefore, we plan to investigate further on the combination of page-view-based rankings and link-graph-based ones. In effect, for humans, rankings of entities are subjective and it is a hard task to approximate “a general notion of importance”.