Model card
This page is an on-wiki machine learning model card.
A model card is a document about a machine learning model that seeks to answer basic questions about the model.
Model Information Hub
Model creator(s)	Aitolkyn Baigutanova, Pablo Aragón, Muniza Aslam, and Diego Saez-Trumper
Model owner(s)	Pablo Aragón and Diego Saez-Trumper
Model interface	Lift Wing API
Code	Inference
Uses PII	No
In production?	TBA
This model uses edit history metadata to predict the likelihood of a reference to survive on a Wikipedia article.

This model card describes a model for summarizing the likelihood of a reference to survive on a Wikipedia article, by using historical editorial activity of revisions involving web domains as references. We use this probability as a proxy for source reliability which we refer to as reference risk. The features are relative to a given language edition.

While the model is deployed on LiftWing and available via the Lift Wing API and Wikimedia Enterprise API, it does not perform machine learning inference at runtime. Instead, it relies on precomputed scores derived from features evaluated in prior machine learning experiments^[1].

Motivation

The credibility of the content presented in Wikipedia articles largely depends on the reliability of references. URL-based references, in particular, can vary widely in quality, ranging from highly credible sources to unreliable or even spam and misleading websites. The burden of verifying the reliability of a source lies on the editors.

Research found that references to risky sources in the English Wikipedia decreased by more than threefold after those sources were added to the perennial sources list^[2]. However, a comparative study of reference reliability across Wikipedia language editions found substantial discrepancies, with many language editions lacking effective mechanisms, such as curated source lists, to identify sources that had already been flagged as unreliable by the corresponding community^[3].

To address this gap, this model provides insights into reference risk based on the historical editorial activity associated with a source. It can guide editors in avoiding the addition of potentially low-quality sources, and assist readers in critically evaluating the information they consume. Because the model is language-agnostic, it can be applied across all Wikipedia language editions.

Users and uses

Ethical considerations, caveats, and recommendations

• The features were derived from the historical editorial activity of a web domain in the target Wikipedia edition. Therefore, the model does not explicitly predict domain reliability but presents a set of characteristics for human evaluation.
• The model does not consider references not supplemented by a URL.
• The same domain can have different feature values across wikis.
• Some domains are assigned multiple labels by the community in the perennial sources list. In such cases, reference risk outputs the "worst" category. Note that widespread sources like forbes.com and guardian.com fall under this category in English wiki .

Model

The model outputs the following characteristics per page:

• Snapshot of data based on which the features were computed
• Wiki code
• Number of URL-based references
• Proportion of deprecated/blacklisted domains
• Minimum survival edit ratio of cited domains
• Mean survival edit ratio of cited domains
• Median survival edit ratio of cited domains
• List of URL-based references. The model outputs the following features per such reference:
  • Parsed URL
  • Domain extracted from the URL
  • Classification of the domain in the local perennial sources list (if any)
  • Classification of the domain in the English perennial sources list (if any)
  • Survival edit ratio (survival), i.e., the proportion of survived edits since domain addition
  • Page count, i.e., in how many pages in a corresponding wiki the domain was added
  • Editor count, i.e., how many distinct editors in a corresponding wiki cited the domain

{
  model_name : <string>
  model_version : <YYYY-MM>
  wiki_db : <string>
  revision_id : <int [0-]>,
  reference_count : <int [0-]>,
  survival_ratio: {
    min : <float [0-1]>, 
    mean : <float [0-1]>, 
    median : <float [0-1]>,
  }
  reference_risk_score : <float [0-1]>, 
  references: <list of dicts>
}

{"rev_id": 1322859357, "lang": "en", "extended_output": "True"}

{
   "model_name":"reference-risk",
   "model_version":"2024-11",
   "wiki_db":"enwiki",
   "revision_id":1322859357,
   "reference_count":104,
   "survival_ratio":{
      "min":0.6391829919620925,
      "mean":0.8608770672439817,
      "median":0.8816057268124078
   },
   "reference_risk_score":0.0,
   "references":[
      {
         "url":"https://www.theguardian.com/music/ng-interactive/2025/nov/07/rosalia-critics-crisis-being-hot-for-god-lux-catalan",
         "domain_name":"theguardian.com",
         "domain_metadata":{
            "ps_label_local":"No consensus",
            "ps_label_enwiki":"No consensus",
            "survival_ratio":0.9094254173213723,
            "page_count":185332,
            "editors_count":50522
         }
      },
      {
         "url":"https://www.metacritic.com/browse/albums/score/metascore/year/filtered?sort=desc&year_selected=2025",
         "domain_name":"metacritic.com",
         "domain_metadata":{
            "ps_label_local":"Generally reliable",
            "ps_label_enwiki":"Generally reliable",
            "survival_ratio":0.9198661789610177,
            "page_count":39922,
            "editors_count":11105
         }
      },
      ...
  ]
}

Data

The model computed features for a given domain are collected from all the historical appearances of that domain in a given wiki. The data extracted from the two tables are available at Wikimedia Data Lake. We use MediaWiki History and Wikitext History tables. The data is updated based on a monthly snapshot of these tables.

Additionally, we collect the reliability classification on the perennial sources list page. Specifically, we retrieve the domain and the corresponding status. We do not consider a source if either domain or status is missing in the current snapshot of the perennial sources list. The status of the source can be one of the following: Blacklisted, Deprecated, Generally Unreliable, No Consensus, or Generally Reliable. For frwiki, an explicit label is not assigned, in that special case, the model retrieves a brief description of the current consensus on source reliability.

Licenses

• Code: MIT License
• Model: No license required. The system relies exclusively on precomputed scores and performs no machine learning model inference.

Citation

Cite this model as:

@misc{baigutanova2024referencerisk,
  title = {Language-agnostic Wikipedia reference risk model card},
  author = {Baigutanova, Aitolkyn and Aragón, Pablo and Aslam, Muniza and Saez-Trumper, Diego},
  year = {2024},
  url = {https://meta.wikimedia.org/wiki/Machine_learning_models/Production/Language-agnostic_reference_risk}
}

References