Machine learning models/Production/Hungarian Wikipedia article topic
| Model card | |
|---|---|
| This page is an on-wiki machine learning model card. | |
 A model card is a document about a machine learning model that seeks to answer basic questions about the model. | |
| Model Information Hub | |
| Model creator(s) | Aaron Halfaker (User:EpochFail) and Amir Sarabadani |
| Model owner(s) | WMF Machine Learning Team (ml@wikimediafoundation.org) |
| Model interface | Ores homepage |
| Code | drafttopic Github, ORES training data, and ORES model binaries |
| Uses PII | No |
| In production? | Yes |
| Which projects? | Hungarian Wikipedia |
| This model uses article text to predict the likelihood that the article belongs to a set of topics. | |
Motivation
How can we predict what general topic an article is in? Answering this question is useful for various analyses of Wikipedia dynamics. However, it is difficult to group a very diverse range of Wikipedia articles into coherent, consistent topics manually.
This model, part of the ORES suite of models, analyzes an article to predict its likelihood of belonging to a set of topics. Similar models (though not necessarily with the same performance level or topics, are deployed across about a dozen other projects. There is also a language agnostic article topic model.
This model may be useful for high-level analyses of Wikipedia dynamics (pageviews, article quality, edit trends) and filtering articles.
Users and uses
Use this model for
- high-level analyses of Wikipedia dynamics such as pageview, article quality, or edit trends — e.g. How are pageview dynamics different between the physics and biology categories?
- filtering to relevant articles — e.g. filter articles only to those in the music category.
Don't use this model for
- definitively establishing what topic an article pertains to
- automated editing of articles or topics without a human in the loop
Current uses
This model is a part of ORES, and generally accessible via API. It is used for high-level analysis of Wikipedia, platform research, and other on-wiki tasks.
Example API call:https://ores.wikimedia.org/v3/scores/huwiki/1234/articletopic
Ethical considerations, caveats, and recommendations
- This model was trained on data that is now several years old (from mid-2020). Underlying data drift may skew model outputs.
- This model uses word2vec as a training feature. Word2vec, like other natural language embeddings, encodes the linguistic biases of underlying datasets — along the lines of gender, race, ethnicity, religion etc. Since Wikipedia has known biases in its text, this model may encode and at times reproduce those biases.
- This model has highly variable performance across different topics — consult the test statistics below to get a sense of inter-topic performance.
Model
Performance
Test data confusion matrix:
| Test data confusion matrix | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Test data sample rates:
| Test data sample rates | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Test data performance:
| Test data performance | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
The training dataset for this model is available here.
Implementation
Model architecture
| Model architecture |
|---|
{
"type": "GradientBoosting",
"params": {
"scale": false,
"center": false,
"labels": [
"Culture.Biography.Biography*",
"Culture.Biography.Women",
"Culture.Food and drink",
"Culture.Internet culture",
"Culture.Linguistics",
"Culture.Literature",
"Culture.Media.Books",
"Culture.Media.Entertainment",
"Culture.Media.Films",
"Culture.Media.Media*",
"Culture.Media.Music",
"Culture.Media.Radio",
"Culture.Media.Software",
"Culture.Media.Television",
"Culture.Media.Video games",
"Culture.Performing arts",
"Culture.Philosophy and religion",
"Culture.Sports",
"Culture.Visual arts.Architecture",
"Culture.Visual arts.Comics and Anime",
"Culture.Visual arts.Fashion",
"Culture.Visual arts.Visual arts*",
"Geography.Geographical",
"Geography.Regions.Africa.Africa*",
"Geography.Regions.Africa.Central Africa",
"Geography.Regions.Africa.Eastern Africa",
"Geography.Regions.Africa.Northern Africa",
"Geography.Regions.Africa.Southern Africa",
"Geography.Regions.Africa.Western Africa",
"Geography.Regions.Americas.Central America",
"Geography.Regions.Americas.North America",
"Geography.Regions.Americas.South America",
"Geography.Regions.Asia.Asia*",
"Geography.Regions.Asia.Central Asia",
"Geography.Regions.Asia.East Asia",
"Geography.Regions.Asia.North Asia",
"Geography.Regions.Asia.South Asia",
"Geography.Regions.Asia.Southeast Asia",
"Geography.Regions.Asia.West Asia",
"Geography.Regions.Europe.Eastern Europe",
"Geography.Regions.Europe.Europe*",
"Geography.Regions.Europe.Northern Europe",
"Geography.Regions.Europe.Southern Europe",
"Geography.Regions.Europe.Western Europe",
"Geography.Regions.Oceania",
"History and Society.Business and economics",
"History and Society.Education",
"History and Society.History",
"History and Society.Military and warfare",
"History and Society.Politics and government",
"History and Society.Society",
"History and Society.Transportation",
"STEM.Biology",
"STEM.Chemistry",
"STEM.Computing",
"STEM.Earth and environment",
"STEM.Engineering",
"STEM.Libraries & Information",
"STEM.Mathematics",
"STEM.Medicine & Health",
"STEM.Physics",
"STEM.STEM*",
"STEM.Space",
"STEM.Technology"
],
"multilabel": true,
"population_rates": null,
"ccp_alpha": 0.0,
"criterion": "friedman_mse",
"init": null,
"learning_rate": 0.1,
"loss": "deviance",
"max_depth": 5,
"max_features": "log2",
"max_leaf_nodes": null,
"min_impurity_decrease": 0.0,
"min_impurity_split": null,
"min_samples_leaf": 1,
"min_samples_split": 2,
"min_weight_fraction_leaf": 0.0,
"n_estimators": 150,
"n_iter_no_change": null,
"presort": "deprecated",
"random_state": null,
"subsample": 1.0,
"tol": 0.0001,
"validation_fraction": 0.1,
"verbose": 0,
"warm_start": false,
"label_weights": {}
}
}
|
Output schema
| Output schema |
|---|
{
"title": "Scikit learn-based classifier score with probability",
"type": "object",
"properties": {
"prediction": {
"description": "The most likely labels predicted by the estimator",
"type": "array",
"items": {
"type": "string"
}
},
"probability": {
"description": "A mapping of probabilities onto each of the potential output labels",
"type": "object",
"properties": {
"Culture.Biography.Biography*": {
"type": "number"
},
"Culture.Biography.Women": {
"type": "number"
},
"Culture.Food and drink": {
"type": "number"
},
"Culture.Internet culture": {
"type": "number"
},
"Culture.Linguistics": {
"type": "number"
},
"Culture.Literature": {
"type": "number"
},
"Culture.Media.Books": {
"type": "number"
},
"Culture.Media.Entertainment": {
"type": "number"
},
"Culture.Media.Films": {
"type": "number"
},
"Culture.Media.Media*": {
"type": "number"
},
"Culture.Media.Music": {
"type": "number"
},
"Culture.Media.Radio": {
"type": "number"
},
"Culture.Media.Software": {
"type": "number"
},
"Culture.Media.Television": {
"type": "number"
},
"Culture.Media.Video games": {
"type": "number"
},
"Culture.Performing arts": {
"type": "number"
},
"Culture.Philosophy and religion": {
"type": "number"
},
"Culture.Sports": {
"type": "number"
},
"Culture.Visual arts.Architecture": {
"type": "number"
},
"Culture.Visual arts.Comics and Anime": {
"type": "number"
},
"Culture.Visual arts.Fashion": {
"type": "number"
},
"Culture.Visual arts.Visual arts*": {
"type": "number"
},
"Geography.Geographical": {
"type": "number"
},
"Geography.Regions.Africa.Africa*": {
"type": "number"
},
"Geography.Regions.Africa.Central Africa": {
"type": "number"
},
"Geography.Regions.Africa.Eastern Africa": {
"type": "number"
},
"Geography.Regions.Africa.Northern Africa": {
"type": "number"
},
"Geography.Regions.Africa.Southern Africa": {
"type": "number"
},
"Geography.Regions.Africa.Western Africa": {
"type": "number"
},
"Geography.Regions.Americas.Central America": {
"type": "number"
},
"Geography.Regions.Americas.North America": {
"type": "number"
},
"Geography.Regions.Americas.South America": {
"type": "number"
},
"Geography.Regions.Asia.Asia*": {
"type": "number"
},
"Geography.Regions.Asia.Central Asia": {
"type": "number"
},
"Geography.Regions.Asia.East Asia": {
"type": "number"
},
"Geography.Regions.Asia.North Asia": {
"type": "number"
},
"Geography.Regions.Asia.South Asia": {
"type": "number"
},
"Geography.Regions.Asia.Southeast Asia": {
"type": "number"
},
"Geography.Regions.Asia.West Asia": {
"type": "number"
},
"Geography.Regions.Europe.Eastern Europe": {
"type": "number"
},
"Geography.Regions.Europe.Europe*": {
"type": "number"
},
"Geography.Regions.Europe.Northern Europe": {
"type": "number"
},
"Geography.Regions.Europe.Southern Europe": {
"type": "number"
},
"Geography.Regions.Europe.Western Europe": {
"type": "number"
},
"Geography.Regions.Oceania": {
"type": "number"
},
"History and Society.Business and economics": {
"type": "number"
},
"History and Society.Education": {
"type": "number"
},
"History and Society.History": {
"type": "number"
},
"History and Society.Military and warfare": {
"type": "number"
},
"History and Society.Politics and government": {
"type": "number"
},
"History and Society.Society": {
"type": "number"
},
"History and Society.Transportation": {
"type": "number"
},
"STEM.Biology": {
"type": "number"
},
"STEM.Chemistry": {
"type": "number"
},
"STEM.Computing": {
"type": "number"
},
"STEM.Earth and environment": {
"type": "number"
},
"STEM.Engineering": {
"type": "number"
},
"STEM.Libraries & Information": {
"type": "number"
},
"STEM.Mathematics": {
"type": "number"
},
"STEM.Medicine & Health": {
"type": "number"
},
"STEM.Physics": {
"type": "number"
},
"STEM.STEM*": {
"type": "number"
},
"STEM.Space": {
"type": "number"
},
"STEM.Technology": {
"type": "number"
}
}
}
}
}
|
Example input and output
Input:
https://ores.wikimedia.org/v3/scores/huwiki/1234/articletopic
Output:
| Example output |
|---|
{
"huwiki": {
"models": {
"articletopic": {
"version": "1.4.0"
}
},
"scores": {
"1234": {
"articletopic": {
"score": {
"prediction": [
"Geography.Regions.Asia.East Asia",
"Geography.Regions.Europe.Europe*",
"STEM.STEM*"
],
"probability": {
"Culture.Biography.Biography*": 0.3423533920926803,
"Culture.Biography.Women": 0.14025995403786506,
"Culture.Food and drink": 0.06340726339907797,
"Culture.Internet culture": 0.011131262002287472,
"Culture.Linguistics": 0.04803618594888942,
"Culture.Literature": 0.022006683732088216,
"Culture.Media.Books": 0.002187089040235113,
"Culture.Media.Entertainment": 0.15309575643241796,
"Culture.Media.Films": 0.034026455544947425,
"Culture.Media.Media*": 0.20341149710773634,
"Culture.Media.Music": 0.0497533083246081,
"Culture.Media.Radio": 0.32875096388132496,
"Culture.Media.Software": 0.013034094135796932,
"Culture.Media.Television": 0.0031210534375715336,
"Culture.Media.Video games": 0.00030343872173972046,
"Culture.Performing arts": 0.005608603834940765,
"Culture.Philosophy and religion": 0.08066624396157131,
"Culture.Sports": 0.06433894067592255,
"Culture.Visual arts.Architecture": 0.03750949389175476,
"Culture.Visual arts.Comics and Anime": 0.0064514547243535985,
"Culture.Visual arts.Fashion": 0.05307244012171412,
"Culture.Visual arts.Visual arts*": 0.11900694660877364,
"Geography.Geographical": 0.032374625359398314,
"Geography.Regions.Africa.Africa*": 0.15894948721508503,
"Geography.Regions.Africa.Central Africa": 0.016011778190990485,
"Geography.Regions.Africa.Eastern Africa": 0.004741053756007876,
"Geography.Regions.Africa.Northern Africa": 0.0244144581524136,
"Geography.Regions.Africa.Southern Africa": 0.0019768365502830605,
"Geography.Regions.Africa.Western Africa": 3.962206086248686e-05,
"Geography.Regions.Americas.Central America": 0.00522749251093348,
"Geography.Regions.Americas.North America": 0.014687908835933414,
"Geography.Regions.Americas.South America": 0.002403901278264949,
"Geography.Regions.Asia.Asia*": 0.20487191607636618,
"Geography.Regions.Asia.Central Asia": 0.09112897244334056,
"Geography.Regions.Asia.East Asia": 0.9998386100569907,
"Geography.Regions.Asia.North Asia": 0.0027527477451360595,
"Geography.Regions.Asia.South Asia": 0.020285604017724872,
"Geography.Regions.Asia.Southeast Asia": 0.03250982686466025,
"Geography.Regions.Asia.West Asia": 0.05840215484971228,
"Geography.Regions.Europe.Eastern Europe": 0.03298542948603228,
"Geography.Regions.Europe.Europe*": 0.5315974960652942,
"Geography.Regions.Europe.Northern Europe": 0.007914585235952828,
"Geography.Regions.Europe.Southern Europe": 0.09252560636319572,
"Geography.Regions.Europe.Western Europe": 0.021413181244936737,
"Geography.Regions.Oceania": 0.026964542276338526,
"History and Society.Business and economics": 0.11345817838571329,
"History and Society.Education": 0.018546943483473882,
"History and Society.History": 0.16482995511902204,
"History and Society.Military and warfare": 0.057955265993041155,
"History and Society.Politics and government": 0.0422117254411273,
"History and Society.Society": 0.20169635418355597,
"History and Society.Transportation": 0.015610557061165244,
"STEM.Biology": 0.037673572379875016,
"STEM.Chemistry": 0.010794729056374292,
"STEM.Computing": 0.04398691739762987,
"STEM.Earth and environment": 0.0039011304821686735,
"STEM.Engineering": 0.16717936084988483,
"STEM.Libraries & Information": 0.022633922980662434,
"STEM.Mathematics": 0.08494932357374003,
"STEM.Medicine & Health": 0.010577875534037367,
"STEM.Physics": 0.0009472592704866106,
"STEM.STEM*": 0.7982423641999686,
"STEM.Space": 0.003438360400580638,
"STEM.Technology": 0.08261264399841732
}
}
}
}
}
}
}
|
Data
Data pipeline
The data to train was fetched from a set of revision IDs. Then various pieces of information about the revision were extracted using automated processes, and the revision text was fed into word2vec to get an article embedding. Finally, labels are derived from the mid-level WikiProject categories that the article is associated with.
Training data
If available, the training dataset can be found here. Training data was automatically and randomly separated from test data during training using the drafttopic git repository (which trains both drafttopic and articletopic models).
Test data
Test data was automatically and randomly split off from train data using the drafttopic git repository (which trains both drafttopic and articletopic models). The model then makes a prediction on that data, which is compared to the underlying ground truth to calculate performance statistics.
Licenses
- Code: MIT license
- Model: MIT license
Citation
Cite this model card as:
@misc{
Triedman_Bazira_2023_Hungarian_Wikipedia_article_topic,
title={ Hungarian Wikipedia article topic model card },
author={ Triedman, Harold and Bazira, Kevin },
year={ 2023 },
url={ https://meta.wikimedia.org/wiki/Machine_learning_models/Production/Hungarian_Wikipedia_article_topic }
}