Fix support for the uselang=qqx URL parameter

{{Edit request}} Currently, when a page using {{Autotranslate}} is viewed with the uselang=qqx URL parameter, then Module:Autotranslate crashes with:

Lua error in Module:Autotranslate at line 72: No fallback page found for autotranslate (base=⧼template-name⧽, lang=(lang)).

— ExE Boss (talk) 18:20, 26 June 2021 (UTC)

@ExE Boss: This is an edge case no reader will ever face with. However, handling it would require at least one additional branch, which, considering how many pages the module’s used on (47M+) and how many times it will actually help, isn’t worth it IMO. When one uses &uselang=qqx, the goal is usually to find out the name of a message that’s output by the MediaWiki software, not by some template, so broken page content doesn’t matter. —Tacsipacsi (talk) 21:16, 27 June 2021 (UTC)

Add support for skipping parameter normalisation

{{edit request|technical=1}} Please add a way to skip parameter normalisation (lowercase conversion and removal of blank parameters), so that {{country showdata}} can be converted to use Module:Autotranslate. (see also: Template:Country showdata?diff=577480314, Module:Autotranslate/sandbox?diff=577610033, and Template:Country showdata?diff=577610221) — ExE Boss (talk) 17:45, 30 July 2021 (UTC)

ExE Boss, why does {{country showdata}}, need that? --Jarekt (talk) 05:13, 15 January 2022 (UTC)

 Not done No usecase provided. --AntiCompositeNumber (talk) 04:25, 18 February 2022 (UTC)

Typo in comment at line 61

{{Edit request}} "than assume it is a template" is incorrect, should be "then" instead of "than" OmegaFallon (talk) 23:09, 27 February 2023 (UTC)

 Done Awesome! Thank you! —‍Mdaniels5757 (talk • contribs) 15:53, 14 March 2023 (UTC)

@OmegaFallon and Mdaniels5757: , we should avoid cosmetic edits which cause refresh on 71M pages using this module. A better way would be to wait for some substantial edit, and do them together. --Jarekt (talk) 16:53, 1 February 2024 (UTC)

Avoid using mw.title.exists

@Jarekt: I prepared a change that avoids using mw.title.exists so that no rows are created in the pagelinks table (only in templatelinks). Could you review it and deploy it if it’s okay? All tests that use the sandbox are green. Thanks in advance! —Tacsipacsi (talk) 21:29, 10 February 2024 (UTC)

@Tacsipacsi: , I looked at the code and it seems fine. I find it a bit harder to read than the original but that would be small price to pay if it helped with the DB performance. As for DB performance I do not know enough about which lua function causes what effect on the DB, and I think it would be good to get a second opinion if this version will make a difference. Trying to apply common logic I find it hard to believe that expanding a template (like English subtemplate) is is better for the DB than just checking if it exists. I am also a little concerned about some unforeseeable (by me) side effects of expanding an otherwise unused English subtemplate. What if /en is much more expensive than other versions for some reason (unlikely). I did look at "NewPP limit report" and the new version doubles "Template argument size" and "Post‐expand include size" for some reason (not sure what that means), but it does lower "Expensive parser function count" by 2. So I am fine with the change if others think that the new version will be gentler on the DB. --Jarekt (talk) 03:37, 11 February 2024 (UTC)

@Jarekt:

I find it a bit harder to read than the original but that would be small price to pay if it helped with the DB performance.

If you have a better idea, I’m open to it.

Trying to apply common logic I find it hard to believe that expanding a template (like English subtemplate) is is better for the DB than just checking if it exists.

My thinking is that

• Apart from that English template expansion (is it really needed? You didn’t explain in the edit summary why the change is needed), the first and only expanded template is also used, so the current code not “just checks if it exists”, but checks if it exists and then expands it – dropping the prior check removes a DB access without causing any new one;
• In the so-called canonical parse (the one that populates the pagelinks and templatelinks tables), {{int:lang}} is always English, so the two expansions are the same; and
• Looking at the PHP code, it’s smart enough to avoid expanding the same template twice.

Did you check the limit report using English interface language? If yes, my third point probably doesn’t stand (or at least it doesn’t entirely avoid expanding the same template twice). If I was indeed wrong, then the current code should not be deployed. There are a few options:

• Just drop the extra English check entirely.
• Track the language that was actually used. If it’s not English, do the extra English check using mw.title.exists; if it’s English, do nothing (the English check has already been done). This avoids expanding the English template unnecessarily, but also avoids the rows in the pagelinks table (as in canonical parse, the actually used language will always be English).
• Cache the parse result on our own. This avoids the double parses, but doesn’t avoid the unnecessary parses.

Tacsipacsi (talk) 01:27, 12 February 2024 (UTC)

@Tacsipacsi: , About my most recent improvement to Module:Autotranslate, which kind of broke your otherwise nice solution. I do a lot of clean up of Category:Pages with script errors and there is always bunch of templates like Template:Senatinform.ru which are missing /en but have some other language, like /ru in this case, and as long as the people creating them are always using their language as a default they do not know that everybody else is getting lua errors. I do not speak most of those languages and do not know what English version should be. This edit was an attempt to alert people writing the template that something is wrong, so they could fix it before it is being used and without me getting involved. I think I can accomplish the same goal with just the test at the template page that is calling autotranslate. I will give it a shot in sandbox. --Jarekt (talk) 02:01, 12 February 2024 (UTC)

@Tacsipacsi: , I made some changes to the /sandbox page. Mostly moved /en check to be only called at the base calling template to avoid all the complications. I also updated Module talk:Autotranslate/testcases. --Jarekt (talk) 14:30, 22 February 2024 (UTC)

@Jarekt:

• I see your point in checking the /en page for existence. Doing so only on the page itself is a good idea, thanks!
• I don’t like that you restored the colon-based check for adding the namespace – page titles can contain colons other than the namespace prefix. For example if I left out the Template: prefix on Template:Editnotices/Page/Commons:Inappropriate images of children, your code would signal an error: while frame:expandTemplate() correctly handles this page as a template, that extra English-subpage-check would check for a gallery and thus fail.
• The assert-based solution is a tiny bit less performant than mine: the string.format() call is executed in the 99% of cases where its result is unused. It may not be noticeable, though.
• Why did you change expandTemplate from a function declaration to a function expression and an assignment? I think it’s less readable. (I don’t care if it’s in one line or three lines, only whether it’s a named or anonymous function.)

Tacsipacsi (talk) 09:20, 14 March 2024 (UTC)

@Tacsipacsi: , I undid some of my changes, since now I understand better the code:

• I removed restored the colon-based check. I see that it is not necessary and the code works fine without it.
• I tweaked my assert-based solution so it is a little more efficient. Might not change execution time but efficiency is a good practice.
• I undid the change of expandTemplate from a function declaration to a function expression and an assignment. I changed it not to shrink code, but because it looked more like a variable assignment which "belongs" in a function, as oppose to local sub-function. It was my attempt to "pritify" the code, as the code looked more "natural" to me, but it makes very little difference. Your solution is probably more readable, and that is important.

Thanks for helping with this module. --Jarekt (talk) 03:53, 18 March 2024 (UTC)

Fallback de-at -> de not respected

There has been a report at Commons:Forum#Sprachanzeige that the automatic template translation of {{Personality rights}} does not respect the language fallback chain for de-at correctly. It should show the de version, however it shows the en version. You can verify this via these links: uselang=de shows that a de version exists, uselang=de-at shows en version but should show de version. I assumed that this might be an issue with incorrect fallback handling for non-English fallback languages. However, I added a unit test to check for this and it seems fine. Does anyone have any idea what could be the issue here? Thanks and best regards, ChrisiPK (Talk|Contribs) 14:30, 20 June 2025 (UTC)

It seems that not all templates are affected by this. For example, {{Nazi symbol}} seems to display the German version with uselang=de-at. Maybe this is due to the fact that {{Personality rights}} uses the Translate extension to translate the template content, whereas {{Nazi symbol}} uses simple subpages of the template? Could it be that the fallback is somehow not handled correctly when the Autotranslate module tries to expand a template that uses the Translate extension? Regards, ChrisiPK (Talk|Contribs) 01:03, 25 June 2025 (UTC)

ChrisiPK, that is kind of odd. Module:Autotranslate uses Lua's mw.language.getFallbacksFor function to get proper fallback. That function is being tested as part of Module:test and {{#invoke:test|fallback|de-at}} lists all the fallbacks for de-at language. which are de followed by en. So {{Personality rights}} should work properly. --Jarekt (talk) 02:53, 25 June 2025 (UTC)

Jarekt, thanks a lot for your reply. I agree that it should work, but unfortunately it doesn't. Having had a second look at that, my guess is as follows: The code in the module tries to find out whether a translation for a given language exists by expanding the template with that language suffix and checking whether this was successful. If not, it continues to the next language in the fallback chain. It seems to work correctly for templates that are translated by creating subpages for the different languages, but seems to fail for templates that are translated via the Translate extension, such as {{Personality rights}}. (Note that there are also translated subpages present on this template, such as {{Personality rights/de}}, but these are no longer used. Instead, {{Personality rights/i18n}} is the base page for translations and the subpages of this are used, such as {{Personality rights/i18n/de}}. These subpages are generated by the Translate extension.)

Keeping this in mind, could it be possible that the expansion of a language subpage that is managed by the Translate extension is always successful, even if no translation for this language exists? This would make sense in my mind, because the Translate extension is probably supposed to do fallback handling internally when no translation for a specific language exists. And experiments seems to confirm this: Even though {{Personality rights/i18n/de-at}} does not exist, it expands into the English language version:

Personality rights Personality rights warning Although this work is freely licensed or in the public domain, the person(s) shown may have rights that legally restrict certain re-uses unless those depicted consent to such uses. In these cases, a model release or other evidence of consent could protect you from infringement claims. Though not obliged to do so, the uploader may be able to help you to obtain such evidence. See our general disclaimer for more information.

For comparison, {{Personality rights/i18n/de}} exists and correctly expands into the German language version:

Personality rights Bitte Persönlichkeitsrechte beachten Obwohl dieses Werk unter einer freien Lizenz veröffentlicht wurde oder gemeinfrei ist, können durch die Persönlichkeitsrechte der abgebildeten Person(en) bestimmte Nachnutzungen eingeschränkt oder von deren Zustimmung abhängig sein. Ein Modelvertrag oder ein anderer Nachweis einer solchen Zustimmung kann dich vor rechtlichen Schwierigkeiten schützen. Auch wenn er dazu nicht verpflichtet ist, kann der Uploader dir eventuell dabei helfen, solche Nachweise einzuholen. Bitte beachte dazu auch unseren allgemeinen Haftungsausschluss.

I am unsure what is the best way forward here. Probably, {{Personality rights/i18n/de-at}} should expand into the German language version, so maybe this is a bug in the Translate extension? Or is this expected behavior? We will probably need to find someone familiar with the extension to comment on this. Knowing this, is there some way that this module could work around this behavior to provide the correct language fallback? Thanks and best regards, ChrisiPK (Talk|Contribs) 10:52, 25 June 2025 (UTC)

ChrisiPK, thank you for excellent analysis, you are the first person who noticed and diagnosed the issue that Module:Autotranslate does not work well with pages generated through translate extension. I guess there are multiple issues here:

1. translate extension is not handling fallbacks correctly. We should file a ticket related to that.
2. Module:Autotranslate does not work well with pages generated through translate extension. That is a new issue and a side effect of last year's change proposed by User:Tacsipacsi. Tacsipacsi, come up with much more efficient version of Module:Autotranslate, which skips a step of checking is a sub-page exists and attempts to render it instead. This approach works, for all the other cases, but fails here. I recreated previous version at Module:Autotranslate/old which is used by {{Personality rights/sandbox}} and that one works fine. {{Personality rights/sandbox|lang=de-at}} gives

Personality rights Bitte Persönlichkeitsrechte beachten Obwohl dieses Werk unter einer freien Lizenz veröffentlicht wurde oder gemeinfrei ist, können durch die Persönlichkeitsrechte der abgebildeten Person(en) bestimmte Nachnutzungen eingeschränkt oder von deren Zustimmung abhängig sein. Ein Modelvertrag oder ein anderer Nachweis einer solchen Zustimmung kann dich vor rechtlichen Schwierigkeiten schützen. Auch wenn er dazu nicht verpflichtet ist, kann der Uploader dir eventuell dabei helfen, solche Nachweise einzuholen. Bitte beachte dazu auch unseren allgemeinen Haftungsausschluss.

While {{Personality rights|lang=de-at}} gives

Personality rights Personality rights warning Although this work is freely licensed or in the public domain, the person(s) shown may have rights that legally restrict certain re-uses unless those depicted consent to such uses. In these cases, a model release or other evidence of consent could protect you from infringement claims. Though not obliged to do so, the uploader may be able to help you to obtain such evidence. See our general disclaimer for more information.

One way, to fix it would be to merge both versions of Module:Autotranslate and allow user to choose either old or new version. We would than recoment to use the old version with translate extension. --Jarekt (talk) 15:13, 27 June 2025 (UTC)

Jarekt, thanks for confirming my suspicion. Regarding the issue of the Translate extension, I have now created phabricator:T398085. Let's hope that this is an easy fix that will be implemented soon. In the mean time, I would suggest to rollback to the old way of checking whether a translation exists to fix all templates that are currently broken. Here is my reasoning for this thought:

• This is a bug, so I think we should not expect the user of the Autotranslate module to make a distinction between which method of checking they want to use.
• I understand that the new method was chosen due to performance improvements, but was there an actual issue with the performance of this module that had any adverse effects? If not, I think we should accept the worse performance and not make the user of the module understand another parameter.
• Adding a parameter would create a lot of additional work: Currently, Commons has a mix of templates translated with subpages and templates translated with the Translate extension. If one of these categories needs to have a parameter added, we would need to go through all of them individually and add the parameter. Do we even have a way of distinguishing between those programmatically, i.e. can we even create a list of all templates that use the Autotranslate module and the Translate extension?

What do you think, should we switch back to the old way of detecting translation subpages? This change could be reverted as soon as the bug in the Translate extension is fixed. The big advantage would be that all this is completely transparent to the users of the module and it would fix all the currently broken templates right away. Best regards, ChrisiPK (Talk|Contribs) 21:39, 27 June 2025 (UTC)

ChrisiPK I am reluctant to be changing Module:Autotranslate which is used on 74M pages on Commons and cloned on 13 other projects. You are right, that this is a bug, but the bug is with Translate Extension, and if we wait until the bug is fixed than no changes to Module:Autotranslate are needed. The issue with performance is important since it is so widely used, and any change triggers 74M pages to be updated, to less efficient version. phabricator:T343131 ticket focuses on some of those performance issues, as we are trying to limit number of expensive operations performed on each page and number of pages expanded in the process of rendering other pages. --Jarekt (talk) 17:05, 29 June 2025 (UTC)

Sorry about the wrong proposal! Unfortunately the fix is not straightforward, there are two problems with simply respecting the fallback chain:

• Real usage of this feature (so-called translation-aware transclusion or language-aware transclusion) traditionally hasn’t used the language fallback, either on Commons or on other wikis. This feature is used on translatable pages (like some Commons- and Help-namespace pages on Commons), not on file description pages; it uses the page language, not the user language, so varieties like Austrian German rarely come into play (if neither the template nor the page transcluding it has Austrian German translation, Austrian users will simply go to the German (Germany) version and get a consistent result there). On the other hand, the fallback chain sometimes contains completely different languages (e.g. Komi-Permyak, an Uralic language, falls back to Russian, an Indo-European language, because it happens to be spoken in Russia), and the goal is to have a consistent experience (a Komi-Permyak-language page can have bits in Komi-Permyak and English – the latter are those that haven’t been translated yet –, but shouldn’t have bits in Komi-Permyak, English and Russian). So introducing the fallback chain would probably cause backlash, even if it was specific to Commons.
• Even if it had complete community consensus, the way this feature currently works allows a maximum number of three “hops”. When I try to transclude {{OTRS Navigation}} on a Hungarian-language page, the first hop is resolving the redirect to Template:VRT Navigation, the second one is the translation-aware transclusion going to Template:VRT Navigation/hu, and the third one is it falling back to Template:VRT Navigation/en. If Template:VRT Navigation/de didn’t exist, the introduction of a fallback chain tried to transclude that non-existing German page anyway on a de-at page, as it’d run out of hops. It’s not documented why this limit exists, but I guess it’s for performance reasons (which are different from what I tried to solve: this ones are about how long it takes to parse a page, while mines are about how much RAM database servers need).

Since it’s not real use of this feature, I’d rather turn it off while this module tries to find the right subtemplate, both to avoid interfering with real usage and to ensure consistency between templates using Translate and templates using manual translation. However, I don’t know how this “turn it off temporarily” could work: the module doesn’t interact directly with the Translate code, so this piece of information should travel through components that aren’t aware of how Translate works.

Do we even have a way of distinguishing between those programmatically, i.e. can we even create a list of all templates that use the Autotranslate module and the Translate extension?

There’s Special:PageTranslation (warning: huge page!), although it has no action API or wikitext-accessible equivalent, so the only way to get data out of it is copying and pasting. There’s a magic word {{TRANSLATABLEPAGE}}, but that returns information only on the current page (and the /i18n subpages themselves usually don’t call Autotranslate). Both methods use a database table, which can be queried through Quarry (example query). A semi-reliable way of finding pages using the Translate extension is that they by convention have the /i18n suffix, but exceptions do exist.

was there an actual issue with the performance of this module that had any adverse effects

Yes, as Jarekt said, the number of database entries caused an actual performance issue. This performance issue affects only English translations 99.9% of the time (the so-called canonical parse always uses uselang=en on Commons, so the English translation is loaded unless overridden with the |lang= parameter), and English has no fallback issues, so it would probably be okay from a performance viewpoint to keep the current way for loading English pages and return back to the old method for other languages, but that would complicate the code a lot. Tacsipacsi (talk) 18:59, 29 June 2025 (UTC)