for older archives, see User talk:Fæ/2021, User talk:Fæ/2020, User talk:Fæ/2019, et seq.

Old projects

• Exemplars for historical documents and maps projects
• 
  Scan released last year at the Internet Archive
• 
  New upload of 1887 address book for Riga, Latvia. (800mb).
• 
  Example map, (created 1774) high resolution map of Africa, approx 373 megapixels in size, similar to a 5 foot square printed poster.

I had a look back at the IA upload project and realized that none of these scripts run because of Python, Pywikibot and internetarchive changes. It turns out I find it quite difficult to remember almost anything about these projects too. So don't be surprised if I'm testing it out and there are flaws. I'll do my best to repair any oddities. I'm seeing '+99' notices from my account which doesn't clear, could be a wm bug for big numbers, so I might not notice changes. It's not deliberate. For the moment please don't ask me to take on large projects, I'd rather pace myself at 'slow'. Fæ (talk) 13:07, 2 June 2026 (UTC)

Thanks, and welcome back!   — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 08:11, 3 June 2026 (UTC)

Another new "feature" of the Wikimedia API is ratelimiting. Added a couple of slow down precautions including slapping down multiprocessing, but it's definitely dogging uploads despite being visible to the API as an established user. It may be necessary to revisit the throttling system rather than bumping into it. It's sad this creates extra work for volunteers.

For the first time the queries found restricted items at Internet Archive like 1826histoirenumismatiquedelare, where the Washington University appears to be claiming copyright in a 200 year old publication. Good grief, I hope this is not a trend that IA is tolerating. --Fæ (talk) 07:09, 5 June 2026 (UTC)

I have updated a process for finding the text of Google cover pages in the recent uploads. The 'pending' queue is at Category:OCR detected cover page. Yet to update the page removal process. There's no hurry and I will get to this slowly. --Fæ (talk) 06:29, 8 June 2026 (UTC)

Note that larger files, like the 800mb PDF transcluded, are now possible thanks to limits changing. The larger files might cause things to break, sometimes in predictable ways like the SHA check from mediawiki taking some time to process and be available on the system. Behind the scenes, these uploads do not behave well and invariably fail to report back that they are successfully uploaded. Fæ (talk) 14:26, 17 June 2026 (UTC)

Extra note about maps - though there is a fairly quick upload of jpegs of maps to Category:David_Rumsey_Historical_Map_Collection, which may take a couple of days, there is a much, much slower process to recover high resolution versions. A rough estimate is that this could take 2 or perhaps 4 months, partly due to the new WMF API throttling limits more than processing time. --Fæ (talk) 12:44, 20 June 2026 (UTC)

@Fæ: Couldn't you technically request permission for the upscale to run on Faebot so it is exempt from the API rate limits? --Nintendofan885^{T&Cs apply} 13:30, 20 June 2026 (UTC)

From what is available to easily read about it, probably not. Faebot has a bot flag, but a brief experiment shows that the API throttle is using IP address. Consequently the map uploads being mentioned here are being forced to sleep 90s before the upload is allowed to complete and then there's another 90s for the filepage to be updated with formatting or a category. These events are in a multiprocessing queue but it makes no difference for the API whether it's a bot or not, or whether Pywikibot can set a bot flag for the action.

Keep in mind, I'm trying not to spend hours at a keyboard, so not looking for complex extra volunteer work or phabricator requests for a fairly modest 50,000 files. Fæ (talk) 13:41, 20 June 2026 (UTC)

Faebot has got running on the somewhat revised toolforge, so the mass link fixing has been moved to the WMF servers and that at least is 6x faster as a result and reduced local processing by about 7%. Other stuff might move there eventually. Fæ (talk) 14:50, 21 June 2026 (UTC)

Fresh eye this morning, and the map upgrading is running around 4x faster locally and probably is obeying the new WMF API throttling rules. The run might be 6 weeks rather than 4 months, which is fine. Fæ (talk) 06:39, 22 June 2026 (UTC)

@Fæ: Nice. FYI the bot flags on Faebot and Noaabot were removed when you went inactive so it shows up as a regular user on Special:RecentChanges. --Nintendofan885^{T&Cs apply} 19:15, 2 July 2026 (UTC)

Darn, forgot about that. Thanks for pointing it out. Maybe that is why there's been such bad throttling on these projects. Fæ (talk) 07:55, 3 July 2026 (UTC)

Welcome back

As you probably have worked out, I've been following this talk page, checking whether DRs of your uploads are valid or not, and responding to the DR with a rationale if I think something should be kept. May I assume you are sufficiently "back" that I can let go of monitoring that?

Welcome back, in any case. - Jmabel ! talk 14:06, 3 June 2026 (UTC)

If you wish, though I'll probably continue to ignore most DRs and let others chip in. After your first million images, the DRs that matter are ones that are a meaningful case study for thousands of others and could be collated with some automated method.

I have noticed your patient work and very much appreciate it! Fæ (talk) 15:21, 3 June 2026 (UTC)

• Yes, welcome back! So glad to see this. Krok6kola (talk) 16:06, 3 June 2026 (UTC)
• It's been almost 5 years. I hope you had a good time! Welcome back. - Alexis Jazz ^{ping plz} 10:03, 5 June 2026 (UTC)

@Fæ Yes, happy to see you here again :) --PantheraLeo1359531 😺 (talk) 09:32, 8 June 2026 (UTC)

Good to see you again, also from me. -- Deadstar (msg) 10:27, 9 June 2026 (UTC)

Marvellous to see you editing again. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 15:15, 9 June 2026 (UTC)

Happy you are back! Nosferattus (talk) 18:15, 29 June 2026 (UTC)

It is great to have your presence here again on Commons Fæ. All the best -- Ooligan (talk) 16:12, 11 July 2026 (UTC)

Welcome back!

Looks like you were gone for a bit. Are you working on any new large chunks of uploads? RAN (talk) 21:44, 17 June 2026 (UTC)

Slightly updated the internetarchive PDF uploading, so these are being refreshed:

Category:Scans from University of Toronto

Category:Old books from American Libraries - this in particular needs contributors to surf the content and imagine categories which can better break up the 150,000 books.

Category:David Rumsey Historical Map Collection - new content but a few thousand jpegs of interesting maps. Some review will be needed as the source library curators may have presumed their collections are public domain by age when it can be more complicated.

Category:Images dezoomed by Fæ is being ever so slowly populated by overwriting the small versions from IA. This is probably why this was not done years ago. Hopefully the maps will not be moved before this dezooming task finishes in several days time.

Category:Books in the Prelinger Library

Category:Genealogy books from the Internet Archive modest number of IA books under this collection but some are eye wateringly large, one has been having an upload attempt of over an hour for a 1,781.2 MB PDF; it may continue to fail though several that succeeded are at the 1gb size.

Category:IA books documents with one page is retrospectively populating. As far as I know, over several years, no volunteers have ever investigated these exceptions...

Quietly restarted the slow job of trimming Google cover pages off pdfs which is harder and slower on local processing. Also been experimenting with nccommons, behind the scenes that's been the ghastly issue of getting timedtext to work on videos and battling anti-bot tools for a site that in theory wants you to take their publications.

There is plenty of gnomic work, so for the moment not looking for anything controversial or technically too challenging. Little projects that do not mean sitting at a keyboard for more than an hour at a time are best. Fæ (talk) 05:20, 18 June 2026 (UTC)

• Are we ever going to get something like "search inside this book" when you are at the pdf page, like we get at Google Books? The only time is see the ASCII text is when I do a general search for all of Commons and I see a snippet, or when the book is transcribed at Wikisource. --RAN (talk) 18:59, 18 June 2026 (UTC)

  The search can be used to find PDF contents matches and is a quick way to find prospective matches if doing copyright or cover page statement testing. For example ("This is a digital copy of a book that was preserved for generations" filetype:pdf intitle:IA) will find PDFs with google cover pages. However there's no way of using the mediawiki to tell you what pages the text is on, or do anything else really smart with it. Well no easy designed way; vaguely remember doing something smarter but can't remember how that worked.

A solution for the IA uploads alone would be to navigate from the commons IA upload, back to IA and then interrogate either the djvu.txt file there, or the xml version. If one needed to filter through, say, 100,000 files, that would be way more efficient than having to download each one and run it through another OCR. Fæ (talk) 19:13, 18 June 2026 (UTC)

There is another way of using a wikimedia tool to do it, but not leaving breadcrumbs for it, as if overused it might get switched off. It solves specific issues of examining a page, but is not intended for a million entire books. --Fæ (talk) 13:15, 27 June 2026 (UTC)

+1. Very good to see you back. Jheald (talk) 13:10, 27 June 2026 (UTC)

Purging thumbnails

Due to old glitches with uploads and other inscrutable problems with WMF operations, some of the very large categories of upload projects have multiple un-thumbnailed files with apparent 0x0 dimensions. These are fairly easy to search and repair individually by purging the cache, forcing a thumbnail refresh, but mass finding and repairing does not have an easy tool for non sysop users. Here's my current pywikibot based solution for fast purging given a category name from a terminal prompt. You just need to know how to install pywikibot, or have a toolforge session.

• User:Fæ/code/thepurge

Fair warning, this is deliberately very fast but API throttling *probably* does not count purge actions as they are non-edits. Worth noting that in the example of incategory:Old_books_from_American_Libraries filewidth:<1 before the programme there were 2,433 files with zero width and after running (a couple of minutes duration) the same files show a correct thumbnail but the search results on-wiki incorrectly show the same number of 0x0 thumbnails. Presumably the database lag takes several minutes or maybe hours. Fæ (talk) 08:35, 29 June 2026 (UTC)

Removing "Historic, Archive Document" cover pages

The refresh of Biodiversity Heritage Library documents from the internet archive has highlighted again the old scans with an extra unattractive cover page with this warning notice. These cover pages were not used at the BHL original archive site, but seem to have been added by the scanners for the IA transfer. When the links to IA are followed, interestingly the cover page is skipped and the document is displayed from the real first page. Policies on these warning notices also seem to have changed and it probably fell out of fashion 9 years ago. Faebot is running through what will be around 100,000 of these pdfs to attempt to trim these manually added cover pages which vary in colour, size and even text between documents. It carefully tries to see if there is a blank second page, often faded to a cream colour, to trim off based on shade and the absence of text, but due to caution may leave these in place in around 25% of cases. The end result is that categories like Category:Henry G. Gilbert Nursery and Seed Trade Catalog Collection/1929 are made fantastically more attractive to volunteers and the pdfs far more likely to be used in articles or elsewhere as is.

The task is attempting a batch to process daily via toolforge in alphabetical order and may take, say, 6 weeks. Fæ (talk) 09:03, 29 June 2026 (UTC)

For 'technical' reasons, this is no longer running in alphabetical order. It was simpler to let the housekeeping run on a random walk basis. --Fæ (talk) 14:55, 12 July 2026 (UTC)

Upload on demand

Hi, I I thought to create a upload-on-demand service where contributors could request a file in posting the ID on a page on Commons (which could be semi-protected to avoid issues, like Commons Delinker). I started coding for Gallica which needs some magic because they deny access to bots. Then I got banned from Gallica... 😥 Now you already have all the tools to do it for Internet Archive. You would not have to check for copyright status. The users requesting uploads would have to do it. What do you think? Regards, Yann (talk) 19:09, 13 July 2026 (UTC)

This exists at https://ia-upload.wmcloud.org/. Never used it myself, but others seem to.

My feeling is that it's a hostage to fortune because the licensing questions can be complex, however looking that the upload log, it's generally used mostly for uncontroversial items.

For my IA projects, I'm happy to see folks make requests at Commons:IA_books#Suggestions_for_Collection_Mirrors, so long as the suggestions are for at least a thousand or more works where copyright is easy to check. There's been some good suggestions so far, even if I'm very, very unhurried. --Fæ (talk) 19:51, 13 July 2026 (UTC)

For what I know, ia-upload was created mainly to create DjVu files, but it is now broken, as OCR is misaligned with the images. See answers to your post on s:en:Wikisource:Scriptorium#Commons internet archive based high resolution djvu creation. Yann (talk) 20:38, 13 July 2026 (UTC)

I believe it works for pdfs. As IA dropped djvu format, then a tool like this should be supported by wikisourcerers, if that is to be the preferred format. Though I just created a way to compile a very high resolution djvu or pdf from IA scans, it's really intended for specific collections rather than general use.

@Tpt: is there a plan to address the complaints about the IA uploader not rendering the text layer correctly? Folks seems quite reliant on it and I suspect the bug is something related to the way tools like pytesseract mark word coordinates versus how the djvu tools expect it to work. I'm also not sure about the wikisource workflow, so that might introduce issues too. Thanks Fæ (talk) 05:16, 14 July 2026 (UTC)

(Aside) - https://en.wikisource.org/wiki/User:Inductiveload/jump_to_file is the script at Wikisource which pulls hi-res scans from IA directly ( As it's accessing a third party site, it's unlikely to be considered for core inclusion or gadget status), you might wish to examine it, to see if it can be improved, or that the JSON approaches could be automated to supplement the PDF copies (with 'archival grade' djvu).. I am the user at Wikisource that asked Inductiveload to look into this, because for some of the CCE volumes PDF (especeially some earlier ones), and other works the overcompression present made the presented scans unreliable for transcription purposes. In addition sometimes the nature of the PDF generation at IA, and the display method used on the Wikimedia side, interacted to reduce the nominal display resolution unexpectedly ( A Technical issue that sometimes cropped up as the cover page was using a different background layer resolution compared to later pages, which the Wikimedia display back-end was applying to the whole document rather than just the cover. ) - Mis-clipped viweboxes, are something that cannot be resolved on the Wikimedia side. ShakespeareFan00 (talk) 16:51, 14 July 2026 (UTC)

IA-Upload mentioned above also assumes that uploaders are competent, in checking certain details, If a high-res Index creator was under development, than it should ideally also consider meta-data the IA JOSN and IIIF manifests provide, possibly cross referenced into Wikidata, so that 'known' works with 'incompatible' licenses can be 'filtered' at the upload stage more easily, by warning uploaders to (re)check the status of a work. As we previously found, however sometimes the IA metadata is not fool-proof. ShakespeareFan00 (talk) 16:51, 14 July 2026 (UTC)

DPLA bot concerns...

The method used for certain meta-data doesn't seem to be compatible with the previous style of finding copyvios.

A serach query like - deepcat:"Media contributed by the Digital Public Library of America" insource:/Date\=19[6-8][1-9]/ hastemplate:NoC-US will not produce results, because the DPLA bot is using SDC exclusively.

Not a deal-breaker, but it means a more complex search enquiry..

The IA uploads by comparison were easy to check by a simple regex.. ShakespeareFan00 (talk) 17:36, 14 July 2026 (UTC)

Correct. As said about ten times now, the DPLA bot has no consensus to erase file page text. If in doubt raise for deletion, it seems to be what the DPLA want as they have given us little choice. Frankly it's bizarre as it literally costs less to leave old text on the page, even marking it as superseded by SDC info if they wish or even hiding it from view with html tags but staying visible in searches.

Terrible, I have zero interest in helping the project. Fæ (talk) 17:38, 14 July 2026 (UTC)

I am just a long-term contributor, and using "tabloid tactics' would be below the academic decorum required by the code of conduct. SDC is a good idea, if it can be queried.. For example: File:Mr._Bryan_in_1899-"I_Stand_Just_Where_I_Stood_Three_Years_Ago."_-_DPLA_-_9e31b287474bfea2ed2ca8f76a335ff7.pdf for example - is Public Domain, the source instiution feels it is. However, there is no simple date in the metadata, (nor indication of which original publication it came from.) As I said elswehere expecting Commons contributors to do academic level bibliogrpahic competence, which would be reasonably expected of a respected GLAM organisation, is a big ask.

(It's also a single page PDF, only containing raster data) which is generally discouraged on Commons.). ShakespeareFan00 (talk) 18:04, 14 July 2026 (UTC)

Courtesy : User_talk:DPLA_bot#This_bot_removed_metadata_to_replace_it_with_SDC. ShakespeareFan00 (talk) 18:23, 14 July 2026 (UTC)

Notification about possible deletion

Bundle DR:
Commons:Deletion requests/Reggio Emilia AV Mediopadana train station
Friniate (talk) 18:33, 19 July 2026 (UTC)

The DR is about two files uploaded by you:

• File:Concrete ribs in a facade (Unsplash).jpg
• File:Reggio Emilia AV Mediopadana, Italy (Unsplash).jpg

Friniate (talk) 18:34, 19 July 2026 (UTC)

DPLA DR casebook

Category:Detected blank

Hi @Fæ

I was just on a bit of a deletion spree. The category is empty now, but the files were useless anyway. There might still be old redirects left over that are now broken. Could your bot possibly find them and nominate them for speedy deletion? Best regards, זיו「Ziv」 • For love letters and other notes 16:44, 27 July 2026 (UTC)

The only files in there had DRs relating to them, so it should not be an issue. The main thing that holds me back on these is double checking whether any individual page scanned from a book or report is in use. It seems very unlikely, but it would complicate a DR. Fæ (talk) 16:47, 27 July 2026 (UTC)

There could well be more, see deletion log as just one example. However, what wasn't possible during a mass deletion was finding all of them. זיו「Ziv」 • For love letters and other notes 16:57, 27 July 2026 (UTC)

Interesting, I did not understand at first. The DPLA argument to keep all redirects is slightly bizarre, like the bot task can't verify unique matches either to the unique ID or the SHA value on upload attempts.

Anyway, looking at it now, but searching the deletion log is slow via the API. It might be worth swapping to SQL, but we are not in a hurry. Fæ (talk) 19:01, 27 July 2026 (UTC)

There seem to be none, if the logic is right. I ran via the API but did not trust it, so this way too. https://quarry.wmcloud.org/query/107732. I suspect there is a housekeeping task that sorts out broken redirects faster than one could search for them... probably.

The caught 'double redirect' is interesting as the image was deleted and the redirect goes to a different format. That's messy. The redirects should have been deleted, keeping these is hard to defend. File:President Richard Nixon bowling at the White House bowling alley.gif. --Fæ (talk) 19:52, 27 July 2026 (UTC)

File tagging File:Google Reader Chinese (2861195992).jpg

shizhao (talk) 03:13, 28 July 2026 (UTC)

Help with automating wikitext

Hello, @Richard Arthur Norton (1958- ) suggested I reach out to you for advice on automating the process of entering wikitext on the works of art uploaded at Category:Images from Frick Art Research Library Photoarchive. I have been entering text manually on the artwork template and it's very time consuming. See File:Allen Bowie Davis-Edward Caledon Bruce-FARL37586.jpg as an example of one already done. Data is from records in the FARL library catalog, example https://library.frick.org/permalink/01NYA_INST/1qqhid8/alma991000022199707141. Any advice is much appreciated! EmbPA101 (talk) 15:10, 30 July 2026 (UTC)

The example page has pretty open html formatting. The solution I use for these is using Python code and BeautifulSoup to pull the data out of the html source code. Each of the fields one would put in an information box is marked up by name on the web page. If that is then formatted as a wiki text page and saved with the image upload by Pywikibot, the one programme could scrape the whole website catalogue.

Alternatively the library.frick.org has json formatted data, try this link. Any tool that can parse this raw json data can also do the job, which oddly enough could be a Google spreadsheet or similar.

If you don't know how to do any of this stuff, it might take you a few months to play around with it and make it work.

As far as I'm aware, there's no generic tool that can do all this for you without some programming. --Fæ (talk) 17:58, 30 July 2026 (UTC)

For the easy Google sheets approach, see https://parsea.io/blog/google-sheets-import-json Fæ (talk) 05:53, 31 July 2026 (UTC)

File:Schackmonster.jpg

File:Schackmonster.jpg (edit|talk|history|links|watch|logs)
Commons:Deletion requests/File:Schackmonster.jpg Dronebogus (talk) 19:12, 30 July 2026 (UTC)