Files that now use the {{DPLA metadata}} template.

Good morning @Dominic: .

While deleting duplicates, I noticed that some files now have the template {{DPLA metadata}} instead of a complete file description. This made me wonder what happens if a user decides to crop the image and upload it as a new version, perhaps to remove a distracting frame or to crop a single person. And that's exactly what I did with File:St. John high school class, St. John, Washington, 1952 - DPLA - ed7c1321cfcf84324824e25c7a66c17c.jpg and reduced the image to three students. File:St. John high school class, St. John, Washington, 1952 - DPLA - ed7c1321cfcf84324824e25c7a66c17c (cropped).jpg. As I correctly suspected, the file description and license were not transferred to the cropped version; this is completely empty. This is not intended and can lead to problems. Best regards, זיו「Ziv」 • For love letters and other notes 06:50, 21 June 2026 (UTC)

I forgot to mention that I had already cropped images from the DPLA bot before. Example File:Steinitz, William-4 - DPLA - 9eeab9ac335135c24b69ab7e36fc5354 (cropped).jpg, everything worked perfectly here, as the original file uses the {{Artwork}} template. זיו「Ziv」 • For love letters and other notes 07:29, 21 June 2026 (UTC)

Further update: As expected, the file has now been tagged by @AntiCompositeBot: because a license is missing. I understand the idea behind {{DPLA metadata}}, it makes your files less maintenance-intensive, but if the template for derived works doesn't work properly, then it's completely unsuitable. I'd rather have more edits from DPLAbot, but users still have the option to create derived works without having to search high and low, possibly having to manually add their own templates and licenses. זיו「Ziv」 • For love letters and other notes 08:13, 21 June 2026 (UTC)

Understood. This is definitely an unintended situation. It's not an error by the bot, but certainly an undesireable interaction with the CropTool. We didn't invent the idea of template metadata pulled from SDC. For example, User:GeographBot has already been doing it for far longer. I tested how those are handled here, and it also loses most of its SDC when run through CropBot, the main difference being that its copyright license seems to be hardcoded so it doesn't get flagged by AntiCompositeBot. I think there is a solution here, but I need to work through the Lua. For example, if File:A is extracted from File:B, is it possible for Module:DPLA to fallback on File:A to File:B's SDC on any fields where they are blank? That would be the solution that works for both the bot workflow and the users using CropTool. Dominic (talk) 20:49, 21 June 2026 (UTC)

@Ziv: I was able to get it to work as I was hoping, via Lua. You can see on your test file. No further edits were made to that file's SDC, but it is emitting the SDC from the {{Extracted from}} file as a fallback (until any metadata is added directly on the file, which would take precedence). This means all CropTool files that copy the template will work natively, as long as the linkage through {{Extracted from}} is kept in place. Dominic (talk) 01:11, 22 June 2026 (UTC)

Dominic! You did it exactly the way I imagined and was hoping for. Yes, I'm familiar with GeographBot. I also once cropped a file uploaded by that bot. I had the same result as you. However, I then went and added the missing information myself. Perhaps your solution would also be something for @Multichill: , who runs the GeographBot? Anyway. Thank you so much for fixing this. Great job! Best regards, זיו「Ziv」 • For love letters and other notes 08:05, 22 June 2026 (UTC)

Maybe on the extracted file consider a "subst" so that the wikitext gets populated, instead of relying on a different file's SDC? - Jmabel ! talk

@Jmabel: The Lua module can only affect the template display, but not make an actual edit. The issue here is that these are uploads being made by CropTool (via user accounts), not DPLA bot. DPLA bot has no knowledge of these files and isn’t maintaining them, since it also doesn’t really k ow what the user has cropped the images to and if it would be accurate to continue syncing the original metadata to it. The template is designed so that using wiki text parameters will override the SDC fallback. But actually hardcoding it by bot would need to be done on the CropTool side. Or we could run a periodic scan for these, but that is a lot of architecture and still leaves them in the state they are in now in the interim anyway. Dominic (talk) 19:48, 22 June 2026 (UTC)

CropTool has been under continuous active development lately, so it's plausible that side could be addressed. Doc James, any thoughts? - Jmabel ! talk 23:37, 22 June 2026 (UTC)

Sure will ask User:Bawolff to add support for this work flow if possible. Doc James (talk · contribs · email) 00:57, 23 June 2026 (UTC)

@Doc James: I think the answer might be in the "Metadata to copy" feature that CropTool already has. Can all the DPLA-sourced SDC (we have a standard reference statement for all) be available options for the user to copy in there? Or, if those are pre-programmed, you can see the properties we add here: COM:DPLA/MOD. Dominic (talk) 16:15, 29 June 2026 (UTC)

@Dominic: This doesn't seem to be working again File:St. John high school class, St. John, Washington, 1952 - DPLA - ed7c1321cfcf84324824e25c7a66c17c (cropped).jpg. The DPLA description is missing once more, and AntiCompositeBot has tagged it again. Why was this changed back when the problem seemed to have been solved? זיו「Ziv」 • For love letters and other notes 12:20, 5 July 2026 (UTC)

I’ve identified the problem: the DPLA bot has since moved the file from which the crop was created to a new name. The issue, however, is that the cropped version isn't being updated accordingly, which in turn has resulted in the license information being missing. Ultimately, this could result in crops that were created years ago—and held a valid license—suddenly losing that license simply because the DPLA decides to move file in 2032?? to a new ID. Solution: The bot should recognize that a crop exists and then adjust it accordingly to the new ID. Best regards, זיו「Ziv」 • For love letters and other notes 12:32, 5 July 2026 (UTC)

@Ziv: As long as the extracted-from file is still a redirect, seems like it would work in Lua. Let me look into it. Dominic (talk) 14:52, 5 July 2026 (UTC)

@Dominic: Yes, but only because I changed it manually to the new fielname. However, this shouldn't be imposed on the average user, who is now wondering how to solve the problem and whether they did something wrong. זיו「Ziv」 • For love letters and other notes 15:35, 5 July 2026 (UTC)

Addendum: The bot overwrite the old redirect for the new file. זיו「Ziv」 • For love letters and other notes 15:36, 5 July 2026 (UTC)

Please tag fewer files as duplicates in batches

Hello @Dominic

I myself wrote that I can manage several hundred per day, that certainly includes administrator @Túrelio: , who processes the category correctly, who works a lot in this category and always creates a redirect, unlike other adminstrators, who simply deletes images without creating a redirect; I personally don't think that's right, and it certainly shouldn't be the goal to simply just delete files. However, it is unacceptable for other bots, such as the one run by @Don-vip: , to stop tagging duplicates because the limit of 190 has been exceeded. Yet those duplicates ought to be processed at some point, which becomes impossible when the DPLA-bot tags several thousand files at once. However, you mustn't forget that it's not just bots using this category, but also ordinary users who pay no regard to processing limits. As far as I'm concerned, the bot can be programmed to detect when a limit has been fallen below and then tag new duplicates, like OptimusPrimeBot does. Best regards, זיו「Ziv」 • For love letters and other notes 11:47, 5 July 2026 (UTC)

Another solution would, of course, be for OptimusPrimeBot to stop adhering to a limit as well, or for that limit to be raised from 190 to ???. זיו「Ziv」 • For love letters and other notes 11:57, 5 July 2026 (UTC)

Thanks, I understood the need to not overload the admins maintaining the category, but didn't take into consideration that if the bots have different thresholds, DPLA_bot could crowd out others. And I also manually tagged more recently when you were active jut because I was trying to get through my backlog. Part of the problem is when the duplicates number in the thousands, I also get anxiety that another Commons editor will come across it in the wild and complain about it. I am trying to conceptualize a new approach for this where I can fully tag the backlog, but they are only released to Category:Duplicate a few at a time. Here is what I am working on: I made {{DPLA duplicate}} and Category:DPLA duplicates for deletion. The purpose is to allow my bot to fully tag the backlog, without crowding Category:Duplicate at all. Then I have a bot job that checks Category:Duplicate on a regular cadence, and only fills it up to 100 DPLA files, refilling from the backlog (using a timestamp-based moving window). This makes it a backlog that the bot can fully self-maintain without other inputs, aside from the one-time initial tag run. Dominic (talk) 18:09, 5 July 2026 (UTC)

It's nice, thanks a lot @Dominic <3 vip (talk) 21:06, 5 July 2026 (UTC)

Very well done @Dominic. Thanks a lot. Best regards, זיו「Ziv」 • For love letters and other notes 07:50, 6 July 2026 (UTC)

Okay, great! I set the threshold at 100, and the check runs every hour. So once per hour, if the bot detects fewer than 100 DPLA files in Category:Duplicate, it fills it back up to 100. I figure since it is filling it back up to the limit every time, if I use 190, then it still crowds out the other bots, because it will use up the 190 constantly. That way the DPLA files can churn and be taken care of but without ever overfilling the backlog. Of course, you won't see it trigger until the current backlog drains back down to that level, but with thresholds set it means DPLA will back off let you work through those. Category:DPLA duplicates for deletion is up to 3000, but I haven't added the main queue I have there yet, until we give it some time to see it working to everyone's satisfaction. Dominic (talk) 16:01, 6 July 2026 (UTC)

bad changes

I do categorization work. This bot is making bad changes to the categorization of images--undoing my careful work. For example, look at the history of File:Heppner United Methodist Church, Heppner - DPLA - 7efbcd83689af0bd5caa71d1299bcea6 (page 1).jpg Thanks Hmains (talk) 02:27, 11 July 2026 (UTC)

convenience link: File:Heppner United Methodist Church, Heppner - DPLA - 7efbcd83689af0bd5caa71d1299bcea6 (page 1).jpg - Jmabel ! talk 02:48, 11 July 2026 (UTC)

If you look at the thread immediately above, I have been trying to gracefully fix some of the community edits that were overwritten in migrating templates previously. If you made edits in the meantime, between when the bot started to roll back its own edits to the previous state, then those new edits got accidentally rolled back too, as it was working off of a list of files. Rather than accidentally continue to revert-war, if you would be okay waiting a day, I will just run a sweep after this current batch of edits finishes to restore any Hmains edits that were overwritten. Thanks for letting me know! Dominic (talk) 02:33, 11 July 2026 (UTC)

I can't say that I understood the chain of comments you point out. I just thought I should mention what I saw. Not interested in war; not here for that. Thanks. Hmains (talk) 02:40, 11 July 2026 (UTC)

(talk page stalker) @Hmains: you sort of got caught in the cross-fire here. Dominic made some large bot edits that were basically sane but not fully thought through as to their consequences; he's now running more bot changes to fix those. I think he's probably on the right track, but he's having to have his bot recover some wikitext content it accidentally lost on that prior go-round, and it has to go back to older versions to pick up that content. When it does that, sadly, recent edits (from after that prior go-round) inevitably get lost. I'd recommend not trying to edit any of the DPLA stuff for another couple of days. After that, I think it should be OK. - Jmabel ! talk 02:55, 11 July 2026 (UTC)

Understood. Thanks Hmains (talk) 03:12, 11 July 2026 (UTC)

@Hmains: I ran a subsequent sweep to fix any edits of yours or others that got overwritten. See examples , , . Please let me know if I missed anything. Thanks! Dominic (talk) 17:11, 12 July 2026 (UTC)

Thanks, I will look around. Hmains (talk) 17:58, 12 July 2026 (UTC)

A glitch

Looks like something went quite wrong with your recent bot edit at File:Seattle waterfront at Pier 4, ca. 1910 - DPLA - 885aa5020b31023925ce94b55e801b8d.jpg, including loss of ImageNotes. - Jmabel ! talk 04:38, 10 July 2026 (UTC)

In that file, and also in File:Tidelands SE from Occidental Hotel, ca. 1884 - DPLA - dcdd800c897fb9d1de321c352ae79db9.jpg (no ImageNotes there), my notes in the description got completely dropped. In both cases, my notes in the description are more or less incorporated by SPL's updates, so that is not disastrous, but I presume that the bot dropped the content without knowing that.

FWIW, normally when I see an edit summary like "Update description after title drift correction (DPLA ID dcdd800c897fb9d1de321c352ae79db9)" from a bot, I don't check the edit, because I assume it is just a bot doing something routine. Also, I don't necessarily watchlist every file I touch. Consequently, I might have failed to notice any number of similar errors by the bot. I'm definitely concerned that it may have killed ImageNotes on many other files, and somewhat concerned that in discarding my notes in the description for some files, significant content might have been lost. - Jmabel ! talk 05:05, 10 July 2026 (UTC)

I’ll definitely look more closely into this. I can already see the cause, since this looks like it’s after a rename. The “after title drift” is a different code path than the regular one, that is invoked when the file has to be renamed too (i.e. in the years since upload the name or ID changed). It’s different because in those situations we don’t preserve all the original data (e.g. the old DPLA ID and URL are wrong and need to be overwritten). It looks like the community metadata preservation code I built didn’t touch this flow. Luckily, everything is recoverable from the edit histories, and I’ll make sure to repair these. Dominic (talk) 11:04, 10 July 2026 (UTC)

Thanks. Looking more closely, there is info in my notes for File:Tidelands SE from Occidental Hotel, ca. 1884 - DPLA - dcdd800c897fb9d1de321c352ae79db9.jpg that did not make it into the SPL rewrite. - Jmabel ! talk 20:39, 10 July 2026 (UTC)

File:Ethel, ca. 1910 - DPLA - 4bea96f9f7d38804ad6f7038571a6169.jpg: my correction on the "Creator" info has been lost, and older content from SPL, now dropped, is misidentified as additional user-contributed metadata. - Jmabel ! talk 21:43, 10 July 2026 (UTC)

Thanks. I don't know if you're seeing my edits on the other examples, but I am testing a script that will run through all of these over again and rescue everything that was lost. I am just trying to test all possible cases still. Dominic (talk) 21:46, 10 July 2026 (UTC)

I take it that means I don't need to keep checking these one by one? That would be very welcome, because, to paraphrase Lewis Carroll, this is an awful lot of running to stay in one place. - Jmabel ! talk 21:54, 10 July 2026 (UTC)

This is one that has been rescued: File:Ethel, ca. 1910 - DPLA - 4bea96f9f7d38804ad6f7038571a6169.jpg. I will test some more cases, including everything you have given me, before running it across the whole set. Just wanted you to know I am working through it! But yes, if you can hold on for a day or two, I think I can take a pass on everything and then give you a summary of the result, so you don't need to keep checking for the time being, I don't want you to have to tak up your time with this. Dominic (talk) 21:58, 10 July 2026 (UTC)

Sorry, I meant for that link to be this one: . I'm looking at the others. Dominic (talk) 23:00, 10 July 2026 (UTC)

File:Greater White Fronted Geese in South Park pond, ca. 1890 - DPLA - ed99c4a1b626ff35d4dd9fd6e008cd8e.jpg lost an ImageNote and also lost my explicit indication of unknown photographer. - Jmabel ! talk 21:54, 10 July 2026 (UTC)

File:Helix, v.4, no.10, Oct. 10, 1968 - DPLA - c0c34376a3f9eab625c173e036b6238c (page 1).jpg lost my accurate information on who did the artwork here, inappropriately returning full "Creator" credit to the editor. Assuming that is a systematic error, that will come up heavily on Helix pages. - Jmabel ! talk 22:09, 10 July 2026 (UTC)

File:James St. and Yesler Way, 1880 - DPLA - dd06267e68254c9697fbda33fa5ca044.jpg lost all "other versions" info. I bet that is also a systematic error. - Jmabel ! talk 22:33, 10 July 2026 (UTC)

By the way: as you move away from having any of this in Wikitext, what are intended to be the correct ways, going forward, for someone to:

• add to or contest the description?
• refine, contest, or outright change the author info?
• refine, contest, or outright change the date?

Also: I notice that for File:Tidelands SE from Occidental Hotel, ca. 1884 - DPLA - dcdd800c897fb9d1de321c352ae79db9.jpg, even your latest update fails to use {{Creator:Theodore E. Peiser}} or equivalent, it just gives his name. Jmabel ! talk 00:35, 11 July 2026 (UTC)

@Jmabel: The Tidelands photo actually looks correct for me. I think you just need to purge your browser cache. The way I have constructed {{DPLA metadata}}, all the regular wikitext parameters that have always worked in the templates we are migrating from can still be newly applied. Also, anu user-added statements in the SDC for the template parameter-based properties also render in the yellow box. Rather than make changes to the institutional metadata, the user's contributions sit alongside them now. This is the only feasible way to ensure we sync/update the instition's own metadata as it changes over time without accidentally overwriting the Wikimedia user contributions. It also is good for attribution of where the data is coming from. This means the "contest" or "outright change" parts of your question are made more difficult, admittedly. I think the the point here is that there is the wikitext parameters that afford Wikimedians the opportunity to call out corrections in free-text, so you can specifically reference the institutions metadata and state explicitly if your contribution is a correction. I tried to document all of these usage notes at Template:DPLA metadata/doc#Editing DPLA-uploaded files, but you're free to edit or provide feedback on that as well.

Aside from that, there is a real side opportunity here, which is that this allows us to surface the Commons user enhancements to the metadata. And that's a big reason I am doing this as well, because it means we can have a much easier time with data roundtripping, by actually being able to isolate and package up these changes to submit them back to the institutions for correction.

Right now I am running my first pass at a remediation script. Some of them take multiple edits because SDC and wikitext can't be edited together. There are thousands so I will check on its progress tomorrow. Thanks! Dominic (talk) 02:22, 11 July 2026 (UTC)

I haven't fully analyzed that, but two immediate questions:

• In Template:DPLA metadata/doc The values appear in the yellow user-contributed box, alongside (or instead of) the DPLA-authored values: I don't follow "alongside (or instead of)". Obviously something determines which it is, "alongside" or "instead of", but the template page is unclear.
• May I assume that even though it says pass one or more {{Other version}} calls, there is nothing special about {{Other version}}, and that applies just as well for {{Extracted files}}, etc.? You might want to reword to make that clearer.

Jmabel ! talk 02:34, 11 July 2026 (UTC)

That’s a good point. “Instead” is not currently accurate, though SDC does have a way to rank values, so I had been considering making this a possibility. I’ve been actively editing the template/module, so some of the documentation will need to be proofread again. And yes, theoretically you can put any templates inside the “other versions” field, I just wasn’t aware it was used for other templates like “extracted files”. I will update it! Dominic (talk) 02:43, 11 July 2026 (UTC)

• Re: Creator: space, do I understand that you have no plans to integrate with that at all, and that the only way they can be added and maintained is user overrides? - Jmabel ! talk 02:45, 11 July 2026 (UTC)
• Hm, it's not "no plans to integrate", per se, it's just that our source data does not already have Wikidata identifiers, or even a way to add them back in right now. It's on the road map, and maybe with some reconciliation it would be possible, but we also don't want to introduce data errors by putting a Creator pointing at the wrong Wikidata item, so that is why it's not possible right now. Dominic (talk) 20:46, 13 July 2026 (UTC)

@Jmabel: following up on the problems you flagged: I've now worked back through all the Northwest Digital Heritage records that bug affected and restored the community content it had dropped from what I can tell now (restoring ImageNote annotations, front/back {{Other version}} galleries, transcribed captions and editorial/dating notes, and community-curated categories). The process wasn't perfectly clean, it took a few tries. Feel free to spot-check a few and flag anything that still looks off or if I didn't get all the affected ones. A few examples of restored records: , , , , , . Dominic (talk) 20:46, 13 July 2026 (UTC)

All looking reasonable. I realize that we have two potentially conflicting goals to meet here (keep things sync'd with our sources, allow Commons editors to edit). At least in data terms this solves it moderately well. I'm still not sure it's an optimal solution in terms of presentation, but it's OK. - Jmabel ! talk 21:54, 13 July 2026 (UTC)

@Jmabel: Yes, it's a bit of a challenge, especially because humans act in non-standard ways, which makes it difficult to respect both goals sometimes (we need to overwrite the outdated institutional metadata each sync, without confusing the changes made by humans with the out-of-sync institutional metadata). I think there is most risk on the initial migration, but once all DPLA data is moved to the SDC, it becomes harder for a human to make changes that could be mistakenly overwritten, since our bot is only ever supposed to edit the SDC statements with DPLA references—which is part of the reason for migrating it all to SDC. In terms of the presentation, I think the data part was more the higher priority to come fist, but now we can continue to make changes to the presentation just by editing the template/module. I'm happy to iterate that over time. I appreciate your feedback! Dominic (talk) 22:05, 13 July 2026 (UTC)

Files uploaded by Fæ

Hello @Dominic: .

Unfortunately, there is another batch of duplicate files, ones that @Fæ: uploaded to Commons years ago. We are facing this issue yet again. The DPLA needs to find a different way to handle this. I have no desire to delete user uploads from 2017 or earlier via a bot upload. Best regards, זיו「Ziv」 • For love letters and other notes 08:32, 12 July 2026 (UTC)

I forgot to mention: I won't be manually modifying the files from Fæ again, as I did with the last batch. by assigning them DPLA descriptions and moving them to the desired DPLA filenames. Instead, I simply delete the DPLA upload as the "newer" duplicate. זיו「Ziv」 • For love letters and other notes 08:42, 12 July 2026 (UTC)

Thank you. Just a reminder to anyone editing the millions of files I have worked on for many years, deleting the categories or text information from the file pages and substituting SDC fields that can break maintenance tasks is not acceptable. Adding SDC information is not controversial, and nobody has any objection.

There is no consensus to "erase, delete, blank, substitute" (by whatever pseudonym) plain text information on image file pages and the SDC is not a WMF guaranteed long term service. The last thing anyone wants is to make life more difficult for future volunteers or reusers that rely on the internal search to find content that may become effectively hidden by reliance on SDC and Lua transcoding. Fæ (talk) 09:08, 12 July 2026 (UTC)

Thank you very much for your detailed reply @Fæ.

As I (and a few others) already tried to explain in strange dupe-tagging, the DPLA upload isn't actually necessary at all.

• File descriptions and reputable upload sources from other sites and institutions should be respected and do not necessarily have to be replaced by a DPLA description, because the DPLA has the same image in its collection.
• File names that users assigned to their files years ago, and which were uncontroversial, should not be changed and do not require a DPLA ID number.

One solution would be as follows: as soon as the DPLA-bot detects an identical user upload for a given file, it tags the new upload as a duplicate. The file itself is placed on an "internal blacklist" from the DPLA so that the DPLA-bot does not upload it again. Consequently, neither the description nor the filename needs to be changed or adjusted. A DPLA version of the file is then not necessary. However, it would be helpful if the bot added the DPLA link under "other version" for such files, so that it is clearly evident that the file can also be found in the DPLA archive. Something like File:Angus F. Bartlett, Captain, Company H, 3rd Missouri State Militia Cavalry.jpg, or better, of course. זיו「Ziv」 • For love letters and other notes 09:39, 12 July 2026 (UTC)

That example '22412cd0994d36f03c4fcf549db2b8e5' raises additional questions. Why did the bot upload a second version of the same resolution but smaller file size 2 years later? Why did that second version match the filesize of the version I uploaded many years earlier? Not sure I want to be the one checking for root causes, however someone may wish to double check the SHA values from the API. There may or may not be 'digitally identical' duplicates as well as visual/colour/resolution duplicates. Admittedly this example appears fully identical, however if there were scans made at different times, or even different physical photographs made from the same negative, there could be reasons to keep other non-digitally-identical duplicates.

If the DPLA bot is creating duplicates with identical SHA values, that is very easy to fix and something the bot creator will no doubt retroactively fix if this has been happening for some time. I'm resisting the desire to investigate myself, there are more useful rabbit holes to spend time on.

Tip for mass uploaders - the IA has files that appear to have changed encoding over time, despite the identical file name. This means that an identical image or pdf might be a slightly different filesize and no different quality than one uploaded years ago. A simple SHA check may not be enough for the bot to act for hundreds of thousand of images at a time without more sophisticated measures. For my own code the IA ID is checked in addition to anything else, and duplicates with different file sizes and file names but using the same IA source are detected and skipped as a precaution against unnecessary duplicates. I suggest other large project maintainers take such cautious steps and test and test again before rolling out big changes. Fæ (talk) 09:52, 12 July 2026 (UTC)

@Fae: The technical answer—as to why it uploaded these files 2 years ago that are now being replaced with new, usually higher-resolution versions—is that we were uploading the files their IIIF image server gave to us at the time, and that for reasons unknown to me, those files have now changed in a way that matches these prior uploads. The prior uploads were not detected before, since they were non-identical. I don't have any direct evidence, but my guess is that they were previously had a limit imposed that was later removed. So this is not an attempt to roll out a big change, but to reduce the duplication. Dominic (talk) 16:55, 12 July 2026 (UTC)

It's not clear you have examined this example case. Neither of the 2 versions uploaded was "higher-resolution". The reasoning given here is untrue. This bot uploaded the same file twice, two years apart, on both occasions they were duplicates and this was not fixed. The bot should cease operating until this is tested and fixed correctly. Fæ (talk) 17:04, 12 July 2026 (UTC)

@Ziv: I may have misunderstood our prior conversation, but I thought we had already reached an understanding. Namely, in these situations with the more-established Fae uploads or equivalent cases, I already said I am happy for you to resolve these as you see fit, and if that means you are deciding to deleting the newer DPLA upload, that is fine. (The only thing I didn't do was change the code's logic for which file to apply the tag to, because I didn't think that you were asking for it.) I do still have an interest in cleaning up the duplication, but I don't have any problem with how you are handling it—so, just to be clear, please let me know if there is anything I should be doing differently.

I just want to clarify one thing, since you have mentioned "a DPLA version" as if one of our goals might be trying to get our own version or ID/metadata on Commons. Just to be clear, DPLA is a partnership that the Missouri Historical Society is part of, and I am doing this work together with their staff (who I also trained on editing Wikipedia to add their images). There is no such as a "DPLA version," they are providing their actual media and data directly to us, which is the whole point of this project: to add all the current media files and their data, which may have changed from other uploads that are almost 10 years old. Until we can't correct old metadata until we standardize them so we can actually detect a file reliably. None of this is motivated by trying to enforce a certain way of doing things or promote DPLA versions of images, just striving for accuracy. Dominic (talk) 16:55, 12 July 2026 (UTC)

"You saw the problem, you fix it"?

Please disable the bot until the problem it creates is fixed and has been convincingly tested as fixed. The bot is "misbehaving" and should be blocked if the operator is unwilling to handle this problem per COM:Bots. It is the bot operator's problem to fix past mistakes, nobody else should be spending their volunteer time doing this maintenance. Fæ (talk) 17:09, 12 July 2026 (UTC)

Fae, you are twisting words. I did not say that, and what I did say was in teh context of telling Ziv I am agnostic as to how the problem should be solved, just that I am happy to implement however he says so. You seem to reflexively think everything is is bad. What, specifically, are you asking for or calling misbehaving? I am trying to be responsive but instead of answering me you just keep saying it is misbehaving. The idea that no one else on Commons volunteers to help maintain bulk uploads is wrong, and you know it. The bot is emitting tags to reduce duplication ("the problem"), but you want to disable it until it? The bot cannot delete files, so that part, of course, does require assistance of Commons admins. Perhaps one could ask why your bot introduced data errors, such as mass-uploading files which say "Copyright undetermined" in their permission field—which the institution was not happy about when I met with them—and which my time is being spent trying to fix. But Commons is a collaborative project, and we recognize both your original contribution, and that no one owns files. What you are saying is self-contradictory, and I am not interested in arguing on a weekend. If you have something that needs to be fixed, please say so. Dominic (talk) 17:27, 12 July 2026 (UTC)

It is reasonable to expect the bot operator to repair mistakes and to analyse root causes, not expect other volunteers to do this for them. Here's the analysis I suggested above that I did not want to do. It demonstrates that the DPLA bot was, and still is based on the upload date of yesterday, overriding automatic duplicate warnings.

SHA1 from 2017 - 9719e05ab718aac6d400b239792ceeb45a766954 File:Angus F. Bartlett, Captain, Company H, 3rd Missouri State Militia Cavalry.jpg.

Analysis of the duplicate file history:

Timestamp | Contributing User | SHA-1 Checksum

------------------------------------------------------------------------------------------

2026-07-11T06:00:00Z | DPLA bot | 9719e05ab718aac6d400b239792ceeb45a766954

2024-12-17T09:31:24Z | DPLA bot | 9639178e1cacf0ddddb51666dc94e0c07076e153

This demonstrates that the DPLA bot by design is ignoring API responses warning that the upload was a duplicate. Fæ (talk) 17:33, 12 July 2026 (UTC)

"If you have something that needs to be fixed, please say so."

The DPLA bot is uploading duplicates. Please fix it and disable the bot uploads until it is fixed. Fæ (talk) 17:34, 12 July 2026 (UTC)

Again, you are twisting things. These files were already duplicates in the sense that Commons cares about ("an exact duplicate or scaled-down version"). You can look at the prior version and see they are visually identical and were in that state for years. Making the upload did not change that (for the better or for the worse), it is just surfaced the existing issue and then tries to solve the issue identified by emitting a {{Duplicate}} tag. It is not ignoring the API warning, it is using it as a signal to follow a logic path to resolve the issue such that the duplicates are not left in place. Which is motivated by wanting to ensure one of the two is cleaned up—at which point it no longer matters whether two visually-identical files had a SHA1 match or not. That is what you claim to want. What is happening now is reparative. This is the fix, but you are still saying to disable it. If you have anything constructive to say about how the workflow could be improved, I am listening. Dominic (talk) 17:48, 12 July 2026 (UTC)

The "workflow" is easily improved. My uploads never ignore the API warnings of SHA1 duplicates, it is the default behavior of the system that such uploads are aborted.

Follow the standard workflow.

There is no excuse to upload SHA1 duplicates as an upload bot or manual upload needs to deliberately override the system's default behaviour. Just stop doing that rather than arguing that somehow this is desirable for Commons or our volunteer colleagues to be forced to respond to an indefinite number of duplicate uploads done by a bot that flags its own digitally duplicate uploads with a duplicate template rather than avoiding uploading the file in the first place by taking notice of the API warnings. Fæ (talk) 17:58, 12 July 2026 (UTC)

Sorry am I missing something. Is the DPLA bot not uploading duplicates since yesterday as shown in this example? My apologies if this is what "reparative" means in this case. Fæ (talk) 17:59, 12 July 2026 (UTC)

Can you not look at the prior file as I stated and see it is a pre-existing duplicate? You yourself said not all duplicates are SHA matches, which is what we are trying to fix, and now you seem confused by me agreeing this is a problem and tagging the 2-year-old duplicates. The upload yesterday is not what is creating the duplicate, as I think we all agree. It just happened to graft a SHA match onto a file that already needed to be removed as a duplicate, which is not creating any strain on volunteer resources. Dominic (talk) 18:20, 12 July 2026 (UTC)

This is incorrect. The new upload yesterday by the DPLA bot was the SHA1 duplicate (the "Analysis of the duplicate file history" could not be clearer).

The upload by the DPLA bot in 2024 was not a SHA1 duplicate and so would not have been rejected by the API. As such the 2024 upload was a "non identical" duplicate with a significant file size difference to the 2017 upload. Such non-identical duplicates are normal on Commons precisely because they are missed by the API. We have created housekeeping tasks in similar situations, such as for milhist / dvids uploads.

Regardless of other processes which might identify non-identical duplicates by checking ID numbers or similar, no automated upload process should allow files with identical SHA1 values to be uploaded. It is not just trivially easy to avoid, but it is a deliberate design decision to force an override of the API. Fæ (talk) 18:57, 12 July 2026 (UTC)

At this point, you are just upset that an upload went out changing our own file to a visually identical but slightly re-scaled version? An action that causes no actual strain additional work to volunteers (it’s getting requested for deletion either way). Just making sure I understand. Dominic (talk) 19:17, 12 July 2026 (UTC)

Could you read the facts please?

The DPLA bot uploaded a non-identical duplicate in 2024. That is a mistake but is *absolutely fine*. The correct way to find and fix those if necessary is to analyse the source references and unique IDs. In truth, you don't have to rush to fix those, they can be merged at any time and often might sit on Commons for years.

On Saturday the DPLA bot overrode the MediaWiki standard API message for SHA1 *identical* duplicates, which almost all bot uploaders would never do. This is easily avoidable, it actually takes extra work and extra design to ignore the API warnings.

The claim that this creates "no additional work" is weird when contrasted with the simple facts spelled out several times above. Fæ (talk) 04:16, 13 July 2026 (UTC)

Non-identical duplicates are undesirable, and I have heard this from most other editors besides you. That is the situation this process tries to fix. The exact same file (our own) that had an extra upload overwrite is the one we request deletion of, so it only exists in that state while sitting in the deletion queue. The bot intentionally does not have a path that would have done such an upload without a deletion request. It is, again, no more work for an admin to delete a duplicate at the same resolution as a slightly different one, so I don't understand what that specific aspect is such a concern. Most Commons editors I have spoken to do not agree with you about visually-identical duplicates being fine. Dominic (talk) 14:20, 13 July 2026 (UTC)

I no longer understand your responses. Please understand what I have written. There is a world of difference from fixing urgent things and looking to improve things which are not urgent.

No doubt you will sort all this out and stop designing bots which create identical duplicates which are trivially easy to avoid. There's no need for me to repeat the facts already laid out. Fæ (talk) 17:20, 13 July 2026 (UTC)

@@Ziv: For what it's worth, I reversed the directionality of the duplicate tags like the one you mentioned, and also in the code for the future. 02:01, 13 July 2026 (UTC)

Thank you. However, the goal should indeed be for the bot to produce fewer duplicates in the future, and not just when the old file relates to a user upload. No, files that the bot has already uploaded should also be recognized, so that these files would only need the new description and new DPLA ID. זיו「Ziv」 • For love letters and other notes 05:07, 13 July 2026 (UTC)

Before I forget, as a benchmark based on a database query, the DPLA bot created c. 9,000 digitally identical duplicates over the weekend depending on timezone. In 3 months over 120,000. The last 24 hours have seen no identical duplicates created, hopefully indicating that the standard API upload warnings are as useful as they are for other upload bots.

These numbers may be misinterpreted, for example it's easy to double count when running queries. Happy to be corrected if there are better measurements. Thanks --Fæ (talk) 18:18, 13 July 2026 (UTC)

If those numbers are accurate, that's a little worrisome. 120,000 redundant files averaging between 1MB and 10MB per file is between 120 GB and 1.2 TB of wasted storage, more if we also count the lower-res versions that were there before. I understand "storage is cheap", but even so (and especially if it is toward the higher end), that's enough to be worth some thought, especially if it is likely not to be a one-time problem. - Jmabel ! talk 22:06, 13 July 2026 (UTC)

Minor correction, though 'higher resolution' has been claimed before, the precise example above was an identical resolution and a slightly lower filesize. I am unclear why the claim was made giving an impression that the DPLA bot goal was better resolution when the case example did not support it.

BTW, counting over 90 days, this adds up to 472 GB. Fæ (talk) 05:20, 14 July 2026 (UTC)