From the Darling Downs to Dortmund, cities are reckoning with how to clean up bloated digital collections — and Toowoomba's approach is drawing quiet attention.
Our reporters are based in Toowoomba and cover local government, business and community. We are independently owned and editorially independent. Stories are produced and reviewed by the Toowoomba editorial desk. Read about our newsroom → Read our editorial standards →
Toowoomba Regional Council's library and records division has been working since early 2026 to systematically remove duplicate images from its digitised historical collections, a task that sounds mundane until you learn the city's archive holds more than 340,000 scanned items, a sizeable chunk of which were processed during a federally funded digitisation push between 2019 and 2022. The problem of duplicate image files — created when batches are scanned multiple times, migrated between storage systems, or ingested from community donations without deduplication checks — is one that digital archivists across the world are now treating as a live infrastructure problem, not a housekeeping footnote.
The timing is not accidental. The State Library of Queensland rolled out updated digital preservation standards for regional councils in February 2026, requiring collections receiving state funding to demonstrate active deduplication protocols before the next funding round, which closes in October. For Toowoomba, one of Queensland's largest inland cities and home to the region's most extensive agricultural and pastoral photographic records, that deadline has sharpened attention considerably.
The council's records team, operating out of the Toowoomba City Library on Herries Street in the CBD, has been using open-source perceptual hashing tools to flag visually similar images for human review — a hybrid approach that keeps staff in the loop rather than deleting files algorithmically. The process is being run alongside the Local History Collection housed at the same site, which includes photographic material from the Darling Downs dating back to the 1870s. The University of Southern Queensland's library faculty, based at the West Street campus, has provided informal technical guidance on the methodology, though no formal contract has been publicly announced.
Importantly, Toowoomba is not simply deleting flagged duplicates. Items are quarantined in a separate repository and cross-referenced against provenance records before any file is permanently removed. That caution reflects a lesson learned from other institutions: a duplicate is not always a duplicate. The same photograph printed from two different negatives, or scanned from both an original and a copy, may carry distinct archival value.
Peer cities of comparable size and archival complexity tell a mixed story. Bendigo in Victoria completed a deduplication project across its digitised goldfields collection in 2024 using automated deletion, later discovering that roughly 800 images flagged as duplicates were actually variant prints with different crop or tonal histories — a finding that prompted a partial rebuild of the removed files. In Dortmund, Germany, the city's municipal archive adopted a tiered review system in 2023 broadly similar to Toowoomba's current model, requiring a second human check before deletion, and reported a 12 percent reduction in storage overhead within eighteen months. Fresno, California, which manages a regional agricultural archive not unlike the Darling Downs collection in subject matter, outsourced deduplication to a commercial vendor in 2022; the contract cost the city's library system approximately USD $180,000, and local archivists have since raised concerns about metadata integrity in the processed files.
Toowoomba's in-house model keeps costs lower but depends heavily on the availability of trained staff — a perennial challenge for regional Queensland. The council has not publicly disclosed the budget allocated to the current project, but digital preservation work of this scale at comparable regional institutions typically runs between $40,000 and $120,000 depending on collection size and staff hours involved.
The broader context matters here. Globally, cultural heritage institutions are sitting on digital storage backlogs that grew rapidly during COVID-era digitisation grants, many of which did not include deduplication as a funded activity. The result is archives carrying significant redundancy — and the storage and licensing costs that come with it.
For residents and researchers using the Local History Collection at Herries Street, the most practical near-term outcome is a more searchable, accurately catalogued archive. The council has indicated the current deduplication phase is expected to run through to the end of 2026. Anyone who has donated photographic material to the collection and wants to check how their items are catalogued can contact the library directly — the records team accepts enquiries in person or by phone during standard business hours.
Spread the word
About this article
Published by The Daily Toowoomba
Daily brief
Free, in your inbox before 7am. Weekdays.