How to find duplicate files across cloud & on-prem

Why duplicate files proliferate

Nobody sets out to store the same file ten times. It just happens. A report gets emailed round and everyone saves the attachment to their own OneDrive. A shared PDF gets uploaded to three different team sites. Someone copies a folder "to be safe" before editing, and the copy never gets deleted. Sync clients duplicate files during conflicts. Migrations copy entire trees into a new system without retiring the old one.

The result is predictable: identical bytes scattered across repositories, plus a long tail of near-duplicates: the same document saved as v1, v2 and v2-final, or the same image exported at three sizes. Every copy consumes storage, gets backed up, and muddies search results. Duplicates are the "redundant" in ROT data, and usually the fastest waste to reclaim.

They also carry a hidden cost beyond storage. When the same document exists in five places, nobody knows which copy is authoritative, edits get made to the wrong version, and search returns a wall of near-identical hits. Removing duplicates is not only about reclaiming space, it is about restoring a single source of truth so people can trust what they find.

Manual approaches and where they break

The instinct is to clean up by hand, and for a single folder that can work. Sort a directory by name to catch "Budget (copy).xlsx" next to "Budget.xlsx"; sort by size to spot suspiciously identical files; scan a document library visually. Built-in tools help a little: file explorers, PowerShell scripts and OS search can list large files or match names.

But manual methods break down fast:

• They rely on names. Two identical files called "report.pdf" and "final.pdf" look unrelated by name, so name-based sorting misses them entirely.
• They are false-positive prone. Same name and size does not guarantee two files are actually the same. You risk deleting the wrong copy.
• They do not cross silos. A file duplicated between SharePoint and a NAS will never show up in a single folder view. You would have to check each system separately and reconcile by hand.
• They do not scale. Eyeballing works for hundreds of files, not the millions in a real organization.

What reliably catches duplicates

The dependable way to catch true duplicates is a tool that recognizes the same file wherever it lives, regardless of its name, folder, system or timestamp. Two files can share a name and be completely different; two files with different names, in different folders or even different systems, can be one and the same. What you want is an approach that groups those real duplicates together and never mistakes two genuinely different files for one.

Done properly, every copy of a file ends up in the same group no matter where it lives or what it is called, so you can safely collapse each group to a single authoritative copy and reclaim the space the rest were wasting.

Beyond exact copies: near-duplicates

Exact copies are only half the problem. A document with one line changed, or a photo re-saved at a different quality, is not identical but still wastes space and muddies your source of truth. The copies worth surfacing therefore include near-duplicates (renamed, lightly edited or repeatedly re-saved versions of the same file), so you can decide which single version to keep.

Doing it across every system at once

The real value comes from checking your whole estate in one pass, not one silo at a time. Duplicates most often hide between systems: the deck that lives in SharePoint, on a shared NAS and in three people's OneDrive folders. Only a scan that covers SharePoint, OneDrive, NetApp, NAS and local drives together can see that a single logical file exists in five places and tell you how much space collapsing it would free.

Once duplicates are grouped, remediation should stay deliberate: keep one authoritative copy, replace the rest with links where useful, and archive or delete the remainder under rules you control. The goal is a single source of truth, not blind deletion. It is also worth keeping an eye on where the retained copy lives. Consolidating everything into one governed library is usually better than leaving the survivor on an unmanaged personal drive.

Finally, treat de-duplication as ongoing, not a one-time event. New duplicates appear every day through the same routine actions that created the old ones, so a scan you run once and forget will drift back to where it started within months. Re-running it on a schedule keeps the win in place and turns duplicate control into a background habit rather than a periodic fire drill.

Find every duplicate with TerraBytes

TerraBytes does exactly this. A single serverless scan spans SharePoint, OneDrive, NetApp, NAS and local drives (your data never leaves your environment and stays encrypted), then groups exact and near-duplicates and shows the storage, euro and CO2 you would reclaim by collapsing each set. Nothing is deleted automatically: you define the keep, archive and delete rules and stay in control. To hunt duplicates alongside outdated, oversized and risky files across your whole estate, start with the storage cleaner and optimizer.