Excessive Non VM Space in Datastore on NetApp

Recently I started to get alarms from a vCenter server stating one of its datastores was exceeding 75% capacity.  I noticed that the datastore in question had less space consumed and less free space than the other datatstore in the cluster of the same size.  I also had roughly 285GB of space consumed in the first datastore listed as “Non VM Space”.

datastores

Using SSH to one of the ESXi hosts revealed different outcomes of the df and du commands to check the space.

  • df -h – 1.1TB consumed on the datastore volume in question
  • du -h – 875GB used on datastore in question

I took into account the thin provisioned VMs, and removed all snapshots but things still didn’t add up.  I eventually found this bug report from NetApp (657692) affecting DataONTAP 8.1.2.  Fortunately the workaround is easy until you can upgrade your filer to a version of ONTAP that has the bug fixed.  Run the deduplication job (after hours, its IO intensive) with the -s switch to force it to scan the stale fingerprint metadata (sis start -s /vol/volname).  After the deduplication job finished all of the missing space had been reclaimed.