Preview Only, does not only preview. It performs the operation
There is a separate Preview Only flag for Fix Songs and Delete Duplicates, but AFAIK if you have Preview only enabled for Delete Duplicates then nothing is actually deleted or moved, if you are seeing something different please run Create Support Files.