Hi @paultaylor,
Remeber I told you the processing of files using the dockerized version of songkong seems the be abnomaly slow ?
Well, I made a finding.
As you might know, unraid uses parity drives, and has several write modes you can activate from its settings menu.
I recently wanted to get rid of an old SMR drive that clearly appears to be a bottleneck in my array, and started a copy between this disk (located in my array) and a new one (also located in my array). I noticed that the copy between these disks was capped at approx 30MB/s.
I started searching for solutions, and found this forum post : https://forums.unraid.net/topic/50397-turbo-write/#comment-496144
It clearly explain there is a way to enable a turbo mode, but this also mean all drives has to be spinning in order to enable it. I activated that mode, and went ahead, starting a new copy. The test was not that successful, as I still had terrible copy speeds (around 40/MB/s).
I then tried something else, and copied the content of my SMR drive to my cache (nvme) drive. As the parity is totally out of the loop during this copy process, I reached a stable 150MB/s, which is not that bad regarding the SMR drive I was copying data from.
Bad news is, that as soon I tried to copy the same folder from the nvme back to the array, I got capped at 40MB/s again, even with the “rewrite” mode enabled that is supposed to speedup things.
So what does that mean for songkong ? Well, it means there is. a terrible bottleneck that will make file processing from the array ultimately slower.
I am discussing this topic with the unraid community now, and will get back to you if we figure out a way or process that would improve speeds. It could then be needed to add some sort of option in SK, that would allow to workaround this unraid inherent transfer speed issue.
I’ll keep you posted.