Hello,
My solution consists of 21 files and 2 of these files are quite large, (9GB and 42GB). I was excited to see the changes to OttoFMS over Otto and was hoping that the under the hood changes would enable me to use OttoFMS where Otto had failed me in the past due to issues with HTTP timeouts. I was also excited to see the sub-deployments feature, as this definitely helps with one of my use cases for Otto (refreshing my dev environment with live data).
Prod specs: 16 vCPU with 32GB ram running Windows Server 2019
Dev specs: 6 vCPU with 12GB ram running Windows Server 2019
Both servers are running NVMe SSD’s and 1Gbe ethernet.
I queued up a dev refresh deployment using the guide here and kicked it off using my largest file. The initial backup took about 10 minutes, but once the file started from prod to dev started, everything started to fall apart.
The CPU utilization in is currently sitting at about 30%, but the network utilization is abysmal and barely passing 120Kbps on average. The file transfer estimated time remaining is currently sitting at 23 hours and 41 minutes 
Hello,
I think you are experiencing the very slow network file transfers that happens when you try to pull large files off windows. It’s not the “Migration” part of the process that is slow. It’s fetching part.
This is a known issue related to Windows IIS aggressively throttling large file transfers. We have a doc about it.
The doc also has a work-a-round and a video that hopefully will help you get around this issue.
We continue to look for ways to turn off IIS file transfer throttling, but so far haven’t come been able to increase the limit above about 120Kbps. This happens even if you just serve a large file out of the HTTP fdoc. The limit has nothing to do with OttoFMS. It’s pretty frustrating frankly.
For now there is a work-a-round.
Let me know if you think I understood your scenario.
Thanks
Todd
1 Like
Awesome, I’ll give this a whirl. Thanks Todd!
A couple of the notes
I misread your download speeds. I thought you said 130MB/s but you said 120KBS that is really slow. You should be able to get 120 to 130 Mbs.
We suspect that you may be in a virtual machine and running IPv6. There are some settings you can adjust on network interface.
We will update the docs with this. But in the meantime, this windows setting on the server that is serving the file makes a big difference, if the server is a Windows VM
Could you let us know of that helps or applies to you.
Thanks
Todd
Well, looks like it didn’t really help. I cranked up the compression and it managed to get the file down to 6GB, but the file transfer speed is still abysmal…
Resource monitor says it’s averaging about 20-50Kbps…
Lets make sure we are talking about numbers on the same scale. Because we see 120MBS, which would mean our servers are over 2000 times faster. I find that hard to believe.
Our standard small file test is a 5 GB file. It usually takes about 5 to 7 minutes to download from a windows server 2022
A 30 GB file takes about 30 minutes to download.
How long does your file say it will take?
Also have you just tried downloading a large file from the HTTP directory. ( IE removing OttoFMS from the mix ). What speed does that show.
Also do you have anti virus software that is running on the either machine that might be trying to scan the zip files?
Todd
Sorry, the screenshot shows 180Kbps or so, but it really does fluctuate between 5Kbps and like 50Kbps most of the time. Eventually, the processes crashes after about 8 hours or so…
If I transfer the file using SMB, I get a solid 100-125Mbps. It takes about 12-15 minutes for the full uncompressed file (~46GB) to transfer off the server. We are running Windows Defender ATP but have exclusions for the entire OttoFMS installation path. You can see the MsMpEng.exe process is sitting idle in the screenshot.
Hi James,
SMB says something about the network, but not really HTTP with IIS. That is what we are using.
Can you try that? Just throw the file in FileMaker Server HTTPS directory and download via a browser. Ideally from the other machine.
Todd
Todd
I regularly move files around using IIS to download files of 30GB+ and see download speeds of saturating my 1 gigabit connection. And when going server to server with 10 gigabit files taking only a few mins to download via IIS.
Can you explain more of what the issue is.
Daniel
Hi Daniel,
Based on a lot of googling and our own experience, Windows IIS has problems with downloading large files through a reverse proxy, which is how OttoFMS connects through port 80/443. We know it isn’t OttoFMS since it doesn’t have the same issues on Mac or Linux. And if we tweak OttoFMS to go back to using a Port, by passing IIS on windows the speed is fine.
We could enable using Port to bypass IIS, as an option. But it would be over HTTP not HTTPS and it would mean having to opening a special port for that.
Todd
1 Like
