Otto 3 to OttoFMS - things go wrong

OttoFMS
1

I have been using Otto 3 for about 5 years at a customer site where it worked gradually better and better over the years, the last 20 deployments were done in under 10 minutes and gave 0 errors.
So I had quite a bit of confidence in version 4, and the first few tests - with smaller customers - gave good results.

So…followed the instructions: uninstalled Otto v3, and installed OttoFMS. Ran some Windows updates and restarted both the dev and prod servers.
Defined a deployment, reviewed it and started it.

Observation 1: the “build phase” takes about as long as the whole deployment in Otto V3.
Observation 2: fetching the files is very slow. Typically about 200 KB/sec throughput on a 10 Gigabit network. 443 MB is 43% complete after 18 minutes.
Obeservation 3: deployment crashes after 68,8% trough the fetching phase.

I restart the OttoFMS services on both ends and retried the deployment, but I can see the same timings.
Mind you that this deployment ran on Otto V3 in under 10 minutes, everything included.

This is version 4.3.1, I saw earlier reports of slowliness and fixes in subsequent OttoFMS releases.

I think it was a mistake to upgrade, and I am now forced to downgrade again, because the customer site is already down for a few hours.

I just hope that OttoFMS wil be ready when the Otto license expires in november. It would be very cool if it would continue to function, until these kind of problems with the new version have been fixed.

This might sound a bit negative, but I must also say I am very happy with OttoFMS for all other customers, and I am telling just about every customer they should have it installed on their FMS. It’s just not ready for the big deployments yet.

[EDIT] trying a re-install of v3, so first task is downloading it:

image
image869×177 4.48 KB

Just saw that the second deployment crashed as well during the fetch phase, around the same percentage. As I cannot download the old version, I think I know how I will spend the rest of my Saturday now… :frowning:
The macOS version is downloadable. Not really helpful if you have Windows servers…

2

EDIT:

All versions of Otto3 are available for download on the Otto3 version history page.

Hi Peter,

Sorry you are having problems. I am going to guess that you are using Windows.

It is true that Windows remains slower than either mac or Linux. We have traced the issue to windows IIS. It is throttling downloads. We have two work-arounds that we are testing that we hope will improve just in time builds.

But I wonder, have you looked into our documentation on how to handle larger files on windows.? Specifically have you looked at building ahead of time? It is relatively easy to do, and it removes the slowest part of the process from the deployment, reducing the time the files are closed on the production server.

Mac and Linux are both very fast across the board. Windows is slower for Just in Time Builds, but otherwise as fast as the others.

I am sorry that you feel you made a mistake in upgrading too soon. We purposefully gave 18 months of overlap between Otto3 reaching end of life and OttoFMS to make sure there was enough time to for everyone to get comfortable with the change. We still have over a year left on that. So if downgrading seems like the right thing for you to do now, I hope you will keep an eye on developments here. I expect that we will be able to resolve this last issue with Windows servers soon.

Thank you

Todd

3

Oh and I forgot. We already have something that will help coming next week.

Version 4.3.0 of OttoFMS includes closeFilesAfterBuild option for the builds. This will delay the closing of Files on the production server until after the files are built and fetched. We don’t yet have that option exposed in OttoDeploy. That will come next week.

Once you can use that, the time that the files are closed on the production server will only be the time for the actual data-migration, even when you do a Just In Time build, on windows server.

in that case it may be that the impact on the production server in terms of downtime, should be less than with Otto 3, which doesn’t have this option.

Hope that helps

Todd

1 Like
4

I did not expect any response over the weekend.
Also mailed support, and got both very helpful replies from you and Angelo.
Thank you guys so much for that!

It is indeed the Windows version that has issues. I have tested the Ubuntu and macOS version, and assumed that the WIndows version would be OK as well.
Yeah, Mother Assumption.

And I did not read the documentation thoroughly enough. What is written there about large file deployments on Windows makes complete sense, and I’m sorry I missed that.

I prepared a build, and created a deployment with a build url. In the build, I included all files, but I saw there was only a “clone” option. Which I did not expect.
In the deploymen options I saw the “Install/Replace” option, so thought that was the way to go for development files that did not have to be migrated, but just copied onto the production server.
I was wrong. After deployment they were empty. Make sense in a way, the clones are just copied. Doesn’t make much sense to give the option, because that does not make any difference.

The problem is not with files being migrated, but with files being copied. But the build option can only use clones. So there is no point making a build IMHO.
Also there is not much time saved in making a build, because it only makes clones. If it were to make compressed files, this would have been the way to go.

So OK, not a big deal. I migrated 19 files using OttoFMS and just copied the 6 remaining files manually.

I revisited the build screen over and over again to see if I was missing something, since there is only a clone option. But there is now way to include a full file.
I hope this is something your team is planning, it would make the build phase make more sense, especially as a workaround as long as copying data over the network using OttoFMS on Windows is still slow.

I am NOT going to downgrade for now, because I know how to get around the existing problems, even if it means more work. OttoFMS is just to cool :slight_smile:

Thanks – peter

5

Hi Peter,

So sorry. We clearly need to improve the build UI. The option to switch between Clones or copies is in the File Overrides.

TG2024-04-21 at 05.58.36@2x
TG2024-04-21 at 05.58.36@2x2160×948 145 KB

TG2024-04-21 at 05.57.40@2x
TG2024-04-21 at 05.57.40@2x1536×1334 213 KB

We’ll see what we can do to make that clearer.

Sorry for the confusion.

Let us know if you have any other issues

Thanks

Todd

6

Morning,
Got it. Thanks. Makes sense now.

Unfortunately, I cannot make OttoFMS build without crashing.
When it crashes, it always gives the same error in the log:

2024-04-22T04:34:38.671Z	error	Unhandled Rejection: Error: Local Admin API error: Fetch failed: Not Found code: 1700 - Resource doesn't exist; Path: /fmi/admin/api/v2/schedules/65
    at Mue (C:\snapshot\ottofms-server\index.js)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Qzn (C:\snapshot\ottofms-server\index.js)
    at async C:\snapshot\ottofms-server\index.js
2024-04-22T04:48:38.635Z	error	Unhandled Rejection: Error: Local Admin API error: Fetch failed: Not Found code: 1700 - Resource doesn't exist; Path: /fmi/admin/api/v2/schedules/65
    at Mue (C:\snapshot\ottofms-server\index.js)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Qzn (C:\snapshot\ottofms-server\index.js)
    at async C:\snapshot\ottofms-server\index.js
2024-04-22T05:06:18.691Z	error	Unhandled Rejection: Error: Local Admin API error: Fetch failed: Not Found code: 1700 - Resource doesn't exist; Path: /fmi/admin/api/v2/schedules/65
    at Mue (C:\snapshot\ottofms-server\index.js)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Qzn (C:\snapshot\ottofms-server\index.js)
    at async C:\snapshot\ottofms-server\index.js

(BTW local time zone timestamps would be nice to have)
It does work though for a smaller number of files.
I have 6 files that have to be copied, and having only them in the build works.
When I add the extra clones, the build crashes.
I can see in the manifest file that it crashes after doing the 6 copies, while adding a clone file to the archive, while it already added several clone files earlier to it without a problem.

I then split the build into 2 builds, one for the copies, one for the clones.
That one crashes after cloning, moving and zipping 13 files, between timeElapsed 286020 and timeElapsed 310220.
Same error in the log.
I’m starting to think this is one of these API key expiries, but the web interface has no interface to the build progress, so the trick of keeping the session active does not work here, and neither does keeping the OttoDeploy screen open.

So I split the build into 3 builds, and that works.

The deployment has been constructed, but I will only be able to run it at the end of the week.
It has 3 subdeployments, because of the 3 builds and looks kinda fancy :slight_smile: .
I will let you know how things turn out, looks like this is going to work though.

The OttoDeploy UI needs some more love indeed. Taking some options out of the value lists, that are not applicable, might be less confusing.

  • a build URL option in the build interface is not applicable
  • an install/replace vs an install shouldn’t be an option in the menu. I cannot imagine a deployment where you do not want to replace a possible existing file. Just bury it somewhere in the options.
  • the manifest of a build knows which files are clones and which ones are not. Do not give the option to install a clone. Again, in case this is really wanted, the option can be buried somewhere.
  • even if you do not agree with these simplifications, having the deployment default to “install” and “migrate” based on the “clone” flag per file in the manifest, would be very cool. And logical. And time saving.

As for the build, it would be nice if a deployment can choose a build from another OttoFMS instance. It would again simplify things, and more important: make them more secure, as the manifest files contain passwords.

7

Hi Peter,

I’d like to get to the bottom of those crashes. It isn’t a key expiry. It’s trying to find this resource in the admin api /fmi/admin/api/v2/schedules/65. And it can’t. That seems pretty odd. If that schedule didn’t exist then I would expect a 404 not a 1700. Either way we can prevent the crash, but I am not sure why that error.

We will dig into that, and see what we can find.

We’ll review the rest of your feedback,

Thanks for taking the time to send it.

Todd

8

Hey Peter,

We believe we’ve handled the source of those crashes with version 4.3.2, let me know if you continue seeing the crashes with new builds after an upgrade. Thank you!

-Kyle

9

You did!
A build with all copies and clones in one go, now worked without any problem.

Just more feedback:
In the PLEASE-READ-FIRST.txt file, the link to upgrading OttoFMS is wrong and results in a 404 error.
It says
https://docs.ottofms.com/managing-ottofms
but should say:
https://docs.ottofms.com/managing-ottofms/updating

It also only mentions option 2.

Thanks for all the help so far. Proof+Geist rocks…

1 Like
10

Hey :slight_smile:

Thanks for feedback.

We’ll get the text file changed in the next release.

Todd