NoMatchError on Deploy after fresh install

cgesell · April 26, 2024, 11:08am

We have installed OttoFMS on multiple servers, 2 mac minis and 2 Linux VM (Ubuntu 22, SSD, 8GB) coming from Otto v3. All fine unless we cannot run deployments on the Linux VMs.

While deployment of a test file from one server to another (simply ‘install’) works fine on the mac installs. Deployment crashes right away.

Otto Error log:

2024-04-26T09:46:31.730Z    error    Unhandled Rejection: NoMatchError
    at n (/snapshot/ottofms-server/index.js)
    at s (/snapshot/ottofms-server/index.js)
    at async URe (/snapshot/ottofms-server/index.js)
    at async Nht (/snapshot/ottofms-server/index.js)
    at async /snapshot/ottofms-server/index.js

The “detailed log” doesn’t give a lot of information:

message    phase    level    timeElapsed
deployment queued    queued    info    00:00.000
Starting  deployment: "TestFonts"    starting    info    00:00.000
1 files    starting    info    00:00.004
Unknown deployment error: NoMatchError    unknown phase    error    00:00.042
Deployment crashed    unknown phase    error    00:01.824

Any thoughts? Thanks! c

toddgeist · April 26, 2024, 2:20pm

Hello,

Thanks for reporting this. We’ll take a look at this issue this morning.

In the meantime, could you give us some more information:

Do the Linux servers have a swap file setup?
What version of OttoFMS are you using?

Thanks

Todd

cgesell · April 29, 2024, 8:54am

Thanks @toddgeist for the response.
it not a swap file setup and we have OttoFMS 4.3.2 installed.
Let me know if I can provide any more details.

Thanks, Christian

toddgeist · April 29, 2024, 1:27pm

Hello Christian

OttoFM requires a swap file. FileMaker Sever will require one in the future as well. I am not sure if that is the cause of your problem, but it very well could be.

Here are the minimum requirements for OttoFMS. Pleas make sure your servers meet them.

Thanks

Todd

cgesell · April 29, 2024, 3:29pm

Ok, thanks! We’ll change that and report back.

cgesell · May 2, 2024, 7:33am

Hi, swap file is set up. Unfortunately this did not help.
Test deployment (installing a 360KB file) crashes before sending the build request to source server.
Any more ideas? Any more info I can provide?

john_r · May 2, 2024, 11:13am

@toddgeist is it worth linking to a good tutorial on how to set up the swap file, including the recommendation to set Swappiness to 10 rather than the default 60?

toddgeist · May 2, 2024, 1:00pm

Could be. Do you have such a link?

btw The next version of FMS is going to do it for you on install if it is’t setup

Todd

john_r · May 2, 2024, 6:13pm

This is one I have used… includes stuff about swappiness at end

cgesell · May 2, 2024, 7:16pm

Thanks for this guys. I am sure this helps the community.
How to create the swap file is not the issue for us. It is set up and we are still facing the same error. Any ideas on the original issue?

toddgeist · May 4, 2024, 5:18pm

Hello

We did make some changes in the latest release that we hope will at least give some more information about what is happening here. Try upgrading to the latest version and running a small deployment again.

If it doesn’t work, send us the otto-info.log from deployment server. You can direct message them to me if you want.

Let us know what happens

Todd

cgesell · May 7, 2024, 11:25am

Alright. Upgrade is done and I ran a small deployment. Here’s the otto-info.log on that:

2024-05-07T11:18:54.524Z	info	GET: /otto/api/info
2024-05-07T11:19:00.222Z	info	POST: /otto/api/deployment
2024-05-07T11:19:00.440Z	error	Unhandled Rejection: NoMatchError
    at n (/snapshot/ottofms-server/index.js)
    at s (/snapshot/ottofms-server/index.js)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async cir (/snapshot/ottofms-server/index.js)
    at async pFe (/snapshot/ottofms-server/index.js)
    at async ogt (/snapshot/ottofms-server/index.js)
    at async /snapshot/ottofms-server/index.js
2024-05-07T11:19:01.477Z	info	otto internal database up to date
2024-05-07T11:19:01.485Z	info	Listening on http://localhost:3061
2024-05-07T11:19:01.485Z	info	environment "production"
2024-05-07T11:19:01.485Z	info	version "4.3.4"
2024-05-07T11:19:01.486Z	info	node version v20.11.1
2024-05-07T11:19:01.495Z	info	started watching offsite backup
2024-05-07T11:19:01.495Z	info	started watching for hostname changes
2024-05-07T11:19:01.499Z	info	Reverse Proxy already installed
2024-05-07T11:19:01.768Z	info	License status: valid

hope this helps…

toddgeist · May 7, 2024, 4:02pm

Thanks for sending in more info. Unfortunately we still can’t locate the problem. We have no other reports of this NoMatchError, and we can’t reproduce it on our Linux Vms.

If you’d like we could take a look with you at one of our office hours sessions.

Office Hours | OttoFMS.

Maybe something will turn up there we can zero in on.

Todd

cgesell · May 8, 2024, 9:58am

Thanks for checking and too bad this didn’t bring you closer to reproducing the issue.

Meanwhile we have found this in the syslog - it is not directly related to a deployment but might be something related:

May  8 10:25:34 kiosque ottofms-linux-x64[1630]: ❌ tRPC failed on deployments.getAll: No auth token
May  8 10:25:34 kiosque ottofms-linux-x64[1630]: Ul [TRPCError]: No auth token
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:     at /snapshot/ottofms-server/index.js
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:     at c (/snapshot/ottofms-server/index.js)
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:     at t (/snapshot/ottofms-server/index.js)
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:     at yHe (/snapshot/ottofms-server/index.js)
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:     at A5i (/snapshot/ottofms-server/index.js)
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:     at /snapshot/ottofms-server/index.js
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:     at Array.map (<anonymous>)
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:     at p$r (/snapshot/ottofms-server/index.js)
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:     at async /snapshot/ottofms-server/index.js {
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:   code: 'UNAUTHORIZED',
May  8 10:25:34 kiosque ottofms-linux-x64[1630]:   [cause]: undefined
May  8 10:25:34 kiosque ottofms-linux-x64[1630]: }

We are now setting up another VM for checking the configuration of the virtualisation. If that won’t help I will definitely book an office hour.

Christian

toddgeist · May 10, 2024, 2:11pm

Thanks again for looking.

the error you are seeing is likely just from authorization needing to be refreshed.

If I google for NoMatchError, i do find some results related to Vms and VMWare. But It wasn’t clear to me that they were related.

See you at office hours maybe

Todd

cgesell · May 13, 2024, 9:56am

Our hosting provider honds.de found the error.

FileMaker and Otto use the Linux tool “df”.
However, FileMaker or the Admin Console cannot interpret the output correctly if the data is on a ZFS fileset and not on a “classic volume”. The path there does not start with a “/”.
In the Admin Console, the allocation is then permanently 0%

Therefore, there is a wrapper for df on the FileMaker VMs under /usr/local/bin, which places a / before the output of df. E.g.:

rpool/data/subvol-9999-disk-0 41943040 3058688 38884352 8% /
vs
/rpool/data/subvol-9999-disk-0 41943040 3058688 38884352 8% /

OttoFM uses “df” but with the path to the FileMaker installation as a parameter - and this contains spaces. The wrapper has not taken this into account.
“df” then returns an error and OttoFM crashes.

The fix is applied and Otto v4 works like a charm.
Thanks @toddgeist for the assistance!!

Christian

toddgeist · May 13, 2024, 1:07pm

Thank you so much for reporting the solution. This is very helpful

Todd

toddgeist · May 13, 2024, 1:36pm

Can you say a bit more about what the fix was that you applied?

With this information we will be able to stop OttoFMS from crashing, but we won’t be able to read the available disk space. So it would be good to know if there is anything more we can do on this side.

Thanks

Todd

cgesell · May 14, 2024, 1:04pm

I am forwarding the answer that Dennis Dawari of Honds IT sent:

The need for the wrapper originates from a “bug”… maybe more of an oversight in FileMaker itself. My former colleague described that on the claris forum years ago (Claris Community (English)).

The “fix” iself is pretty trivial and not very elegant. df’s output is piped to sed which just adds an / before the ZFS pool name. In our case “rpool”.

/usr/local/bin/df:

#!/bin/sh
/bin/df $@ | /bin/sed "s#rpool#/rpool#g"

If $@ is not quoted ("$@") the string "*/opt/FileMaker/FileMaker Server/*" will be interpreted as two parameters, which will result in an error:

root@test:~# */usr/local/bin/df /opt/FileMaker/FileMaker\ Server/*
/bin/df: /opt/FileMaker/FileMaker: No such file or directory
*/bin/df: Server/*: No such file or directory

However - if I remember correctly OttoFM will crash as well, if no output is generated at all. So an empty script at /usr/local/bin/df will also result in a crash.

The reason for the crash does not seem to be the return code. In the example above the exit code will always be 0, since pipefail (or similar) was not used and “sed” always will exit with 0.