I followed the Deployment Pattern: Refresh Dev guide but unfortunately, I was not able to successfully perform the deployment.
The source server is a Windows server while the destination is Ubuntu. Both have OttoFMS 4.1.0. I tried multiple times and restarted both servers but it’s still giving an error on the copying portion of the sub-deployment.
I received the following error message from deployment log (destination server): Build server error: failed - backup or copy failed: Horizon Party.fmp12. Error: Schedule timed out
And below are from otto-error.log on the source server:
backup or copy failed: Horizon Party.fmp12. Error: Schedule timed out
Unhandled Rejection in Promise: TypeError: Cannot read properties of undefined (reading 'split')
UnauthorizedError: username or password is incorrect - {"service":"express-error-handler"}
backup or copy failed: Horizon Party.fmp12. Error: Schedule timed out
backup or copy failed: Horizon Party.fmp12. Error: Schedule timed out
Thanks for reporting this. We are going to dig into this and see if your error logs help us find anything.
Could you send us the full Deployment JSON? You can send as a direct message to me or to @kduval Here is how you can get the full deployment JSON from your deployment in OttoDeploy
We are making a bit of progress on what might be happening. Could you send the otto_info.log file from the source server as well. You can Direct message it to me.
A fix for this issue was included in OttoFMS version 4.1.1!
If a backup takes longer than 10 minutes it is still possible to get this error, if you have a problem with this for a very large file, reach out and we will work with you to provide some solutions! We’re working on finding the sweet spot between “My backup is big and needs to finish” and “This backup has hung for some reason and needs to error out”. Thank you!
@toddgeist I am not sure how to send you a direct message here. I will try to update OttoFMS to 4.1.1 and see if I can perform the deployment successfully using it.
Update: I still can’t perform this deployment even after updating OttoFMS to 4.1.1. Both servers were restarted after the updating.
otto-info.log from destination server shows the following lines:
2024-02-22T05:41:49.492Z error DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error
2024-02-22T05:44:22.337Z info GET: /otto/api/info
2024-02-22T05:44:22.372Z info GET: /otto/api/build/list
2024-02-22T05:46:30.045Z info otto internal database up to date
2024-02-22T05:46:30.052Z info Listening on http://localhost:3061
2024-02-22T05:46:30.052Z info environment "production"
2024-02-22T05:46:30.053Z info version "4.1.1"
2024-02-22T05:46:30.085Z info started watching offsite backup
2024-02-22T05:46:30.086Z info started watching for hostname changes
2024-02-22T05:46:30.130Z info Reverse Proxy already installed
2024-02-22T05:46:30.468Z info License status: valid
2024-02-22T05:46:32.245Z error UnauthorizedError: username or password is incorrect - {"service":"express-error-handler"}
2024-02-22T05:46:32.369Z info GET: /otto/api/info
2024-02-22T05:46:32.455Z info GET: /otto/api/info
2024-02-22T05:46:56.576Z info GET: /otto/api/info
2024-02-22T05:47:02.190Z info GET: /otto/api/build/list
2024-02-22T05:47:02.297Z info refreshed the session token
2024-02-22T05:47:02.340Z info refreshed the session token
2024-02-22T05:47:12.813Z info POST: /otto/api/deployment
2024-02-22T05:47:25.544Z error Failed to send event batch_started: fetch failed
2024-02-22T05:47:25.547Z error Failed to send event deployment_started: fetch failed
2024-02-22T05:47:26.601Z error DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error
2024-02-22T05:47:43.621Z error Failed to send event deployment_failed: fetch failed
2024-02-22T05:49:29.583Z info GET: /otto/api/info
2024-02-22T05:49:29.620Z info GET: /otto/api/build/list
2024-02-22T05:50:26.108Z info POST: /otto/api/deployment
2024-02-22T05:51:00.167Z error DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error
2024-02-22T06:02:33.497Z info GET: /otto/api/info
2024-02-22T06:02:33.536Z info GET: /otto/api/build/list
2024-02-22T06:03:39.518Z info GET: /otto/api/info
2024-02-22T06:03:39.556Z info GET: /otto/api/build/list
2024-02-22T06:04:51.368Z info GET: /otto/api/info
2024-02-22T06:04:51.413Z info GET: /otto/api/build/list
2024-02-22T06:05:51.542Z info GET: /otto/api/info
otto-error.log from destination server shows the following lines:
2024-02-22T02:05:39.890Z error UnauthorizedError: Token is missing - {"service":"express-error-handler"}
2024-02-22T05:41:15.317Z error UnauthorizedError: Token is missing - {"service":"express-error-handler"}
2024-02-22T05:41:49.491Z error DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error
2024-02-22T05:46:32.245Z error UnauthorizedError: username or password is incorrect - {"service":"express-error-handler"}
2024-02-22T05:47:25.543Z error Failed to send event batch_started: fetch failed
2024-02-22T05:47:25.547Z error Failed to send event deployment_started: fetch failed
2024-02-22T05:47:26.601Z error DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error
2024-02-22T05:47:43.621Z error Failed to send event deployment_failed: fetch failed
2024-02-22T05:51:00.167Z error DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error
Also, I tried copying the deployment json from OttoDeploy and I can’t seem to do it. The copy button did not copy the json to my clipboard. I can’t select the json manually either.
It looks like your server is unable to reach out via the network. Every one of the http calls that gets made is failing with the “fetch failed” error, which is normally a network error.
Does the server have a valid SSL cert set up? Are you able to reach out to any other external sources from the server?
The community has a feature that doesn’t give new users too much permission (protecting us from spam), looks like that was what was getting you. I’ve just upped your permissions, you should be able to see it now.
I am again dealing with a backup/copy that takes longer than 10 mins for a backup to run. What are your thoughts on how to get around this. In Otto v3, I would just use a copy dropped into the saved folder.
My current use-case is copying production data from another server back to the development server, then doing the 2 step process to migrate to a staging file, then to dev. So in the end, I have production data in a dev file that is using the dev file’s accounts.
@kduval the similar issue I had occurred again today. I can’t copy files from one remote server to another. The error message is “Build server is unreachable”.
The Otto console of both servers is accessible. I tried restarting the Otto service and re-running the deployment but was unsuccessful.
This seems to happen only on one side of the deployment. For example:
mx to sg (this has the issue)
sg to mx (no issue)
I included some screenshots of the Otto console of the server I’m trying to copy file “TO”.
Do you have anything from the logs on the source server? usually the error you’re getting is related to the source server OttoFMS crashing mid-build or something similar.
That is a weird one. We don’t get many details from that aggregate error, I’ll have to see if I can get handling for that into the build poller. It looks like your build worked fine, the call to the source just failed for some reason. I’ll make the build status polling hardier to errors like this so it will retry a few times on a failure.
As for what you can do right now, running this build as a pre-build should work well for you. If you have not configured a pre build before you can configure and run them from OttoDeploy. Running the build ahead of time should bypass this error. Thank you!!!
