We remove our custom puma.service.erb template located at
lib/capistrano/templates, as the upstream capistrano3-puma gem (v6.2.0)
now supports all the adjustments we previously needed.
This template was originally copied and modified in
PR #5842 (Bump capistrano3-puma from 5.2.0 to 6.0.0),
with the following relevant changes:
1. WatchdogSec=0
We disabled systemd's watchdog feature by setting WatchdogSec=0.
This is now covered upstream since v6.1.0 via:
> <% if fetch(:puma_systemd_watchdog_sec) && fetch(:puma_systemd_watchdog_sec) > 0 %>
> WatchdogSec=<%= fetch(:puma_systemd_watchdog_sec) %>
> <% end %>
To preserve the same behavior, we now explicitly set:
> set :puma_systemd_watchdog_sec, 0
in config/deploy.rb, instead of relying on the template override.
2. SyslogIdentifier=puma
In our custom template, we replaced the default line:
> SyslogIdentifier=<%= fetch(:puma_service_unit_name) %>
with:
> SyslogIdentifier=puma
This was done to avoid introducing a new config variable
in the installer. However, now that we want to remove the overridden template,
we must accept the default behavior from the gem:
> set :puma_service_unit_name, -> { "puma_#{fetch(:application)}_#{fetch(:stage)}" }
As a result, the generated SyslogIdentifier will change from "puma" to something
like "puma_consul_staging".
The only implication is that we'll open a PR in the installer
to replace the hardcoded "puma" with the dynamic default,
so everything stays consistent.
Another option would be to override puma_service_unit_name
with "puma" to keep the SyslogIdentifier exactly as before.
However, this would also affect the names of the systemd services and sockets
(e.g., puma.service instead of puma_consul_staging.service),
which may lead to conflicts or require manual cleanup steps like disabling
and removing the old units. For now, we prefer to avoid that.
3. After=network.target
We had removed the "syslog.target" from the "After=" directive,
as we didn't depend on syslog.
However, keeping "After=syslog.target network.target" does not negatively impact us,
since Puma logs to files directly using StandardOutput=append:....
So we can also adopt the default value from template here.
4. ExecStart
In version 6.2.0, a small change was made to the template logic for the ExecStart line.
This change does not affect us. Since we don't define puma_use_login_shell,
the behavior remains exactly the same as before.
With these changes already supported in the gem, we no longer need
a local copy of the puma service template.
capistrano3-puma v6.0.0 updated the defaults for puma_access_log and
puma_error_log to use a single file based on puma_env, like:
> set_if_empty :puma_access_log, -> { File.join(shared_path, 'log', "#{fetch(:puma_env)}.log") }
> set_if_empty :puma_error_log, -> { File.join(shared_path, 'log', "#{fetch(:puma_env)}.log") }
However, our installer expect:
- puma_access.log
- puma_error.log
To keep the existing behavior aligned with the installer, we define
the Puma log paths in config/deploy.rb
In capistrano3-puma v6.0.0, the default for 'puma_service_unit_name' changed to:
> "#{application}_puma_#{stage}"
But the installer uses the older convention:
> "puma_#{application}_#{stage}"
To ensure consistency and avoid unit name conflicts when switching between
versions or deploying older branches, we now define the variable explicitly
in config/deploy.rb:
> set :puma_service_unit_name, -> { "puma_#{fetch(:application)}_#{fetch(:stage)}" }
This commit copies the default puma.service.erb template from the
capistrano3-puma gem into lib/capistrano/templates. This allows us to
customize the generated systemd unit file during deploy.
Note that we are also removing the `:puma_conf` variable from `config/deploy.rb`,
as the new ExecStart line in the systemd template (based on capistrano3-puma 6.0.0)
does not rely on a separate Puma config file. The command now directly invokes:
ExecStart=<%= expanded_bundle_command %> exec puma -e <%= fetch(:puma_env) %>
This replaces the older format used in 5.2.0:
ExecStart=<%= expanded_bundle_command %> exec --keep-file-descriptors puma -C <%= fetch(:puma_conf) %>
which required explicitly setting the Puma config path.
In earlier versions of capistrano3-puma, the puma_bind has a default value to:
unix://.../tmp/sockets/puma.sock via set_if_empty in lib/capistrano/puma.rb.
This default was removed in 6.0.0, requiring to explicitly set :puma_bind in deploy.rb.
This caused the following runtime error during deploy:
> Failed to restart consul_puma_staging.service: Unit
> consul_puma_staging.socket has a bad unit file setting.
capistrano3-puma 6.0.0 removed the `puma:systemd:config` and
`puma:systemd:enable` tasks. This commit updates the deploy script to use
the new `puma:install` and `puma:enable` tasks instead.
This task was necessary when updating to version 2.1.0, when we
integrated Puma with Systemd. Now that all Consul Democracy
installations are using Systemd, we no longer need it.
By default, when starting a Rails application, ExecJS checks whether there's
a runtime available in the system. Since on production we now use FNM to
manage node versions, this meant that we had to run Puma using the
`fnm exec bundle exec puma` command.
However, this causes issues when integrating Puma with Systemd. For instance,
when using `fnm exec`, the PID for the main process is no longer the Puma
process but the FNM process, meaning that stopping and restarting Puma will
no longer behave as expected and might last 90 seconds (Systemd's timeout to
stop a service). Furthermore, using a Puma Systemd socket in order to
implement hot or phased restarts won't work either, and so we'll get 502
errors while restarting the service.
So we're using the `EXECJS_RUNTIME=Disabled` environment variable in the
Systemd service, which means we can remove the `fnm exec` prefix in the
ExecStart command (which starts Puma) and now the PID for the main process
is the Puma process, stopping/restarting the service no longer times out,
and we'll be able to implement Puma socket activation and hot restarts.
However, removing `bundle` from the list of commands which use `fnm exec`
(the `fnm_map_bins`) meant that we got a `RuntimeUnavailable` exception when
running `bundle exec bin/delayed_job`. We tried to add `delayed_job` to the
`fnm_map_bins` variable but couldn't make it work.
Since we only need the ExecJs runtime when precompiling the assets but we
don't need it in order to start Puma or Delayed Job, we're adding
`{ EXECJS_RUNTIME: "Disabled" }` to Capistrano's default environment and
we're only using the default Runtime when `fnm exec` is used, which we've
done by setting the environment variable in the prefix of the fnm command.
This way, when running `EXECJS_RUNTIME='' fnm exec bundle exec rake assets:precompile`,
the right ExecJS Runtime will be loaded, but it won't be loaded (since it
isn't needed) when starting Puma or Delayed Job.
Puma 4 is no longer maintained, and Puma 5 removed the option to run
Puma as as daemon. That means that, in order to upgrade, we need to rely
in a process monitoring tool. We're using systemd because it's installed
by default in most (all?) operating systems running Consul Democracy on
production and it's supported by both Puma and the capistrano3-puma gem.
Using systemd means Puma will be automatically started whenever it
crashes, so we no longer need the changes done in commit 40b3c9f2c and
we can now use the default tasks configuration in capistrano3-puma and
capistrano3-delayed-job.
Since Puma is also started automatically when booting the system, we
don't need a cron job to do so anymore and can remove it from the
`schedule.rb` file.
The previous version worked fine when using Capistrano to deploy a new
release, but it did not work for other commands, like `cap npm install`
or `cap puma:start`, as they do not create a release. Therefore, the
`git:create_release` was never called, and the `map_node_bins` was not
invoked.
Since we need node binaries available to the deploy user when executing
any process that runs the application, like puma, delayed_job, and rake
tasks, among others, it makes sense to load FNM to use the correct Node
version in the same places where Capistrano loads RVM to load the correct
Ruby version.
With dynamic loading of the `fnm_setup_command`, we get the correct path
when deploying a new release and also when running any other capistrano
command that does not deploy a new release.
This code is based on what the rvm1-capistrano and capistrano-nvm gems
do, but simplified a bit to take advantage of the `fnm exec` command.
Since ExecJS will check for a JavaScript runtime every time the
application is started, we need to include commands like `bundle` (used
when running `bin/delayed_job`), `puma` and `rake`, so Node.js is found
when running these commands. I'm not sure whether `pumactl` is also
necessary, but I've added it for consistency.
We're also including the default commands gems like capistrano-nvm use
because we will add the `npm` command in the near future.
Note that, in order to setup FNM, we need to enter the actual release
folder (using `within release_path` isn't enough). So we have to run
this task after the actual release is created; otherwise many commands
would run in the previous release's folder.
We use FNM instead of NVM. Reason: the setup seems easier with just
`eval "$(fnm env)"`.
So now, we try to install Node.js; if the command fails, it could be
because FNM isn't installed (and we need to install it) or because the
version of Node.js cannot be installed with the current version of FNM
(and we need to update FNM). After installing/updating FNM, we try to
install Node.js again.
Note we're using `fnm env` in the middle of the `fnm_setup_command`.
That way, the command will raise a `SSHKit::Command::Failed` exception
if `fnm` isn't installed, and it will give us an indicator that we need
to actually install it.
Note we updated the `mail` gem in commit 103742847, which is necesary
for Ruby 3.1 because it adds the net-smtp dependency. The net-smtp
library was removed from Ruby in Ruby 3.1, and if we don't include it,
we get an error:
```
cannot load such file -- net/smtp (LoadError)
```
We're also updating the Bundler version in the Gemfile.lock so it's the
one included in Ruby 3.1. Without updating it, we get a warning:
```
Calling `DidYouMean::SPELL_CHECKERS.merge!(error_name => spell_checker)'
has been deprecated. Please call `DidYouMean.correct_error(error_nam e,
spell_checker)' instead.
```
Finally, in order to make Capistrano work, we need to add a couple more
changes:
* Make the net-ssh gem compatible with SSL 3.0; done in commit b2eec088b
* Explicitly allow aliases in the `deploy-secrets.yml` file because
Psych 4.x (included in Ruby 3.1) doesn't load aliases without this
option
Note we're excluding a few files:
* Configuration files that weren't generated by us
* Migration files that weren't generated by us
* The Gemfile, since it includes an important comment that must be on
the same line as the gem declaration
* The Budget::Stats class, since the heading statistics are a mess and
having shorter lines would require a lot of refactoring
DelayedJob offers the `--monitor` (aliased as `-m`) option to create a
process that monitors the workers and restarts them when they crash.
This change implies that, in order to stop the delayed job workers, we
now need to pass the `-n` option when running `bin/delayed_job stop`:
`RAILS_ENV=production bin/delayed_job -n 2 stop`.
We were using `server` on staging but `server1` and `server2` on
preproduction and production.
The reason behind it is we've always used one server on staging but
sometimes we've used several servers on preproduction and production.
However, this is a bit of a mess on installations which have only one
server on preproduction or production and need to use the `server` key
for the staging environments but `server1` for other environments.
So, in order to keep compatibility with existing Consul installations,
we're now allowing either `server` or `server1` on any environment.
We were installing RVM every time we were deploying, which meant
deployments took a few extra seconds and generated a lot of unneeded
output even when we didn't have to update Ruby.
We need to update RVM when a new Ruby version needs to be installed
(otherwise we could be asking RVM to install a Ruby version which the
current RVM version cannot handle), but in every other case we're fine
using the already installed RVM version.
We're also considering the case where the RVM command is not available.
Even though the CONSUL installer installs RVM, some people might prefer
to install it through Capistrano.
The server_name variable isn't used since we removed the Apache task in
commit 012d5297e, db_server and config_files aren't used since we
removed the capistrano templates in that same commit, and full_app_name
isn't used since commit 94a7e13dc.
The 2.0.0 release of capistrano-bundler creates a configuration file
under `.bundle/config`. It was creating a new configuration file for
each release because we weren't using a shared folder.
Besides, quoting the capistrano-bundler README [1]:
> In order for Bundler to work efficiently on the server, its project
> configuration directory (<release_path>/.bundle/) should be persistent
> across releases. You need to add it to the linked_dirs Capistrano
> variable
[1] https://github.com/capistrano/bundler/blob/v2.0.1/README.md
We were getting an error when restarting Puma after upgrading Ruby. Even
if the restart command was sent successfully, Puma silently crashed and
the log had the following error:
/home/deploy/.rvm/rubies/ruby-2.4.9/lib/ruby/site_ruby/2.4.0/bundler/spec_set.rb:91:in
`block in materialize': Could not find rake-13.0.1 in any of the sources
(Bundler::GemNotFound)
So it looks like the crash happens because Puma was started when the
application used Ruby 2.4 and now when it's restarted it still tries to
use Ruby 2.4, even if the application now uses Ruby 2.5.
I haven't found a proper way to configure Puma so we can avoid this, so
as a workaround I've added the `puma:start` task after restarting Puma.
If Puma was successfully restarted, `puma:start` will do nothing; if
Puma crashed, `puma:start` will start it.
To guarantee the tasks will be executed in the proper order, the tasks
introduced by capistrano3-delayed_job and capistrano3-puma are cleared,
and then we configure the order so first we restart Puma, then restart
the Delayed Jobs processes (so there's enough time for Puma to crash if
Ruby was upgraded) and then start Puma.
These tasks are not needed for new installations, and in existing
installations they've already been executed when upgrading to version
1.1.
One of them also raises a warning in Rails 5.2:
DEPRECATION WARNING: Dangerous query method (method whose arguments are
used as raw SQL) called with non-attribute argument(s): "MIN(id) as id".
Non-attribute arguments will be disallowed in Rails 6.0. This method
should not be called with user-provided values, such as request
parameters or model attributes. Known-safe values can be passed by
wrapping them in Arel.sql()
We're now using the same version we used to generate our Gemfile.lock.
Using the latest bundler we got a deprecation warning, which might turn
into an error in the future:
[DEPRECATED] The `--deployment` flag is deprecated because it relies on
being remembered across bundler invocations, which bundler will no
longer do in future versions.
We could also upgrade to bundler 2.x, but since we're using Ruby 2.4 and
Ruby 2.6 comes with bundler 1.17, we've decided to keep this version.
The task `deploy:restart` was doing nothing since we moved from unicorn
to puma.
Now we're also restarting delayed jobs on `deploy:restart`, which is
probably what's expected in most cases.
We need to download the most recent CONSUL and check its `.ruby-version`
file before installing Ruby.
The `rvm1-capistrano` gem knows it and was invoking the `updating` task
before installing Ruby. So we were getting a warning in Capistrano about
the `updating` task being executed twice.
New CONSUL instances who didn't use the newest installer and haven't
done any deployments with capistrano would get an exception because the
`current` capistrano folder doesn't exist yet.
We were using Capistrano's shared folder because it was the default
folder used by the capistrano-puma gem. However, it's easier to manage
it if it's under version control.
So we're moving the old `puma.rb` to `puma/development.rb`, and we use
the new `puma.rb` file for all environments except development. Anyone
installing CONSUL can change these files at will or change the specific
files for preproduction, production and staging environments.
Deploy environment files were loaded after config/deploy.rb was loaded,
meaning if we used settings like `deploy_to` or `current_path` to set
variables inside the config/deploy.rb file, we'd be using the default
settings instead of the ones dictated by the deploy environment files.
This is the same configuration we had with unicorn.
With several workers, we've got two basic configuration options:
* Preload the application and use a hot restart
* Don't preload the application and use a phased restart
I've decided to preload the application because using a hot restart
changes are available immediately, while with a phased restart there are
a few seconds when both workers for the old code and workers for the new
code exist.
Using a phased restart also has advantages, so some forks might want to
disable the `puma_preload_app` setting in order to use it.
Old versions of the installer created this file as root, making it
impossible to change it as a regular user.
So for old installations we need to make sure we've got write access to
this file.
We're using `sudo` because in these applications the installer gives
`sudo` access to the deploy user, so everything works fine with the
default configuration.
While this is not a secret and in theory should be in a file under
version control, currently the CONSUL installer disables delayed jobs by
default, meaning we were keeping two versions of the delayed jobs
configuration file, and some existing configurations have their settings
defined in a file in capistrano's `shared` folder.
So we're moving existing settings to the secrets file.
We were copying the current SMTP and SSL settings to the secrets file
after overwriting them, but we need to copy them before overwriting
them.
The workaround I've found is to copy the tasks to the folder of the
previous release and execute them there.
Existing installations having their configuration settings in the
capistrano shared folder needed this migration.
Note we can't just use `YAML.load` because we'd lose the anchors defined
in the file. So we have to parse the file the hard way.
