Some institutions using CONSUL have expressed interest in this feature
since some of their tenants might already have their own domains.
We've considered many options for the user interface to select whether
we're using a subdomain or a domain, like having two separate fields,
using a check box, ... In the end we've chosen radio buttons because
they make it easier to follow a logical sequence: first you decide
whether you're introducing a domain or subdomain, and then you enter it.
We've also considered hiding this option and assuming "if it's got a
dot, it's a domain". However, this wouldn't work with nested subdomains
and it wouldn't work with domains which are simply machine names.
Note that a group of radio buttons (or check boxes) is difficult to
style when the text of the label might expand over more than one line
(as is the case here on small screens); in this case, most solutions
result in the second line of the label appearing immediately under the
radio button, instead of being aligned with the first line of the label.
That's why I've added a container for the input+label combination.
This way we'll avoid having all tenant organizations named "CONSUL" by
default, and we'll also use different email senders per tenant, which
will reduce the change of the emails being marked as spam.
We were using `consul.dev` as default, which might be fine for the
development environment, but it's something that definitely needs to be
changed on production.
Now we're providing a better default setting: using the same domain
which is used to generate URLs in emails. This also makes it easier to
configure new tenants, since the setting will default to whatever host
the new tenant is using.
The `|| "consul.dev"` part might not be needed; I'm keeping it because
I'm not sure production environents would not experience any new issues
if we don't add it, but we might remove it in the future.
Note that the `sitemap:refresh` task only pings search engines at the
end, so it only does so for the `Sitemap.default_host` defined last. So
we're using the `sitemap:refresh:no_ping` task instead and pinging
search engines after creating each sitemap.
Note we're pinging search engines in staging and preproduction
environments. I'm leaving it that way because that's what we've done
until now, but I wonder whether we should only do so on production.
Since we're creating a new method to get the current url_options, we're
also using it in the dev_seeds.
Until now, running `db:dev_seed` created development data for the
default tenant but it was impossible to create this data for other
tenants.
Now the tenant can be provided as a parameter.
Note that, in order to be able to execute this task twice while running
the tests, we need to use `load` instead of `require_relative`, since
`require_relative` doesn't load the file again if it's already loaded.
Also note that having two optional parameters in a rake task results in
a cumbersome syntax to execute it. To avoid this, we're simply removing
the `print_log` parameter, which was used mainly for the test
environment. Now we use a different logic to get the same result.
From now on it won't be possible to pass the option to avoid the log in
the development environment. I don't know a developer who's ever used
this option, though, and it can always be done using `> /dev/null`.
When running this task we truncate all tables; however, doing so doesn't
execute the `after_destroy` callback which drops the generated schema.
That meant we could run into a situation where there are schemas in the
database with no associated tenant, leading to data inconsistencies.
So we're now destroying the tenants (alongside their schemas) before
truncating the rest of the database tables.
This way all tenants will be able to access them instead of just the
default one.
The apartment gem recommends using a rake task instead of a migration,
but that's a solution which is primarily meant for new installations.
Migrations are easier to execute on existing installations.
However, since this migration doesn't affect the `schema.rb` file, we
still need to make sure the shared schema is created in tasks which do
not execute migrations, like `db:schema:load` or `db:test:prepare`, just
like the apartment gem recommends. That's why we're enhancing these
tasks so they execute this migration.
Note that there might be cases where the database user isn't a superuser
(as it's usually the case on production environments), meaning commands
to create, alter or drop extensions will fail. There's also the case
where users don't have permissions to create schemas, which is needed in
order to create the shared extensions schema and the schemas used by the
tenants. For these reasons, we're minimizing the number of commands, and
so we only alter or create extensions when it is really necessary.
When users don't have permission, we aren't running the commands but
showing a warning with the steps needed to run the migration manually.
This is only necessary on installations which are going to use
multitenancy; single-tenant applications upgrading don't need to run
this migration, and that's why we aren't raising exceptions when we
can't run it.
For new installations, we'll change the CONSUL installer so extensions
are automatically created in the shared schema.
Also note the plpgsql extension is not handled here. This is a special
extension which must be installed on the pg_catalog schema, which is
always in the search path and so is shared by all tenants.
Finally, we also need to change the `database.yml` file in order to
search for shared extensions while running migrations or model tests,
since none of our enabled extensions are executed during migrations;
we're also adding a rake task for existing installations. Quoting the
apartment documentation:
> your database.yml file must mimic what you've set for your default and
> persistent schemas in Apartment. When you run migrations with Rails,
> it won't know about the extensions schema because Apartment isn't
> injected into the default connection, it's done on a per-request
> basis.
This rule was added in Rubocop 1.18.0, but we didn't add it back then.
Since we're applying it most of the time, we might as well be consistent
and apply it everywhere.
In the dev seeds, we were using `Setting["url"]/proposals`, but we can
use `proposals_url` instead, similar to what we do in the rest of the
application.
We can do a similar thing in the sitemap. This way the sitemap will also
work on installations who haven't manually set the "url" setting.
Since we aren't using `Setting["url"]` anywhere anymore, we're removing
it.
This setting was mainly redundant, since we already had the
`server_name` in the secrets. Furthermore, `server_name` is
automatically configured when running the installer, while
`Setting["url"]` had to be manually set in the admin section the
application was installed.
Note we're using `ActionMailer::Base` setting to generate URLs. Sounds a
bit strange, but it's a standard way Rails provides to generate URLs
outside the context of a request.
We forgot to do so in commit fc757428e, when we upgraded rubocop-rails.
In this upgrade, this rule was changed to include the `change_column`
method.
Just like we did in commit df959b74f, I'm raising an
`ActiveRecord::IrreversibleMigration` exception in migrations done
before version 1.0.0, since making all of them reversible would be too
much work for little benefit.
This method was introduced in Rails 6.0. It can be used to take an array
and create a hash where the elements of the array are the indexes of the
hash.
Performance tests show both methods of truncating the database take
about the same time, so we can remove one dependency and we don't lose
anything in the process.
All the code in the `bin/` and the `config/` folders has been generated
running `rake app:update`. The only exception is the code in
`config/application.rb` where we've excluded the engines that Rails 6.0
has added, since we don't use them.
There are a few changes in Active Storage which aren't compatible with
the code we were using until now.
Since the method to assign an attachment in ActiveStorage has changed
and is incompatible with the hack we used to allow assigning `nil`
attachments, and since ActiveStorage now supports assigning `nil`
attachments, we're removing the mentioned hack. This makes the
HasAttachment module redundant, so we're removing it.
Another change in ActiveStorage is files are no longer saved before
saving the `ActiveStorage::Attachment` record. This means we need to
manually upload the file when using direct uploads. We also have to
change the width and height validations we used for images; however,
doing so results in very complex code, and we currently have to write
that code for both images and site customization images.
So, for now, we're just uploading the file before checking its
dimensions. Not ideal, though. We might use active_storage_validations
in the future to fix this issue (when they support a proc/lambda, as
mentioned in commit 600f5c35e).
We also need to update a couple of tests due to a small change in
response headers. Now the content disposition returns something like:
```
attachment; filename="budget_investments.csv"; filename*=UTF-8''budget_investments.csv
```
So we're updating regular expression we use to check the filename.
Finally, Rails 6.0.1 changed the way the host is set in integration
tests [1] and so both `Capybara.app_host` and `Capybara.default_host`
were ignored when generating URLs in the relationable examples. The only
way I've found to make it work is to explicitely assign the host to the
integration session. Rails 6.1 will change this setup again, so maybe
then we can remove this hack.
[1] https://github.com/rails/rails/pull/36283/commits/fe00711e9
When we added this setting in commit 9979b5399, we disabled it by
default so it would be compatible with existing installations.
Since then, we've released version 1.4, which adds the settings to
existing databases. That means we can now enable it by default and
existing installations won't be affected.
We had the same texts four times, with slight variations in the case of
the management section.
We're unifying them under the "verification" i18n namespace, since the
texts are about actions which can be done depending on whether users are
verified or not.
Note the names of the i18n keys aren't very consistent, since we use
"debates" in plural but "proposal" in singular. We're leaving it like
this so existing translations aren't affected.
We were saying that actions marked with an asterisk were only possible
when users were verified. However, there were no actions marked with an
asterisk; instead, we didn't show these actions to non-verified users.
Besides, the concept of Census doesn't exist in many CONSUL
installations, where verification is done through other means, so the
text saying that only users on Census could do certain things wasn't
correct in these cases.
Due to that, we're removing the asterisk in the Spanish version as well.
We're also removing the asterisk in the default welcome pages, since
we're already saying which action can't be done until the account is
verified.
In the case of the residence verification page, we had asterisks but we
didn't explain what the asterisk stood for, so we're also removing it.
We were using this hack in order to allow `File.new` attachments in
tests files. However, we can use the `fixture_file_upload` helper
instead.
Just like it happened with `file_fixture`, this helper method doesn't
work in fixtures, so in this case we're using `Rack::Test::UploadedFile`
instead.
This fixes a few issues we've had for years.
First, when attaching an image and then sending a form with validation
errors, the image preview would not be rendered when the form was
displayed once again. Now it's rendered as expected.
Second, when attaching an image, removing it, and attaching a new
one, browsers were displaying the image preview of the first one. That's
because Paperclip generated the same URL from both files (as they both
had the same hash data and prefix). Browsers usually cache images and
render the cached image when getting the same URL.
Since now we're storing each image in a different Blob, the images have
different URLs and so the preview of the second one is correctly
displayed.
Finally, when users downloaded a document, they were getting files with
a very long hexadecimal hash as filename. Now they get the original
filename.
Giving any user a direct link to edit another user's account settings doesn't seem like a
great idea. Instead we'll generate a random secure hash string to help keep things
more secure. We'll store these hashes on each user so that we have a way to find
them during this public query. To do this we need to add a column to the user table.
We're going to make it dynamic using the geozones. Besides, class
methods can be overwritten using custom models, while constants can't be
overwritten without getting a warning [1].
Makes the definition of segments with geozones a little cleaner. I
think it’s worth it, compared to the slight memory gain of using a
constant [2].
[1] warning: already initialized constant UserSegments::SEGMENTS
[2] https://stackoverflow.com/questions/15903835/class-method-vs-constant-in-ruby-rails#answer-15903970
Some developers work on CONSUL installations where Spanish and/or
English aren't part of the available locales. In those cases, the
`dev_seed` task was crashing because we were using attributes like
`name_en` and `name_es`.
So we're using attributes for random locales instead.
We're using a proc so we don't have code like this all over the place:
random_locales.map do |locale|
I18n.with_locale(locale) do
phase.name = I18n.t("budgets.phase.#{phase.kind}")
phase.save!
end
end
This would make the code harder to read and would execute a `save!` once
per locale, which would make the task much slower.
We could also avoid the procs writing something like:
def random_locales_attributes(**attribute_names_with_values)
random_locales.each_with_object({}) do |locale, attributes|
I18n.with_locale(locale) do
attribute_names_with_values.each do |attribute_name, (i18n_key, i18n_args)|
value = I18n.t(i18n_key, (i18n_args || {}).merge(language: I18n.t("i18n.language.name")))
attributes["#{attribute_name}_#{locale.to_s.underscore}"] = value
end
end
end
end
And calling the method with with:
random_locales_attributes(name: ["seeds.budgets.name", year: Date.current.year - 1])
However, this code would also be different that what we usually do, we'd
have to apply some magic to pass the `language:` parameter, and the
strings wouldn't be recognized by i18n-tasks, so we aren't sure we're
really gaining anything.
These locales are officially maintained by CONSUL developers, so we're
using them when available.
This way we'll be able to use `random_locales` in places where we're
manually using English and Spanish, giving support to other locales
while maintaining compatibility with the current version.
This column wasn't used in any released Consul version since it was only
used during development. For the same reason, the task to migrate the
information in the `link` column to the `links` table isn't needed
either.
We didn't enable it by default in commit 676adfcb3 so existing
installations didn't suddenly get a new section without expecting it.
But since the setting already exists for installations using version
CONSUL 1.3, now it will only be enabled for new installations.
In order to migrate existing files from Paperclip to ActiveStorage, we
need Paperclip to find out the files associated to existing database
records. So we can't simply replace Paperclip with ActiveStorage.
That's why it's usually recommended [1] to first run the migration and
then replace Paperclip with ActiveStorage using two consecutive
deployments.
However, in our case we can't rely on two consecutive deployments
because we have to make an easy process so existing CONSUL installations
don't run into any issues. We can't just release version 1.4.0 and 1.5.0
and day and ask everyone to upgrade twice on the same day.
Instead, we're following a different plan:
* We're going to provide a Rake task (which will require Paperclip) to
migrate existing files
* We still use Paperclip to generate link and image tags
* New files are handled using both Paperclip and ActiveStorage; that
way, when we make the switch, we won't have to migrate them, and in
the meantime they'll be accessible thanks to Paperclip
* After we make the switch, we'll update the `name` column in the active
storage attachments tables in order to remove the `storage_` prefix
Regarding our handling of new files, the exception are cached
attachments. Since those attachments are temporary files used while
submitting a form and we have to delete them afterwards, we're only
handling them with Paperclip. We'll handle these ones in version 1.5.0.
Note the task creating the dev seeds was failing after these changes
with an `ActiveStorage::IntegrityError` exception because we were
opening some files without closing them. If the same file was attached
twice, it failed the second time.
We're solving it by closing the files with `File.open` and a block. Even
though we didn't get any errors, we're doing the same thing in the
`Attachable` concern because it's a good practice to close files after
we're done with them.
Also note we have to change the CKEditor Active Storage code so it's
compatible with Paperclip. In this case, I haven't been able to write a
test to confirm the attachment exists; I was getting the same
`ActiveStorage::IntegrityError` mentioned above.
Finally, we're updating the site customization image controller to use
`update` so the image and the attachment are updated within the same
transaction. This is also what we do in most controllers.
[1] https://www.youtube.com/watch?v=tZ_WNUytO9o
The `column` method in ActiveRecord::ConnectionAdapters::TableDefinition
supports adding the `index:` option. The documentation says:
> Instantiates a new column for the table. See connection.add_column for
> available options.
>
> Additional options are:
>
> :index - Create an index for the column. Can be either true or an
> options hash.
So basically the `connection.add_column` method silently ignores the
`index:` option, and whenever we intended to create an index this way,
we didn't.
We're creating a new migration where we properly add the indexes that
weren't added when we intended to.
Thanks to the rubocop-rails team, who added this cop in version 2.11.0
and helped us notice this bug.
Since version 2.0 introduced many breaking changes, we're upgrading to
it first.
The changes have been done by installing the rubocop-faker gem and
running:
```
rubocop \
--require rubocop-faker \
--only Faker/DeprecatedArguments \
--auto-correct
```
Currently it is not necessary to include the link_url field.
When we display these cards without link_url, they create an empty link that
redirects to the same page. I understand that this is not a desired behavior, so I
think it is better to add a validation in this case and force administrators to add a
link_url when creating a card.
