We were using two different sets of extensions but, since the markdown
code is always written by administrators, IMHO it makes sense to be
consistent and always render markdown code the same way.
* Add Tables option to Redcarpet in Legislation draft
* Allow table tags in Admin Legislation Sanitizer
* Add Test to render markdown tables in Legislation drafts
* Add Test for Admin Legislation Sanitizer
We include test for image, table and h1 to h6 tags and additional tests to strengthen the allowed and disallowed parameters
* Add Table from markdown test in System and Factories
* Add test to render tables for admin user
* Remove comment line about Redcarpet options
* Edit custom css for legislation draft table to make it responsive
This methods performs much better when we need to load a lot of
globalized models translations and returns the best fallback translation
for the current language.
Note that in the budgets wizard test we now create district with no
associated geozone, so the text "all city" will appear in the districts
table too, meaning we can't use `within "section", text: "All city" do`
anymore since it would result in an ambiguous match.
Co-Authored-By: Julian Herrero <microweb10@gmail.com>
Co-Authored-By: Javi Martín <javim@elretirao.net>
These tests were failing with Ruby 3.0 because we were getting an error
when loading the `translations` association of the dummy class:
```
NameError:
uninitialized constant
DummyBanner::#<Class:0x000055630e4dccd8>::Translation
```
Specifying the `class_name` of the `translations` association solves the
issue.
It has been detected that for the :pt-BR, :zh-CN and :zh-TW locales,
the translate button was being displayed, but when requesting the
translation, the remote translation validation failed due to:
'''
validates :locale, inclusion: { in: ->(_) {
RemoteTranslations::Microsoft::AvailableLocales.available_locales }}
'''
That available_locales method did not contemplate these 3 languages
in the format used by the application.
To solve this problem the api response is mapped to return all
locales in the format expected by the application.
Add remote translation model test to ensure that a remote translation
is valid when its locale is pt-BR.
Co-Authored-By: Javi Martín <35156+javierm@users.noreply.github.com>
When accessing the valuation area, we were only displaying the
investments directly assigned to the current valuator, but we weren't
displaying the investments assigned to that valuator's group.
Using the `assigned_investments_ids` method, which takes the valuator
group into account, solves the issue.
We've also found an issue on our development machines: since we don't
have a unique index per `investment_id` and `valuator_id` in the
`budget_valuator_assignments` table, we've found duplicate records on
this table. When that happened, we were displaying the same investment
several times.
Since now we no longer join this table in the query returning the
investment, this issue is also solved, and we're adding a test for it.
We can now remove the call to the `distinct` method when calculating the
number of investments per heading.
To get the heading where a user voted, we were relying on the
`balloted_heading_id` field.
Our guess is this was done so the total number of users is the same as
the sum of users who voted on a heading. That is, if 2000 people voted
just on the "All city" heading, 1000 voted just on the "North district"
heading, and 500 people voted on both, instead of showing "3500 people
voted in total, 2500 voted in all city, 1500 voted in north district",
we show something like "3500 people voted in total, 2250 voted in all
city, and 1250 voted in north district".
However, this approach has some disadvantages.
The first disadvantage is, the stats aren't correct. In the case above,
2500 voted on the "All city heading", so the statistics for this heading
don't show reality.
The second one is we weren't considering the last heading where users
voted inside the budget being displayed, but the last heading where
users voted, period. That means that, if all the people above voted on a
later budget, the stats for the budget above would become "3500 people
voted in total, 0 voted in all city, and 0 voted in north district".
That also means we were including headings from previous budgets in the
statistics for more recent budgets when people hadn't voted on the
recent ones.
So we're removing the `balloted_heading_id` since its data is lost once
people vote on a new budget. And, in order to show the right stats and
simplify the code, we're no longer trying to add votes just to one
heading when users vote on several headings.
Co-Authored-By: Julian Nicolas Herrero <microweb10@gmail.com>
It was added in rubocop-performance 1.13.0. We were already applying it
in most places.
We aren't adding it for performance reasons but in order to make the
code more consistent.
Note we could use `acts_as_paranoid` with the `without_default_scope`
option, but we aren't doing so because it isn't possible to consider
deleted records in uniqueness validations with the paranoia gem [1].
I've added tests for these cases so we don't accidentally add
`acts_as_paranoid` in the future.
Also note we're extracting a `RowComponent` because, when
enabling/disabling a tenant, we're also enabling/disabling the link
pointing to its URL, and so we need to update the URL column after the
AJAX call.
[1] See issues 285 and 319 in https://github.com/rubysherpas/paranoia/
Some institutions using CONSUL have expressed interest in this feature
since some of their tenants might already have their own domains.
We've considered many options for the user interface to select whether
we're using a subdomain or a domain, like having two separate fields,
using a check box, ... In the end we've chosen radio buttons because
they make it easier to follow a logical sequence: first you decide
whether you're introducing a domain or subdomain, and then you enter it.
We've also considered hiding this option and assuming "if it's got a
dot, it's a domain". However, this wouldn't work with nested subdomains
and it wouldn't work with domains which are simply machine names.
Note that a group of radio buttons (or check boxes) is difficult to
style when the text of the label might expand over more than one line
(as is the case here on small screens); in this case, most solutions
result in the second line of the label appearing immediately under the
radio button, instead of being aligned with the first line of the label.
That's why I've added a container for the input+label combination.
Creating a schema takes about 3-4 seconds on my machine, so omitting
the callbacks makes tests much faster.
To do so, we're using the `insert!` method added in Rails 6.0, which
inserts a record without executing callbacks or validations. To make the
tests look consistent, we're adding a FactoryBot strategy which uses
`insert!` instead of `create!`.
Note this strategy is useless in most cases because it doesn't work when
models have translatable attributes or associations. However, IMHO it's
worth it even if we only use it for tenants.
We could also use `Tenant.insert!` instead, but then we would have to
add all the mandatory attributes, and in this case the code is clearer
if we only add the attributes we need for the test.
This way we'll avoid having all tenant organizations named "CONSUL" by
default, and we'll also use different email senders per tenant, which
will reduce the change of the emails being marked as spam.
We were using `consul.dev` as default, which might be fine for the
development environment, but it's something that definitely needs to be
changed on production.
Now we're providing a better default setting: using the same domain
which is used to generate URLs in emails. This also makes it easier to
configure new tenants, since the setting will default to whatever host
the new tenant is using.
The `|| "consul.dev"` part might not be needed; I'm keeping it because
I'm not sure production environents would not experience any new issues
if we don't add it, but we might remove it in the future.
We were using the same logic in many different places, so we're
simplifying the code. I'm not convinced about the method names, though,
so we might change them in the future.
Note using this method for the default tenant in the `TenantDiskService`
class resulted in a `//` in the path, which is probably harmless but
very ugly and it also generates a different key than the one we got
until now. I've added an extra test to make sure that isn't the case.
We're using the "tenants" subfolder for consistency with the folder
structure we use in ActiveStorage and because some CONSUL installations
might have folders inside the `data` folder which might conflict with
the folders created by tenants.
Note that the Python scripts have a lot of duplication, meaning we need
to change all of them. I'm not refactoring them because I'm not familiar
enough with these scripts (or with Python, for that matter).
Also note that the scripts folder is still shared by all tenants,
meaning it isn't possible to have different scripts for different
tenants. I'm not sure how this situation should be handled; again, I'm
not familiar enough with this feature.
Right now this is configured using the `secrets.yml` file, which is the
file we've used in the past to configure SMTP settings.
Note that, in the `current_secrets` method, the `if default?` condition
is there so in single-tenant applications it returns the exact same
object as `Rails.application.secrets`, and it makes it immediately clear
for developers reading the code. We're also caching the tenant secrets
(using `||=`) so they behave the same way as Rails secrets; for this to
work properly 100% of the time (for example, in tests) we need to expire
these cached secrets whenever the Rails secrets change.
A similar `unless Tenant.default?` condition is present in the
ApplicationMailer because there's a chance some CONSUL installations
might not be using secrets to define the SMTP settings(they might be
using environment variables, for example) and so in this case we don't
want to force settings based on the secrets.yml file because it would
break the application.
The structure of the SMTP settings in the secrets file should be:
```
production:
tenants:
name_of_the_tenant_subdomain:
smtp_settings:
address:
(...)
```
While this is not strictly necessary, it can help moving the data of one
tenant to a different server or removing it.
Note we're using subfolders inside the `tenants` subfolder. If we simply
used subfolders with the schema names, if the schema names were, for
instance, language codes like `es`, `en`, `it`, ... they would conflict
with the default subfolders used by Active Storage.
Note we aren't allowing to delete a tenant because it would delete all
its data, so this action is a very dangerous one. We might need to add a
warning when creating a tenant, indicating the tenant cannot be
destroyed. We can also add an action to delete a tenant which forces the
admin to write the name of the tenant before deleting it and with a big
warning about the danger of this operation.
For now, we're letting administrators of the "main" (default) tenant to
create other tenants. However, we're only allowing to manage tenants
when the multitenancy configuration option is enabled. This way the
interface won't get in the way on single-tenant applications.
We've thought about creating a new role to manage tenants or a new URL
out of the admin area. We aren't doing so for simplicity purposes and
because we want to keep CONSUL working the same way it has for
single-tenant installations, but we might change it in the future.
There's also the fact that by default we create one user with a known
password, and if by default we create a new role and a new user to
handle tenants, the chances of people forgetting to change the password
of one of these users increases dramatically, particularly if they
aren't using multitenancy.
We had some of the logic in the ApplicationMailer. Since we're going to
use this logic in more places, we're moving it to the Tenant model,
which is the model handling everything related to hosts.
Some tasks don't have to run on every tenant. The task to calculate the
TSV is only done for records which were present before we added the TSV
column, and that isn't going to happen in any tenants because we added
the TSV column before adding the tenants table. Similarly, the migration
needed for existing polls isn't necessary because there weren't any
tenants before we allowed to set the starting/ending time to polls.
We aren't adding any tests for these tasks because tests for rake tasks
are slow and tests creating tenants are also slow, making the
combination of the two even slower, particularly if we add tests for
every single task we're changing. We're adding tests for the
`.run_on_each` method instead.
The subdomain elevator we were using, which is included in apartment,
didn't work on hosts already including a subdomain (like
demo.consul.dev, for instance). In those cases, we would manually add
the subdomain to the list of excluded subdomains. Since these subdomains
will be different for different CONSUL installations, it meant each
installation had to customize the code. Furthermore, existing
installations using subdomains would stop working.
So we're using a custom method to find the current tenant, based on the
host defined in `default_url_options`.
In order to avoid any side-effects on single-tenant applications, we're
adding a new configuration option to enable multitenancy
We're enabling two ways to handle this configuration option:
a) Change the application_custom.rb file, which is under version control
b) Change the secrets.yml file, which is not under version control
This way people prefering to handle configuration options through
version control can do so, while people who prefer handling
configuration options through te secrets.yml file can do so as well.
We're also disabling the super-annoying warnings mentioning there are no
tenants which we got every time we run migrations on single-tenant
applications. These messages will only be enabled when the multitenancy
feature is enabled too. For this reason, we're also disabling the
multitenancy feature in the development environment by default.
Note we're using the `:HOST` regular expression since subdomains can
contain the same characters as domains do. This isn't 100% precise,
though, since subdomains have a maximum length of 63 characters, but is
good enough for our purposes.
This way all tenants will be able to access them instead of just the
default one.
The apartment gem recommends using a rake task instead of a migration,
but that's a solution which is primarily meant for new installations.
Migrations are easier to execute on existing installations.
However, since this migration doesn't affect the `schema.rb` file, we
still need to make sure the shared schema is created in tasks which do
not execute migrations, like `db:schema:load` or `db:test:prepare`, just
like the apartment gem recommends. That's why we're enhancing these
tasks so they execute this migration.
Note that there might be cases where the database user isn't a superuser
(as it's usually the case on production environments), meaning commands
to create, alter or drop extensions will fail. There's also the case
where users don't have permissions to create schemas, which is needed in
order to create the shared extensions schema and the schemas used by the
tenants. For these reasons, we're minimizing the number of commands, and
so we only alter or create extensions when it is really necessary.
When users don't have permission, we aren't running the commands but
showing a warning with the steps needed to run the migration manually.
This is only necessary on installations which are going to use
multitenancy; single-tenant applications upgrading don't need to run
this migration, and that's why we aren't raising exceptions when we
can't run it.
For new installations, we'll change the CONSUL installer so extensions
are automatically created in the shared schema.
Also note the plpgsql extension is not handled here. This is a special
extension which must be installed on the pg_catalog schema, which is
always in the search path and so is shared by all tenants.
Finally, we also need to change the `database.yml` file in order to
search for shared extensions while running migrations or model tests,
since none of our enabled extensions are executed during migrations;
we're also adding a rake task for existing installations. Quoting the
apartment documentation:
> your database.yml file must mimic what you've set for your default and
> persistent schemas in Apartment. When you run migrations with Rails,
> it won't know about the extensions schema because Apartment isn't
> injected into the default connection, it's done on a per-request
> basis.
The scripts crashed when the `data` folder wasn't present, which is the
common situation in development environments or production environments
not using Capistrano, since this folder isn't under version control.
In case we receive consecutive requests we are locking the poll author record
until the first request transaction ends, so the author answers count during
subsequent requests validations is up to date.
Note that the `create` action doesn't create an image but updates an
answer instead. We're removing the references to `:create` in the
abilities since it isn't used.
In the future we might change the form to add an image to an answer
because it's been broken for ages since it shows all the attached
images.
Adding, modifiying, and/or deleting questions for an already started
poll is far away from being democratic and can lead to unwanted side
effects like missing votes in the results or stats.
So, from now on, only modifiying questions will be possible only if
the poll has not started yet.
We need to update a couple of tests because a poll is created in the
tests with a timestamp that includes nanoseconds and in the form to edit
the time of the poll the nanoseconds are not sent, meaning it was
detected as a change.
We were already saving it as a time, but we didn't offer an interface to
select the time due to lack of decent browser support for this field
back when this feature was added.
However, nowadays all major browsers support this field type and, at the
time of writing, at least 86.5% of the browsers support it [1]. This
percentage could be much higher, since support in 11.25% of the browsers
is unknown.
Note we still need to support the case where this field isn't supported,
and so we offer a fallback and on the server side we don't assume we're
always getting a time. We're doing a strange hack so we set the field
type to text before changing its value; otherwise old Firefox browsers
crashed.
Also note that, until now, we were storing end dates in the database as
a date with 00:00 as its time, but we were considering the poll to be
open until 23:59 that day. So, in order to keep backwards compatibility,
we're adding a task to update the dates of existing polls so we get the
same behavior we had until now.
This also means budget polls are now created so they end at the
beginning of the day when the balloting phase ends. This is consistent
with the dates we display in the budget phases table.
Finally, there's one test where we're using `beginning_of_minute` when
creating a poll. That's because Chrome provides an interface to enter a
time in a `%H:%M` format when the "seconds" value of the provided time
is zero. However, when the "seconds" value isn't zero, Chrome provides
an interface to enter a time in a `%H:%M:%S` format. Since Capybara
doesn't enter the seconds when using `fill_in` with a time, the test
failed when Capybara tried to enter a time in the `%H:%M` format when
Chrome expected a time in the `%H:%M:%S` format.
To solve this last point, an alternative would be to manually provide
the format when using `fill_in` so it includes the seconds.
[1] https://caniuse.com/mdn-html_elements_input_type_datetime-local
