In commit 96ae69fe9, we stopped using cookies to track Ahoy visits and
started using a combination of the IP and the browser agent instead.
However, since we're still using the legacy logic from Ahoy 1.x to track
visits (which we had to add in commit b5220effd), this way of tracking
visits doesn't work and counts every page visited by a user as an
independent visit.
Maybe we could migrate existing data, which uses the `visitor_id` column
so it uses the new `visit_token` and `visitor_token` columns, but
there's no mention in the Ahoy documentation regarding how to do so.
While deciding what to do about this, we found something interesting.
For two years, we've been seeing random failures in the
`system/admin/tenants_spec.rb` tests, with messages like:
```
1) Tenants Create Tenant with subdomain
Failure/Error:
raise TenantNotFound, <<~EXCEPTION_MESSAGE
Could not set search path to schemas, they may be invalid:
"#{tenant}" #{full_search_path}.
Original error: #{exception.class}: #{exception}
EXCEPTION_MESSAGE
Apartment::TenantNotFound:
Could not set search path to schemas, they may be invalid:
"earth" "public", "shared_extensions".
Original error:
ActiveRecord::StatementInvalid: Could not find schema earth
```
And we've found one of the causes: the AJAX requests done by Ahoy to
track visits. Sometimes a test that creates or updates a tenant finishes
but the Ahoy AJAX request to, say, `earth.lvh.me/ahoy/visits`, is
handled by the next test, when the `earth` schema no longer exists, thus
raising an `Apartment::TenantNotFound` exception.
So by disabling these AJAX requests and tracking the visits in the
server instead, we're killing two birds in one stone: we're fixing the
bug regarding the visits count and we're reducing the flakiness in our
test suite. It looks like we're also removing the "phantom ahoy cookie"
we were getting since the mentioned commit b5220effd: an ahoy cookie was
quickly set and unset in the browser.
Note that, even though we aren't migrating any data, we're still adding
the new fields, because some tests started to fail because, when
tracking visits in the server without cookies, Ahoy expects the Visit
model to have a `visit_token` field.
Note: Since we update to 1.80.1 deprecation warnings are appear when execute the assets:precompile command.
In order to silence this deprecation, we add silence_deprecation option in sass.rb initializer.
The code has also been updated to remove the deprecation warnings that appeared related to the function
darken(), lighten() and "Using / for division" instead of the function calc().
Bumps [sassc-embedded](https://github.com/sass-contrib/sassc-embedded-shim-ruby) from 1.70.1 to 1.80.1.
- [Commits](https://github.com/sass-contrib/sassc-embedded-shim-ruby/compare/v1.70.1...v1.80.1)
---
updated-dependencies:
- dependency-name: sassc-embedded
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
When the `multitenancy_management_mode` is enabled.
In order to avoid infinite redirects when regular users try to access
the admin section, we're redirecting to the account page in this case.
Otherwise, the admin section would redirect to the root path, which
would redirect to the admin section, which would redirect to the root
path, and so on.
We only want to render the account link and login items in the header.
And we want only render the Multitenancy and Administrators sections in
the admin sidebar.
We include the administrators management so it's possible to give
permissions to other users to manage tenants.
In order to restrict access to other sections by typing the URL or
following a link, we're only enabling the rest of the routes when we
aren't in the multitenancy management mode.
We're going to add some constraints in the routes file, and if we add a
`resolve` clause inside a constraints block, we get an error saying that
"The resolve method can't be used inside a routes scope block" when
starting the application.
There are many possible ways to implement this feature:
* Adding a custom middleware
* Using rack-attack with a blocklist
* Using routes constraints
We're choosing to use a controller concern with a redirect because it's
what we do to handle unauthorized cancancan exceptions.
Using a checkbox wasn't very intuitive because checkboxes are
checked/unchecked when clicked on even if there's an error in the
request. Usually, when checkboxes appear on a form, they don't send any
information to the server unless we click a button to send the form.
So we're using a switch instead of a checkbox, like we did to
enable/disable phases in commit 46d8bc4f0.
Note that, since we've got two switches that match the default
`dom_id(record) .toggle-switch` selector, we need to find a way to
differentiate them. We're adding the `form_class` option for that.
Also note that we're now using a separate action and removing the
JavaScript in the `update` action which assumed that AJAX requests to
this action were always related to updating the `visible_to_valuators`
attribute.
This is consistent to what we usually do. Also, we're applying the same
criteria mentioned in commit 72704d776:
> We're also making these actions idempotent, so sending many requests
> to the same action will get the same result, which wasn't the case
> with the `toggle` action. Although it's a low probability case, the
> `toggle` action could result in [selecting an investment] when trying
> to [deselect] it if someone else has [deselected it] it between the
> time the page loaded and the time the admin clicked on the
> "[Selected]" button.
Just like it happened with proposals, the button to select/deselect an
investment wasn't very intuitive; for example, it wasn't obvious that
pressing a button saying "selected" would deselect the investment.
So we're using a switch control, like we do to enable/disable features
since commit fabe97e50.
Note that we're making the text of the switch smaller than in other
places because the text in the investments table it is also smaller
(we're using `font-size: inherit` for that purpose). That made the
button look weird because we were using rems instead of ems for the
width of the button, so we're adjusting that as well.
Also note we're changing the width of the switch to `6em` instead of
`6.25em` (which would be 100px if 1em is 16px). We're doing so because
we used 100 for the minimum width because it's a round number, so
now we're using another round number.
This is consistent to what we usually do. Also, we're applying the same
criteria mentioned in commit 72704d776:
> We're also making these actions idempotent, so sending many requests
> to the same action will get the same result, which wasn't the case
> with the `toggle` action. Although it's a low probability case, the
> `toggle` action could result in [selecting a proposal] when trying to
> [deselect] it if someone else has [deselected it] it between the time
> the page loaded and the time the admin clicked on the "[Selected]"
> button.
The button to select/deselect a proposal wasn't very intuitive; for
example, it wasn't obvious that pressing a button saying "selected"
would deselect the proposal.
So we're using a switch control, like we do to enable/disable features
since commit fabe97e50.
We had three files that were almost identical, and we can use
environment variables to specify the differences.
Note we're using the `PGUSER` and `PGPASSWORD` variables, since these
variables will automatically be used by the PostgreSQL client when we
have a blank `username` and `password` keys in the `database.yml` file
(which we did until now). The difference between these variables and the
`POSTGRES_USER` and `POSTGRES_PASSWORD` variables is that the `PG`
variables are used by the client connecting to the database, while the
`POSTGRES_` variables are used by the Docker postgresql image when
creating the database superuser.
For consistency with the code in our github workflows (and everywhere
else in the postgres world), we're respecting this double standard. The
fact that there are two different names for what's basically the same
thing makes the code confusing, though, particularly when running the
docker-compose commands, since we get the password from an environment
variable but we have to assign two different environment variables with
it.
So we're accepting both `PGPASSWORD` and `POSTGRES_PASSWORD` variables
in the database configuration file. This way, developers using
docker-compose can use `POSTGRES_PASSWORD` for everything and it'll work
fine. We're also making `PGPASSWORD` default to `POSTGRES_PASSWORD` so
we don't get a warning if we only set `POSTGRES_PASSWORD`:
```
WARN[0000] The "PGPASSWORD" variable is not set. Defaulting to a blank
string.
```
Also note we're using `DB_HOST` instead of `PGHOST` because that's the
variable Rails currently uses by default for new applications [1].
Finally, note we're using `.presence` in the `ENV` calls in the
database.yml file. The `PGPASSWORD` variable was set to an empty string
when running docker-compose, so using `ENV["PGPASSWORD"] ||` wouldn't
work.
[1] https://github.com/rails/rails/blob/c90a8701e5/railties/lib/rails/generators/rails/app/templates/config/databases/postgresql.yml.tt#L22
The keys to configure Omniauth for WordPress were only added
to the production environment, so we unified all the keys across
the staging and preproduction environments.
We're using version 13 because it's the one included in Debian Bullseye,
which is the operating system we currently use in our Dockerfile.
For consistency, we're using the same version in GitHub Actions.
Note this image requires setting a password. Otherwise we get an error:
> Database is uninitialized and superuser password is not specified.
> You must specify POSTGRES_PASSWORD to a non-empty value for the
> superuser. For example, "-e POSTGRES_PASSWORD=password" on
> "docker run".
Since now we're setting a password in the postgres service, we also need
to provide the `PGPASSWORD` environment variable (or to specify the
password in the `database.yml` file, which we do for GitLab since it
uses a separate database configuration file). Otherwise we get an error:
```
PG::ConnectionBad: connection to server at "::1", port 5432 failed:
fe_sendauth: no password supplied (PG::ConnectionBad)
```
When running delayed_job on reboot using Consul Democracy 2.2.0, we were
getting an error:
```
`require': cannot load such file -- sassc-embedded (LoadError)
from <internal:rubygems/core_ext/kernel_require.rb>:86:in `require'
from config/application.rb:1:in
```
No idea why this gem isn't detected when running `bin/delayed_job`.
When deploying with Capistrano, we use `bundle exec bin/delayed_job`,
and that works fine, so we're using it here as well.
We had already fixed this issue in the Cosul Democracy installer [1],
but forgot to configure the Cron job to do the same.
[1] https://github.com/consuldemocracy/installer/commit/3dc69eb6
Until now, people had to edit the original route files in order to add
custom routes.
This was inconsistent with the other customizations, since we use custom
folders or files for customizing controllers, components, views, ...
(which you usually customize as well when adding a new route).
So now we're providing a file for custom routes, which will make it
easier to know which routes are not present in Consul Democracy by
default.
When we upgraded to Rails 7 in commit 8596f1539, we broke the custom
locales since now all nested folders in `config/locales/` are loaded by
default [1]. This meant that the custom folder was now loaded before any
languages whose code alphabetically goes after the word "custom".
As a workaround, we're overwriting the default locales paths so they
don't include the custom folder. We're doing this step before the
`add_locales` initializer; that is, before the default locales paths are
used.
Unfortunately, I haven't found a way to add a test for this behavior,
since we would need to add a file in `config/locales/custom` that
overwrites an internationalization key for an existing language, but
only during a specific test, and the i18n load path is evaluated when
the application starts up.
[1] See pull request 41872 in https://github.com/rails/rails/
It was confusing to have the action to create an answer in
`QuestionsController#answer` while the action to destroy it was
`AnswersController#destroy`.
The routes for poll questions were accidentally deleted in commit
5bb831e959 when deleting the `:show` action, and restored in commit
9871503c5e. However, the deleted code was:
```
resources :questions, only: [:show], controller: 'polls/questions' (...)
```
While the restored code was:
```
resources :questions, controller: 'polls/questions' (...)
```
Meaning we forgot to add the `only: []` option when restoring the
routes.
We also forgot to remove the `before_action` code when deleting the
`:show` action, so we're removing it now.
Having a class named `Poll::Question::Answer` and another class named
`Poll::Answer` was so confusing that no developer working on the project
has ever been capable of remembering which is which for more than a few
seconds.
Furthermore, we're planning to add open answers to polls, and we might
add a reference from the `poll_answers` table to the
`poll_question_answers` table to property differentiate between open
answers and closed answers. Having yet another thing named answer would
be more than what our brains can handle (we know it because we did this
once in a prototype).
So we're renaming `Poll::Question::Answer` to `Poll::Question::Option`.
Hopefully that'll make it easier to remember. The name is also (more or
less) consistent with the `Legislation::QuestionOption` class, which is
similar.
We aren't changing the table or columns names for now in order to avoid
possible issues when upgrading (old code running with the new database
tables/columns after running the migrations but before deployment has
finished, for instance). We might do it in the future.
I've tried not to change the internationalization keys either so
existing translations would still be valid. However, since we have to
change the keys in `activerecord.yml` so methods like
`human_attribute_name` keep working, I'm also changing them in places
where similar keys were used (like `poll_question_answer` or
`poll/question/answer`).
Note that it isn't clear whether we should use `option` or
`question_option` in some cases. In order to keep things simple, we're
using `option` where we were using `answer` and `question_option` where
we were using `question_answer`.
Also note we're adding tests for the admin menu component, since at
first I forgot to change the `answers` reference there and all tests
passed.
We've been ignoring what the Bullet gem reports for at least 6 years
(maybe more), but we were still updating the gem and maintaining the
code in `config/environments/` (which caused conflicts every time we run
`rails app:update` to upgrade to a new Rails version). Maintaining it
isn't a huge effort, but it's infinitely bigger than the benefits we get
from it, which are zero.
Adding `includes` everywhere we query for records would be a huge
maintenance effort and would make the code less readable, so I don't
think it's worth it. We might do it occasionally if we detect a
performance bottleneck.
We could also use a gem to automatically avoid the N+1 queries problem,
like Goldiloader [1], ArLazyPreload [2] or JitPreload [3]. Benchmarks
show that the performance improvements obtained by using these gems is
about less than 10% (it depends a lot on the page being loaded, though),
which IMHO doesn't justify adding yet another gem that patches
ActiveRecord and that could be incompatible with other gems doing so.
There are a couple of open pull requests (at the time of writing,
they've been open for about two years) in the Rails repository [4][5] to
automatically avoid N+1 queries as well. For now, we'll hope something
similar is integrated in Rails itself in the future.
[1] https://github.com/salsify/goldiloader
[2] https://github.com/DmitryTsepelev/ar_lazy_preload
[3] https://github.com/clio/jit_preloader/
[4] Pull request 45231 in https://github.com/rails/rails/
[5] Pull request 45413 in https://github.com/rails/rails/
The text wasn't gramatically correct in English, since it had
the verb before the subject.
Since I'm not a native English speaker, I'm not sure about the
correct way to mention the Census part. I'm using "residents
in" since it sounds OK to me, but I might be wrong.
When this code was added, in commit 1a20a3ce4, we had no validation
rules checking the presence of the start and end dates of a poll. Now we
do, so we don't have to check this condition in the view.
Since they're related, we're making them part of the same list. Instead
of finding a way to have the `Select` prefix they had as a label for the
list, we're including the "prefix" they had inside their texts, so the
text of a button doesn't need any additional context.
This way we can simplify the view, particularly the form. However, we're
still adding some complexity to the form so inputs are inside labels and
so the collection is easier to style with CSS.
We're using different controls depending on the number of available
locales.
When there are only a few locales, the solution is obvious: radio
buttons to select the default language, and checkboxes to select the
available ones are simple and intuitive.
With many languages, showing two consecutive lists of 30 languages could
be confusing, though, particularly on small devices, where scrolling
through both lists could be hard.
So, in this case, we're rendering a <select> to choose the default
language. For selecting the available languages, however, we're sticking
to checkboxes because all the other existing options (like multiple
selects) are hard to use. We think it's OK because the form doesn't have
any additional fields, and there's only one big list of options to
scroll through.
While testing the application, we noticed that if we use the
`admin-fieldset-separator` styles when there's only one fieldset, it's
harder to notice that there's an additional field to select the default
language. So we're only using the `admin-fieldset-separator` styles when
all the fields are grouped in fieldsets.
Regarding the help text for the fieldset, if we leave the help text
outside the <legend> tag, people using screen readers won't hear about
this content. However, if we include it inside the <legend> tag, some
screen readers might read it every time they move to a different
checkbox (or radio button), which can be annoying. Since I don't think
these help messages are really essential, I'm leaving them out of the
<legend> tag. It's also easier to style them if they're outside the
<legend> tag.
Note we're using `display: table` for the labels, for the reasons
mentioned in commit 923c2a7ee.
Also note that, when there's only one available locale, this section is
useless. In this case, we aren't disabling it for now because there's a
chance people see it in the official Consul Democracy demo and then
wonder why it isn't available on their installation. We might disable it
in the future, though.
This is the admin section; it's obvious that every link in the menu will
take you to a page to manage something.
We're going to add a new item to either the "Settings" or the "Site
content" section, so it's a good time to improve what's already there.
We were defining one builder in the `app/lib/` folder and another one
inside a helper module.
So now we're grouping them together. This way we're following the "one
class per file" convention that we follow most of the time. And, by
extracting the `TranslatableFormBuilder` class to its own file, it'll be
easier to add tests for it.
Note that, for consistency, we're renaming the
`TranslationsFieldsBuilder` class so it ends in `FormBuilder`.
Since now generating stats (assuming the results aren't in the cache)
only takes a few seconds even when there are a hundred thousand
participants, as opposed to the several minutes it took to generate them
when we introduced the Cron job, we can simply generate the stats during
the first request to the stats page.
Note that, in order to avoid creating a temporary table when the stats
are cached, we're making sure we only create this table when we need to.
Otherwise, we could spend up to 1 second on every request to the stats
page creating a table that isn't going to be used.
Also note we're using an instance variable to check whether we're
creating a table; I tried to use `table_exists?`, but it didn't work. I
wonder whether `table_exists?` doesn't detect temporary tables.
As mentioned in Ahoy's README [1]:
> Ahoy provides a number of options to help with GDPR compliance.
> Update config/initializers/ahoy.rb with:
>
> class Ahoy::Store < Ahoy::DatabaseStore
> def authenticate(data)
> # disables automatic linking of visits and users
> end
> end
>
> Ahoy.mask_ips = true
> Ahoy.cookies = :none
As also mentioned in the README:
> If Ahoy was installed before v5, add an index before making this
> change.
> (...)
> For Active Record, create a migration with:
> add_index :ahoy_visits, [:visitor_token, :started_at]
However, the `visitor_token` doesn't exist in our table, since we
generated the `visits` table when Ahoy used the `visitor_id` column. So
we're using this column for the index.
Note we also need to change the `visit` method, since otherwise we get
an exception [2]. As mentioned on the issue reporting the exception:
> you'll need to copy the latest version of that method and adapt it to
> your model. I believe you'll want to replace:
>
> where(visit_token: ahoy.visit_token) with
> where(id: ensure_uuid(ahoy.visit_token))
>
> where(visitor_token: ahoy.visitor_token) with
> where(visitor_id: ensure_uuid(ahoy.visitor_token))
So we're copying the latest version of that method and changing it
accordingly.
[1] https://github.com/ankane/ahoy/blob/v5.0.2/README.md
[2] Issue 549 in https://github.com/ankane/ahoy
This page isn't linked from anywhere and most Consul Democracy
installations don't even know it exists, so it's useless for most
people.
If we ever bring it back, we should at least add a link pointing to this
page.
Now that we've moved the logic to generate the events data to a model,
and we've got access to the model in the component rendering the chart,
we can render the data inside the chart instead of doing an extra AJAX
request to get the same data.
Originally, this was problaby done this way so the page wouldn't take
several seconds to load while preparing the data for the chart when
there are thousands of dates being displayed. With an AJAX call, the
page would load as fast as usual, and then the chart would render after
a few seconds. However, we can have an even better performance
improvement in this scenario if we use a Set instead of an Array. The
method `Array#include?`, which we were calling for every date in the
data, is much slower that `Set#merge`. So now both the page and the
chart load as fast as expected.
We could also use something like:
```
def add
(...)
shared_keys.push(*collection.keys)
end
def build
(...)
shared_keys.uniq.each do |k|
(...)
end
def shared_keys
@shared_keys ||= []
end
```
Or other approaches to avoid using `Array#include?`. The performance
would be similar to the one we get when using `Set`. We're using a `Set`
because it makes more obvious that `shared_keys` is supposed to contain
unique elements.
We've had some tests failing in the past due to these AJAX requests
being triggered automatically during the tests and no expectations
checking the requests have finished, so now we're reducing the amount of
flaky tests.
We were always displaying the event names in English.
Note we're changing the `user_supported_budgets` key because it didn't
make much sense; the investments are supported, and not the budgets.
We're also adding "created" to most of the event names in order to make
the texts more explicit, since not all the events refer to created data.
We were tracking some events with Ahoy, but in an inconsistent way. For
example, we were tracking when a debate was created, but (probably
accidentally) we were only tracking proposals when they were created
from the management section. For budget investments and their supports,
we weren't using Ahoy events but checking their database tables instead.
And we were only using ahoy events for the charts; for the other stats,
we were using the real data.
While we could actually fix these issues and start tracking events
correctly, existing production data would remain broken because we
didn't track a certain event when it happened. And, besides, why should
we bother, for instance, to track when a debate is created, when we can
instead access that information in the debates table?
There are probably some features related to tracking an event and their
visits, but we weren't using them, and we were storing more user data
than we needed to.
So we're removing the track events, allowing us to simplify the code and
make it more consistent. We aren't removing the `ahoy_events` table in
case existing Consul Democracy installations use it, but we'll remove it
after releasing version 2.2.0 and adding a warning in the release notes.
This change fixes the proposal created chart, since we were only
tracking proposals created in the management section, and opens the
possibility to add more charts in the future using data we didn't track
with Ahoy.
Also note the "Level 2 user Graph" test wasn't testing the graph, so
we're changing it in order to test it. We're also moving it next to the
other graphs test and, since we were tracking the event when we were
confirming the phone, we're renaming to "Level 3 users".
Finally, note that, since we were tracking events when something was
created, we're including the `with_hidden` scope. This is also
consistent with the other stats shown in the admin section as well as
the public stats.
Using <h3> headings for the links had two disadvantages.
First, it was the wrong heading level to use, since there was no <h2>
tag before it.
Second, headings are supposed to be followed by content associated to
that heading; here, we had no content following the headings.
So we're using a list of links and giving it a heading. We're adding
styles so the page still looks like it used to, although these styles
are certainly asking for improvements.
In commit b3f570512, we changed the key generator hash digest class, and
we wrote:
> Since we haven't seen any Consul Democracy applications using
> encrypted messages and these messages become invalid with this change
> (...)
We didn't realize that ActiveStorage also used the old hash digest class
to generated the signed URLs used to access an image. This doesn't
affect us when we generate images using `image.variant`, because that
generates a new URL on the fly using the new hash digest class. However,
URLs referencing the images generated using the old hash digest class,
like the ones in the HTML content generated with CKEditor, would result
in 404 errors.
So we're rotating the signed IDs generated by earlier versions of
ActiveStorage. This way both new and old images will be correctly
displayed.
Note that, unlike cookies, which will keep working once rotated even if
we delete the code to rotate them, old ActiveStorage URLs will always
need the code rotating them in order to keep working.
Note that we used to have the link to delete images inside the same
<form> tag as the button to update the image. However, using a button
means we're adding a new <form> tag for the action to delete the image.
This isn't valid HTML and, in some browsers, might result in the button
sending the request to the wrong URL.
As explained in commit 5311daadf, to avoid this, we'd need to replace
`button_to` with `button_tag` in the action in order to generate a
button without a form. Then, we could add either a `form` or a
`formaction` attribute to the button.
However, I thik it's easier to move the delete button outside the update
button <form> tag. On the minus side, since the buttons no longer share
a parent, they're harder to style. So we're using a mix of nested flex
layouts with one of the nested elements using a container unit as width.
Since we're at it, we're also improving the styles on small and medium
screens by making sure the "Update" button wraps before the "Delete"
button does (using a container query), by giving enough width to the
column containing this actions on small screens as well (removing
`small-12` and giving it two-thirds of the width on all screen sizes)
and by having a gap between elements.
Note that, at the time of writing, container queries are only supported
by about 91%-93% of the browsers, meaning that some administrators will
see all from controls displayed vertically, one on top of the other, on
all screen sizes. We think this is acceptable, and the page remains
fully functional in this case.
