We've been ignoring what the Bullet gem reports for at least 6 years
(maybe more), but we were still updating the gem and maintaining the
code in `config/environments/` (which caused conflicts every time we run
`rails app:update` to upgrade to a new Rails version). Maintaining it
isn't a huge effort, but it's infinitely bigger than the benefits we get
from it, which are zero.
Adding `includes` everywhere we query for records would be a huge
maintenance effort and would make the code less readable, so I don't
think it's worth it. We might do it occasionally if we detect a
performance bottleneck.
We could also use a gem to automatically avoid the N+1 queries problem,
like Goldiloader [1], ArLazyPreload [2] or JitPreload [3]. Benchmarks
show that the performance improvements obtained by using these gems is
about less than 10% (it depends a lot on the page being loaded, though),
which IMHO doesn't justify adding yet another gem that patches
ActiveRecord and that could be incompatible with other gems doing so.
There are a couple of open pull requests (at the time of writing,
they've been open for about two years) in the Rails repository [4][5] to
automatically avoid N+1 queries as well. For now, we'll hope something
similar is integrated in Rails itself in the future.
[1] https://github.com/salsify/goldiloader
[2] https://github.com/DmitryTsepelev/ar_lazy_preload
[3] https://github.com/clio/jit_preloader/
[4] Pull request 45231 in https://github.com/rails/rails/
[5] Pull request 45413 in https://github.com/rails/rails/
This way we remove a bit of duplication.
These changes also affect the way geozones are rendered in a couple of
minor ways, making them more consistent:
* No empty list of geozones is rendered when there are no geozones
(before these changes, an empty list was rendered in the index action
but not in the show action)
* The text clarifying the geozone restriction is always shown (before
these changes, it was shown in the index action but not in the show
action)
We've added tests for these cases.
This is consistent with the way we've got partials to render debates,
proposals and legislation processes on their index pages.
Note that, while adding the tests for the status icon, we're keeping one
system test because it also tests the process of voting. We're adding a
new, similar component test, where the voter is created in the database,
so all possible statuses are tested in the component.
With this parameter, Vimeo no longer uses cookies that identifies users
browsing our site.
They do still store some cookies, though; quoting from Vimeo player
parameters overview:
> When DNT is enabled, Vimeo deploys one essential cookie via the
> embeddable player:
> The __cf_bm cookie, which is part of Cloudflare's Bot Management
> service and helps mitigate risk associated with spam and bot traffic.
Not sure whether this counts as essential cookies in our case; they're
essential for Vimeo, but for us, they're third-party cookies, after all.
[1] https://help.vimeo.com/hc/en-us/articles/12426260232977-Player-parameters-overview
When embedding a video in our site YouTube stores cookies in the user's
computer that aren't necessary to watch the video, so we'd have to make
people accept those cookies before letting them watch the video.
Using a URL that doesn't use cookies, like mentioned in YouTube Help
[1], is easier, though, and respects people's privacy without affecting
the user experience.
That I've found some references saying that youtube does store cookies
once you hit the "play" button even when using the nocookie server [2].
Not sure whether that's an old behavior or I'm doing something wrong,
but I don't see this is the case; even after playing the video, cookies
aren't stored on my browser.
[1] https://support.google.com/youtube/answer/171780#zippy=%2Cturn-on-privacy-enhanced-mode
[2] https://www.cnet.com/news/privacy/youtubes-new-nocookie-feature-continues-to-serve-cookies/
Now we're also testing that there's an iframe with the URL; before this
change, the test would pass even if the JavaScript generating the iframe
wouldn't work.
Although most Consul Democracy installations will only have a few
available languages using `config.i18n.available_locales`, there's a
chance some installation will keep every language as available and will
enable the desired ones using the admin interface. In these cases,
enabling (or disabling) every language would be tedious, particularly
when casually experimenting in a staging environment or while using the
official Consul Democracy demo.
So we're adding buttons to simplify the process. Since some
installations might have only a couple of available languages, and in
this case these buttons would be pretty much useless, we're only showing
them when there are many languages available.
Since they're related, we're making them part of the same list. Instead
of finding a way to have the `Select` prefix they had as a label for the
list, we're including the "prefix" they had inside their texts, so the
text of a button doesn't need any additional context.
People using screen readers usually expect links to take them somewhere
else in the page on to a different page, while they expect buttons to
change something on the page.
Since we're in the latter scenario, using a button is more accessible.
It's also more natural; with a button, we don't need to provide `#` as
the URL or stop the default event when the button is clicked. And,
unlike links, buttons can be activated with either the space or the
enter key. Finally, clicking a link pointing to `#` with the middle
mouse button opens a useless new tab, while buttons do nothing in this
case.
Now that we only have one "All" link on the page, we no longer need to
specify which "All" link we're clicking or which "All" link we are
checking, so we're simplifying the code doing so.
Since we don't usually style HTML classes starting with `js-`, we're
renaming it, so it's consistent with the `CheckAllNone` name used in the
`check_all_none.js` file.
Since we were on the "Pending review" filter, and there were no records
pending review, the code checking all checkboxes were checked/unchecked
didn't test anything because there were no checkboxes on the page.
So we're clicking on the "All" filter first.
This way we can simplify the view, particularly the form. However, we're
still adding some complexity to the form so inputs are inside labels and
so the collection is easier to style with CSS.
We're using different controls depending on the number of available
locales.
When there are only a few locales, the solution is obvious: radio
buttons to select the default language, and checkboxes to select the
available ones are simple and intuitive.
With many languages, showing two consecutive lists of 30 languages could
be confusing, though, particularly on small devices, where scrolling
through both lists could be hard.
So, in this case, we're rendering a <select> to choose the default
language. For selecting the available languages, however, we're sticking
to checkboxes because all the other existing options (like multiple
selects) are hard to use. We think it's OK because the form doesn't have
any additional fields, and there's only one big list of options to
scroll through.
While testing the application, we noticed that if we use the
`admin-fieldset-separator` styles when there's only one fieldset, it's
harder to notice that there's an additional field to select the default
language. So we're only using the `admin-fieldset-separator` styles when
all the fields are grouped in fieldsets.
Regarding the help text for the fieldset, if we leave the help text
outside the <legend> tag, people using screen readers won't hear about
this content. However, if we include it inside the <legend> tag, some
screen readers might read it every time they move to a different
checkbox (or radio button), which can be annoying. Since I don't think
these help messages are really essential, I'm leaving them out of the
<legend> tag. It's also easier to style them if they're outside the
<legend> tag.
Note we're using `display: table` for the labels, for the reasons
mentioned in commit 923c2a7ee.
Also note that, when there's only one available locale, this section is
useless. In this case, we aren't disabling it for now because there's a
chance people see it in the official Consul Democracy demo and then
wonder why it isn't available on their installation. We might disable it
in the future, though.
This is the admin section; it's obvious that every link in the menu will
take you to a page to manage something.
We're going to add a new item to either the "Settings" or the "Site
content" section, so it's a good time to improve what's already there.
Note that, for everything to work consistently, we need to make sure
that the default locale is one of the available locales.
Also note that we aren't overwriting the `#save ` method set by
globalize. I didn't feel too comfortable changing a monkey-patch which
ideally shouldn't be there in the first place, I haven't found a case
where `Globalize.locale` is `nil` (since it defaults to `I18n.locale`,
which should never be `nil`), so using `I18n.default_locale` probably
doesn't affect us.
Note that, currently, we take these settings from the database but we
don't provide a way to edit them through the admin interface, so the
locales must be manually introduced through a Rails console.
While we did consider using a comma-separated list, we're using spaces
in order to be consistent with the way we store the allowed content
types settings.
The `enabled_locales` nomenclature, which contrasts with
`available_locales`, is probably subconsciously based on similar
patterns like the one Nginx uses to enable sites.
Note that we aren't using `Setting.enabled_locales` in the globalize
initializer when setting the fallbacks. This means the following test
(which we could add to the shared globalizable examples) would fail:
```
it "Falls back to an enabled locale if the fallback is not enabled" do
Setting["locales.default"] = "en"
Setting["locales.enabled"] = "fr en"
allow(I18n.fallbacks).to receive(:[]).and_return([:fr, :es])
Globalize.set_fallbacks_to_all_available_locales
I18n.with_locale(:fr) do
expect(record.send(attribute)).to eq "In English"
end
end
```
The reason is that the code making this test pass could be:
```
def Globalize.set_fallbacks_to_all_available_locales
Globalize.fallbacks = I18n.available_locales.index_with do |locale|
((I18n.fallbacks[locale] & Setting.enabled_locales) + Setting.enabled_locales).uniq
end
end
```
However, this would make it impossible to run `rake db:migrate` on new
applications because the initializer would try to load the `Setting`
model but the `settings` table wouldn't exist at that point.
Besides, this is a really rare case that IMHO we don't need to support.
For this scenario, an installation would have to enable a locale, create
records with contents in that locale, then disable that locale and have
that locale as a fallback for a language where content for that record
wasn't created. If that happened, it would be solved by creating content
for that record in every enabled language.
We were defining one builder in the `app/lib/` folder and another one
inside a helper module.
So now we're grouping them together. This way we're following the "one
class per file" convention that we follow most of the time. And, by
extracting the `TranslatableFormBuilder` class to its own file, it'll be
easier to add tests for it.
Note that, for consistency, we're renaming the
`TranslationsFieldsBuilder` class so it ends in `FormBuilder`.
There was an edge case where a user could configure a locale and then
the application would change the locales to that one would no longer be
available. In that case, we were getting a `I18n::InvalidLocale`
exception when accessing the subscriptions page.
So now, we're defaulting to `I18n.locale`. Note we're using
`I18n.locale`instead of `I18n.default_locale` because `set_user_locale`
is called inside the `switch_locale` block in `ApplicationController`,
which already sets `I18n.locale` based on `I18n.default_locale`.
This way it'll be easier to change it while checking we haven't broken
existing behavior.
While writing the tests, I noticed we were sometimes storing a symbol in
the session while sometimes we were storing a string. So we're adding a
`to_s` call so we always store a string in the session.
When calculating the stats on, say, January 1st 2024, and using a group
age of, say, between 20 and 24 years, we were considering that everyone
born between 2000 and 2004 had between 20 and 24 years. This wasn't
accurate, since most people born in 2004 would have 19 years at that
point, and most people born in 1999 would have 24 years.
The age of the participants should refer to the time where the voting
took place, and not the time when the budget was created.
Note that now the tests pass even when the budget is created in a year
but the balloting phase takes place in a different year. By default,
each budget phase lasts for one month, so when we create a budget in a
test, its balloting phase takes place a few months after the date when
the budget is created. Since the `between_ages` scope uses the date of
the balloting phase as a reference, and the test was using the date when
the budget was created as a reference, the test failed depending on
whether those dates took place in the same year or not.
In commit e51e03446, we started using the same code to show stats in the
public area and in the admin area. However, in doing so we introduced a
bug, since stats in the public area are only shown after a certain part
of the process has finished, meaning the stats appearing on the page
never change (in theory), so it's perfectly fine to cache them. However,
in the admin area stats can be accessed while the process is still
ongoing, so caching the stats will lead to the wrong results being
displayed.
We've thought about expiring the cache when new supports or ballot lines
are added; however, that means the methods calculating the stats for the
supporting phase would expire when supports are added/removed but the
methods calculating the stats for the voting phase would expire when
ballot lines are added/removed. It gets even more complex because the
`headings` method calculates stats for both the supporting and the
voting phases.
So, since loading stats in the admin section is fast even without the
cache because they only load very basic statistics, we're taking the
simple approach of disabling the cache in this case, so everything works
the same way it did before commit e51e03446.
Co-authored-by: Javi Martín <javim@elretirao.net>
When we first started caching the stats, generating them was a process
that took several minutes, so we never expired the cache.
However, there have been cases where we run into issues where the stats
shown on the screen were outdated. That's why we introduced a task to
manually expire the cache.
But now, generating the stats only takes a few seconds, so we can
automatically expire them every day, remove all the logic needed to
manually expire them, and get rid of most of the issues related to the
cache being outdated.
We're expiring them every day because it's the same day we were doing in
public stats (which we removed in commit 631b48f58), only we're using
`expires_at:` to set the expiration time, in order to simplify the code.
Note that, in the test, we're using `travel_to(time)` so the test passes
even when it starts an instant before midnight. We aren't using
`:with_frozen_time` because, in similar cases (although not in this
case, but I'm not sure whether that's intentional), `travel_to` shows
this error:
> Calling `travel_to` with a block, when we have previously already made
> a call to `travel_to`, can lead to confusing time stubbing.
Debugging shows that the bottleneck in the stats calculation is the
number of times we're querying the users table using the same array of
IDs in the `where` condition but each time combined with other
conditions.
So we're inserting the results of querying the users table with the
array of IDs in a temporary table and using this temporary table for the
other calculations. When querying this temporary table, there's no need
to filter for IDs anymore.
For budget stats, the `generate` method is now about 10-20 times faster
for a budget with 20,000 participants. For budgets with only a few dozen
participants, there's no significant difference in performance.
I thought about modifying the `participants` method and use the
temporary table there. The problem, however, is that in this case it
isn't clear when to drop the temporary table, and we could end up with
thousands of temporary tables in the database if we don't do it right.
Creating and dropping the temporary table in the same transaction, on
the other hand, guarantees that won't be the case.
Note there's no risk of duplicate tables since they're created and
dropped inside a transaction, so we're always using the same table name
for the same resource. We're adding a test that fails with a
`PG::DuplicateTable: ERROR: relation "participants__1"` error if we
don't use a transaction.
Back in commit 383909e16, we said:
> Even if this class looks very simple now, we're trying a few things
> related to these stats. Having a class for it makes future changes
> easier and, if there weren't any future changes, at least it makes
> current experiments easier.
Since there haven't been any changes in the last 5 years and we've found
cases where using the GeozoneStats class results in a slightly worse
performance, we're removing this class. The code is now a bit easier to
read, and is consistent with the way we calculate participants by age.
This reverts commit 383909e16.
We were calculating the age stats based on the age of the users who
participated... at the moment where we were calculating the stats. That
means that, if 20 years ago, 1000 people who were 16 years old
participated, they would be shown as having 36 years in the stats.
Instead, we want to show the stats at the time when the process took
place, so we're implementing a `participation_date` method.
Note that, for polls, we could actually use the `age` column in the
`poll_voters` table. However, doing so would be harder, would only work
for polls but not for budgets, and it wouldn't be statistically very
relevant, since the stats are shown by age groups, and only a small
percentage of people would change their age group (and only to the
nearest one) between the time they participate and the time the process
ends.
We might use the `poll_voters` table in the future, though, since we
have a similar issue with geozones and genders, and using the
information in `poll_voters` would solve it as well (only for polls,
though).
Also note that we're using the `ends_at` dates because some people but
be too young to vote when a process starts but old enough to vote when
the process ends.
Finally, note that we might need to change the way we calculate the
participation date for a budget, since some budgets might not enabled
every phase. Not sure how stats work in that scenario (even before these
changes).
We were already testing it as part of the statisticable concern, but
it's easier to understand when we have specific tests for it as well.
Note that we're using `eq` insteab of `be` because it's what we use most
often in the test. For consistency, we're also changing the tesst in
budget stats so it uses `eq`.
According to the README [1]:
> To mask previously collected IPs, use:
> Ahoy::Visit.find_each do |visit|
> visit.update_column :ip, Ahoy.mask_ip(visit.ip)
> end
We're adapting the code with our version, since we use the `Visit` model
instead of the `Ahoy::Visit` model.
[1] https://github.com/ankane/ahoy/blob/v5.0.2/README.md#ip-masking
This page isn't linked from anywhere and most Consul Democracy
installations don't even know it exists, so it's useless for most
people.
If we ever bring it back, we should at least add a link pointing to this
page.
The test was supposed to be testing notifications without a link, but it
was adding a link. It was only passing because the expectations was
checking whether there was a link whose text was a URL, but what we
want to check here is whether the URL is present in the `href`
attribute.
We're about to delete the page showing public stats and, since this link
was using the `/stats` path, we're fixing it.
Now that we've moved the logic to generate the events data to a model,
and we've got access to the model in the component rendering the chart,
we can render the data inside the chart instead of doing an extra AJAX
request to get the same data.
Originally, this was problaby done this way so the page wouldn't take
several seconds to load while preparing the data for the chart when
there are thousands of dates being displayed. With an AJAX call, the
page would load as fast as usual, and then the chart would render after
a few seconds. However, we can have an even better performance
improvement in this scenario if we use a Set instead of an Array. The
method `Array#include?`, which we were calling for every date in the
data, is much slower that `Set#merge`. So now both the page and the
chart load as fast as expected.
We could also use something like:
```
def add
(...)
shared_keys.push(*collection.keys)
end
def build
(...)
shared_keys.uniq.each do |k|
(...)
end
def shared_keys
@shared_keys ||= []
end
```
Or other approaches to avoid using `Array#include?`. The performance
would be similar to the one we get when using `Set`. We're using a `Set`
because it makes more obvious that `shared_keys` is supposed to contain
unique elements.
We've had some tests failing in the past due to these AJAX requests
being triggered automatically during the tests and no expectations
checking the requests have finished, so now we're reducing the amount of
flaky tests.
We were always displaying the event names in English.
Note we're changing the `user_supported_budgets` key because it didn't
make much sense; the investments are supported, and not the budgets.
We're also adding "created" to most of the event names in order to make
the texts more explicit, since not all the events refer to created data.
Note we're delegating the `t` method because i18n-tasks doesn't detect
code like `ApplicationController.helpers.t` and so reports we aren't
using the `admin.stats.graph` translations.
We were tracking some events with Ahoy, but in an inconsistent way. For
example, we were tracking when a debate was created, but (probably
accidentally) we were only tracking proposals when they were created
from the management section. For budget investments and their supports,
we weren't using Ahoy events but checking their database tables instead.
And we were only using ahoy events for the charts; for the other stats,
we were using the real data.
While we could actually fix these issues and start tracking events
correctly, existing production data would remain broken because we
didn't track a certain event when it happened. And, besides, why should
we bother, for instance, to track when a debate is created, when we can
instead access that information in the debates table?
There are probably some features related to tracking an event and their
visits, but we weren't using them, and we were storing more user data
than we needed to.
So we're removing the track events, allowing us to simplify the code and
make it more consistent. We aren't removing the `ahoy_events` table in
case existing Consul Democracy installations use it, but we'll remove it
after releasing version 2.2.0 and adding a warning in the release notes.
This change fixes the proposal created chart, since we were only
tracking proposals created in the management section, and opens the
possibility to add more charts in the future using data we didn't track
with Ahoy.
Also note the "Level 2 user Graph" test wasn't testing the graph, so
we're changing it in order to test it. We're also moving it next to the
other graphs test and, since we were tracking the event when we were
confirming the phone, we're renaming to "Level 3 users".
Finally, note that, since we were tracking events when something was
created, we're including the `with_hidden` scope. This is also
consistent with the other stats shown in the admin section as well as
the public stats.
The only way to use campaigns is to manually insert them in the
database, which IMHO isn't very practical.
We're going to change every piece of code that generates an Ahoy event
and, in this case, the easiest way to change the Campaing model so it
doesn't use Ahoy events is to simply remove it.
Note we're keeping the database tables until we release a new version,
just in case some Consul Democracy installations are using them. We'll
inform in the release notes that we'll remove the campaigns table after
the release, so existing installations using the `campaigns` table can
move the data somewhere else before we remove the table.
Using <h3> headings for the links had two disadvantages.
First, it was the wrong heading level to use, since there was no <h2>
tag before it.
Second, headings are supposed to be followed by content associated to
that heading; here, we had no content following the headings.
So we're using a list of links and giving it a heading. We're adding
styles so the page still looks like it used to, although these styles
are certainly asking for improvements.
The JavaScript required to display the chart wasn't loaded on the admin
stats page.
We're not adding a test because we're going to move the budgets graph to
a different page on the pull request containing this commit.
Note we're changing the "Go back" link, since using a turbolinks refresh
broke this link when using the Chromium browser. Besides, there was an
inconsistency where some of the "Go back" links in admin stats pointed
to the admin stats page but other links pointed to `:back`.
We were using Foundation's accordion menu to open/close nested lists of
links. Unfortunately, Foundation's accordion makes it impossible to
access links in nested links using the keyboard [1] (note the issue is
closed, but in the latest version of Foundation, 6.8.1, it's still
present, and Foundation's development is mostly discontinued).
Furtheremore, it adds the `menuitem` role to links, but ARIA menus are
not ment for navigation but for application behavior and, since it
doesn't add the `menubar` or `menu` roles to the parent elements, it
results in accessibility issues for people using screen readers (also
reported by the Axe accessibility testing engine).
So we need to implement our own solution. We're using the most commonly
used pattern: a buttton with the `aria-expanded` attribute. And, for
people using browsers where JavaScript hasn't loaded, we're keeping the
submenus open at all times (just like we were doing until now), and
we're disabling the buttons (since they do nothing without JavaScript).
This might not be an ideal solution, but it's probably good enough, and
way better than what we had until now.
We've also considered using the <details> and <summary> elements instead
of using buttons to open/close items on the list. However, these
elements still present some accessibility issues [2], and the transition
between open and closed can't be animated unless we overwrite the
`click` event with JavaScript. The pattern of using these elements to
open/close a nested list of links isn't common either, and some people
using screen readers might get confused when entering/leaving the nested
list.
We tried other approaches to get the animation effect, all of them based
on adding `[aria-expanded="false"]:not([disabled]) + * { display: none;
}` to the CSS file.
Unfortunately, animation using CSS isn't feasible right now because
browsers can't animate a change form `height: 0` to `height: auto`.
There are some hacks like animating the `max-height` or the `flex-grow`
property, but the resulting animation is inconsistent. A perfect
animation can be done using the `grid-template-rows` property [3], but
it requires adding a grid container and only works in Firefox and recent
versions of Chrome and similar browsers.
Getting to a solution with JavaScript was also tricky. With the
following approach, `slideToggle()` opened the menu the first time, even
if it was already open (not sure why):
```
toggle_buttons.on("click", function() {
$(this).attr("aria-expanded", !JSON.parse($(this).attr("aria-expanded")));
$(this).next().slideToggle();
});
```
This made the arrow turn after the menu had slided instead of doing it
at the same time:
```
toggle_buttons.on("click", function() {
var button = $(this);
button.next().slideToggle(function() {
button.attr("aria-expanded",
!JSON.parse(button.attr("aria-expanded")));
});
}
```
With this, everything disappeared quickly:
```
toggle_buttons.on("click", function() {
var expanded = JSON.parse($(this).attr("aria-expanded"));
if (expanded) {
$(this).next().slideUp();
} else {
$(this).next().slideDown();
}
$(this).attr("aria-expanded", !expanded);
}
```
So, in the end, we're hiding the nested link lists with JavaScript
instead of CSS.
[1] Issue 12046 in https://github.com/foundation/foundation-sites
[2] https://www.scottohara.me/blog/2022/09/12/details-summary.html
[3] https://css-tricks.com/css-grid-can-do-auto-height-transitions
Note that we used to have the link to delete images inside the same
<form> tag as the button to update the image. However, using a button
means we're adding a new <form> tag for the action to delete the image.
This isn't valid HTML and, in some browsers, might result in the button
sending the request to the wrong URL.
As explained in commit 5311daadf, to avoid this, we'd need to replace
`button_to` with `button_tag` in the action in order to generate a
button without a form. Then, we could add either a `form` or a
`formaction` attribute to the button.
However, I thik it's easier to move the delete button outside the update
button <form> tag. On the minus side, since the buttons no longer share
a parent, they're harder to style. So we're using a mix of nested flex
layouts with one of the nested elements using a container unit as width.
Since we're at it, we're also improving the styles on small and medium
screens by making sure the "Update" button wraps before the "Delete"
button does (using a container query), by giving enough width to the
column containing this actions on small screens as well (removing
`small-12` and giving it two-thirds of the width on all screen sizes)
and by having a gap between elements.
Note that, at the time of writing, container queries are only supported
by about 91%-93% of the browsers, meaning that some administrators will
see all from controls displayed vertically, one on top of the other, on
all screen sizes. We think this is acceptable, and the page remains
fully functional in this case.
