Using `>` and `<` for dates is a bit confusing because it usually
requires mentally translating `<` into "before" and `>` into "after",
meaning it takes a few seconds to check which is the starting date and
which is the ending date. When using a range, it's easier to see which
is which.
Note that we were using `>` and `<`, but now we're using the equivalent
of `>=` and `<=`. This makes sense IMHO, since the beginning of the year
and the end of the year should also be included in this case.
We were calculating the age stats based on the age of the users who
participated... at the moment where we were calculating the stats. That
means that, if 20 years ago, 1000 people who were 16 years old
participated, they would be shown as having 36 years in the stats.
Instead, we want to show the stats at the time when the process took
place, so we're implementing a `participation_date` method.
Note that, for polls, we could actually use the `age` column in the
`poll_voters` table. However, doing so would be harder, would only work
for polls but not for budgets, and it wouldn't be statistically very
relevant, since the stats are shown by age groups, and only a small
percentage of people would change their age group (and only to the
nearest one) between the time they participate and the time the process
ends.
We might use the `poll_voters` table in the future, though, since we
have a similar issue with geozones and genders, and using the
information in `poll_voters` would solve it as well (only for polls,
though).
Also note that we're using the `ends_at` dates because some people but
be too young to vote when a process starts but old enough to vote when
the process ends.
Finally, note that we might need to change the way we calculate the
participation date for a budget, since some budgets might not enabled
every phase. Not sure how stats work in that scenario (even before these
changes).
This was added in commit 02f19aa4b, before we started tracking events.
I don't think we ever used it; in any case, we now use the `Ahoy::Chart`
class to deal with the stats Ahoy used to generate.
This page isn't linked from anywhere and most Consul Democracy
installations don't even know it exists, so it's useless for most
people.
If we ever bring it back, we should at least add a link pointing to this
page.
Now that we've moved the logic to generate the events data to a model,
and we've got access to the model in the component rendering the chart,
we can render the data inside the chart instead of doing an extra AJAX
request to get the same data.
Originally, this was problaby done this way so the page wouldn't take
several seconds to load while preparing the data for the chart when
there are thousands of dates being displayed. With an AJAX call, the
page would load as fast as usual, and then the chart would render after
a few seconds. However, we can have an even better performance
improvement in this scenario if we use a Set instead of an Array. The
method `Array#include?`, which we were calling for every date in the
data, is much slower that `Set#merge`. So now both the page and the
chart load as fast as expected.
We could also use something like:
```
def add
(...)
shared_keys.push(*collection.keys)
end
def build
(...)
shared_keys.uniq.each do |k|
(...)
end
def shared_keys
@shared_keys ||= []
end
```
Or other approaches to avoid using `Array#include?`. The performance
would be similar to the one we get when using `Set`. We're using a `Set`
because it makes more obvious that `shared_keys` is supposed to contain
unique elements.
We've had some tests failing in the past due to these AJAX requests
being triggered automatically during the tests and no expectations
checking the requests have finished, so now we're reducing the amount of
flaky tests.
We were always displaying the event names in English.
Note we're changing the `user_supported_budgets` key because it didn't
make much sense; the investments are supported, and not the budgets.
We're also adding "created" to most of the event names in order to make
the texts more explicit, since not all the events refer to created data.
Note we're delegating the `t` method because i18n-tasks doesn't detect
code like `ApplicationController.helpers.t` and so reports we aren't
using the `admin.stats.graph` translations.
We were tracking some events with Ahoy, but in an inconsistent way. For
example, we were tracking when a debate was created, but (probably
accidentally) we were only tracking proposals when they were created
from the management section. For budget investments and their supports,
we weren't using Ahoy events but checking their database tables instead.
And we were only using ahoy events for the charts; for the other stats,
we were using the real data.
While we could actually fix these issues and start tracking events
correctly, existing production data would remain broken because we
didn't track a certain event when it happened. And, besides, why should
we bother, for instance, to track when a debate is created, when we can
instead access that information in the debates table?
There are probably some features related to tracking an event and their
visits, but we weren't using them, and we were storing more user data
than we needed to.
So we're removing the track events, allowing us to simplify the code and
make it more consistent. We aren't removing the `ahoy_events` table in
case existing Consul Democracy installations use it, but we'll remove it
after releasing version 2.2.0 and adding a warning in the release notes.
This change fixes the proposal created chart, since we were only
tracking proposals created in the management section, and opens the
possibility to add more charts in the future using data we didn't track
with Ahoy.
Also note the "Level 2 user Graph" test wasn't testing the graph, so
we're changing it in order to test it. We're also moving it next to the
other graphs test and, since we were tracking the event when we were
confirming the phone, we're renaming to "Level 3 users".
Finally, note that, since we were tracking events when something was
created, we're including the `with_hidden` scope. This is also
consistent with the other stats shown in the admin section as well as
the public stats.
The only way to use campaigns is to manually insert them in the
database, which IMHO isn't very practical.
We're going to change every piece of code that generates an Ahoy event
and, in this case, the easiest way to change the Campaing model so it
doesn't use Ahoy events is to simply remove it.
Note we're keeping the database tables until we release a new version,
just in case some Consul Democracy installations are using them. We'll
inform in the release notes that we'll remove the campaigns table after
the release, so existing installations using the `campaigns` table can
move the data somewhere else before we remove the table.
Using <h3> headings for the links had two disadvantages.
First, it was the wrong heading level to use, since there was no <h2>
tag before it.
Second, headings are supposed to be followed by content associated to
that heading; here, we had no content following the headings.
So we're using a list of links and giving it a heading. We're adding
styles so the page still looks like it used to, although these styles
are certainly asking for improvements.
The JavaScript required to display the chart wasn't loaded on the admin
stats page.
We're not adding a test because we're going to move the budgets graph to
a different page on the pull request containing this commit.
Note we're changing the "Go back" link, since using a turbolinks refresh
broke this link when using the Chromium browser. Besides, there was an
inconsistency where some of the "Go back" links in admin stats pointed
to the admin stats page but other links pointed to `:back`.
We were only using it in one place: the debates controller. All the
other controllers including CommentableActions were overwriting this
action, except the ones in the admin area, where creating proposals,
debates or investments isn't possible.
Note this means that, most of the times, we weren't tracking events
creating a resource.
Also note that since, as mentioned in commit 3752fef6b, there are no
geozones in the debates form, we don't have to load them when creating a
debate fails due to validation rules.
We were using Foundation's accordion menu to open/close nested lists of
links. Unfortunately, Foundation's accordion makes it impossible to
access links in nested links using the keyboard [1] (note the issue is
closed, but in the latest version of Foundation, 6.8.1, it's still
present, and Foundation's development is mostly discontinued).
Furtheremore, it adds the `menuitem` role to links, but ARIA menus are
not ment for navigation but for application behavior and, since it
doesn't add the `menubar` or `menu` roles to the parent elements, it
results in accessibility issues for people using screen readers (also
reported by the Axe accessibility testing engine).
So we need to implement our own solution. We're using the most commonly
used pattern: a buttton with the `aria-expanded` attribute. And, for
people using browsers where JavaScript hasn't loaded, we're keeping the
submenus open at all times (just like we were doing until now), and
we're disabling the buttons (since they do nothing without JavaScript).
This might not be an ideal solution, but it's probably good enough, and
way better than what we had until now.
We've also considered using the <details> and <summary> elements instead
of using buttons to open/close items on the list. However, these
elements still present some accessibility issues [2], and the transition
between open and closed can't be animated unless we overwrite the
`click` event with JavaScript. The pattern of using these elements to
open/close a nested list of links isn't common either, and some people
using screen readers might get confused when entering/leaving the nested
list.
We tried other approaches to get the animation effect, all of them based
on adding `[aria-expanded="false"]:not([disabled]) + * { display: none;
}` to the CSS file.
Unfortunately, animation using CSS isn't feasible right now because
browsers can't animate a change form `height: 0` to `height: auto`.
There are some hacks like animating the `max-height` or the `flex-grow`
property, but the resulting animation is inconsistent. A perfect
animation can be done using the `grid-template-rows` property [3], but
it requires adding a grid container and only works in Firefox and recent
versions of Chrome and similar browsers.
Getting to a solution with JavaScript was also tricky. With the
following approach, `slideToggle()` opened the menu the first time, even
if it was already open (not sure why):
```
toggle_buttons.on("click", function() {
$(this).attr("aria-expanded", !JSON.parse($(this).attr("aria-expanded")));
$(this).next().slideToggle();
});
```
This made the arrow turn after the menu had slided instead of doing it
at the same time:
```
toggle_buttons.on("click", function() {
var button = $(this);
button.next().slideToggle(function() {
button.attr("aria-expanded",
!JSON.parse(button.attr("aria-expanded")));
});
}
```
With this, everything disappeared quickly:
```
toggle_buttons.on("click", function() {
var expanded = JSON.parse($(this).attr("aria-expanded"));
if (expanded) {
$(this).next().slideUp();
} else {
$(this).next().slideDown();
}
$(this).attr("aria-expanded", !expanded);
}
```
So, in the end, we're hiding the nested link lists with JavaScript
instead of CSS.
[1] Issue 12046 in https://github.com/foundation/foundation-sites
[2] https://www.scottohara.me/blog/2022/09/12/details-summary.html
[3] https://css-tricks.com/css-grid-can-do-auto-height-transitions
Note that we used to have the link to delete images inside the same
<form> tag as the button to update the image. However, using a button
means we're adding a new <form> tag for the action to delete the image.
This isn't valid HTML and, in some browsers, might result in the button
sending the request to the wrong URL.
As explained in commit 5311daadf, to avoid this, we'd need to replace
`button_to` with `button_tag` in the action in order to generate a
button without a form. Then, we could add either a `form` or a
`formaction` attribute to the button.
However, I thik it's easier to move the delete button outside the update
button <form> tag. On the minus side, since the buttons no longer share
a parent, they're harder to style. So we're using a mix of nested flex
layouts with one of the nested elements using a container unit as width.
Since we're at it, we're also improving the styles on small and medium
screens by making sure the "Update" button wraps before the "Delete"
button does (using a container query), by giving enough width to the
column containing this actions on small screens as well (removing
`small-12` and giving it two-thirds of the width on all screen sizes)
and by having a gap between elements.
Note that, at the time of writing, container queries are only supported
by about 91%-93% of the browsers, meaning that some administrators will
see all from controls displayed vertically, one on top of the other, on
all screen sizes. We think this is acceptable, and the page remains
fully functional in this case.
As mentioned in commits 5311daadf and bb958daf0, using links combined
with JavaScript to generate POST (or, in this case, PUT) requests to the
server has a few issues.
We were already using buttons to destroy pages from the pages index.
As mentioned in commits 5311daadf and bb958daf0, using links combined
with JavaScript to generate POST (or, in this case, DELETE) requests to
the server has a few issues.
This way we can simplify setting the title and styling the link in the
header. We're also fixing the unnecessary padding introduced by the
`column` classes, which caused the header not to be aligned with the
rest of the elements surrounding it. We're still keeping it the margin
used in the `row` classes so it's aligned with the rest of the form;
ideally, we would remove the `row` classes in the rest of the form and
in the whole admin section, but this isn't something we can tackle right
now.
Note that, in the CSS, the `margin-left: auto` property needs to be
included after `@include regular-button` because that mixin overwrites
the `margin-left` property. Since we're modifying this code, we're
making it compatible with RTL text, using `$global-left` instead of
`left`.
We were already doing that when deleting content blocks from the index
page, and we also ask for confirmation in almost every page in the admin
section.
We were already using button to destroy content blocks from the content
blocks index.
As mentioned in commits 5311daadf and bb958daf0, using links combined
with JavaScript to generate POST (or, in this case, DELETE) requests to
the server has a few issues.
As mentioned in commits 5311daadf and bb958daf0, using links combined
with JavaScript to generate POST (or, in this case, DELETE) requests to
the server has a few issues.
Note that the AJAX response stopped working after replacing the link
with a button. Not sure about the reason, but, since this is one of the
very few places where we use AJAX calls to delete content, the easiest
solution is to stop using AJAX and be consistent with what we do in the
rest of the admin section.
As mentioned in commits 5311daadf and bb958daf0, using links combined
with JavaScript to generate POST (or, in this case, DELETE) requests to
the server has a few issues.
As mentioned in commits 5311daadf and bb958daf0, using links combined
with JavaScript to generate POST (or, in this case, DELETE) requests to
the server has a few issues.
Using the standard `confirm` parameter, we can remove all the custom
code we added to do the same thing.
Since the code is similar, we're doing the same when asking for
confirmation to send notifications.
We moved this file to `app/lib/` in commit cb477149c so it would be in
the autoload_paths. However, this class is loaded by ActiveStorage, with
the following method:
```
def resolve(class_name)
require "active_storage/service/#{class_name.to_s.underscore}_service"
ActiveStorage::Service.const_get(:"#{class_name.camelize}Service")
rescue LoadError
raise "Missing service adapter for #{class_name.inspect}"
end
``
So this file needs to be in the $LOAD_PATH, or else ActiveStorage won't
be able to load it when we disable the `add_autoload_paths_to_load_path`
option, which is the default in Rails 7.1 [1].
Moving it to the `lib` folder solves the issue; as mentioned in the
guide to upgrade to Rails 7.1 [2]:
> The lib directory is not affected by this flag, it is added to
> $LOAD_PATH always.
However, we were also referencing this class in the `Tenant` model,
meaning we needed to autoload it as well somehow. So, instead of
directly referencing this class, we're using `respond_to?` in the Tenant
model.
We're changing the test so it fails when the code calls
`is_a?(ActiveStorage::Service::TenantDiskService)`. We need to change
the active storage configurations in the test because, otherwise, the
moment `ActiveStorage::Blob` is loaded, the `TenantDiskService` class is
also loaded, meaning the test will pass when using `is_a?`.
Note that, since this class isn't in the autoload paths anymore, we need
to add a `require` in the tests. We could add an initializer to require
it; we're not doing it in order to be consistent with what ActiveStorage
does: it only loads the service that's going to be used in the current
Rails environment. If somebody changed their production environment in
order to use (for example), S3, and we added an initializer to require
the TenantDiskService, we would still load the TenantDiskService even if
it isn't going to be used.
[1] https://guides.rubyonrails.org/v7.1/configuring.html#config-add-autoload-paths-to-load-path
[2] https://guides.rubyonrails.org/v7.1/upgrading_ruby_on_rails.html#autoloaded-paths-are-no-longer-in-$load-path
This way we simplify the code a little bit and we create a method unique
to the `TenantDiskService` class, which can be used to check whether
we're using this class without using `is_a?` or similar.
This way we'll add an extra layer of protection from attacks that might
cause our application to redirect to an external host.
There's one place where we're allowing redirects to external hosts,
though: administrators can link external resources in notifications, and
we're redirecting to them after marking the notification as read.
Since the tests for the remote translations controller were
(accidentally) using an external redirect, we're updating them to use a
relative URL.
In Rails 6.1 and earlier, `button_to` generated a <button> tag when it
received the content as a block, but an <input> tag when receiving
the content as the first parameter.
That's why we were using blocks with `button_to` most of the time; for
starters, <button> tags accept pseudocontent and so are easier to style.
In Rails 7.0, `button_to` always generates a <button> tag [1], so we're
simplifying the code what uses `button_to`, passing the content as a
first parameter instead of passing it as a block.
[1] https://guides.rubyonrails.org/v7.1/configuring.html#config-action-view-button-to-generates-button-tag
The config.file_watcher option still exists but it's no longer included
in the default environtment file. Since we don't use it, we're removing
it.
The config.assets.assets.debug option is no longer true by default [1],
so it isn't included anymore.
The config.active_support.deprecation option is now omitted on
production in favor of config.active_support.report_deprecations, which
is false by default. I think it's OK to keep it this way, since we check
deprecations in the development and test environments but never on
production environments.
As mentioned in the Rails upgrade guide, sprockets-rails is no longer a
rails dependency and we need to explicitly include it in our Gemfile.
The behavior of queries trying to find an invalid enum value has changed
[2], so we're updating the tests accordingly.
The `favicon_link_tag` method has removed the deprecated `shortcut`
link type [3], so we're updating the tests accordingly.
The method `raw_filter` in ActiveSupport callbacks has been renamed to
`filter` [4], so we're updating the code accordingly.
[1] https://github.com/rails/rails/commit/adec7e7ba87e3
[2] https://github.com/rails/rails/commit/b68f0954
[3] Pull request 43850 in https://github.com/rails/rails
[4] Pull request 41598 in https://github.com/rails/rails
While using `require_dependency` to load original Consul Democracy code
from custom code works with the classic autoloader, this method was
never meant to be used this way. With zeitwerk, the code (apparently)
works in the test, development and production environments, but there's
one important gotcha: changing any `.rb` file in development will
require restarting the rails server since the application will crash
when reloading.
Quoting zeitwerk's author Xavier Noria, whom we've contacted while
looking for a solution:
> With the classic autoloader, when the Setting constant is autoloaded,
> the autoloader searched the autoload paths, found setting.rb in
> app/models/custom, and loaded it. With zeitwerk, the autoloader scans
> the folders in order and defines an autoload (Module#autoload) in
> Object so Ruby autoloads Setting with app/models/custom/settings.rb.
> Later, when app/models/setting.rb is found, it's ignored since there's
> already an autoload for Setting.
>
> That means the first file is managed by the autoloaders, while the
> second is not.
>
> So require_dependency worked, but it was pure luck, since the purpose
> of require_dependency is forcing the load of files managed by the
> autoloaders and, as we've seen, app/models/settings.rb isn't one of
> them.
>
> With your current pattern for custom files, the best solution is using
> Kernel#load.
So we're using `load` instead of `require_dependency`. Note that, with
`load`, we need to add the `.rb` extension to the required file, and we
no longer have to convert the Pathname to a string with `to_s`.
