We were calling `parse_remote_to_hash` in the Devise initializer, which
runs when the application starts.
That meant that, if we got an exception when calling that method, the
application wouldn't start. We got exceptions if the single sign-on
(SSO) URL isn't available or we aren't providing the right credentials.
So we're moving the call to `parse_remote_to_hash` to
`OmniauthTenantSetup`, which is only called when actually trying to sign
in with SAML.
Since we're moving the code there, we're also unifying the code so SAML
settings are configured the same way for the main tenant and other
tenants, like we did for OpenID Connect in commit c3b523290.
In order to keep the existing behavior, we're caching the result of
`parse_remote_to_hash` in an instance variable. Not sure about the
advantages and disadvantages of doing so over parsing the remote URL
metadata on every SAML-related request.
Note that the SAML tests in `OmniauthTenantSetup` use the `stub_secrets`
method. But this method is called after the application has started,
meaning it doesn't stub calls to `Rails.application.secrets` in
`config/initializers/`. So, before this commit, the code that parsed the
IDP metadata URL wasn't executed in the tests. Since now we've moved the
code but we don't want to depend on external URLs when running the
tests, we need to stub the call to the external URL. Since we're now
stubbing the call, we're adding expectations in the tests to check that
we correctly use the settings returned in that call.
The `issuer` setting was renamed to `sp_entity_id` in omniauth-saml [1],
and it's been deprecated in ruby-saml since version 1.11.0, released on
July 24, 2019 [2].
The ruby-saml code currently uses:
```
def sp_entity_id
@sp_entity_id || @issuer
end
```
So setting `issuer` to the same value as `sp_entity_id` if
`sp_entity_id` is present, as we were doing, has no effect.
On the other hand, neither omniauth-saml nor ruby-saml use the
`idp_metadata_url` and `idp_metadata` settings.
[1] https://github.com/omniauth/omniauth-saml/commit/74ed8dfb3aed
[2] https://github.com/SAML-Toolkits/ruby-saml/releases/tag/v1.11.0
When we first added OIDC support, we were configuring the redirect URI
in the devise initializer, just like we did for other providers.
Thanks to the changes in the previous commit, that code is no longer in
the devise initializer, which means we can use `url_helpers` to get the
redirect URI.
This means we no longer need to define this URI in the secrets. This is
particularly useful for multitenancy; previously, we had to define the
redirect URI for every tenant because different tenants use different
domains or different subdomains.
We were using the `client_options` hash for the default tenant, defined
in the Devise initializer, but we forgot to include that key in the
multitenant code. This means OIDC wasn't working when different tenants
used different configurations.
Adds rake task "polls:remove_duplicate_partial_results" to delete duplicated
rows in "poll_partial_results" made before the DB was strict about duplicates.
Duplicates are considered only for records without "option_id", grouping by:
(question_id, booth_assignment_id, date, answer). We keep the first one and
delete the rest, per tenant.
The controller use:
Poll::PartialResult.find_or_initialize_by(booth_assignment_id, date, question_id, answer)
which is not a strong protection against race conditions. Without a unique
index at the DB level, duplicates could be created. This task cleans up any
existing duplicates.
Similar to what we did in PR "Avoid duplicate records in poll answers" 5539,
specifically in commit 503369166, we want to stop relying on the plain text
"answer" and start using "option_id" to avoid issues with counts across
translations and to add consistency to the poll_partial_results table.
Note that we also moved the `possible_answers` method from Poll::Question to
Poll::Question::Option, since the list of valid answers really comes from the
options of a question and not from the question itself. Tests were updated
to validate answers against the translations of the assigned option.
Additionally, we renamed lambda parameters in validations to improve clarity.
- name: :oidc → Identifier for this login provider in the app.
- scope: [:openid, :email, :profile] → Tells the provider we want the user’s ID (openid), their email, and basic profile info (name, picture, etc.).
- response_type: :code → Uses Authorization Code Flow, which is more secure because tokens are not exposed in the URL.
- issuer: Rails.application.secrets.oidc_issuer → The base URL of the OIDC provider (e.g., Auth0). Used to find its config.
- discovery: true → Automatically fetches the provider’s endpoints from its discovery document instead of manually setting them.
- client_auth_method: :basic → Sends client ID and secret using HTTP Basic Auth when exchanging the code for tokens.
Add system tests for OIDC Auth
Edit the oauth docs to support OIDC auth
Note that Rails 7.1 changes `find_or_create_by!` so it calls
`create_or_find_by!` when no record is found, meaning we'll rarely get
`RecordNotUnique` exceptions when using this method during a race
condition.
Adding this index means we need to remove the uniqueness validation.
According to the `create_or_find_by` documentation [1]:
> Columns with unique database constraints should not have uniqueness
> validations defined, otherwise create will fail due to validation
> errors and find_by will never be called.
We're adding a test that checks what happens when using
`create_or_find_by!`.
Note that we're creating voters combining `create_with` with
`find_or_create_by!`. Using `find_or_create_by!(...)` with all
attributes (including non-key ones like `origin`) fails when a voter
already exists with different values, e.g. an existing `origin: "web"`
and an incoming `"booth"`. In this situation the existing record is not
matched and the unique index raises an exception.
`create_with(...).find_or_create_by!(user: ..., poll: ...)` searches by
the unique key only and applies the extra attributes only on creation.
Existing voters are returned unchanged, which is the intended behavior.
For the `take_votes_from` method, we're handling a (highly unlikely, but
theoretically possible) scenario where a user votes at the same time as
taking voters from another user. For that, we're doing something similar
to what `create_or_find_by!` does: we're updating the `user_id` column
inside a new transaction (using a new transactions avoids a
`PG::InFailedSqlTransaction` exception when there are duplicate
records), and deleting the existing voter when we get a
`RecordNotUnique` exception.
On `Poll::WebVote` we're simply raising an exception when there's
already a user who's voted via booth, because the `Poll::WebVote#update`
method should never be called in this case.
We still need to use `with_lock` in `Poll::WebVote`, but not due to
duplicate voters (`find_or_create_by!` method will now handle the unique
record scenario, even in the case of simultaneous transactions), but
because we use a uniqueness validation in `Poll::Answer`; this
validation would cause an error in simultaneous transactions.
[1] https://api.rubyonrails.org/v7.1/classes/ActiveRecord/Relation.html#method-i-create_or_find_by
We were getting an exception in this case, which was OK I guess since
this shouldn't happen if the application is used in a normal way, but we
can simplify the code a little bit if we make the `recipients` code
return an empty list of users.
Note that the behavior of the `AdminNotification#list_of_recipients` and
`Newsletter#list_of_recipient_emails` methods is now slightly different;
previously they returned `nil` when given an invalid segment recipient,
while now they return an empty array. I haven't found a place where this
change is relevant. For example, in both of these models, the `deliver`
method used to raise an exception when given an invalid segment while
now it doesn't, but we always check the user segment is valid before
calling the `deliver` method anyway, so it doesn't really affect the
application.
We were getting a warning by CodeQL regarding a possible code injection
in the `send(segment)` code.
In practice, this wasn't a big deal because the `self.recipients` method
is only called in the admin section, meaning only admin users could try
to take advantage of the code injection, and because this code is rarely
called with an invalid segment due to several conditions in the code
checking that the user segment is valid, with the only exception being
the `generate_csv` action in the `Admin::EmailsDownloadController`.
In any case, now we're checking that the segment is valid before calling
the `send` method. Since now we're making sure that the segment is valid
in the `recipients` method itself, we can remove this check from methods
calling it.
Note that, while we're no longer including them as part of the
`execute_release_2.2.0_tasks` task, we're keeping the tasks to remove
duplicate poll voters and poll options just in case there are some
unexpected issues when adding a unique database index while upgrading to
version 2.3.0. We'll remove them in version 2.4.0.
There are many possible ways to implement this feature:
* Adding a custom middleware
* Using rack-attack with a blocklist
* Using routes constraints
We're choosing to use a controller concern with a redirect because it's
what we do to handle unauthorized cancancan exceptions.
Back in commit c984e666f, we reorganized the code related to the GraphQL
API, but we didn't reorganize the tests.
So we're doing it now, since we're going to fix a potential issue and
add some tests for it.
These tests were always passing because they were stubbing the response
of the same method they were testing. For example, we were testing the
result of `Comment.public_for_api` and stubbing it at the same time.
So we're now stubbing the result of the associations; for example, in
order to test `Comment.public_for_api`, we're stubbing the response of
`Debate.public_for_api`. Now the tests fail if, for instance, the
implementation of `Comment.public_for_api` returns all comments.
- added 2 new types
- modified the models to get data through graphQL
- modified the corresponding spec
- also testing that hidden comments do not show up
- modified comments specs bc now it returns comments on budget
investments
Note: to avoid confusion, "answer" will mean a row in the poll_answers
table and "choice" will mean whatever is in the "answer" column of that
table (I'm applying the same convention in the code of the task).
In order make this task perform reasonably on installations with
millions of votes, we're using `update_all` to update all the answers
with the same choice at once. In order to do that, we first need to
check the existing choices and what are the possible option_ids for
those choices.
Note that, in order for this task to work, we need to remote the
duplicate answers first. Otherwise, we will run into a RecordNotUnique
exception when trying to add the same option_id to two duplicate
answers.
So we're making this task depend on the one that removes duplicate
answers. That means we no longer need to specify the task to remove
duplicate answers in the release tasks; it will automatically be
executed when running the task to add an option_id.
Note that, since poll answers belong to a user and not to a voter, we
aren't doing anything regarding poll answers. This is a separate topic
that might be dealt with in a separate pull request.
Also note that, since there are no records belonging to poll voters, and
poll voters don't use `acts_as_paranoia` and don't have any callbacks on
destroy, it doesn't really matter whether we call `destroy!` or
`delete`. We're using `delete` so there are no unintended side-effects
that might affect voters with the same `user_id` and `poll_id` on
Consul Democracy installations customizing this behavior.
We use caching in our application in two different ways:
1. Rails.cache.fetch in models/controllers/libraries
2. Fragment caching in the views
When using Rails.cache.fetch, we usually set an expiration date or
provide a precise way to invalidate it. If the code changes and the
information stored in the cache is different from what the new code
would return, it's usually not a big deal because the cache will expire
in an hour or a day. Until commit a4461a1a5, the statistics were an
exception to this rule, but that's no longer the case. The actual
exception to this rule are the i18n translations, but the code caching
them is very simple and hasn't changed for more than three years (when
it was written for the first time).
When using fragment caching, on the other hand, Rails automatically
invalidates the cache when the associated _view code_ changes. That is,
if a view contains cache, the view renders a partial, and the partial
changes, the cache is correctly invalidated. The cache isn't invalidated
when the code in helpers, models or controllers change, though, which
the Rails team considers a compromise solution.
However, we've been moving view partials (and even views) to components,
and the cache isn't invalidated when a component changes (it doesn't
matter whether the component Ruby file or the component ERB file
changes). That means that the cache will keep rendering the HTML
generated by the old code.
So, now, we're clearing the cache when upgrading to a new version of
Consul Democracy, as part of the release tasks. That way, institutions
upgrading to a new version don't have to worry about possible issues
with the cache due to the new code not being executed.
I was thinking of adding it to a Capistrano task, but that would mean
that every time people deploy new code, even if it's a hotfix that
doesn't affect the cache at all, the cache would be cleared, which could
be inconvenient. Doing it in a release, that usually changes thousands
of lines of code, sounds like a good compromise.
Note that, for everything to work consistently, we need to make sure
that the default locale is one of the available locales.
Also note that we aren't overwriting the `#save ` method set by
globalize. I didn't feel too comfortable changing a monkey-patch which
ideally shouldn't be there in the first place, I haven't found a case
where `Globalize.locale` is `nil` (since it defaults to `I18n.locale`,
which should never be `nil`), so using `I18n.default_locale` probably
doesn't affect us.
We were defining one builder in the `app/lib/` folder and another one
inside a helper module.
So now we're grouping them together. This way we're following the "one
class per file" convention that we follow most of the time. And, by
extracting the `TranslatableFormBuilder` class to its own file, it'll be
easier to add tests for it.
Note that, for consistency, we're renaming the
`TranslationsFieldsBuilder` class so it ends in `FormBuilder`.
Back in commit 383909e16, we said:
> Even if this class looks very simple now, we're trying a few things
> related to these stats. Having a class for it makes future changes
> easier and, if there weren't any future changes, at least it makes
> current experiments easier.
Since there haven't been any changes in the last 5 years and we've found
cases where using the GeozoneStats class results in a slightly worse
performance, we're removing this class. The code is now a bit easier to
read, and is consistent with the way we calculate participants by age.
This reverts commit 383909e16.
According to the README [1]:
> To mask previously collected IPs, use:
> Ahoy::Visit.find_each do |visit|
> visit.update_column :ip, Ahoy.mask_ip(visit.ip)
> end
We're adapting the code with our version, since we use the `Visit` model
instead of the `Ahoy::Visit` model.
[1] https://github.com/ankane/ahoy/blob/v5.0.2/README.md#ip-masking
We moved this file to `app/lib/` in commit cb477149c so it would be in
the autoload_paths. However, this class is loaded by ActiveStorage, with
the following method:
```
def resolve(class_name)
require "active_storage/service/#{class_name.to_s.underscore}_service"
ActiveStorage::Service.const_get(:"#{class_name.camelize}Service")
rescue LoadError
raise "Missing service adapter for #{class_name.inspect}"
end
``
So this file needs to be in the $LOAD_PATH, or else ActiveStorage won't
be able to load it when we disable the `add_autoload_paths_to_load_path`
option, which is the default in Rails 7.1 [1].
Moving it to the `lib` folder solves the issue; as mentioned in the
guide to upgrade to Rails 7.1 [2]:
> The lib directory is not affected by this flag, it is added to
> $LOAD_PATH always.
However, we were also referencing this class in the `Tenant` model,
meaning we needed to autoload it as well somehow. So, instead of
directly referencing this class, we're using `respond_to?` in the Tenant
model.
We're changing the test so it fails when the code calls
`is_a?(ActiveStorage::Service::TenantDiskService)`. We need to change
the active storage configurations in the test because, otherwise, the
moment `ActiveStorage::Blob` is loaded, the `TenantDiskService` class is
also loaded, meaning the test will pass when using `is_a?`.
Note that, since this class isn't in the autoload paths anymore, we need
to add a `require` in the tests. We could add an initializer to require
it; we're not doing it in order to be consistent with what ActiveStorage
does: it only loads the service that's going to be used in the current
Rails environment. If somebody changed their production environment in
order to use (for example), S3, and we added an initializer to require
the TenantDiskService, we would still load the TenantDiskService even if
it isn't going to be used.
[1] https://guides.rubyonrails.org/v7.1/configuring.html#config-add-autoload-paths-to-load-path
[2] https://guides.rubyonrails.org/v7.1/upgrading_ruby_on_rails.html#autoloaded-paths-are-no-longer-in-$load-path
This way we simplify the code a little bit and we create a method unique
to the `TenantDiskService` class, which can be used to check whether
we're using this class without using `is_a?` or similar.
We used to open these links in new tabs, but accidentally stopped doing
so in commit 75a28fafc.
While, in general, automatically opening a link in a new tab/window is a
bad idea, the exception comes when people are filling in a form and
there are links to pages that contain information which will help them
fill in a form.
There are mainly two advantages of this approach. First, it makes less
likely for people to accidentally lose the information they were filling
in. And, second, having both the form and a help page open at the same
time can make it easier to fill in the form.
However, opening these links in new tabs also has disadvantages, like
taking control away from people or making it harder to navigate through
pages when using a mobile phone.
So this is a compromise solution.
This rule was added in rubocop-rspec 2.9.0.
We were using `be_nil` 50% of the time, and `be nil` the rest of the
time. No strong preference for either one, but IMHO we don't lose
anything be being consistent.
Note we're excluding a few files:
* Configuration files that weren't generated by us
* Migration files that weren't generated by us
* The Gemfile, since it includes an important comment that must be on
the same line as the gem declaration
* The Budget::Stats class, since the heading statistics are a mess and
having shorter lines would require a lot of refactoring
For the HashAlignment rule, we're using the default `key` style (keys
are aligned and values aren't) instead of the `table` style (both keys
and values are aligned) because, even if we used both in the
application, we used the `key` style a lot more. Furthermore, the
`table` style looks strange in places where there are both very long and
very short keys and sometimes we weren't even consistent with the
`table` style, aligning some keys without aligning other keys.
Ideally we could align hashes to "either key or table", so developers
can decide whether keeping the symmetry of the code is worth it in a
case-per-case basis, but Rubocop doesn't allow this option.
We were already applying these rules in most cases.
Note we aren't enabling the `MultilineArrayLineBreaks` rule because
we've got places with many elements whire it isn't clear whether
having one element per line would make the code more readable.
Since IRB has improved its support for multiline, the main argument
towars using a trailing dot no longer affects most people.
It still affects me, though, since I use Pry :), but I agree
leading dots are more readable, so I'm enabling the rule anyway.
* Add Tables option to Redcarpet in Legislation draft
* Allow table tags in Admin Legislation Sanitizer
* Add Test to render markdown tables in Legislation drafts
* Add Test for Admin Legislation Sanitizer
We include test for image, table and h1 to h6 tags and additional tests to strengthen the allowed and disallowed parameters
* Add Table from markdown test in System and Factories
* Add test to render tables for admin user
* Remove comment line about Redcarpet options
* Edit custom css for legislation draft table to make it responsive
It has been detected that for the :pt-BR, :zh-CN and :zh-TW locales,
the translate button was being displayed, but when requesting the
translation, the remote translation validation failed due to:
'''
validates :locale, inclusion: { in: ->(_) {
RemoteTranslations::Microsoft::AvailableLocales.available_locales }}
'''
That available_locales method did not contemplate these 3 languages
in the format used by the application.
To solve this problem the api response is mapped to return all
locales in the format expected by the application.
Add remote translation model test to ensure that a remote translation
is valid when its locale is pt-BR.
Co-Authored-By: Javi Martín <35156+javierm@users.noreply.github.com>
