The `parameterize` method uses the `I18n.transliterate` method, whose
documentation says:
```
I18n.transliterate("Ærøskøbing")
=> "AEroskobing"
I18n.transliterate("日本語")
=> "???"
```
That means we can't use it for dictionaries where characters don't have
a transliteration to the latin alphabet.
So we're changing the code in order to only transliterate characters
with a transliteration to the latin alphabet.
Note the first example ("Česká republika") already worked with the
previous code; the test has been added to make sure accented characters
are handled properly.
This way we don't have to use the `send` method in other places, like
the AdminNotification class, and we can change the internal
implementation at any point.
We're going to add geozones as user segments, so it's handy to have the
method in the UserSegments class.
We're also changing the `user_segment_emails` parameter name for
consistency and simplicity.
We're going to make it dynamic using the geozones. Besides, class
methods can be overwritten using custom models, while constants can't be
overwritten without getting a warning [1].
Makes the definition of segments with geozones a little cleaner. I
think it’s worth it, compared to the slight memory gain of using a
constant [2].
[1] warning: already initialized constant UserSegments::SEGMENTS
[2] https://stackoverflow.com/questions/15903835/class-method-vs-constant-in-ruby-rails#answer-15903970
The `open` method can be used to open files or URLs and it's deprecated
in Ruby 2.7. In this case, it's clear we're dealing with a URL, so we
can use `URI.parse`.
The code was a bit strange, since it returned a value and had a side
effect: opening the URL. I'm not sure about the intention of the code;
my best guess is we wanted to test the URL exists and was accessible
before returning it (and, if that's the case, IMHO the code should be a
bit more explicit in order to show the intention behind it), but it
could also be an unintended side effect which was there by accident.
Now the URL is no longer opened; if the URL isn't accessible, we'll find
out when trying to connect to it with the Savon client.
In commit 5a4921a1a we replaced `URI.parse` with `URI.open` due to some
issues during our tests with S3.
However, there are some security issues with `URI.open` [1], since it
might allow some users to execute code on the server.
So we're using `URI.parse#open` instead.
[1] https://docs.rubocop.org/rubocop/cops_security.html#securityopen
This column wasn't used in any released Consul version since it was only
used during development. For the same reason, the task to migrate the
information in the `link` column to the `links` table isn't needed
either.
It was never used and its data is based on Madrid's requirements for
successful proposals. It was written during the development of the
dashboard, probably for testing purposes.
There could be inconsistencies in the database and an attachment might
have a `record_id` pointing to a record which no longer exist. We were
getting an exception in this case.
We were getting a duplicate prepared statement error when trying to
execute more than one test using this task. So we're checking whether
the prepared statement exists before defining it.
Just like we add the `storage_` prefix for new records so we can use
both Active Storage and Paperclip at the same time.
Now the migration actually works, at least for basic cases.
This model is added by the devise-security gem, but we don't use it. We
were getting an error while migrating:
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "old_passwords" does not exist
LINE 8: WHERE a.attrelid = '"old_passwords"'::regclas...
^
SELECT a.attname, format_type(a.atttypid, a.atttypmod),
pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
c.collname, col_description(a.attrelid, a.attnum) AS comment
FROM pg_attribute a
LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
LEFT JOIN pg_type t ON a.atttypid = t.oid
LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
WHERE a.attrelid = '"old_passwords"'::regclass
AND a.attnum > 0 AND NOT a.attisdropped
ORDER BY a.attnum
lib/tasks/active_storage.rake:27:in `block (4 levels) in <top (required)>'
lib/tasks/active_storage.rake:26:in `each'
lib/tasks/active_storage.rake:26:in `block (3 levels) in <top (required)>'
lib/tasks/active_storage.rake:25:in `block (2 levels) in <top (required)>'
Caused by:
PG::UndefinedTable: ERROR: relation "old_passwords" does not exist
We can use the `ActiveStorage::Blob` class to find where the file is
supposed to be stored instead of manually setting the path. This is also
more robust because it works with Active Storage configurations which
don't store files in the default folder.
In order to migrate existing files from Paperclip to ActiveStorage, we
need Paperclip to find out the files associated to existing database
records. So we can't simply replace Paperclip with ActiveStorage.
That's why it's usually recommended [1] to first run the migration and
then replace Paperclip with ActiveStorage using two consecutive
deployments.
However, in our case we can't rely on two consecutive deployments
because we have to make an easy process so existing CONSUL installations
don't run into any issues. We can't just release version 1.4.0 and 1.5.0
and day and ask everyone to upgrade twice on the same day.
Instead, we're following a different plan:
* We're going to provide a Rake task (which will require Paperclip) to
migrate existing files
* We still use Paperclip to generate link and image tags
* New files are handled using both Paperclip and ActiveStorage; that
way, when we make the switch, we won't have to migrate them, and in
the meantime they'll be accessible thanks to Paperclip
* After we make the switch, we'll update the `name` column in the active
storage attachments tables in order to remove the `storage_` prefix
Regarding our handling of new files, the exception are cached
attachments. Since those attachments are temporary files used while
submitting a form and we have to delete them afterwards, we're only
handling them with Paperclip. We'll handle these ones in version 1.5.0.
Note the task creating the dev seeds was failing after these changes
with an `ActiveStorage::IntegrityError` exception because we were
opening some files without closing them. If the same file was attached
twice, it failed the second time.
We're solving it by closing the files with `File.open` and a block. Even
though we didn't get any errors, we're doing the same thing in the
`Attachable` concern because it's a good practice to close files after
we're done with them.
Also note we have to change the CKEditor Active Storage code so it's
compatible with Paperclip. In this case, I haven't been able to write a
test to confirm the attachment exists; I was getting the same
`ActiveStorage::IntegrityError` mentioned above.
Finally, we're updating the site customization image controller to use
`update` so the image and the attachment are updated within the same
transaction. This is also what we do in most controllers.
[1] https://www.youtube.com/watch?v=tZ_WNUytO9o
ActiveStorage support was added to CKEditor in version 5.1.0. However,
we can't upgrade to version 5.0.0 or later because it only supports
loading CKEditor via CDN.
So we're copying the code related to ActiveStorage instead. I tried to
move it to the `vendor/` folder, but couldn't find a way to load it from
there and if I found one I wouldn't be sure it'd work on production
environments.
We're not adding the rule because it would apply the current line length
rule of 110 characters per line. We still haven't decided whether we'll
keep that rule or make lines shorter so they're easier to read,
particularly when vertically splitting the editor window.
So, for now, I'm applying the rule to lines which are about 90
characters long.
Since version 2.0 introduced many breaking changes, we're upgrading to
it first.
The changes have been done by installing the rubocop-faker gem and
running:
```
rubocop \
--require rubocop-faker \
--only Faker/DeprecatedArguments \
--auto-correct
```
Our previous system to delete cached attachments didn't work for
documents because the `custom_hash_data` is different for files created
from a file and files created from cached attachments.
When creating a document attachment, the name of the file is taken into
account to calculate the hash. Let's say the original file name is
"logo.pdf", and the generated hash is "123456". The cached attachment
will be "123456.pdf", so the generated hash using the cached attachment
will be something different, like "28af3". So the file that will be
removed will be "28af3.pdf", and not "123456.pdf", which will still be
present.
Furthermore, there are times where users choose a file and then they
close the browser or go to a different page. In those cases, we weren't
deleting the cached attachments either.
So we're adding a rake task to delete these files once a day. This way
we can simplify the logic we were using to destroy cached attachments.
Note there's related a bug in documents: when editing a record (for
example, a proposal), if the title of the document changes, its hash
changes, and so it will be impossible to generate a link to that
document. Changing the way this hash is generated is not an option
because it would break links to existing files. We'll try to fix it when
moving to Active Storage.
People who have already upgraded to version 1.3.0 don't need to execute
them again.
We're not deleting the tasks yet in case some people would like to
upgrade from version 1.2.0 to version 1.3.1. In this case, they'll have
to execute the tasks manually.
Note we're making the validation rule dynamic so it's affected by the
way we stub the constant in the tests to emulate data created in old
applications.
Co-Authored-By: Javi Martín <javim@elretirao.net>
By using real XML responses developers will be able to understand better
how the integration works (the data flow), and the correspondency between
`remote_census` settings and their place at a real XML response.
As `stubbed_responses` methods were removed from the model layer now the
stubbing part should be managed from the test environment code so also
added a new helper module `RemoteCensusSetup` that can be used anywhere
where we need to call the web service.
Co-Authored-By: Javi Martín <javim@elretirao.net>
By simplyfing the responses the configuration for specs can be simpler too.
We're also using more generic terms instead of the ones used in Madrid's
Census API.
Co-Authored-By: Javi Martín <javim@elretirao.net>
Otherwise the variants returned for document_type="1" and
document_number="" will be
`["0", "00", "000", "0000", "00000", "000000", "0000000", "00000000"]`
which seems to be useless.
Probably this case is not real for production environments where those
arguments will always be fullfilled but seems to be interesting for
testing environment where this method is being called when those
paremeters where empty.
We've had to add a couple of hacks in order to make jQuery UI datepicker
work with Turbolinks, and one of our tests is failing because the
datepicker changes its height when changing from a month with 5 weeks to
a month with 6 weeks.
We could add a workaround so the test still passes (jQuery UI doesn't
provide a configuration option to always displays 6 weeks in the
datepicker), but I think it's easier to just use the HTML5 native date
input field, which also allows us to simplify the code a bit and IMHO it
improves the user experience, particularly when using mobile phones.
Since date fields are not supported in Safari and Internet Explorer,
we're still using the jQuery UI datepicker on those browsers (and on any
other browser not supporting date fields).
Due to these changes, we're moving the tests checking datepicker's
behaviour to the dashboard. I've choosing not to change the public pages
because I'm not 100% sure everybody would like this change (some people
prefer the datepicker because we can configure the way it looks).
Note we're using a new sanitizer. Ideally we'd reuse the
`AdminWYSIWYGSanitizer`, but then code that would be correctly shown by
markdown-it (like the <h1> tag) wouldn't be shown on the web, which is
confusing. Ideally we would configure markdown-it to only allow the tags
present in the `AdminWYSIWYGSanitizer` and provide some kind of help
showing which tags are allowed.
