After a `visit`, we were checking for content or filling in fields that
were already there before the `visit`, so we weren't 100% sure that the
request had finished before the test continued.
In the case of the verification tests, we were clicking the submit
buttons over and over without and then checking or interacting with
elements that were already there. Even though the button was disabled
between requests, meaning there wouldn't be simultaneous requests, it
was possible to interact with a form field before waiting for the
request to finish.
Some of these tests have recently failed on our CI, and it might be
because of that:
```
1) Admin budgets Edit Changing name for current locale will update the
slug if budget is in draft phase
Failure/Error: raise ex, cause: cause
Selenium::WebDriver::Error::UnknownError:
unknown error: unhandled inspector error: {"code":-32000,
"message":"Node with given id does not belong to the document"}
(Session info: chrome=134.0.6998.35)
1) Budgets creation wizard Creation of a multiple-headings budget by
steps
Failure/Error: expect(page).to have_content "Heading created
successfully!"
Selenium::WebDriver::Error::UnknownError:
unknown error: unhandled inspector error: {"code":-32000,
"message":"Node with given id does not belong to the document"}
(Session info: chrome=134.0.6998.35)
1) Custom information texts Show custom texts instead of default ones
Failure/Error: raise ex, cause: cause
Selenium::WebDriver::Error::UnknownError:
unknown error: unhandled inspector error: {"code":-32000,
"message":"Node with given id does not belong to the document"}
(Session info: chrome=134.0.6998.35)
1) Users Regular authentication Sign in Avoid username-email collisions
Failure/Error: raise ex, cause: cause
Selenium::WebDriver::Error::UnknownError:
unknown error: unhandled inspector error: {"code":-32000,
"message":"Node with given id does not belong to the document"}
(Session info: chrome=134.0.6998.35)
2) Verify Letter Code verification 6 tries allowed
Failure/Error: raise ex, cause: cause
Selenium::WebDriver::Error::UnknownError:
unknown error: unhandled inspector error: {"code":-32000,
"message":"Node with given id does not belong to the document"}
(Session info: chrome=134.0.6998.35)
2) Valuation budget investments Valuate Finish valuation
Failure/Error: raise ex, cause: cause
Selenium::WebDriver::Error::UnknownError:
unknown error: unhandled inspector error: {"code":-32000,
"message":"Node with given id does not belong to the document"}
(Session info: chrome=134.0.6998.35)
1) Users Delete a level 2 user account from document verification page
Failure/Error: raise ex, cause: cause
Selenium::WebDriver::Error::UnknownError:
unknown error: unhandled inspector error: {"code":-32000,
"message":"Node with given id does not belong to the document"}
(Session info: chrome=134.0.6998.35)
```
There were many cases where we were clicking on a link or (most of the
time) a button and then calling the `visit` method. In the past, it
worked just fine because clicking on buttons usually results in non-AJAX
requests, meaning that the test waited for the request to finish before
continuing.
That's no longer the case, though. In the last few months/years (not
sure since when) we're getting sporadic failures because the test
doesn't wait for the request to finish before making another request
with the `visit` method. This sometimes results in flaky tests.
Some of these tests have recently failed in our CI. Here are a few
examples (note the numbers don't follow an order because these tests
failed in different jobs):
```
1) Admin edit translatable records Current locale translation does not
exist For ActivePoll Shows first available fallback
Failure/Error: expect(page).to have_content "Sondage en Français"
expected to find text "Sondage en Français" in "Language: \n
\nEnglish\nDeutsch\nEspañol\nFrançais\nNederlands\nPortuguês
brasileiro\n中文\n Go back to CONSUL DEMOCRACY\nCONSUL
DEMOCRACY\nADMINISTRATION\nMenu\nNotifications\nMy content\nMy
account\nSign out\nProposals\nDebates\nComments\nPolls\n
Collaborative Legislation\nParticipatory budgets\nVoting booths
\nSignature Sheets\nMessages to users\nSite content\nModerated
content\nProfiles\nStatistics\nSettings\nProposals dashboard\n×
\nPolls description updated successfully.\nList of polls\nPolls
description\nCreate poll\nThere are no polls."
2) Public area translatable records Existing records Update a
translation With valid data Changes the existing translation
Failure/Error: expect(page).to have_field "Debate title",
with: "Title in English"
expected to find field "Debate title" that is not disabled but
there were no matches
2) Admin collaborative legislation Update Edit milestones summary
Failure/Error: expect(page).to have_content "There is still a long
journey ahead of us"
expected to find text "There is still a long journey ahead of us"
in "Language: \n
\nEnglish\nDeutsch\nEspañol\nFrançais\nNederlands\nPortuguês
brasileiro\n中文\n Go back to CONSUL DEMOCRACY\nCONSUL
DEMOCRACY\nADMINISTRATION\nMenu\nNotifications\nMy content\nMy
account\nSign out\nProposals\nDebates\nComments\nPolls\n
Collaborative Legislation\nParticipatory budgets\nVoting booths
\nSignature Sheets\nMessages to users\nSite content\nModerated
content\nProfiles\nStatistics\nSettings\nProposals dashboard\n×
\nProcess updated successfully. Click to visit\nBack\nAn example
legislation process\nInformation\nHomepage\nDebate\nProposals\n
Drafting\nFollowing\n1 language in use\nCurrent language\n
English\nSummary\n Format\n ◢\n Milestone\nManage progress
bars\nDon't have defined milestones\nCreate new milestone".
(However, it was found 1 time including non-visible text.)
3) Admin collaborative legislation SDG related list create Collaborative
Legislation with sdg related list
Failure/Error:
within("tr", text: "Legislation process with SDG related content") do
expect(page).to have_css "td", exact_text: "17"
end
Capybara::ElementNotFound:
Unable to find css "tr"
4) Valuation budget investments Valuate Feasibility can be marked as
pending
Failure/Error: expect(find("#budget_investment_feasibility_undecided"))
.not_to be_checked
Capybara::ElementNotFound:
Unable to find css "#budget_investment_feasibility_undecided"
3) Custom information texts Show custom texts instead of default ones
Failure/Error:
within("#section_help") do
expect(page).to have_content "Custom help with debates"
expect(page).not_to have_content "Help with debates"
end
4) Admin budgets Update Deselect all selected staff
Failure/Error: expect(page).to have_link "Select administrators"
expected to find link "Select administrators" but there were no
matches
3) Admin polls SDG related list edit poll with sdg related list
Failure/Error:
within("tr", text: "Upcoming poll with SDG related content") do
expect(page).to have_css "td", exact_text: "17"
end
Capybara::ElementNotFound:
Unable to find css "tr"
4) Admin polls SDG related list create poll with sdg related list
Failure/Error:
within("tr", text: "Upcoming poll with SDG related content") do
expect(page).to have_css "td", exact_text: "17"
end
Capybara::ElementNotFound:
Unable to find css "tr"
5) Admin custom images Image is replaced on admin newsletters
Failure/Error:
within(".newsletter-body-content") do
expect(page).to have_css("img[src*='logo_email_custom.png']")
end
Capybara::ElementNotFound:
Unable to find css ".newsletter-body-content"
6) Admin custom images Image is replaced on front views
Failure/Error:
within("#map") do
expect(page).to
have_css("img[src*='custom_map.jpg'][alt='Districts list']")
end
Capybara::ElementNotFound:
Unable to find css "#map"
```
Note we're excluding a few files:
* Configuration files that weren't generated by us
* Migration files that weren't generated by us
* The Gemfile, since it includes an important comment that must be on
the same line as the gem declaration
* The Budget::Stats class, since the heading statistics are a mess and
having shorter lines would require a lot of refactoring
Since this test was added in commit 78c6a30424 the
"current_password" was filled with an incorrect value.
This test did not fail because in previous versions of the
"devise-security" gem if this "current_password" was not valid
a "self.valid?" was performed which caused the error we are testing.
In version 0.17 (as can be seen in the following commit
https://github.com/devise-security/devise-security/pull/340/commits/41a99b67fe0)
if the "current_password" is not valid, related errors are still added
but the "self.valid?" code is removed, which is what causes the
expected error "must be different than the current password"
to appear in the "new password" field.
By adding a correct "current_password" in the test, we avoid this validation
for the "current_password" (which no longer includes the expected error)
and follow the natural flow of devise that does end up intercepting the error
for the new password entered.
One of these tests has failed once because there wasn't a user with the
right confirmation token. While I haven't been able to reproduce the
issue, there's a chance it's caused by a `visit` call to the
confirmation path which might start before the redirect request to the
successful sign up page has finished.
I'm not sure this is the case, though, but, worst case scenario, if the
test fails again we'll know it isn't because of a missing expectation.
The Google response contains an `email_verified` field instead of a
`verified_email` field, and so we weren't treating verified Google
accounts as verified.
Currently the application does not send any email to confirm the
account for already confirmed users. But we show a notice message
that may look like you will recive one:
"If your email address exists in our database, you will receive
an email with instructions for how to confirm your email address
in a few minutes."
In this commit we keep the original message, but send an email to
the user informing them that their account is now registered.
This way no one can know if someone else's account is confirmed and
we don't have to worry about GDPR either.
Co-Authored-By: taitus <sebastia.roig@gmail.com>
When we try to register with omniauth and the email or username already exists,
we use the finish_signup and do_finish_signup actions to allow the user to choose
another email or username.
The do_finish_signup action of the registration controller calls the
send_oauth_confirmation_instructions method which is responsible for sending the
confirmation email.
In this method we were only validating the case that the email is duplicated. Now
we add one more condition that allows us to send the instructions for the case in
which we have had to change our username.
The controller provided by the `devise-security` gem which tests
password is expired does not execute the `before_action` we have in our
application controller. That means it doesn't set the current locale.
We were having issues in the tests checking this behavior if the
previous test had set the current locale to a different one. This meant
the process running the browser had one locale while the process running
the test had a different one, which resulted in a page in English (as
expected), only the flash message notifying users their password expired
was in a different language.
To reproduce this behavior, run:
```
rspec './spec/system/welcome_spec.rb[1:1:2:2:1]' spec/system/users_auth_spec.rb:623 --order defined
```
I'm not sure whether this is a bug or it's a problem with the tests. In
theory it might be possible to reproduce a similar behavior in
production due to what we mention about the controller not executing the
`set_current_locale` method. But I haven't been able to reproduce the
situation, particularly since the password expiration seems to be
checked exclusively at login time (that is, if you stay logged in for 10
years, your password doesn't seem to expire).
So for now I'm just making the tests pass by using the login form
instead of using `login_as`.
JavaScript is used by about 98% of web users, so by testing without it
enabled, we're only testing that the application works for a very
reduced number of users.
We proceeded this way in the past because CONSUL started using Rails 4.2
and truncating the database between JavaScript tests with database
cleaner, which made these tests terribly slow.
When we upgraded to Rails 5.1 and introduced system tests, we started
using database transactions in JavaScript tests, making these tests much
faster. So now we can use JavaScript tests everywhere without critically
slowing down our test suite.
ChromeDriver only supports characters in the BMP (Basic Multilingual
Plane). One test fails with ChromeDriver because it was entering emojis.
I'm using kanji characters instead, although I must admit I'm not sure
why such an unusual login was used in the first place.
