diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md index 074dbeced..b80a40074 100644 --- a/.github/ISSUE_TEMPLATE.md +++ b/.github/ISSUE_TEMPLATE.md @@ -2,29 +2,27 @@ [comment]: # (As a , I want to so that .) - ### Why is this Needed? [comment]: # (Describe the problem and why this task is needed. Provide a description of the current state, what you would like to happen, and what actually happened) - ### Description [comment]: # (Feature or Bug? eg. Type: Bug) *Type*: - ### Current Behavior + [comment]: # (Describe what actually happened.) - ### Expected Behavior + [comment]: # (Describe what you expected to happen.) - ### Steps to reproduce + [comment]: # (Describe how we can replicate the bug step by step.) - ### Definition of Done + [comment]: # (Any other information that would be useful, bullets are helpful.) diff --git a/.github/ISSUE_TEMPLATE/flakies.md b/.github/ISSUE_TEMPLATE/flakies.md index c253bc3cd..6b0a02a0b 100644 --- a/.github/ISSUE_TEMPLATE/flakies.md +++ b/.github/ISSUE_TEMPLATE/flakies.md @@ -1,4 +1,5 @@ # What + Tests that fail randomly are called "flakies", this one seems to be one: **Randomized seed:** FILL_WITH_RANDOM_SEED @@ -12,13 +13,16 @@ Tests that fail randomly are called "flakies", this one seems to be one: ``` # How + - [ ] Explain why the test is flaky, or under which conditions/scenario it fails randomly - [ ] Explain why your PR fixes it ## Tips for flaky hunting ### Random values issues + If the problem comes from randomly generated values, running multiple times a single spec could help you reproduce the failure by running at your command line: + ```bash for run in {1..10} do @@ -28,6 +32,7 @@ done You can also try running a single spec: Add option `:focus` to the spec and push your branch to Github, for example: + ```ruby scenario 'Show', :focus do ``` @@ -35,17 +40,20 @@ scenario 'Show', :focus do But remember to remove that `:focus` changes afterwards when submitting your PR changes! ### Test order issues + Running specs in the order they failed may discover that the problem is that a previous test sets an state in the test environment that makes our flaky fail/pass. Tests should be independent from the rest. After executing rspec you can see the seed used, add it as an option to rspec, for example: `bin/rspec --seed 55638` (check **Randomized seed** value at beginning of issue) ### Other things to watch for + - Time related issues (current time, two time or date comparisons with miliseconds/time when its not needed) - https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests ### Final thoughts -The true objective of this issue is not "to fix" a flaky spec, but to change a spec that randomly fails into one that we can trust when it passes (or fails). + +The true objective of this issue is not "to fix" a flaky spec, but to change a spec that randomly fails into one that we can trust when it passes (or fails). That means you've to think "What are we testing here?" and "Can we test the same thing in another way?" or "Are we already testing this somewhere else at least partially?".