Skip to content

Add documentation for failing builds on unmet thresholds #173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
charlesdemers wants to merge 1 commit into from
Closed

Add documentation for failing builds on unmet thresholds #173

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,54 @@ Configuration is optional. It should be put in a file at `config/coverage.js` (`
}
```

## Fail a build when coverage doesn’t meet defined thresholds

To analyze coverage stats you will need the [`nyc`](https://github.com/istanbuljs/nyc) command line tool. The first step is to install it:

`npm install nyc --save-dev`

Then you will need to define the thresholds you would like by adding a `.nycrc` file to the root of your app.

For example:

```
{
"all": true,
"check-coverage": true,
"per-file": true,
"temp-directory": "coverage",
"branches": 60,
"functions": 60,
"lines": 60,
"statements": 60,
"watermarks": {
"branches": [70, 80],
"functions": [70, 80],
"lines": [70, 80],
"statements": [70, 80]
}
}
```

With this config, `nyc` will validate the coverage percentage for each file individually and fail whenever it’s lower than 60%. The watermarks numbers are used only as guidelines to modify the output color of the reporter.

Next, to execute `nyc` you can define a command to check the coverage in `package.json`:

```json
"scripts": {
"test": "COVERAGE=true ember test",
"check-coverage": "./node_modules/.bin/nyc check-coverage"
Copy link

@JaKXz JaKXz Jan 5, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👋 Istanbul member here - how does ember produce the coverage result?

Does it do the instrumentation? In which case you would want to set instrument: false in your nyc config.

Finally, if it produces a coverage report, I think you would want this command to be nyc report --report-dir ./custom-report-dir, but I don't think you can have it actually fail based on the coverage check/watermarks if nyc is not doing the instrumenting. Correct me if I'm wrong @bcoe @coreyfarrell

Copy link

@coreyfarrell coreyfarrell Jan 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure the nyc report sub-command checks watermarks, I think it just generates reports. I do know that nyc checks watermarks, for example one of my projects vinyl-rollup uses instrument: false and coverage check works. If you add unreachable code to index.mjs it'll fail due to insufficient coverage, nyc exits with a non-zero code.

Copy link
Collaborator

@kategengler kategengler Jan 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JaKXz Thanks for chiming in!

The instrumentation is done using babel-plugin-istanbul at build time.

The reports are generated using istanbul-api (https://github.com/kategengler/ember-cli-code-coverage/blob/master/lib/attach-middleware.js#L35) in a middleware that receives the coverage data from browser-based tests.

It does sound like we'd need the nyc configuration suggested here: https://github.com/istanbuljs/nyc#use-with-babel-plugin-istanbul-for-babel-support

}
```

Finally, to fail the build on Travis if the thresholds are not met, you can add the previously defined command in the "script" section:

```yml
script:
- npm test
- npm run check-coverage
```

## Create a passthrough when intercepting all ajax requests in tests

To work, this addon has to post coverage results back to a middleware at `/write-coverage`.
Expand Down