Regular test output swallowed

[alexkwolfe]

It appears as though regular test output is swallowed when using the reporter in a standard Rails app.

Please see the following sample app for a simple demonstration: https://github.com/alexkwolfe/codeclimate-issue

[brynary]

Thanks for the reproduction case! I believe I've found the issue. This code:

if ENV['CODECLIMATE_REPO_TOKEN']
  require 'codeclimate-test-reporter'
  CodeClimate::TestReporter.start
end

Needs to go at the very top of test_helper.rb. I'm not sure why the output issue is occurring with it's current placement, but that would explain why we haven't seen this before. The CC gem is quite simple so I imagine it's something in the SimpleCov gem under the hood.

Let us know if you have any issues with that.

Best,
-Bryan

[benjaminoakes]

We noticed today that our Travis CI build for acceptance tests was failing, but it was incorrectly reporting success (exit code 0). This happened when using the CodeCimate gem and running rspec with a --tag option.

Strange, but we've disabled the coverage reporting until we figure out why that is.

---

Details

With CodeClimate disabled and failing specs:

$ bundle exec rspec ./spec/features/eula_spec.rb --tag '@type:feature'; echo $?
[...]
1

With CodeClimate enabled and failing specs:

$ bundle exec rspec ./spec/features/eula_spec.rb --tag '@type:feature'; echo $?
[...]
0

The beginning of our spec_helper.rb is:

begin
  require "codeclimate-test-reporter"
  CodeClimate::TestReporter.start
rescue LoadError
end

Disabling means that we commented out that block of code.

Version from our Gemfile.lock:

codeclimate-test-reporter (0.3.0)

And for completeness: we're using Ruby 2.1.0

[jasonroelofs]

@benjaminoakes This is a bug with SimpleCov and tracked here: simplecov-ruby/simplecov#269

You'll want to force SimpleCov 0.7.1 in your Gemfile for the time being.

[benjaminoakes]

@jasonroelofs Thanks for looking into it. We will try the change you suggest. I'd recommend making the change in ruby-test-reporter itself and releasing a new gem, since many users probably don't realize this is happening.