rake spec should have a non-zero return value for failing specs

[fiedl]

I've recently updated to the current version rspec-rails 2.14.2.

Expected Behavior

With failing specs, the task rake spec should return a non-zero return value.

Present Behavior

With failing specs, the task rake spec does return 0.

bundle exec rake
echo $?   # =>  0

Therefore, for example, every build on travis-ci is green, even if specs are failing.

---

Does anyone know if this is a bug or how to fix this?

I've already tried to ensure the setting manually---without success.

# Rakefile
# ...
RSpec::Core::RakeTask.new( :spec ) do |t|
  t.pattern = "{./spec/**/*_spec.rb,./vendor/engines/**/spec/**/*_spec.rb}"
  t.fail_on_error = true
end

Thanks!

---

rake 10.3.1, rails 4.1.1

[myronmarston]

Thanks for reporting this. Do you have an example repro project you can provide? I'm pretty sure there's something specific about your environment causing this as 2.14 has been out for many months and no one's ever reported this before.

You might also want to try upgrading to RSpec 3 as that has some changes related to this area. We're no longer using an at_exit hook (which avoids some nasty ruby bugs like this).

[fiedl]

@myronmarston : Thanks for the quick reply. Then, probably, I've got another issue with this project, since we are in the process of upgrading from Rails 3.2 to Rails 4.

You can find the Rakefile here:
https://github.com/fiedl/wingolfsplattform/blob/sf/rails4/Rakefile

Is it wise to try to upgrade to RSPec 3 simultaneous to the Rails upgrade? This could be the next step.

Or, as a workaround, I could use rspec rather than rake spec in travis, for the moment, since rspec is returning the correct return value.

[myronmarston]

Is it wise to try to upgrade to RSPec 3 simultaneous to the Rails upgrade? This could be the next step.

No. I would do the upgrades one at a time. If/when issues arise, it's too hard to tell which part is at fault when you upgraded two major pieces at the same time.

You can find the Rakefile here:
https://github.com/fiedl/wingolfsplattform/blob/sf/rails4/Rakefile

Thanks. Can the issue be repro'd via that repository? Can you paste step-by-step instructions (e.g. git clone, then git checkout <a particular SHA>, then bundle install, then run x...) to repro your issue?

Or, as a workaround, I could use rspec rather than rake spec in travis, for the moment, since rspec is returning the correct return value.

Personally, I almost never use the rake task. The rspec command is far more flexible. The one nice thing about the rake task is that it can hook into a pipeline of rake tasks when you want to make it part of a larger build.

[myronmarston]

This might also be the same issue reported in rspec/rspec-core#1196. You can try backporting the fix @JonRowe did there and seeing if that fixes your issue.

[fiedl]

Reproduction

To reproduce the original issue of the missing return value when using RSpec::Core::RakeTask.new(:spec), here are the necessary steps:

• (a) use travis: Fork the repo fiedl/wingolfsplattform, make sure it is hooked up in your travis, and push the commit 9894497b91b0fe20ecbea3d30b82f659f3a136b3 to some new branch in order to have it run on travis. This is my result of running this build. As one can see, there are a lot of failing specs, but the exit status is 0.

• (b) locally: Create a database.yml according to your local preferences. Then follow the travis.yml script:

  # bash
  git clone git@github.com:fiedl/wingolfsplattform.git
  cd wingolfsplattform
  git checkout 9894497b91b0fe20ecbea3d30b82f659f3a136b3
  sudo apt-get -y install pwgen libicu-dev
  bundle install
  mkdir -p public/uploads
  bundle exec rake db:create db:migrate db:test:prepare

  And finally, run the specs:

  # bash
  bundle exec rake

rspec rather than RSpec::Core::RakeTask

Personally, I almost never use the rake task. The rspec command is far more flexible. The one nice thing about the rake task is that it can hook into a pipeline of rake tasks when you want to make it part of a larger build.

Thanks for the advise. I've tried changing my rake spec task accordingly to call just rspec:

# Rakefile
# ...
Rake::Task[ :spec ].clear
task :spec do
  exit system("rspec")
end

The exit ensures that the exit status of the system call is passed to the exit status of the rake call. That way, we don't have to change our habit to run the specs just by calling rake. And I don't have to change my travis config.

Unfortunately, this does not solve the problem. But it helped to narrow it down.

Issue running feature specs

Some further testing revealed that also running rspec itself is affected by the issue. I think the issue occurs whenever running feature specs:

# STATUS 1 -- correct behavior
bundle exec rspec spec/models/user_spec.rb
bundle exec rspec spec/models/*_spec.rb
bundle exec rspec spec/models/user_spec.rb spec/models/group_spec.rb
bundle exec rspec spec/models/user_spec*.rb
bundle exec rspec spec/models/**/*_spec.rb

# STATUS 0 -- wrong behavior
bundle exec rspec ./spec/**/*_spec.rb ./vendor/engines/**/spec/**/*_spec.rb
bundle exec rspec ./spec/**/*_spec.rb
bundle exec rspec spec vendor/engines/your_platform/spec
bundle exec rspec spec/features/*_spec.rb
bundle exec rspec vendor/engines/your_platform/spec/features/*_spec.rb
bundle exec rspec vendor/engines/your_platform/spec/features/search_field_spec.rb
bundle exec rspec vendor/engines/your_platform/spec/features/corporate_vita_spec.rb
bundle exec rspec

In all cases with the wrong behavior, at least one feature spec is run.

[fiedl]

SimpleCov

I had a look at rspec/rspec-core#1196. We do have also simplecov in our spec_helper.

Deactivating simplecov and coveralls causes the issue to disappear.

I've just seen a comment in the simplecov readme:

Important Notice: There is currently a bug that affects exit code handling on the 0.8 line of SimpleCov, see simplecov-ruby/simplecov#281. Please use gem 'simplecov', '~> 0.7.1' until this is resolved.

Solution

Modify the Gemfile:

# Gemfile
# ...
gem 'simplecov', '~> 0.7.1', require: false

Thanks for helping!

[bf4]

@fiedl Thanks for the reproducible SimpleCov at_exit bug. It helped me provide evidence SimpleCov is fixed in master :) https://travis-ci.org/bf4/wingolfsplattform/builds