-
Notifications
You must be signed in to change notification settings - Fork 551
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Models in subfolders are reporting 0% coverage #671
Comments
👋 Thanks for the bug report! That's super weird... Can you check if the problem really goes away if you move them outside of the subfolder? Or is it likely messing up autoloading. I'm not sure if it's a problem with the folder, it might be the subclassing or it might be the autoloading of subfolders in rails. It might also be a problem with the files being loaded before you run the tests (often with spring, sometimes while running with rake) - so it's likely an environment problem I think. Because the ruby coverage library takes care of tracking all files loaded after the test was invoked. Without a repo to reproduce it's hard to figure out. |
@PragTob thanks to reply, my mate has created a very simple repository to demonstrate the issue, you can clone it: The repository has three basic models Hope you can replicate the issue with the repository. |
👋 Thanks for the quick turnaround, I'll try to set it up now, but can't guarantee how far I'll get - time being limited and all :) linke for my future self: https://github.com/darkcode85/simplecov-bug Also, how do you usually run the tests? |
@PragTob Thanks! we run the test using: |
Yeah can reproduce |
Works when running This is the rails binstub: #!/usr/bin/env ruby
begin
load File.expand_path('../spring', __FILE__)
rescue LoadError => e
raise unless e.message.include?('spring')
end
APP_PATH = File.expand_path('../config/application', __dir__)
require_relative '../config/boot'
require 'rails/commands' It loads a bunch of stuff before executing the tests, and as a result the coverage library can't track what was loaded and so it's wrong. Might need to talk to the rails team as to how improve the situation or might have to look at something else. |
@PragTob thanks to look into that, I can confirm that it works running |
this is neither really a rails bug nor a simplecov bug - it's a setup bug. As Rails is bigger and we integrate with Rails it's probably more of a documentation bug on our side ;) To be fair, our document is quite vocal about the fact that simplecov needs to be loaded and started first for it to work which is clearly violated here :) I guess I gotta ask the rails folks to see what they think how this should best be handled/where it should be integrated. |
I did another round of testing... removing my old adversary spring via |
@PragTob weird... I can confirm that after do |
yeah spring messes with code coverage greatly due to the nature of what it does... we have specific spring instructions in the README but these don't work for me either - I haven't used spring myself in years as it caused me too much pain and too many lost hours already |
Experiencing the same issue for rspec. |
Yeah, @PragTob, had the same elusive issue with spring. Similarly, I have decided to remove spring for rspec That being said, while trying to recreate this issue again, I noticed another weird interaction with spring.
I dont know how useful that is but I thought I should mention it. Perhaps, someone with deeper understanding of spring might be able to explain this behaviour. |
Debugging SimpleCov file loading, the movie: export DISABLE_SPRING=1
cat <<'EOF' > debug_coverage.rb
require 'pp'
require 'coverage'
Coverage.start
at_exit { pp(Coverage.result) }
Rake::Test['default'].invoke # or whatever is a way to run the above before any requires in your Rakefile or spec or test helper
EOF
bundle exec ruby debug_coverage.rb also looking at #465 #340 #671 (comment) #235 #328 (comment) #281 |
Yeah.. I left this open so far to reinvestigate how we can work with spring as I think last time I tried it I forgot something. I fear we won't work well with spring at all and I don't have much personal experience because spring has caused me more hours of debugging than saving time so I remove it from all my projects immedeatly. |
Just chiming in to say, @saramoohead and I were just trying to report coverage on some new work she is doing and Simplecov was returning some very wacky numbers (0% on models that had a 100% covering spec). Issue went away when re-running with Annoyingly, we actually ran into this problem 6 months earlier. We ended up adding Whilst not ideal, I wonder if it is possible for Simplecov to know if it's running inside Spring, and if so, produce a warning that recommends re-running using |
@andypearson I also disable spring all the time. I have this in my bash profile. I suppose you can try something like The reason this is a problem at all is because you can't restart the Ruby Coverage module. So, with spring, the code is loaded before coverage and so is never added. |
Similar issue with Rails 6, Minitest, Spring and simplecov 0.17.1. |
Use |
It doesn't really do anything in CI anyway, and it messes with our coverage reports: simplecov-ruby/simplecov#671
Rails Version: 5.2.0.rc1
Simplecov Version: 0.15.1
Test Framework: Minitest
Test coverage are running properly for all the models in my application except for these that live in a sub folder, example:
I have a notification.rb model in
app/models/notification.rb
:And also a subclass model called email.rb in
app/models/notification/email.rb
:I have added the initial test coverage for both models. In
test/models/notification_test.rb
:and in
test/models/notification/email_test.rb
:Both tests are running and passing with success and I can force the test to fail but the coverage report always show 0% coverage for these models.
This is my test_helper.rb:
The text was updated successfully, but these errors were encountered: