Unexpected invocation errors with Test::Unit & Rails v4.2.1

[floehopper]

Created on behalf of Benoit Mio based on his post to the mailing list.

I'm working a old project with some test-unit (3.0.9) and mocha (1.1.0). For the moment I'm working with Rails 4.1.10 (with ruby 2.2.2p95,) and all test pass, but when trying to switch to Rails 4.2.1 many tests fails. It seems that tests are not "stable" any more.

For example is it possible that 2 tests in the same test file can fail when they are run together? If one or the other is skipped test pass. Otherwise I get Mocha::ExpectationError: unexpected invocation: FileApplication(id: integer........). I get essentially "unexpected invocation" errors.

I would like to know where I can look to fixe this issue? In the test_helper?

Thanks a lot in advance for your help.

PS: I've read this post https://groups.google.com/forum/#!searchin/mocha-developer/rails4/mocha-developer/9Jcq64UYZf8/eKId7KtxoUoJ but I have a correct Gemfile.

[floehopper]

I've just had a thought. When you say you're using test-unit, can you explain how?

As far as I'm aware, for some time (since Rails v4 or maybe earlier?) ActiveSupport::TestCase has only used Minitest::Test and not Test::Unit::TestCase. Can you start a Rails console (in the test environment) and send me the result of running ActiveSupport::TestCase.ancestors?

Also in order to help me reproduce the problem, can you post your Gemfile and your test/test_helper.rb, or at least the mocha/test-unit-related bits? Either in here or as a Gist. Thanks.

[For my own reference: I've started trying to reproduce the problem here - https://github.com/mocha-rb/mocha-with-rails-4.2.1]

[benoittgt]

Under rails 4.2.2

2.2.2 :001 > ActiveSupport::TestCase.ancestors
 => [ActiveSupport::TestCase, ActiveSupport::Testing::TimeHelpers, ActiveSupport::Testing::Deprecation, ActiveSupport::Testing::Assertions, ActiveSupport::Callbacks, ActiveSupport::Testing::SetupAndTeardown, ActiveSupport::Testing::TaggedLogging, Minitest::Test, Minitest::Guard, Minitest::Test::LifecycleHooks, Minitest::Assertions, Minitest::Runnable, Object, RequireAll, PP::ObjectMixin, Mongoid::Extensions::Object, BSON::Object, Origin::Extensions::Object, ActiveSupport::Dependencies::Loadable, JSON::Ext::Generator::GeneratorMethods::Object, Kernel, BasicObject]

My test_helper :

ENV['RAILS_ENV'] ||= 'test'
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'
require 'webmock/test_unit'
require 'sidekiq/testing'
require 'capybara/rails'
require 'mocha/test_unit'

class ActionController::TestCase
  include Devise::TestHelpers
end

class ActionController::TestRequest
  def set_header(name, value)
    @env[name] = value
  end
end

class ActiveSupport::TestCase
  include ApplicationHelper
  include ActionView::Helpers::TextHelper
  include FactoryGirl::Syntax::Methods
  include WebMock::API

  fixtures :all

  # Add more helper methods to be used by all tests here...

  def assert_false(assertion, msg='')
    assert( ! assertion, msg)
  end

  def log_is_as_admin_user admin_user
    sign_in admin_user
  end

  def log_in_as user
    sign_in :user, user
  end

  def log_in_as_admin admin=nil
    admin ||= create :admin
    log_in_as admin
    admin
  end

end

Test::Unit.at_start do
  FactoryGirl.create :Portofolio
  FactoryGirl.create :FolderApp
end

[floehopper]

If your tests inherit from Rails base classes like ActiveSupport::TestCase then you are not actually using Test::Unit any more. Can you try changing require 'mocha/test_unit' in your test_helper.rb to require 'mocha/mini_test'? Let me know whether that helps.

Note that you may have other problems with code that tries to use Test::Unit e.g. you might need to change require 'webmock/test_unit' and I doubt that the Test::Unit.at_start block is doing anything useful. Ultimately, I'd probably try to remove the test-unit gem from your Gemfile and fix anything that breaks.

[benoittgt]

Thanks @floehopper by changing 'require 'mocha/test_unit' in 'mocha/mini_test' it's far better. No more errors mock's related.

The behavior in my tests was really strange, like if mocks where not teardown between different tests.

I let you close the bug and thank you again for your help.

PS: Thanks for the tips, I've also clean the code.

[floehopper]

@benoittgt: I'm glad that fixed the problem. I think Rails should probably make it clearer that they no longer support test-unit. I'm also going to look at whether I can make mocha fail fast if you ask it to integrate with test-unit, but it fails to do so. In the meantime I'm going to close this issue.

[daqo]

I faced the same problem and changing require "mocha/unit_test" to require "mocha/mini_test" fixed it for me.

[floehopper]

@daqo: I'm glad to hear that it solved your problem too.

[hakanai]

I have the same problem here but it still happens after changing to mocha/mini_test.
If I run the tests multiple times in a row, it only seems to happen about 1/3 of the time.
Every time it fails, it lists a bunch of expectations which don't (or shouldn't) even apply to the current test, so I assume it's leaking mocking expectations between tests somehow.

[floehopper]

@trejkaz:

I suspect the reason it's not happening every time is that the order of your tests is randomized and sometimes the order causes a problem and sometimes it doesn't.

You could confirm this by running the tests until they fail and then finding the seed value at the top of the output. Then run the tests repeatedly with this seed using a command like this: TESTOPTS="--seed=NNNNN" rake. They should now fail every time, because the test order is kept constant. Please can you try this and let me know what happens?

Please can you send me the following so I can try to reproduce the problem:

• Exact Ruby version and platform
• Exact Rails version
• Exact Mocha version
• The whole of your Gemfile & Gemfile.lock files
• The whole of your test_helper.rb or spec_helper.rb file

Please can you also try running the tests with Mocha debugging enabled and send me the output: MOCHA_OPTIONS=debug rake.

Thanks.

[hakanai]

I think I got to the bottom of it. When using test-unit, we were loading mocha up-front from the Gemfile. On switching to minitest, the require for mocha now has to be in test_helper.rb, below the line that requires rails/test_help. I guess the way it hooks requires the test classes to have already been defined, whereas when we were using test-unit it didn't.

What's worrying is that requiring mocha too early doesn't trigger any other, easier to understand error. :)

Anyway, because the hooks didn't run, the teardown hook which clears the expectations wasn't running, which resulted in the expectations piling up, or being wrong for the next test.

[floehopper]

@trejkaz: I'm glad you managed to resolve the problem. Thanks for letting me know. There is an outstanding issue which I think would address your concern: Fail fast when using mocha/test_unit or mocha/mini_test if integration fails. Hopefully I'll find some time to work on this soon.