-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Public API does not provide hooks for server started, all browsers captured, etc #1037
Comments
@lazd is your request mainly in the scope of usage with gulp? If so, could we move this discussion to https://github.com/karma-runner/gulp-karma instead? At the moment I'm doing a number of changes in karma so it exists properly and we can use its public API with callbacks. |
It's related, but it's an issue with Karma's public API so I filed here. We need more than exit callbacks -- we need events for browser capture etc. Any thoughts on how that can be exposed as part of the public API? |
I think that we can extend the public API given a well-defined use-case. At the moment I'm focused on making Karma exit properly, but would be keen on looking into other practical scenarios after that. What is your practical need for those additional hooks? |
I think this is a valid feature request... Maybe we could allow registering events on globalEmitter... |
@pkozlowski-opensource, the practical need is to know when Karma is in certain states so we can trigger test runs (as described above). @vojtajina, registering events on Without knowing if all of these are possible, I imagine the events could be something like this: runner
server
I'm not sure what other events are available or useful, but server.browsersReady is the one that's blocking my use case. |
@lazd OK, I was going over all the workflows I'm using currently and I wonder what is the exact workflow you are trying to address. While I can see value in providing all those events, it seems to me that the real issue is to be able to run tests when a server (started with if I'm not mistaken, you are after the workflow where there is an external watch (grunt, gulp etc.) that watches files and should:
What is your exact use-case? Would the above work for you? |
@pkozlowski-opensource the use case you described above would work nicely for what I'm trying to do, which is basically something like this: https://github.com/lazd/gulp-karma-test/blob/master/gulpfile.js#L17-L26 The above use case would be possible if the runner is responsible for waiting until the server is ready, but it might be of benefit to still expose the server's state via events for more advanced tools. I could imagine the events described above being useful for VM providers (like Sauce and Browserling) that want to connect browsers and know when they're ready/crashed/done/etc. |
@lazd ok, we are on the same page then! I can see that you are doing some work on the gulp + karma integration and wonder if we could join forces to have a no-nonsense integration of the 2. This is why we've started this repo: https://github.com/karma-runner/gulp-karma as atm I'm not sure how much added value we could get from a dedicated plugin. |
@pkozlowski-opensource, yes let's work together on this! From the start, I knew |
@lazd yup, I've noticed that this is not the best candidate for a plugin, this is why I wanted to have this repo with a integration scenario. At the moment my biggest worry is that Karma doesn't exit cleanly, there are 2 PRs waiting to be merged:
as soon as those are in we can look into the server-waiting-thing and see what utils / helpers people might need and where is the best place to implement them. |
@pkozlowski-opensource, those were my biggest concerns as well, resulting in the spawn of an external process in the both the Grunt and gulp plugins. I had to do some very dirty stuff to know when to start running tests for the watch, hence the filing of this issue. This comment elaborates on all of the above. |
Yes, I know, it took me some time to tracks down all those non-exiting-properly issues but I believe that I've nailed all of them. |
@pkozlowski-opensource @lazd Any updates on how this is going? I think there are several of us waiting for this to get through and working or find out what the recommended path forward is for now. |
@abierbaum I must say that for now I was using the gulp+karma integration scenario demonstrated here https://github.com/karma-runner/gulp-karma and I must say that I'm very happy with this so far. As of now there is an issue on Windows (karma-runner/gulp-karma#5) but it is well understood and we've got a PR with a fix waiting to be merged. So my immediate goal is to make the https://github.com/karma-runner/gulp-karma workflow working for everyone, including people on Windows. Now, it is a bit hard for me to say what are the best next steps as I must say that I'm pretty happy with the integration scenario indicated above. I guess you guys got different use cases / favourite workflows so to get things moving I would ideally need the description of your perfect working scenario with an ideal syntax you would like to see. |
@pkozlowski-opensource Thanks for the pointer to this path. That is what I needed. I will give it a try and see how well it works for our case. We have browserify (with watchify) and gulp-watch for sass files. My biggest concern is getting everything working for the "watching" case where I want karma, browserify, and everything else to update automatically on any change. If I run into any issues I will try to create a test case I can share publically to show any problems. |
@abierbaum, the biggest blocker I had for a similar workflow was knowing when Karma was ready to run the first round of tests. I needed to do something like:
Currently, there is no way to know: a. When Karma is ready to run tests (browsers have been captured) Without this information, it's possible that tests are not ran because no browsers have been captured or because a change happened in the middle of the first run. In order to extract that information, I had to do some awful, dirty things in a branch that I'm not comfortable releasing. However, the API resulting from these awful hacks does get the job done, check it out. Now that @pkozlowski-opensource has fixed the exit errors, the dirty hack of spawning another process for Karma is not needed, but until this issue is closed, we'll still be spying on Karma's stdout to figure out what it's doing. |
@lazd I must say that I didn't put much more effort into this topic (apart from the obvious timeout fixes) simply because I'm quite happy / comfortable with my current workflow. The thing is that I never had to to what you are describing, that is, manually start karma, wait for browsers to connect and then run tests. I guess the reason for this is that I'm working as follows:
So, I would be very keen on understanding your workflow where you need to "manually" start Karma, lunch browsers etc. The only use-case I can see is to integrate with external watchers. While I agree that it would be great if we could plug any external watching mechanism, the current one built-in in Karma works for me, so I'm not too pressed to dig into this. What are your real-life workflow scenarios that you are trying to tackle (apart from integrating with external watchers?). |
@pkozlowski-opensource, I understand your workflow and why you wouldn't need this. Running tests immediatelyI like the tests to run immediately before I start watching. If the tests fail, this lets me know that things were broken before I started changing code and prevents me from scratching my head as to why a change in an unrelated file caused tests to fail when, actually, the code was broken in the first place. Yes, we could do a single run with Karma, then when it exits, start watching with Karma, but we're launching/closing browsers all the time and we wouldn't have to do that if we had a Integration with watchersIf you're building your app, say with browserify, you're watching for source file changes, then building a bundle, and then you'd like to run tests on the bundle. Yes, we can let Karma watch the bundle file (output) and get away with it, but I'd rather use one watcher. I can't quite recall what the problem was, but I had trouble getting the server to recognize a new list of files between runs, so if tests were added, it wouldn't accept them. I would have to dig back into it to give any kind of useful troubleshooting information, but something wasn't right. If this is still an issue, it's a separate one for sure, but it serves to highlight the problems we're having unless we let Karma do the watching. |
@lazd thnx for the detailed info, I'm happy that we can focus on concrete use-cases (disclaimer: I'm not saying that Karma shouldn't provide hooks that you are proposing, just trying to understand the situation better!). Now, for the running tests immediately - I'm guess we might have a different setup, but when I do Then, the use-case with Browserify is more interesting. I'm not using it myself but it is very likely that I will need to dive into its usage soon. But I'm using JS written the CommonJS-style at the daily basis and this is why I've spend more time improving https://github.com/karma-runner/karma-commonjs. What I'm trying to say here is that I would probably even try to avoid running browserify bundling while running tests. I do acknowledge that the https://github.com/karma-runner/karma-commonjs plugin doesn't solve everything (for example, it doesn't have any fallback for the native modules provided by browserify). Would this plugin work for you? Once again, I think your requests are valid, I'm just trying to better understand what people are looking for, practically speaking. |
@lazd Have you successfully used gulp, karma, and browserify together? I am currently having trouble figuring out how to get karma and browserify to work correctly in this context. (ie. how to have tests that karma is running use require() to pull in modules that are under test and similarly to evaluate requires within the hierarchy of test files) |
@abierbaum, I haven't given it a fair shot, I've been busy on other projects. I would suggest generating a browserify bundle with your tests and including that in Karma along side your application bundle. Given the fact that browserify transforms could change things, it's best to serve Karma a ready-made bundle. |
@abierbaum your watching issue is probably the same as reported here: #1113 and has nothing to do with the issue discussed here. Would be awesome if you could move your comment over to #1113 in order to not diverge from the original topic. |
@pkozlowski-opensource Will do. I will just delete it from this thread for now. Thanks. |
Return the EventEmitter instance so that it's possible to hook into various lifecycle events. Add a "browsers_ready" event so that it's possible to know when the karma server has finished starting up. Enables and partially resolves karma-runner#1037
Problem
The public API does not provide enough insight into the state of the server.
Why?
Because server tasks that are dependent on its state can be triggered programmatically by outside entities, it is necessary for said entities to have knowledge of the server's state.
For instance, imagine if you wanted to do the following, programmatically:
server.start()
)runner.run()
)runner.run()
)Currently, this is not possible. Yes, this is exactly what
autoWatch
does, but runs cannot being triggered programatically. Yes,singleRun
will run tests when all browsers are connected, but it will kill the server and exit when tests are done running, which makes the third step impossible.Workaround
In order to work around this, implementors are forced watch
process.stdout
for messages to infer what's going on inside of Karma, which is incredibly brittle and hackish.Solution
I tried digging to find a way to get access to the event emitter that triggers some of these events, but it seems it's not currently possible via the public API.
If
server.start
returned and object with a reference theglobalEmitter
object, we'd be halfway there.The text was updated successfully, but these errors were encountered: