Add FastBoot to ember-try

[tomdale]

Many addons don't load in FastBoot, often for reasons that are accidental and easy to fix. It would be awesome if ember-try could add a mode to test addons in FastBoot, so addon authors can detect and fix these issues.

/cc @kategengler

[kategengler]

I have no clue how to use FastBoot myself; is there some way to run headless tests in FastBoot? (ember-try is primarily used to run tests with variations of dependencies on CI.)

[rwjblue]

@tomdale - It isn't as simple as using ember-try. I have a rough idea/plan that I'll ping you about in chat.

[kategengler]

If there's some way to automate testing of addons for FastBoot compatibility -- we can make it an EmberObserver score point if that helps.

[tomdale]

@kategengler The basic approach is:

1. Build for FastBoot: ember fastboot:build (just like ember build but puts things in fastboot-dist instead of dist.
2. Use ember-fastboot-server to load the app.js and vendor.js files. Most importantly, report if there are any failures evaluating the file in Node.

[kategengler]

I decided to try to test FastBoot with my own addon, ember-feature-flags.

1. (Per readme) ember install ember-cli-fastboot

2. (Per readme)

   rm -rf bower_components 
   bower install --save ember#canary 
   bower install
   

3. (Per above)
   ember fastboot:build

4. (My guess) npm install -g ember-fastboot-server

5. (ember-fastboot readme) ember-fastboot --port 80 --app-file fastboot-dist/assets/dummy.js --vendor-file fastboot-dist/assets/vendor.js --html-file fastboot-dist/index.html

And ... boom

DEBUG: -------------------------------
DEBUG: Ember : 2.2.0-canary+aefcf141
DEBUG: -------------------------------
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: listen EACCES
    at exports._errnoException (util.js:746:11)
    at Server._listen2 (net.js:1139:19)
    at listen (net.js:1182:10)
    at Server.listen (net.js:1267:5)
    at EventEmitter.listen (/usr/local/lib/node_modules/ember-fastboot-server/node_modules/express/lib/application.js:617:24)
    at Object.<anonymous> (/usr/local/lib/node_modules/ember-fastboot-server/bin/ember-fastboot:50:20)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)

Does this mean ember-feature-flags is not compatible with FastBoot, or is it something else?

[tomdale]

@kategengler You shouldn't need ember-fastboot-server unless you're running in production. Can you try ember fastboot to boot the dev server? Also, the error you're getting is due to trying to bind to port 80 as non-superuser. If you try --port 3000 or something it should work.

[kategengler]

@tomdale Great, ember fastboot works. What would that command do for an addon that wasn't fastboot compatible? (Can I tell from that command if an addon is compatible?)

[tomdale]

@kategengler You can simulate what an incompatible addon would look like by accessing document, for example, in initialization. When the server boots you should get ReferenceError.

[tomdale]

@kategengler Figured this thread might be a good place to follow up on our conversation from Ember NYC. :) What do you need help with from me to move forward on adding FastBoot compatibility to Ember Observer?

Based on what you said the other night, my thinking right now is that there could be three levels of compatibility, in increasing order of goodness:

1. Addon does not prevent app from booting in Node.js (doesn't touch document at eval time)
2. Addon says that it it compatible (author adds a flag to package.json explicitly saying it is compatible)
3. Addon has automated tests that run in FastBoot

[kategengler]

@tomdale The only thing I need right now is time!

While I still do want to add this to Ember Observer, I've reconsidered whether this also belongs in ember-try. I would be very frustrated if there was some site testing my addon for a feature and I didn't know how to recreate that test locally. Since testing FastBoot requires installing an addon (then removing it if you don't actually need/want it as a devDepedency of your addon) and now that ember-try has npm-support, it makes sense as an ember-try command.

I imagine ember try:fastboot.

For the third level, how can an addon run automated tests "in FastBoot"?

[tomdale]

@kategengler Third level is still being designed and should definitely be punted on for right this second. At a high-level, I want to be able to express a set of acceptance tests that can apply in both environments. At first, that is probably things like:

1. Visit someURL
2. Verify a <p> tag with specified content exists
3. Click the link with ID user-profile-link
4. Verify the URL is now someOtherURL

Over time, it would be cool to run these tests in a browser with JavaScript disabled. Eventually we may get form processing in FastBoot as well, so you could then test things like filling out form fields. But I think that's a ways off. ;)

[kategengler]

@tomdale Gotcha. So, if its just the first level, I should be able to find time to get it at least into ember-try within the next week or so.

[tomdale]

@kategengler Perfect. Ping me if you run into any trouble.

[kategengler]

@tomdale Since ember fastboot starts up a server, how long should the ember try command wait to declare success?

[tomdale]

@kategengler In my experience, loading the app is relatively quick. I'd set the timeout at… maybe 10 seconds? Do note however that the build process may take longer, especially if there's a lot of JavaScript or SCSS to compile.

[kategengler]

See ember-cli/ember-try#53

Went with a 30s default timeout. The failure case I tested with ember-cli-mirage took more than 20s to fail. The timeout for success is less than ideal since it could "succeed" without ever having run the server, but, it's better than nothing. Open for other ideas.