Scoping of acceptance tests - adding CLI acceptance tests

[patrickjahns]

Currently we are having automated acceptance tests for API and UI.

However we are missing some form of acceptance tests for running CLI commands.

I propose to add the additional CLI scope - and start to add some acceptance tests for running certain CLI commands

In the end we will have then 3 major acceptance test suites:

• API
• UI
• CLI

[phil-davis]

Agree - and it will be easy to add tests for occ commands because we already have the test infrastructure for doing such commands and examining their output.

When making tests, we should check the "result" of the command by all of:

• check the command exit status is as expected
• check the command output is as expected - e.g. human-readable stuff like The user "aaa" was created successfully
• check that the thing actually happened - e.g. use the APIs to check that the user/group/file/share/whatever actually exists or is gone... This makes sure that the CLI command is not just a pretty NOOP that emits nice text messages, but that it actually does something in the back-end.

[phil-davis]

@paurakhsharma remind me to talk with you about this. It might not take long to get a start on this. The tests would be quick-running, and would add value against regressions in an area where manual release testing currently only covers "accidental" uses of occ commands that are done when settting up for other tests.

[patrickjahns]

During experimenting with building phars - I wrote a quick CLI tester - https://github.com/patrickjahns/cli-phar-experiments/blob/master/features/bootstrap/FeatureContext.php

Most of it was inspired by behats own FeatureContext
https://github.com/Behat/Behat/blob/master/features/bootstrap/FeatureContext.php

Should get you started fast - recommend to always use the Symfony Process component as its a high enough abstraction to work with output.

If you don't want to use behat - symfony provides also a commandtester class as helper:
http://symfony.com/doc/current/console.html#testing-commands

It might also be worth checking if someone wrote a bridge for commandtester and behat

[phil-davis]

There are already some home-grown acceptance test methods that return the command status and output. I will look if they already use any of the "standard" stuff above, if we can/should change them to be more "standard"...

[patrickjahns]

https://symfony.com/doc/current/components/process.html -> very nice process handling component ;-). Not sure if we used it in the wrapper in the Testing app - but might also be worth using it there

[phil-davis]

cli acceptance test suite type now exists in core. See core issue owncloud/core#33051 for details of core occ commands that need to be tested.

Other apps that implement occ commands of their own will need tests added also.

[phil-davis]

@patrickjahns I added issues in each app repo that seems to have app-specific occ commands. Those can be assigned to future sprints as resources and priority allow.

The infrastructure is now all there in core, and test scenarios are being added day-by-day. So the "bootstrapping" of this acceptance test functionality is done.

Close this issue?