Support for testing Windows Store apps

[vjrantal]

Issue

Windows Store apps can already be built in the AppVeyor servers, but running them doesn't seem to be possible. Based on the discussion at http://help.appveyor.com/discussions/problems/395-no-tests-discovered one of the main remaining issues is the need for a developer license https://msdn.microsoft.com/en-us/library/windows/apps/hh974578.aspx.

Solution ideas

There is a possibility to configure developer license to domain-joined machines https://technet.microsoft.com/en-us/library/hh852635.aspx#BKMK_DeveloperLicense, but an alternative could be supporting a scenario where acquiring the license is automated using an automation framework. A proof-of-concept project can be found at https://github.com/vjrantal/DeveloperLicenseAcquirer , which has been tested on Windows 2012 R2 server (outside of AppVeyor).

The missing piece from AppVeyor servers is the support for the "Desktop Experience"-feature that is required for the developer license UI flow. Currently, that feature doesn't seem to be enabled in AppVeyor - see line 251 from https://ci.appveyor.com/project/vjrantal/cordova-plugin-alljoyn/build/1.0.38#L251.

If that feature would be enabled, it might be possible to automate building and running (in the same AppVeyor build server) a Windows Store app, because it seems like the certificate acceptance can also be automated with a script something like https://github.com/vjrantal/cordova-plugin-alljoyn/blob/appveyor-experiments/tests/InstallCordovaCert.ps1.

To avoid leaking private information, the developer's username and password (used during the license acquisition) could be passed to the AppVeyor server as Secure Variables http://www.appveyor.com/docs/build-configuration#secure-variables .

[FeodorFitsner]

Hi Ville,

Thanks for the details provided!

Looks like Desktop-Experience feature is enabled on Pro environment build workers: https://ci.appveyor.com/project/FeodorFitsner/simple-console

I can switch your account to Pro to test your solution and see if it works. Then we'll add Desktop experience to Azure build images.

[vjrantal]

Thanks!

I can try it out as soon as the account switch is done. Will I see the change reflected at https://ci.appveyor.com/plan?

[FeodorFitsner]

Yeah, I've just changed it to "Pro trial". it will get back to Free in two days.

[vjrantal]

Got the UI Automation up and running in AppVeyor, but for some reason, I don't seem to get forward from a step where the license server is contacted and a after that, a window with embedded Web view is presented to the user.

The things that came to my mind are:

• AppVeyor is not able to connect to the license server
  • Is there any restrictions on where connection can be made?
• The window with a Web view fails to load
  • Again, a connection issue or restrictions about using the Web rendering engine in the AppVeyor servers?

Is there any way to get a screenshot from the build server? That might help seeing what is going on in there 😃

Here is the latest trial as reference https://ci.appveyor.com/project/vjrantal/cordova-plugin-alljoyn/build/1.0.60 but it is probably not very helpful, because those are just internal debugging messages that I write from my UI Automation app.

[FeodorFitsner]

There is no restrictions as far as I know. Maybe if you connect via RDP to the build worker will help you troubleshoot the issue.

UPDATE: Updated wrong link.

[vjrantal]

RDP was super-helpful and I immediately saw what the error message was:

I actually found a workaround to this from https://social.msdn.microsoft.com/Forums/en-US/26835764-8d87-4fa5-8bac-0750528d6fd0/we-couldnt-get-your-developer-license-for-windows-81?forum=csharpgeneral which got me forward all the way to the point that I was able to install my app successfully! See line 644 from https://ci.appveyor.com/project/vjrantal/cordova-plugin-alljoyn/build/1.0.64#L644.

However, the next (perhaps final) obstacle is that Store apps ("modern apps") can't be run by administrator by default. The issue and workaround can be found at https://www.vpsblocks.com.au/support/Knowledgebase/Article/View/211/8/windows-2012-r2-internet-explorer-cant-be-opened-using-administrator-account, but from my perspective, the problem in the solution is that it seems to require a reboot of the machine so can't really test if this works or is the final issue blocking the E2E test flow.

I was wondering would it be possible to configure the policy to allow apps to run by the AppVeyor team? Alternatively, I'll try to find out if there is any way around this without a reboot.

[FeodorFitsner]

That's great news, you're almost there!

So, what policy do you need to configure - could you please elaborate?

[vjrantal]

Basically, I think steps 1-6 under title "Or alternatively modify Windows security policy" from https://www.vpsblocks.com.au/support/Knowledgebase/Article/View/211/8/windows-2012-r2-internet-explorer-cant-be-opened-using-administrator-account would be needed.

[FeodorFitsner]

Well, but AppVeyor build agent runs under appveyor account, not a built-in Administrator account.

[gep13]

@FeodorFitsner said...
Maybe if you connect via RDP to the build worker will help you troubleshoot the issue.

Is this available to everyone using AppVeyor, or only those that are on the higher plans? This is an amazing feature!

@vjrantal sorry for butting into your thread 👍

[FeodorFitsner]

Yes, for now it's available for Pro plan only. For Azure there are some technical questions that must be solved first.

[vjrantal]

Well, but AppVeyor build agent runs under appveyor account, not a built-in Administrator account.

I believe the issue is that it is forbidden to run Store apps with an admin account that has User Account Control (UAC) disabled, i.e., there are no prompts if, for example, you run new command prompt as admin. The appveyor account is such.

I did verify on my own Windows Server 2012 R2 that if the policy is changed according to https://www.vpsblocks.com.au/support/Knowledgebase/Article/View/211/8/windows-2012-r2-internet-explorer-cant-be-opened-using-administrator-account, the Store apps can be run. As mentioned, the reason I can't try this in AppVeyor is that it requires a reboot.

However, the challenge with enabling UAC "globally" is that it might be a breaking change for some automation that doesn't assume UAC prompts.

I tried to find a workaround, for example, creating a temporary user (https://github.com/vjrantal/cordova-plugin-alljoyn/blob/appveyor-experiments/tests/CreateTempUser.ps1) and running my app as that user with something like "runas /user:temp-user ...", but that doesn't seem to work. Perhaps it would be required to first really sign in as that user and then start the app.

I don't currently have good ideas about what to try next so would appreciate if anyone else has any suggestions..

[FeodorFitsner]

Yeap, enabling UAC globally would kill a lot of builds.

Why would you need to login with created user - creation of user profile? If so, it could be easily done with WinAPI. I can give you a working C# code for that if you're willing to try.

Restarting VM is an interesting option too and would give people new scenarios. I added an issue for that: #204

[vjrantal]

I am actually not sure why the app couldn't be run as the temporary user and based on the error message, it was more about permissions than missing user profile. Here is the error I saw:

I think this could be still looked further to figure out why exactly the failure comes and if it would be a supported scenario to run Store apps under different user while logged in as admin. If I am not mistaken, it is this https://msdn.microsoft.com/en-us/library/windows/desktop/hh706903%28v=vs.85%29.aspx that gets called on the line from which the error comes.

Being able to reboot should be one solution since it is easy to script changes needed to registry keys and security policy configs.

[FeodorFitsner]

OK, have you tried adding that user to "Administrators", checked ACLs to app file? You may also try using ProcMon to see where the error comes from.

[vjrantal]

OK, have you tried adding that user to "Administrators", checked ACLs to app file?

Making the temporary user an administrator didn't change the error message.

I did test that if I sign in "normally" as that temporary user, I can start the Store app that I had installed while signed in as the built-in administrator account. Also, I became more confident that this isn't about user profiles since the same error is there even though I had properly signed as the temporary user so profiles and such should be there.

I'll try to find out if I am trying something that is by design not supposed to be possible.

[FeodorFitsner]

Thanks, keep us posted!

In the meantime, I've started thinking on reboot feature design. Seems like we could have something working by the next update (April 3).

[gep13]

@FeodorFitsner said...
Yes, for now it's available for Pro plan only. For Azure there are some technical questions that must be solved first.

No worries, just thought I would ask, as this really would be a killer feature for troubleshooting purposes.

[sreesharp]

Didn't go through the actual issue. But if it is something to do with store app installation error due to policy, please check out this blog http://sreesharp.com/clickonce-deployment-for-enterprise-windows-store-apps/

[vjrantal]

please check out this blog http://sreesharp.com/clickonce-deployment-for-enterprise-windows-store-apps/

I believe this post would contain relevant instructions assuming AppVeyor build machines would support the enterprise sideloading. The blog post says "You should get the sideloading product key for your enterprise Windows licensing team", but to get there, I think the AppVeyor team would have to purchase the required keys and configure the machines according to the requirements (machines must be domain-joined). Based on http://blogs.windows.com/itpro/2014/04/03/windows-8-1-update-sideloading-enhancements/ the price for the keys would potentially be fairly low.

The thing that the enterprise sideloading would solve is that then, there would be no need to go through the developer license acquisition for every build, which would result into more reliability and faster build build times. So overall, it would be a very welcome feature in my opinion.

However, I think the enterprise sideloading would not solve the most burning issue we have at hand which is that Store apps can't be run as the user available on the AppVeyor servers. For that, one idea on this discussion was supporting a "reboot scenario" where we could build and install the app and configure the machine as administrator and then use another user account to run the app.

[vjrantal]

Restarting VM is an interesting option too and would give people new scenarios. I added an issue for that: #204

The support is now there and I tried it out, but it seems like what I am trying to do isn't yet end-to-end working.

The reboot support works on its own, but the issue is that to run Store apps, I need to tweak the security policy, which in turn might affect the AppVeyor "services/agents" from booting up properly after the restart - just a guess though.

Here is an example of a build that times out, because it doesn't continue after the reboot https://ci.appveyor.com/project/AllJoyn-Cordova/cordova-plugin-alljoyn-d3t62/build/1.0.16#L429. The line where I tweak the policy is https://github.com/vjrantal/cordova-plugin-alljoyn/blob/appveyor-testing/appveyor.yml#L22.

If I manually RDP to the build machine after the it is rebooted, I can see that the machine indeed rebooted and I can run Store apps (i.e., the security policy change got applied). However, as said, seems that the AppVeyor build doesn't continue properly after the reboot so my flow doesn't yet work from start to finish.

[FeodorFitsner]

I'd like to fork your repo and investigate why the build isn't restarting. What commands can I safely remove from appveyor.yml to avoid dependency on encrypted variables?

[vjrantal]

investigate why the build isn't restarting

If my guess on why the issue happens is right, you should be able to use your example appveyor.yml from #204 and just add right before Restart-Computer line:

- PowerShell Set-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System" -Name "FilterAdministratorToken" -Value 1

If this doesn't reproduce the issue, then it is something else that I can investigate further and try to reduce as a simpler test case.

[FeodorFitsner]

Cool, will give it a try.

[FeodorFitsner]

I'm running the following sequence on Pro environment (with interactive build worker process): https://github.com/FeodorFitsner/simple-console/blob/reboot-test/appveyor.yml

Without Set-ItemProperty ... eberything works fine, but with it AppVeyor.BuildAgent.Interactive process crashes in few seconds after reboot:

Perhaps, we could return to the idea of creating a separate user account for running Store apps.

[vjrantal]

Perhaps, we could return to the idea of creating a separate user account for running Store apps.

I have now tried pretty much everything I can think of to go via that route, but no success.

@FeodorFitsner Do you happen to know what AppVeyor.BuildAgent.Interactive is trying to do when it crashes? I wonder if that is something that is mandatory for the agent's functionality or could that part be "try-catched" and the agent could still run?

[FeodorFitsner]

Do you see something related in Application event log?

[vjrantal]

Do you see something related in Application event log?

It is hard for me to debug the build agent crash, because to reproduce, I need to reboot the build server, but since the agent doesn't come up, the build fails and the VM gets shut down so can't RDP into it anymore.

[briandunnington]

I know this thread is getting a little old, but what was the final outcome of all of this? I just tried to rebuild/run my original Win8.1 app & tests and it builds completely, but it never discovers any tests (same behavior as the original issue: http://help.appveyor.com/discussions/problems/395-no-tests-discovered).

On a slightly-related note, I saw here (http://help.appveyor.com/discussions/problems/2702-vs2015-uwp-project) that support for UWP was going to be 'installed this week' on Aug 7th. Was that already done, and if so, can Win10/UWP apps be built and tested now?

[FeodorFitsner]

Well, seems like the issue has not been resolved as there are technical difficulties to do that in stateless build environment.

[sgrebnov]

Another thing to be aware of is that Windows Store apps don't run w/o active desktop or remote session out of the box. We use some trick to enable special debug mode to get it working on our CI:

apache/cordova-medic@49da31f#diff-02ebb0a2743750f76789021349d1f4c9R219

https://github.com/apache/cordova-medic/blob/master/lib/patches/EnableDebuggingForPackage.ps1