Six months ago we started looking into finally upgrading our 2010 TFS instance at work to 2013. After several months of research and issues Microsoft released the CTP of TFS 2015 so we decided to go all the way to the bleeding edge. We expected this to be a pretty quick project but like most we battled with many issues. One particularly difficult issue was our Feature Test builds. We are finally migrating this weekend so I though it would be good to share some of our experiences and also document our findings for potential future use.
Changes in TFS 2015
One of the big changes in 2015 was an overhaul of their build system which also effected test components. 2015 does still support the old XAML based builds from previous versions of TFS but also adds a new more feature-rich build system with cross-platform capabilities. They combined build and test agent roles into a new 2015 build agent that handles both replacing Build/Test Controllers/Agents. In general, it seemed like a good change and better system.
Test Agents
Using 2013 Build and Test Controllers/Agents in TFS 2015
Our initial goal however was to defer as much work as possible and migrate as much as we could, as easily as possible. So we thought we might need to setup 2013 Test Controllers and Agents to avoid recreating our builds hoping for the most capability with our old setup. Initially we used the 2015 build system but used 2013 Test Controller and Agent. This produced many different errors but in the end we found a few changes were required.
First we had to use a 2013 Build Controller/Agent to avoid an error which indicated it couldn’t find the built test project. This make sense if you think about it because it was built by 2015 but tests were run by 2013.
Unable to get type MyFeatureTestProject.MyTestClass, MyFeatureTestProject, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null. Error: System.IO.FileNotFoundException: Could not load file or assembly 'file:///C:Builds4MyTeamProjectMyFeatureTestBuildBinariesMyFeatureTestProject.DLL' or one of its dependencies. The system cannot find the file specified.
We also had to update the builds to use the 2013 XAML build template and update our .testsettings files to reference the new test controller.
This was more work than we anticipated so we also started evaluating the new 2015 build agents.
TFS 2015 Build Agents
We found most of our old migrated builds worked perfectly fine in the new system. This included standard compilation, unit test, gated checkin and deployment builds. The outliers were our FeatureTest builds. These used test settings files which pointed to our older test controllers to run the tests. Since 2015 doesn’t have Test Controllers these wouldn’t work.
The good news was we only had 6 Feature Test builds so it wasn’t that big a deal to recreate them using the new build system. Most of the configuration was very similar to the old build configuration.
This approach also had the added benefit of skipping all the 2013 controller/agent setups. This also includes installation of older versions of VS on the test agents.
CodedUI References
Across our six feature test builds we were using CodedUI versions from Visual Studio 2010, 2012, 2013 and 2015. So without any changes the first error we hit referenced not finding an older version of MSTest/CodedUI references:
Could not load file or assembly ‘Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified.
We just needed to update the projects to reference the newer versions of these assemblies using the following assembly bindings in the test projects config file:
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.QualityTools.WindowsStore.CodedUITestFramework" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.TestTools.WindowsStore.UITest.Common" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.TestTools.WindowsStore.UITest.Extension" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.TestTools.UITest.ExtensionUtilities" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.TestTools.WindowsStore.UITesting" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.TestTools.WindowsStore.UITest.Playback" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.TestTools.UITest.WindowsStoreUtility" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.TestTools.WindowsStore.UITest.Framework" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="UiaComWrapper.WindowsStore" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.QualityTools.CodedUITestFramework" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.TestTools.UITest.Common" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.TestTools.UITest.Extension" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.VisualStudio.TestTools.UITesting" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="UiaComWrapper" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
</dependentAssembly>
</assemblyBinding>
</runtime>
Interactive Mode
One of the next issues we hit was about running in interactive mode.
Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: Automation engine is unable to playback the test because it is not able to interact with the desktop. This could happen if the computer running the test is locked or it's remote session window is minimized.
Interactive mode allows CodedUI tests to interact with the desktop to open a web browser for example. Enabling this mode was just a checkbox in the config for older test agents but 2015 didn’t have this option. So, I reached out to Donovan Brown (a DevOps Senior PM at Microsoft) for help on this one and he delivered. Read his blog post to enable interactive mode on a 2015 TFS build agent.
Data Driven Tests
The majority of our CodedUI feature tests are also data driven using the DataSource attribute connected to TFS to pull data from Test Cases. For example,
[TestMethod]
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase", "http://mytfsserver:8080/tfs/MyProjectCollection;MyTeamProject", "1234", DataAccessMethod.Sequential)]
public void service_should_return_valid_json()
{
These tests resulted in the following error:
[error] The unit test adapter failed to connect to the data source or to read the data. For more information on troubleshooting this error, see "Troubleshooting Data-Driven Unit Tests" (http://go.microsoft.com/fwlink/?LinkId=62412) in the MSDN Library. [error]Error details: Unable to find assembly 'Microsoft.TeamFoundation.Common, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
This sent me down a rabbit hole at first trying to figure out why the Microsoft.TeamFoundation.Common assembly couldn’t be found. I found that only the 2010 version of this assembly was in the GAC and new versions weren’t. So at first I thought that was the issue and I created a Nuget package with the assembly to resolve the reference issue but in the end found that didn’t fix it.
I finally figured out that the TFS connection string was still pointed at my old TFS server and that was causing my issue. All I had to do was update this connection string. I am a dummy!
We also found some of our CodedUI feature test classes were decorated with the [TestClass] attribute instead of the [CodedUITest] attribute. This apparently worked fine in 2010 but starting in 2012 caused errors in conjuction with the DataSource attribute.
Lower Integrity Level
We also received the following error randomly on some but not all tests/projects:
Microsoft.VisualStudio.TestTools.UITest.Extension.PlaybackFailureException: Microsoft.VisualStudio.TestTools.UITest.Extension.PlaybackFailureException: The Internet Explorer process appears to be running at a lower integrity level than the current process. Please run Internet Explorer at the same integrity level and try again.
In the end not exactly sure why it seemed to be internment but changing IE to run as Admin on our agents seemed to fix this error. We found some details on the MSDN regarding this.
Additional Tips
Here are few tips we came up with that may help others get to the root causes of their errors more quickly:
Make sure you get latest and overwrite your local files
I spun my wheels for a while on this one where I thought I had changed and checked in a change to the test project in the new TFS instance but Visual Studio was confused on my source between the old and new TFS server and it didn’t check in the changes. I suspect this is because I didn’t change my workspace local path.
Create a sample CodedUI feature test project which mimics your actual tests to test your agents
Most feature test builds take hours to complete in my experience. That feedback loop is FAR too long for testing TFS configuration. We setup a project with a single feature test that
- Opened a web browser – tests interactive mode and browser settings
- Pulled data from a test case using the DataSource attribute – tests data driven tests
- Navigated to one of the sites we want to test – tests network/firewall issues
- Ensured it could login to the site – tests authentication
Run this on your agents before wasting a bunch of time on a long test builds. We eventually plan on writing Chef cookbooks to verify/configure our agents but that’s another post for another day.
Conclusion
Although it took much longer than we thought we are really excited about all the new features in TFS 2015 and plan to stay up to date with further releases of TFS in the future. Hopefully some of this post helps you on your TFS migration journey.
Let me know I you find any other common issues/tips and I will add them here as well.
This is a great post with very useful information. Probably the most important instruction you provided is to stay technically current. Additionally, I must point out that stating “I am a dummy” solidifies that you are at the top of the infrastructure food chain and one of the most valuable leaders in the business! Thanks for taking the time to compile the findings of your efforts.
Happy TFS’ing,
THank you very much for this post. It helped immensely in the conversion from the older tfs to 2015. Much APPRECIATED.
When you ran the coded ui on your testing machine. Did you run the code on the build server or on a test agent. My configuration is requiring me to run it on a windows 10 test machine. So question is the interactive login that donovan speaks to. YOu did that change on your build server because the tests are running there correct? Or did you have to enable what he details on the build agent and the machine being used for test
The CodedUI tests are run on the agent so the interactive login has to be setup on the agent.
Great article! One question on the Coded UI Tests. I’m getting that integrity level message. You noted “changing IE to run as Admin on our agents” – was that just by changing the Internet Options or did you have to change something in the agent?
It wasn’t clear.
Thanks
Never mind – I found the option – it wasn’t immediately available when I right-clicked but it was there the next time. (doh!) Thanks
Glad it worked for you!
Thank you very much! Great article, good job.