TFS 2010 Build ‘Partially Succeeded’ (Failed) without reporting any error

It sounds very strange when a build fails without giving any error. Recently while working with TFS 2010 build, I came across same kind of scenario. It took my lot of time to locate the exact cause of error. But finally, I succeeding in locating it.

Here is how summary report looks like for such a build. Showing no error at all. Still failed. Check-ins rejected. Very Bad.

 

 

 

Now, I thought of exploring detailed build log and look for a keyword ‘Error’. That worked and I found the error. That error was actually in Unit Test Run. As following image shows all tests have passed but still some error is reported by MSTest.

 

Now, I went back to the summary page to open unit test run results.

 

Here, highlighted section on test results window show that some other error occurred as all unit tests passed. So, I intuitively clicked that link and located the exact cause of the error.

 

So, that helped me to answer the strange question i started with.

 

 

Error MSB3482 : An error occurred while signing: Keyset does not exist

This is a build error got while building a .net project. This error basically indicates that certificate (.pfx) used for sighing the assembly has been expired. Make sure that certificate is renewed and error will disappear.

TFS 2010 : Troubleshooting high processor usage by data tier

As the development teams grow, load on TFS also grows. Being a comprehensive ALM solution, it acts as single point of service for all stake holders in your organization. Just like heart TFS pumps in/out information into the veins of organization. So, data tier is most vulnerable to become a bottle neck in case load increases beyond limit.

One fine day in our organization we saw that SQL server started consuming >95% avg processor time. Which was started hitting our whole ALM implementation. Developers were not able to checkin, Builds were damn slow, reports were too slow to fetch data. We were totally clue less about what was happening there. We shot arrows in dark, in all directions. We were also looking into scaling up data tier at one point of time. But after two weeks of trouble shooting we were able to nail the issue, which was primarily due to one of our custom solution we build around TFS.

During this trouble shooting I came across lot of interesting ways to probe in to performance issue with TFS. Here are few very important i would like to share.

Monitoring TFS in coming requests

This can be an initial point one must look at as soon as performance issue strikes. Interestingly, TFS keeps a record of all in-coming requests. This can give a quick input in case issue is due to the increase in requests to TFS due to the increase in number of clients using tfs. But there is a job which runs fortnightly and deletes all request logs older than 14 days. So, that is why I said “as soon as you hit the performance issue”. TFS SQL database table “tbl_Command” keeps log of in-coming tfs requests. You can query it any time to check those interesting details.

SQL Server Performance Dash board

This is a life saver tool, I must say. It can answer all your questions on which query/Stored procedure is the main accuse of creating the performance issue with SQL server. This is from Microsoft. It installs a set of reports on database machine which run on database’s system tables or Database Management Views. Microsoft did not released this for SQL server 2005. So it will not work directly after installation. You have to do some steps to make it work for SQL Server 2008. Just refer to following url. Its easy.

http://blogs.technet.com/b/rob/archive/2009/02/18/performance-dashboard-reports-for-sql-server-2008.aspx

Apart form these two methods we tried lot of other options, as I said “shot arrows in dark”. But, I am very sure that that methods mentioned above will give you enough inputs to resolve performance issue with TFS.

Troubleshoot : Perfmon.exe shows error message “Unable to connect to machine”

Lot of people face this issue which using performance monitor( perfmon.exe) while trying to monitor performance counters from remote machines. Here is the checklist which will help to make sure that you always able to access performance counters from remote location.

Access Permissions

Make sure the windows user who is trying to access performance counters from remote machine is local administrator of that local machine. As following screen shot highlights, just log on to remote machine and make sure the user who will access performance data is part of ‘Administrators’ group on remote machine.

If you have security issue in making user ‘Administrators’ then try to add the users in other highlighted groups. ‘Performance Log Users’ and ‘Performance Monitor Users’. But it is always preferable that user should be part of  ’Administrators’ group.

 

Remote Registry Service

Some times even after having enough access permissions on remote machine you may get an error ‘Unable to connect machine’. Or in case if you are using .net API System.Diagnostics.PerformanceCounter to access you will get ‘Network path not found’ (Win32Exception). So to resolve it just make sure that remote registry service is running on remote machine.

I am sure this will help.

 

Performance Counters Timeouts and Load Testing with Visual Studio

If you are getting following error while running load test and wondering why this happens. Here in this blog entry I highlight ways to fix it.

The performance counter category ‘Memory’ cannot be accessed on computer ‘COMPUTER01’ (Timed out trying to read performance counter category ‘Memory’ on computer ‘COMPUTER01’); check that the category and computer names are correct.

Performance Monitor Tool (perfmon)

This is a well known Windows in-build tool for fetching performance data. First try to fetch the required performance counter from specific machine through this tool. You may notice two things here if your machine is a remote machine. If it takes more time (more than 1/2 minutes) to get the performance counters, that will indicate that you need to change timout values for load test configuration. OR if it does not access the counter then there are permissions issue.

Performance Monitor User Permissions

If you are not able to access performance data from perfmon, then you have to correct that first. As same permission level applies to visual studio load test when it tries to fetch performance data. User who is trying to access performance data (locally/remotely) should be part of  ”Performance Monitor Users” and “Performance Log Users”. If there are not much security concerns make the user part of Local “Administrators” group which will ensure full access on the machine.

Sometimes still there may be some issues. In that case just remove the User or the Group of which the User is part of, from “Users” Group of target machine. Which will provide you fully un-restricted access to machine.

Firewall Exceptions

Rarely you may have to do it. Either turn off the local firewall or enable the rules for Performance Logs and Alerts in the Windows Firewall with Advanced Security snap-in.

Load test timeout settings

If permission levels are all fine. But still you are still running into the same issue. That means you have to set load test’s performance counter initialization time out settings. But where to do that? as there is no provision to do it though VS IDE. For that you have to do some manual configurations in load test’s hosting process’s configuration file. Now this configuration file is different for VS 2008 and VS 2010. Following are the details on that.

For Visual Studio 2008, open the file VSTestHost.exe.config in the folder C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE

OR

For Visual Studio 2010, open the file Devenv.exe.config in the folder C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE

AND then add following entries to it. That will solve the timout issue.

<appSettings> 
     <add key=”LoadTestCounterCategoryReadTimeout” value=”60000″/> 
     <add key=”LoadTestCounterCategoryExistsTimeout” value=”60000″/> 
</appSettings>

 

Debugging an Addin in visual studio

Following are the steps :

1. Delete all bin and obj files
2. Copy paste any snk file where it is required
3. Set appropriate debug type you want for the project
4. Right click on addins project , set “addin project” as startup project
5. Right click on “addins project”, select properties and set as shown in the screen shot below
6. Set Start external program C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\devenv.exe
7. Set Command line arguments /resetaddin <addin lib name without extention>
8. Set working directory C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE
9. Create folder addins in C:\Users\<user>\Documents\Visual Studio 20101\Addins
10. Copy paste the addin definition file and set the parameter <Assembly> as <Assembly>Path to AddIn.dll</Assembly>
11. Make sure that in addin definition file value of <FriendlyName> node should be exactly same as that of addin definition file name.
12. Run the project in debug mode.

Courtesy : http://blogs.msdn.com/b/syedab/archive/2009/06/22/debugging-an-addin-in-visual-studio.aspx

Troubleshoot : TFS Build 2010 : Build agent is currently reserved for a build

I configured three build agents on the same machine with same configuration (lets say A1, A2, and A3). I reserved A1 and A2 for some builds say B1 and B2 using build definition process configuration. Everything worked fine till I deleted those builds (B1, B2) as I was experimenting with them. Now once I deleted them, reserved build agents A1, A2 were there but were not doing any thing. For example if I queued 3 builds at a time they were being processed by only A3. So I thought of deleting these build agents. Adding salt to wound I was not able to delete that, showing a dialog with message

---------------------------
Team Foundation Build Configuration
---------------------------
Cannot delete agent 'A3': The build agent A3 cannot be removed from build controller C1 because it is currently reserved for a build.
---------------------------
OK
---------------------------

On my pursuit to solve this issue, I ended up exploring TFS 2010 db for my project collection and here is what I found.

“tbl_BuildAgentReservation” is the table which retails the reservation details.

“tbl_BuildAgent” the refers to PK of  tbl_BuildAgentReservation for reservation ids against build agent entries.

So, these two tables maintains that blocking link. Once you break it, issue is resolved. And I did the same. Following are the queries you can run to break this reservation.

SELECT * FROM tbl_BuildAgent

Locate the build agent in question then note the value of ReservationId and AgentId and Execute the following SQL statements:

DELETE FROM tbl_BuildAgentReservation WHERE ReservationId = [ReservationId]
UPDATE tbl_BuildAgent SET ReservationId = NULL WHERE AgentId = [AgentId].

Courtesy :

http://sleepcanwait.blogspot.com/2010/07/cannot-remove-build-agent-from-build.html

http://www.anujchaudhary.com/2011/06/tfs-2010-unable-to-stop-build.html

 

 

 

Troubleshoot : Visual Studio Web Test – Response body capture limit

While working with visual studio webtest for load testing, I came across this issue where response of the web request was cut short because of the default limit set for that by visual studio. Actually I was using some extraction rules to fetch some values from the response of that particular request. Following snapshot shows webtest playback window. The highlighted response tab  shows “(Truncated)” to indicate that the response from the web request has been cut short.

Webtest response truncated

Basically, the default response size limit set by visual studio is 1.5 MB. The setting can not be changed from webtest/web request property window. It can be overridden either through coded version of the web test or using web test plugins. Following code snippet shows the webtest plugin method. Though, it will be the same property which should be set while working with coded web test.

public class WebTestPlugIns : WebTestPlugin
{
    public override void PreWebTest(object sender, PreWebTestEventArgs e)
    {
       e.WebTest.ResponseBodyCaptureLimit = 3000000;
    }
}

In this code snippet I am overriding the response capture limit to 3MB.