Introduction

Click here to view the source code

Here is the past article related to Project Aggregation:

• Upgrade Projects: Project Aggregation

So, as I mentioned in my last blog post, I was working on a project involving numerous severely out of date .NET projects that I wanted to update. The caveat was I did not want to meticulously go through each one and upgrade them by hand, but instead to leverage Visual Studio’s SDK to do the upgrading process for me. Fortunately, there is a way to do this through the EnvDTE namespace and at this point I was able to first combine all the projects across various solutions into one single solution. After figuring this out I am now ready to start using this single solution to methodically upgrade each project as I see fit.

Project Details

After aggregating all my projects I was able to start working with the EnvDTE.Project class, but trying to figure out how to sift through the details of every project seem to be a bit problematic. It seems Microsoft does not actually give you deep tutorial details about how to mess with Visual Studio (VS) programmatically, which makes sense considering how easily you are able to shoot yourself in the foot (e.g. instances of VS staying open if not closed properly as mentioned in the last blog post). But I did manage to find a stackoverflow post that describes someone’s attempt to also change the target framework for their projects.

I went ahead and created an enumeration to represent all the possible framework versions that was currently available at the time of this writing (I did not bother to go below 3.5 considering I am trying to move towards progress not go backwards):

public enum TargetFramework
{
    [Description("3.5")]
    v3_5,
    [Description("4.0")]
    v4_0,
    [Description("4.5")]
    v4_5,
    [Description("4.5.1")]
    v4_5_1,
    [Description("4.5.2")]
    v4_5_2,
    [Description("4.6")]
    v4_6
}

At this point we can use much of the same implementation from our last post for manipulating VS projects and solutions. The one thing I included was a check for the Solution User Options (.suo) file. The file actually can become quite a necessity when aggregating a large amount of projects as it severely reduces the amount of loading times and manipulating projects for a solution. But this file is not always created if the application is programmatically opening the solution for the first time. For that reason, I attempt to force VS to create one by closing and reopening the solution:

var sourceDirectory = Path.GetDirectoryName(_solutionName);
var suoFiles = (new DirectoryInfo(sourceDirectory)).GetFiles("*.suo");
if (!suoFiles.Any())
{
    _logger.Log("No .suo file, closing and reopening solution");
    await CloseAsync();
    _dte = EnvDTEFactory.Create(_visualStudioVersion);
    await OpenSolution();
}

Just like with the aggregation process we are going to iterate through the upgrade process multiple times in case certain projects are not able to upgrade on a certain pass. At this point the upgrade process starts:

private void UpdateProject(ProjectWrapper project, TargetFramework framework)
{
    project.AttemptToReload();

    if (project.Project.Kind == Constants.vsProjectKindSolutionItems
        || project.Project.Kind == Constants.vsProjectKindMisc)
    {
        project.IsSpecialProject = true;
    }
    else
    {
        if (SetTargetFramework(project.Project, framework))
        {
            project.Reload();
            _logger.Log("Project Updated: {0}", project.Name);
        }

        lock (_nonUpdatedLocker)
        {
            _nonUpdatedProjects.Remove(project);
        }
    }
}

After initially going through the upgrade process I started receiving errors related to ‘Project Unavailable.’ After sifting online I found a solution to this issue by attempting to reload the project if it ends up unloading. The AttemptToReload() method simply attempts to access a property from the EnvDTE.Project and if it fails we will reload it. Although I found a solution to this problem it was off putting not to know why this problem occurs in the first place. But I did notice something after a project needed to be reloaded that it no longer had to be upgraded. So, it seems VS will automatically upgrade projects dependent on certain ones you are upgrading.

After some various trial and error, I noticed I was coming across special types of projects in the code base related to either a solution item or as miscellaneous. Given these types of projects have no purpose to upgrade, I simply marked them as special and move on (in case later I may want to use this information for some other purpose). Once we filter through the special projects, we can start setting the framework for a given project. The first thing we need to retrieve is the project’s target framework moniker, which is just the string value used to set the framework version, along with whether you want to set the project as a Client Profile:

private string GetTargetFrameworkMoniker(TargetFramework targetFramework, bool isClientProfile = false)
{
    var version = targetFramework.ToDescription();

    var clientProfile = isClientProfile ? ClientProfile : String.Empty;

    return String.Format(TargetMoniker, version, clientProfile);
}

So the return string value for wanting to set a projects framework for 4.6 as a non-Client Profile will be “.NETFramework,Version=v4.6”. From this point we will compare if the new target moniker differs from what is already set. If so, we go ahead and grab the project’s property related to ‘TargetFrameworkMoniker’ and set it with our new target monker:

private bool SetTargetFramework(Project project, TargetFramework targetFramework)
{
    var targetMoniker = GetTargetFrameworkMoniker(targetFramework);
    var currentMoniker = project.Properties.Item(TargetFrameworkMonikerIndex).Value;

    if (!currentMoniker.ToString().Contains("Silverlight")
        && !Equals(targetMoniker, currentMoniker))
    {
        project.Properties.Item(TargetFrameworkMonikerIndex).Value = targetMoniker;

        return true;
    }

	…
}

Once this process succeeds we will go ahead and reload the project again since after changing a project property it will tend to unload itself:

_project = (Project)((Array)(_project.DTE.ActiveSolutionProjects)).GetValue(0);

At this point, we are able to upgrade all projects programmatically. But there were quite a few issues I had to deal with that I could not figure out how to resolve through this process.

Issues

• One of the main issues I still received from my last blog post was the EnvDTE interfaces becoming busy, which would result in a ‘RPC_E_SERVERCALL_RETRYLATER’ exception message. Many AttemptTo methods still had to be used to retry the process to see when the solution would become available again.
• A problem I could not resolve programmatically was attempting to upgrade ASP.NET projects. Although I attempted to try and resolve this issue programmatically, since there were less than a handful of projects that needed upgrading I went ahead and upgraded them manually.
• There was also a weird error I ran into stating ‘Inheritance security rules violated by type: ItemsCollectionEditor. Derived types must either match the security accessibility of the base type or be less accessible.’ This was quite a difficult problem to try and hunt down since even finding the offending project seemed to jump around from time to time. I found this post related to the error, which suggested a couple ideas to manipulate the AssesmblyInfo.cs files. Another error I came across as well during this time was a ‘Operation not supported’ exception. And I did find another post for this error as well, but they mostly described an explanation to the issue rather than providing a solution. In the end, I found the arrangement of the projects seemed to only allow for proper upgrading while using VS 2013.

Conclusion

As long as it took to get to the point of being able to automate the upgrade process for all the projects, it turned out this was actually the easiest part of the entire process. Getting the solutions to build again with all the architectural changes made in .NET and making sure applications were able to run again still were not enough. Despite the effort put in, one of the ORMs used within the projects were left so stagnated that later versions were incompatible staying on the old current framework of .NET, but the latest version made such significant changes to their product that it was close to impossible to modify all the hundreds of files using it. Although there were solutions, unfortunately so much time had be spent at this point that the upgrade process was put to a halt. Fortunately though I managed to gain a greater appreciation for VS’s SDK and came out with a couple of highly useful tools for aggregating and upgrading future projects.

Introduction

Click here to view the source code

So, I came across quite an interesting situation approaching a new project that had been developed for several years. All of the library projects were never updated past .Net 3.5. Apparently, the developers did not want to take on the arduous task of not only updating all their projects, but also not deal with upgrading one of their ORMs, nHibernate, and the changes the software made between full releases. Which is understandable considering there were close to 200 projects and much of the code infrastructure was not encapsulated. This form of neglect seemed quite troubling to realize since these projects had to get upgraded eventually and left an absurd technical debt on future developers. Well, once the original programmers decided to leave I decided to take it upon myself to try and update these projects myself. Partly out of curiosity, but  mostly because I am a bit of a masochist. In order to start this process though, I decided on letting Visual Studio handle the upgrade process for me. In order to do that I first needed to figure out how to place all my projects into a single solution.

Project Creation with EnvDTE

The first thing I decided was, given the disorganization of the code base, manually upgrading projects was not realistic. Knowing that, I started figuring out how to automate searching through all projects and adding them to a single solution. I knew I would have to make use of Visual Studio’s SDK, more specifically the EnvDTE namespace, which is used for Visual Studio’s core automation process. So I went ahead and started creating a separate class library dedicated to using EnvDTE objects. The first thing I had to add was the base EnvDTE projects:

As I was going through the process of building out my code, I ran into an issue with the EnvDTE.Constants references:

With my lack of experience with Interop Types, I fortunately managed to find a solution here. Apparently Visual Studio just has an issue attempting to embed certain assemblies and the way to get around it is to set the property “Embed Interop Types” on the assembly to false. Through a matter of trial and error I managed to find the ‘envdte’ assembly was the one causing my issues and fixed this promptly:

With this I can now begin creating wrappers for the features I want to employ with the EnvDTE namespace.

Wrappers and Factory

Two of the main features I wanted to take advantage of the EnvDTE namespace are the EnvDTE.Project and _DTE.Solution interfaces. So I went ahead and created two class files to represent these; ProjectWrapper and SolutionWrapper. The ProjectWrapper class is fairly straight forward, which merely exposes one property of the EnvDTE.Project. I started out this path to decouple any issues I may run into for future projects:

public class ProjectWrapper
{
     private Project _project;

     public ProjectWrapper(Project project)
     {
         _project = project;
     }

     public string FullName { get { return _project.FullName; } }
}

As for the SolutionWrapper, much of the project aggregation process exists in this class. To start, we first must create our DTE object, which will contain the Visual Studio solution. I went ahead and place the implementation inside a factory:

internal static class EnvDTEFactory
{
    internal static DTE Create(VisualStudioVersion visualStudioVersion)
    {
        var vsProgID = visualStudioVersion.ToDescription();
        var type = Type.GetTypeFromProgID(vsProgID, true);
        var obj = Activator.CreateInstance(type, true);

        return obj as DTE;
    }
}

The VisualStudioVersion is an enumeration I created to represent the different versions of Visual Studio. The enumeration contains a description attribute that represents the Visual Studio version’s ProgID:

public enum VisualStudioVersion
{
    [Description("VisualStudio.DTE.12.0")]
    VisualStudio2013,
    [Description("VisualStudio.DTE.14.0")]
    VisualStudio2015
}

Since this automation process does not always perform consistently when attempting to add old projects into newer versions of visual studio, I had to have the flexibility to either use Visual Studio 2013 or 2015 (the solutions were at least kept fairly up to date).

Before we can begin aggregating all the projects into our solution, first we have to gather all the project files that exist in our root directory:

private IEnumerable<FileInfo> GetProjectsMissingFromSolution(string rootPath)
{
    var projectsInSolution = GetProjectNamesInSolution();
    var projectsInDirectory = GetProjectFilesInDirectory(rootPath);

    foreach (var projectFile in projectsInDirectory)
    {
        if (!projectsInSolution.Any(p => p.Contains(projectFile.Name)))
        {
            yield return projectFile;
        }
    }
}

As the method name GetProjectNamesInSolution suggests, we first look inside the given solution file, if it already exists, and read the contents through a stream to find all the current projects files. Afterwards, we find all projects in our root directory then ignore any projects that are already inside our solution.

At this point, we first need to open the solution to allow for the addition of projects:

_dte.Solution.Open(_solutionName);

When this occurs successfully, you can actually see the solution in your Task Manager Details tab, but not in the Processes tab. This is good to know in case the application closes abruptly. Although I do my best to clean up these solutions on completion, if you happen to force close the process, these instances will continue to exist until you clean them up yourself:

At this point we are ready to iterate through our projects and include them into the solution:

private void AddProjects(FileInfo[] missingProjects)
{
    foreach (var project in missingProjects)
    {
        try
        {
            AttemptTo(() =>
            {
                if (_projectWrappers.All(p => p.FullName != project.FullName))
                {
                    AddProjectFromFile(project.FullName);
                    _logger.Log("\tProject Added: {0}", project.Name);
                }
            }, 3).Wait();
        }
        catch (Exception)
        {
            _logger.Log("Skipping adding this project for now");
        }
    }
}

You will notice the method AttemptTo used throughout the SolutionWrapper class. This method will simply attempt to re-execute a given action an X number of times. The reason I need this is because as quickly as it may seem to execute actions through the EnvDTE interfaces, in the background sometimes the process takes longer than expected to complete your last request. So, when this occurs and you attempt to perform another action, an exception is thrown with this message:

The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))

Although I did find a solution, which involves using the COM interface IMessageFilter. Using this, I could read in the RetryRejectedCall event, or even the Servercall_RetryLater and force the thread to retry the event on its own without throwing an exception. Unfortunately I was never able to get this to work, so instead I accommodated by catching the specific exception, sleeping the thread for a few seconds, then trying again.

For each project, I only need to tell the solution to add it based on the file location:

var project = _dte.Solution.AddFromFile(fileName, false);

From here, I wrap the returned EnvDTE.Project and attempt to go through the aggregation process again if any projects were missed:

for (int i = 0; i < iterations && _missingProjects.Any(); i++) { _logger.Log("************ Attempt {0} ************", i + 1); _logger.Log("Number of projects missing from the solution: {0}", _missingProjects.Count()); AddProjects(_missingProjects); await AttemptTo(() =>
    {
        _missingProjects = _missingProjects.Where(p => _projectWrappers.All(pw => pw.FullName != p.FullName)).ToArray();
    });
}

The reason why I am going through the aggregation process multiple times is because some projects refuse to get loaded unless another dependent project is already included in the solution. When this occurs, sometimes going through the process again with the missing projects will turn out successful after X many iterations.

After all this we simply need to Save the solution:

_dte.Solution.SaveAs(_solutionName);

And finally close it completely, this includes both the solution and the DTE object that contains it:

_dte.Solution.Close();
_dte.Quit();

Then you’re finished. At this point you should be able to go to the location you specified for your solution, open it, and witness all the projects included.

Issues

Although this process works fairly seamlessly with newer, less complicated architectures, when I attempted to apply this solution to this projects code base mentioned earlier I ran into a plethora of problems and tiny gotchas.

• First of all, when attempting to aggregate almost 200 libraries it can take quite a long time to run, not to mention how many times you have to re-iterate over the project files that end up refusing to add because they need X, Y, and Z projects before they can be included.
• I also found a Solution User Options (.suo) file that was sometimes created after the process finished was absolutely necessary to open the solution again in a timely manner. Initially, efficient habits led me to remove this file when attempting to check it into source control. But the next time I tried to open the monolithic solution, it was taking almost twice as long as when I automated the process.
• I had to also specifically ignore certain project files because no matter how much I tried to fix the library itself, Visual Studio refused to include them into my solution. Some projects were just so old and unused that the IDE just could not understand them anymore.
• One of the biggest issues I was surprised I ran into was dealing with visual studio versions. I actually first started working on this problem under Visual Studio 2013, but as 2015 came out I naturally switched to that IDE, but found I could no longer aggregate more than 1/4 of all my projects. Even specifying ‘Visual Studio 2013’ as my vsProgTypeID made no difference. I had to both use Visual Studio 2013 and set my vsProgTypeID to be the same in order for this to work. It makes sense since the current Visual Studio may not know all the differences used from other versions. Nonetheless, this caught me off guard for a moment.

Conclusion

Overall, I am quite happy with the solution I made. I was able to aggregate all of the relevant libraries along with quite a few defunct ones, which I only figured out after noticing no usages for those projects. This was also a great debugging tool since so many common dependencies were sprawled across several solutions. If a change was ever made from one, it was extremely difficult to tell where that change would break elsewhere. With the aggregated solution, I was able to simply find usages of a change and make the appropriate updates. Also, at the time, I was able to leverage Visual Studio 2013 Ultimate’s Architecture Tools, which made weeding out unused libraries. Now that I was able to collect all my libraries into a single solution, the next step was to find a way to automate the upgrade process.