The Atmosera DevOps Toolkit is a set of building blocks that can be leveraged to simplify operations involving various tools commonly found in software development environments. These involve Work/Bug tracking systems, Source Control Systems, Build Systems, and others. It can be considered along the lines of being a bunch of Legos™ with many different blocks of various shapes and sizes, along with the ability to create custom blocks that can be built up to perform simple or complex operations on data stored in various tools including migrations, transformations, archiving, etc.
Typical engagements have unique considerations, so the building blocks are continually expanded and built upon. This block-based approach has a much higher re-use rate [and therefore the potential for cost amortization] than attempting to build out a complete “tool” [see the history section for more information].
Capabilities and Limitations
An important consideration in any migration/transformation effort is the ability to accurately represent the data in the new system, this is referred to as the “fidelity” of the effort. Determining the cost and ROI for differing levels of fidelity is a key part of ensuring level expectations on the outcome [and also why trial migrations are so important for empirical validation].
In many cases, the target system does not expose APIs or other means to import certain data or limit the quality of the data. A simple example is “Build Runs” in nearly every tool…While it is often possible to migrate the build definition, the history of executions simply does not have a means of being imported. In many cases, it is perfectly acceptable to “leave this behind” [low fidelity], but in other cases, creating a type of archive of the information may be desirable.
Even when API(s) exist, there may be differences that are “reductive” and can not be accurately mapped. Consider an input with values 1 through 5 as a priority and a target of only high, medium, and low…. The toolkit can be configured to map these values but resolution will be lost as there are only 3 possible outputs for 5 inputs.
Technical Intro
The toolkit uses several C# projects written using the latest .NET 8.0 with many C# 11 and 12 features. Each tool has an adapter assembly, there is a persistence assembly [currently Entity Framework-based backed by SQL Server, along with the core assembly and appropriate harness and testing assemblies.
The design’s key goal is to encapsulate each tool’s details and the mechanics of communications [often ReST APIs] from the higher-level goal-focused custom application. While C# programming skills are needed, significant focus has been on keeping this aspect as simple as possible to reduce the barrier to usage.
The block diagram below provides a high-level view of the typical data flow when used in a migration situation:
History
Microsoft Team Foundation Server was introduced in 2005. At the time a wide variety of source control systems were in use, including VSS, CVS, SVN, Mercurial, Perforce, and others. Importing from these systems, as well as various sources of work planning and tracking, was a very active area. Microsoft created the “TFS Integration Platform” [TIP] to address this. Unfortunately, the tool had several issues and quickly collapsed on itself.
David Corbin was an independent consultant performing significant work in this area and quickly realized that while something was needed that could amortize development costs [compared to full custom solutions], it was also important to introduce an intermediate generalized/abstracted datastore. Since adapters for each tool interacted with this store, and not with other tools, they could be configured in a manner that facilitated tool->tool endpoints and the introduction of both new capabilities and tool support.
That codebase was used on many engagements over the next (approximately) 13 years. Like many efforts, it accumulated technical debt [quick fixes for a given engagement that were never refactored into stable designs]. It was based on old (often deprecated but still operational) APIs. It was retired (used for the last time in a production capacity) in 2019.
Sample Use Cases
Migration between tools
Jira Issues -> Azure DevOps
Azure DevOps -> GitHub
GitHub -> Azure DevOps
Import of Other Tools
Transformation within a Tool
Azure DevOps Inter-Organization Operations
Archiving of Historical Information
Source Migration TFVC ->GIT, SVN->GIT
Atmosera is currently leveraging the toolkit to provide operational excellence to our clients. We have seen great success in utilizing the toolkit alongside our professional services engagements. Ranging from Bitbucket, and Team Foundation Server to Azure DevOps Services migrations and GitHub-related engagements in automation and services tracking.
A particularly complex engagement around Team Foundation Server migrations allowed Atmosera engineers to put the toolkit to the test. During the engagement, the toolkit was able to scrape together all the required data analytics to perform the migration. This data was then collated and utilized to develop a migration strategy and feasibility study for the organization. This work would normally take several weeks to complete with all parties involved. The toolkit allowed us to gather and collate this information within days and get the project moving forward. The subsequent milestones of the engagement then focused on actioning the data which the toolkit assisted in aggregating. With Atmosera developers customizing the toolkit we then began to action the migration of repositories and related metadata. The ability of our DevOps Engineers to utilize a single toolkit to go from the very beginnings of planning through execution to validation has proven time and time again to be extremely effective and efficient.
The toolkit is actively developed by our DevOps specialists here at Atmosera. We plan to continue to integrate further into the GitHub Ecosystem and Bitbucket Systems as the next phase of its lifecycle. As we build out the toolkit and advance its capabilities we will expand its migration and data analytics capabilities. If you are interested in understanding how you may be able to leverage our DevOps Professionals and gain the advantages of our toolkit, feel free to reach out!