RESOURCES HOME
Microsoft Azure DevOps is a tool that supports software development teams from ideation through deployment and ongoing enhancements.
Here we distill some best practices and tricks of the trade from having used the tool over many years. This guide aims to help teams new to Azure DevOps get started, as well as teams that want to migrate or merge DevOps projects into a clean new instance. This best practices guide for getting started with Azure DevOps will cover:
- Security and Teams
- Project Management
- Code Repos
- Boards and Agile Project Management
- Pipelines and Test Plans and Suites
- Project Migration
Microsoft provides a good set of documentation for Azure DevOps. The parts referenced within this article are provided throughout, as well as at the end of the document.
Security and Teams
Azure DevOps security can be managed within itself. Microsoft recommends connecting Azure DevOps to Microsoft Entra. Microsoft reiterates the need to only give users and services the minimum amount of access needed to perform their business functions.
Summary of Security Considerations:
Disable “Allow public projects” in your organization’s policy settings to prevent every organization user from creating a public project.
Only allow specific users to create new projects.
Block external guest access entirely by disabling the “Allow invitations to be sent to any domain” policy.
Keep groups as small as possible. Access should be restricted, and the groups should be frequently audited.
Team Management Considerations:
- Keep teams small, define a team for each development group of six to 12 developers.
- If you have several development or feature teams, it is recommended that a team is defined for each feature.
- Configure development teams to support roll-up to project management feature teams.
Security and user groups:
See the following recommendations for assigning permissions to security groups and user groups.
Do
- Use Microsoft Entra ID, Active Directory, or Windows security groups when you’re managing lots of users.
- When you’re adding teams, consider what permissions you want to assign to team members who need to create and modify area paths, iteration paths, and queries.
- When you’re adding many teams, consider creating a Team Administrators custom group where you allocate a subset of the permissions available to be Project Administrators.
- Consider granting the work item query folders Contribute permission to users or groups who require the ability to create and share work item queries for the project.
- Take advantage of built-in roles and default to Contributor for developers. Admins get assigned to the Project Administrator security group for elevated permissions, allowing them to configure security permissions.
Don’t
- Don’t change the default permissions for the Project Valid Users group. This group can access and view project information.
- Don’t add users to multiple security groups that contain different permission levels. In certain cases, a Deny permission level may override an Allow permission level.
- Don’t change the default assignments made to the Project Valid Users groups. If you remove or set View instance-level information to Deny for one of the Project Valid Users groups, no users in the group can access whatever project, collection, or deployment you set the permission on.
- Don’t assign permissions that are noted as Assign only to service accounts to user accounts.
Project Management
Code Repos
Each DevOps project can support multiple code repositories. It is easy for a project manager to easily fall into disorganization by creating multiple projects for each technology type or putting too much in one repo. Projects can be configured to support multiple technology types within one team. Separate repos will allow for cleaner deployment options and pipeline configuration.
When using Azure DevOps code repos with Microsoft services, it is recommended to have a separate repo per service. This will allow for the possibility to directly connect the service to the repo or perform deployment-specific action for that service. For example:
- Power BI may be configured to connect to an Azure DevOps repo. Checking in and out reports will automatically deploy those reports to Power BI workspaces.
- Azure Data Factory (data pipelines and orchestration) provides a low/no-code environment for developers. It connects to the Azure DevOps repo to keep track of all the actions and deployments of the service.
- Pro-code resources like Function Apps and Static Web Sites can be deployed directly to the resource from a developer’s Visual Studio instance. This is good for development environments which will then support CI/CD and automated test actions following a code push.
Boards and Agile Project Management
The DevOps workspace is for a product with related projects managed within with separate code repos, teams, and other objects used. Azure DevOps helps with managing projects for a product. Azure DevOps follows the agile methodology of using a product backlog as opposed to user stories (comparison).
CoStrategix Project Managers find task management on Boards helpful and intuitive. Effort is at the Product Backlog Item level, while time is recorded on the associated tasks. Using PBIs and Tasks to track time for invoicing is not recommended for external projects. Azure DevOps is a software development tool, so it makes sense that time-tracking for invoicing purposes is not as easy.
Team for a project, every project is set with a default team, and each team has its own set of tools.
Create different teams for the organization. So in one project:
- Every team owns their own backlog, to create a new backlog you create a new team.
- Every backlog has a corresponding Kanban board you can use to track progress and update status.
- Each team can have one or many repos associated.
- PBIs can be easily moved between teams
The fact that teams have their own tools and that projects can have multiple teams associated, means a project manager can manage simple to complex projects within the same project structure:
- Project management is easier with a nicely contained project.
- Backlog management is easy and intuitive.
- Nice having the documentation with the project/product (although the documentation tool is not as feature-rich.)
CoStrategix development teams take advantage of the tagging functionality Azure DevOps provides. Team Leads and Developers can group and manage Product Backlog Items and Tasks across sub-features and activities by adding and removing Tags. This is helpful for work across features and disciplines.
Pipelines and Test Plans and Suites
Pipelines can be configured for build, deployment, testing, and/or code coverage. At CoStrategix we aim for each project to do the following:
- Build manual (push of a button) or automated (CI/CD) pipelines for all deliverables.
- Build pipelines should include automated testing in some manner for consistency and quality of the deliverables.
- Depending on the technology code coverage should be included to track the depth of the tests.
Note: Not all Microsoft services will support automated deployments and CI/CD. Microsoft has been working to add automated build/publish/deployment functionality within its services. Most recently, Power BI included a connection to code repositories and Azure Data Factory started supporting pipeline deployments from one environment to the next.
Code Coverage can be added to Azure DevOps as an extension. Below are examples from the Cobertura extension. This tool will track the code covered during the testing phase of the pipelines.
There are 4 ways to organize Azure Test Plans:
- Directly add test cases to the root of the test plan
- Add a Static test suite and add your test cases to that (this is like adding folders)
- Create a Query-based test suite that pulls test cases into the test suite via a query
- Create a Requirement-based test suite, which will pull the test cases into the test plan that we attached to the Stories during grooming. (best practice)
An example Test Plan for data quality on a data platform project is included below.
Example screenshots for test case summary and code coverage executed on a .Net module.
Project Migration
Azure DevOps Project content can be migrated with the Azure DevOps Migration Tools. This is a community-built library of tools to manage Azure DevOps projects. This tool set is recommended to CSH as Definity uses multiple DevOps objects and the entire Definity organization does not need to migrate, specific projects will be migrated. If there is a need to move the entire collection, then contact Microsoft for their migration service.
The Azure DevOps Migration tool can migrate Work Items, TestPlans & Suits, Teams, Shared Queries, and Pipelines from one organization to another. For CSH, the recommendation is to only migrate project work items, test plans and suites, and pipelines between organizations. Wiki pages could be moved manually. Confirmation is needed on the depth of the wiki documentation and whether it is cost-effective to write a script based on Azure DevOps APIs to move the content.
The Azure DevOps Migration Tools require a user with knowledge of the organization and the ability to edit the needed XML configuration files. Each file defines the source and destination of the Azure DevOps organizations and what will be migrated. The biggest concern for the migration is any customization that exists on the project. The custom items will need to be identified within the configuration to ensure proper mapping.
When performing a migration, ensure to test, test, and test again. The tools can be executed many times as the source projects should not be deleted until the final migration is complete. The testing will ensure all items are migrated properly.
Note: The migration will cause some rework for permissions. Most likely all users will have to be given access to projects and the appropriate permissions to work in those projects. The migration tool can migrate Teams and permissions but that assumes the Team and permission structure should be retained.
Test Plan Example
Here we provide an example of a Test Plan for Data Quality:
Overview
This outline gives an overview of the best practices that we follow during test plan creation in Azure DevOps for a data project. This assumes the data project is following a 3-Zone data architecture.
In Scope:
- Data quality of the ingestion process.
- Data quality of visualization if any.
Out Scope:
- Data analysis aspect of the project. The framework is documented within the Distributed Application Development Entity (DADE).
Steps in DevOps:
- Test plan creation
- Test suites within test plans
- Test cases within test plans
Project Framework:
- The main test plan name will be the data source name or any interface name. It can be a new data source you are ingesting to your data cloud relating to any domain. Eg – HR data, finance data, operations data, etc.
- Test suites are structurally divided into 3 zones
- Data Quality in source files
- Data Quality in Zone 1
- Data Quality in Zone 2
- Test cases for Data Quality on source files. We mostly apply the basic Data Quality dimension in this phase. It is subjective based on the profiling results and the project needs. This will be executed via Python scripts or custom SQL queries.
- Data completeness – Column-wise checks, missing reference data, etc.
- Data Integrity fundamentals – Dimension checks, column name checks, etc.
- Timeliness – Data recency checks, freshness to be determined
- Test cases for Data Quality on Zone 1 – Achieved via Python scripts or custom SQL queries
- Data truncation check – Applied across entire columns
- Duplication checks – Applied for a single file load
- Data Volume check – If the size is within the limit
- Timeliness
- Test cases for Data Quality on Zone 2 – Achieved via Python scripts or custom SQL queries
- Conformity – Data type checks, date format checks, currency checks, or any other applicable checks
- Data size – If zone 1 maintains the same size as zone 2
- Data accuracy checks
- Consistency and synchronization – Consistent across zone 1 or any other data sources
- Ingestion mode check – If incremental/full load is correctly implemented
- Test cases for Data Quality on Visualization – Achieved via Python scripts or custom SQL queries or Excel
- Here the test plan can be the report/dashboard name
- Data Quality on the visuals within the report
- UI quality aspect – If the functional aspect of the report is working properly
- Test Case for Data Quality on API checks – Achieved via Python scripts or Excel Power Query (Data completeness Check)
- Here the main aim is to compare data between API results and Zone 1
- Python Scripts compares record-level data between the sources and identifies any mismatches
- Excel Power Query can also be used to compare the data
- Separate Python script to monitor the data for count comparison
References
Azure DevOps: Best practices for Agile project management
Azure DevOps Security best practices
Azure DevOps Security – Group Management Dos & Don’ts
Azure DevOps Migration Tools
Azure DevOps documentation home
[Medium] Azure Test Plans: Test Plans
CoStrategix is a strategic data and digital solutions firm that enables organizations to capitalize on today’s business opportunities. We help organizations develop new digital products from strategic ideation, through implementation, and ongoing DevOps support. How can we help you?