Consolidating Application Insights Repos

[TimothyMothra]

Currently

We are shipping 17 packages across 4 repos:

• Base https://github.com/microsoft/ApplicationInsights-dotnet
• Web https://github.com/microsoft/ApplicationInsights-dotnet-server
• Logging https://github.com/microsoft/ApplicationInsights-dotnet-logging
• AspNetCore https://github.com/microsoft/ApplicationInsights-aspnetcore

Challenges

• This creates additional work to synchronize changes across each repo.
  Typically work begins in this (Base) repo and requires a signed build to test those changes in another repo.
  This slows down the dev cycle.
• We take this cost again whenever we do a release. Releases must occur in stages (Base > Web > Logging > Core) and involves a lot of waiting for automation to complete.
  A full release typically costs us a week. In the last 2 years, we've never released one repo without also releasing all others.
• This creates confusion for customers. Most of our customers use "application insights" and don't know which repo to open issues.
• This creates confusion for contributors. It takes some effort to determine the correct repo to contribute for a specific feature.

Proposal

Over the next month, we will merge all 4 repos into this one.

AspNetCore did this exercise last year and CoreClr is doing this now.

Timeframe

I'll be starting this work in October and I expect this can be completed within a month, finishing in early to mid November.

Next Steps

• All PRs will be tagged with this issue for tracking.
• I'll start opening prs to reorganize files before the migration to help prevent conflicts.
• I'll migrate one repo at a time and confirm that builds are successful to prevent blocking any contributor. We want to preserve the entire commit history.
• After the migration, I'll lock the source repo.
• I expect there will several follow up PRs as I attempt to consolidate code.
• I don't know the correct mechanism to migrate issues but I will make every attempt to do so.

Detailed Plan

In Scope

• 4 Repos (Base, Web, AspNetCore, Logging)
• 4 PR Build definitions
• 4 Signing Definitions
• 4 Release Pipelines

End Goal

• Move all code to Base SDK repo, and preserve commit history
• Shared build definitions for PRs, Signing, and Releases

Plan

Prep work

1. DONE Reorganize individual repos. This is to ensure that there are no conflicting directories or file names and to guarantee no conflicts when migrating source. This will require some minor changes to existing build definitions.

Migrate source

2. DONE Migrate a repo as-is to Base. Don't merge dependencies right away. The goal is to migrate the source as quickly as possible so devs can continue working.
3. DONE Clone existing build definitions and point to new directories in Base SDK. The goal is to get builds working again as quickly as possible so devs can continue working. These can be validated against a feature branch before merging.
4. DONE Once all builds are working, need to lock the original repo to prevent new PRs. Issues can be migrated anytime after this.
5. DONE Safe to start the next repo.

Merge source

6. All projects should be merged into a single directory. This can be done as follow up PRs.
   UPDATE This is blocked because each project has unique .props files that need to be consolidated.
7. The newer single directory source will require consolidated build definitions. This can be validated against a feature branch before merging.
   UPDATE Final todo: The two linux builds need to be merged.

Merge dependencies

8. Right now each repo has unique project dependencies (*.props, *.targets). These need to be consolidated to ensure that future code maintenance can be kept reasonable. Once the newer build infra is in place, this will be simple and low risk.
   UPDATE This is in progress, but proving to be very time consuming. Each project has unique and conflicting dependencies that need to be resolved.

Todo

• DONE CONTRIBUTING guide

Stretch Goals

• DONE https://aka.ms/deprecateIconUrl
• Fix code coverage tool
• DONE http://aka.ms/fxcopanalyzers
• fix Perf/XDT.Tests project
• DONE Update MSTest in all projects
• Remove duplicate ApplicationInsightsTypes project (one per reop)
• DONE Can we replace AddXmlLanguage?
• DONE Make build results Public
• Convert build definitions to YML

Risks

Potential Impact due to moving source

Mitigation: Only one repo will be migrated at a time and to completion to reduce the scope of disruptions.

Potential Impact to Releases

Migrating source could impact our ability to publish a new release.
Mitigation: Existing build definitions will not be changed, in the event that we need to rapidly rollback and release immediately.

Potential Impact to Devs

While source is being migrated, I'll need devs to pause contributions.
Mitigation: To keep this impact as low as possible, I plan to migrate source as-is without any reorganization. Keep using existing build infrastructure, just from a single repo.

Existing PRs may not get merged

Mitigation: I'm going to work with the team to get as many existing PRs merged as possible. Unfortunately, any inactive PRs may not get merged before the migration.

[github-actions]

This issue is stale because it has been open 300 days with no activity. Remove stale label or comment or this will be closed in 7 days.