Improve Dependency Resolution for Workflow Optimization

contrib/git-changes-action/README.md

@@ -1,6 +1,6 @@
 # Git Changes Action
 
-This GitHub Action exports a variable that contains the list of Go modules changed in the current pull request, along with any dependent modules. This can be useful for automating build, test, or deployment workflows that involve Go projects.
+This GitHub Action exports a variable that contains the list of Go modules changed in the current pull request, and if desired, along with any dependent modules. This can be useful for automating build, test, or deployment workflows that involve Go projects.
 
 [![Go Reference](https://pkg.go.dev/badge/github.com/synapsecns/sanguine/contrib/git-changes-action.svg)](https://pkg.go.dev/github.com/synapsecns/sanguine/contrib/git-changes-action)
 [![Go Report Card](https://goreportcard.com/badge/github.com/synapsecns/sanguine/contrib/git-changes-action)](https://goreportcard.com/report/github.com/synapsecns/sanguine/contrib/git-changes-action)
@@ -9,16 +9,15 @@ This GitHub Action exports a variable that contains the list of Go modules chang
 
 1. To use this action, add the following steps to your workflow:
 
-    Check out the current pull request using the actions/checkout action. It's recommended to set fetch-depth to 0 and submodules to recursive to ensure that all necessary dependencies are fetched.
-
-
+    Check out the current pull request using the actions/checkout action. It's recommended to set fetch-depth to 0 and submodules to 'recursive' to ensure that all necessary dependencies are fetched.
 
 ```yaml
     steps:
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
           submodules: 'recursive'
+          dependencyLevelResolution: "modules"
 ```
 
 1. Use the synapsecns/sanguine/git-changes-action Docker image to run the git-changes script, which exports a variable that contains the list of changed Go modules.
@@ -29,55 +28,26 @@ This GitHub Action exports a variable that contains the list of Go modules chang
         with:
           github_token: ${{ secrets.github_token }}
           timeout: "1m" # optional, defaults to 1m
+          dependencyLevelResolution: "modules" 
+          # For package level resolution, use "packages" instead of "modules"
 ```
 
 You can customize the behavior of the git-changes script by using the following inputs:
 
  - `github_token`: The token to use for authentication with the GitHub API. This is required to fetch information about the current pull request.
  - `timeout`: The maximum time to wait for the GitHub API to respond. Defaults to 1 minute.
+ - `dependencyLevelResolution`: "modules" or "packages".
 
 The output of the git-changes script is a comma-separated list of Go module paths. You can access this list using the `filter_go` output variable, like so:
 
 ```yaml
-      - run: echo "Changed modules: ${{ steps.filter_go.outputs.changed_modules }}"
-      - run: echo "Unchanged modules: ${{ steps.filter_go.outputs.unchanged_modules }}"
-      - run: echo "Changed modules (including dependencies): ${{ steps.filter_go.outputs.changed_modules_deps }}"
-      - run: echo "Unchanged modules (including dependencies): ${{ steps.filter_go.outputs.unchanged_modules_deps }}"
-```
-
-## Example
-
-Here's an example workflow that uses the `git-changes` action to run tests for changed Go modules:
-
-
-```yaml
-name: Test Go Modules
-
-on:
-  pull_request:
-    types: [opened, synchronize]
-
-jobs:
-  test_go_modules:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-        with:
-          fetch-depth: 0
-          submodules: 'recursive'
+      - run: echo 'Changed modules:' ${{ steps.filter_go.outputs.changed_modules }}
+      - run: echo 'Unchanged modules:' ${{ steps.filter_go.outputs.unchanged_modules }}
+      - run: echo 'Changed modules (including dependencies):' ${{ steps.filter_go.outputs.changed_modules_deps }}
+      - run: echo 'Unchanged modules (including dependencies):' ${{ steps.filter_go.outputs.unchanged_modules_deps }}
 
-      - uses: docker://ghcr.io/synapsecns/sanguine/git-changes-action:latest
-        id: filter_go
-        with:
-          github_token: ${{ secrets.github_token }}
-          timeout: "1m"
-
-      - name: Run tests
-        run: go test -v ${{ steps.filter_go.outputs.changed_modules }}
 ```
 
-This workflow will run tests for all changed Go modules and their dependencies whenever a pull request is opened or synchronized.
-
 ## How It Works
 
 First a change tree is calculated between the ref and the base (as passed in by the user or inferred by the event type) based on the flow described below and [here](https://github.com/dorny/paths-filter/blob/4067d885736b84de7c414f582ac45897079b0a78/README.md#supported-workflows). This is a file tree that contains all the files that have changed between two refs.
@@ -94,7 +64,9 @@ graph TB
     F --> G
 ```
 
-Each module in the `go.work` is visited. If any changes were detected by the previous step, the module is added to the list of changed modules. If `include_dependencies` is on and the module has a dependency that is also in the `go.work`, the dependency is added to the list of changed modules as well. This process is repeated until all modules in the `go.work` have been visited.
+## Module Level Resolution
+
+Each module in the `go.work` is visited. If any changes are detected, the module itself is added to the list of changed modules. By setting `include_dependencies` to `modules`, changes to their direct or indirect module dependencies will flag the module as changed. This process is repeated until all modules in the `go.work` have been visited.
 
 ```mermaid
 sequenceDiagram
@@ -126,3 +98,42 @@ sequenceDiagram
   end
   GW->>GW: Continue Until All Modules Visited
 ```
+
+## Package Level Resolution
+
+Each module in the `go.work` is visited. If any changes are detected, the module itself is added to the list of changed modules. By setting `include_dependencies` to `packages`, changes to its direct or indirect package dependencies will flag the module as changed. This process is repeated until all package dependencies for the module been visited.
+
+```mermaid
+sequenceDiagram
+  participant GW as go.work
+  participant M as Module
+  participant P as Package
+  participant CML as Changed_Module_List
+  participant UML as Unchanged_Module_List
+  participant D as Dependency
+
+  GW->>M: Visit Module
+  M->>P:Extract all packages for each Module
+  P->>M: Returns [module]: [package1,package2]
+M->>M: Changes detected?
+M->>P: Loop over packages
+  P-->>GW: Changes Detected?
+  alt Changes Detected
+    GW->>CML: Add Module to Changed_Module_List
+    M->>D: Has Package Dependency?
+    alt Has Dependency
+      GW->>CML: Add Dependency to Changed_Module_List
+    else No Dependency
+      M-->>GW: No Dependency to Add
+    end
+  else No Changes Detected
+    GW->>UML: Add Module to Unchanged_Module_List
+    M->>D: Has Dependency in go.work?
+    alt Has Dependency
+      GW->>UML: Add Dependency to Unchanged_Module_List
+    else No Dependency
+      M-->>GW: No Dependency to Add
+    end
+  end
+  GW->>GW: Continue Until All Modules Visited
+```

contrib/git-changes-action/detector/git/doc.go

@@ -0,0 +1,2 @@
+// Package git provides utilities to interact with git.
+package git

contrib/git-changes-action/detector/detector.go → contrib/git-changes-action/detector/git/git.go

@@ -1,72 +1,21 @@
-package detector
+package git
 
 import (
 	"encoding/hex"
 	"encoding/json"
 	"errors"
 	"fmt"
-	"github.com/ethereum/go-ethereum/common"
 	"github.com/go-git/go-git/v5"
 	"github.com/go-git/go-git/v5/plumbing"
 	"github.com/go-git/go-git/v5/plumbing/object"
 	"github.com/google/go-github/v41/github"
 	"github.com/synapsecns/sanguine/contrib/git-changes-action/detector/actionscore"
 	"github.com/synapsecns/sanguine/contrib/git-changes-action/detector/tree"
 	"github.com/synapsecns/sanguine/core"
-	"golang.org/x/mod/modfile"
 	"os"
-	"path"
 	"strings"
 )
 
-// DetectChangedModules is the change detector client.
-// nolint: cyclop
-func DetectChangedModules(repoPath string, ct tree.Tree, includeDeps bool) (modules map[string]bool, err error) {
-	modules = make(map[string]bool)
-
-	goWorkPath := path.Join(repoPath, "go.work")
-
-	if !common.FileExist(goWorkPath) {
-		return nil, fmt.Errorf("go.work file not found in %s", repoPath)
-	}
-
-	//nolint: gosec
-	workFile, err := os.ReadFile(goWorkPath)
-	if err != nil {
-		return nil, fmt.Errorf("failed to read go.work file: %w", err)
-	}
-
-	parsedWorkFile, err := modfile.ParseWork(goWorkPath, workFile, nil)
-	if err != nil {
-		return nil, fmt.Errorf("failed to parse go.work file: %w", err)
-	}
-
-	depGraph, err := getDependencyGraph(repoPath)
-	if err != nil {
-		return nil, fmt.Errorf("could not get dep graph: %w", err)
-	}
-
-	for _, module := range parsedWorkFile.Use {
-		changed := false
-		if ct.HasPath(module.Path) {
-			changed = true
-		}
-
-		if includeDeps {
-			deps := depGraph[module.Path]
-			for _, dep := range deps {
-				if ct.HasPath(dep) {
-					changed = true
-				}
-			}
-		}
-
-		modules[module.Path] = changed
-	}
-
-	return modules, nil
-}
-
 // getChangeTreeFromGit returns a tree of all the files that have changed between the current commit and the commit with the given hash.
 // nolint: cyclop, gocognit
 func getChangeTreeFromGit(repoPath string, ghContext *actionscore.Context, head, base string) (tree.Tree, error) {
@@ -76,7 +25,7 @@ func getChangeTreeFromGit(repoPath string, ghContext *actionscore.Context, head,
 		return nil, fmt.Errorf("could not open repository %s: %w", repoPath, err)
 	}
 
-	head, err = getHead(repository, ghContext, head)
+	head, err = GetHead(repository, ghContext, head)
 	if err != nil {
 		return nil, fmt.Errorf("could not get head: %w", err)
 	}
@@ -272,9 +221,9 @@ func getHeadBase(repo *git.Repository) (head string, base string, err error) {
 	return co.Hash().String(), lastCommit.Hash.String(), nil
 }
 
-// getHead gets the head of the current branch.
+// GetHead gets the head of the current branch.
 // it attempts to mirror the logic of  https://github.com/dorny/paths-filter/blob/0ef5f0d812dc7b631d69e07d2491d70fcebc25c8/src/main.ts#L104
-func getHead(repo *git.Repository, ghContext *actionscore.Context, head string) (string, error) {
+func GetHead(repo *git.Repository, ghContext *actionscore.Context, head string) (string, error) {
 	if head != "" {
 		return head, nil
 	}

contrib/git-changes-action/detector/ref.go → contrib/git-changes-action/detector/git/ref.go

@@ -1,4 +1,4 @@
-package detector
+package git
 
 import (
 	"context"

contrib/git-changes-action/detector/depgraph.go → contrib/git-changes-action/detector/module/depgraph.go

@@ -1,4 +1,4 @@
-package detector
+package moduledetector
 
 import (
 	"fmt"
@@ -36,25 +36,25 @@ func getDependencyGraph(repoPath string) (moduleDeps map[string][]string, err er
 	}
 
 	// map of module->dependencies + replaces
-	var dependencies map[string][]string
-	// bidirectional map of module->module name
-	var moduleNames *bimap.BiMap[string, string]
+	var dependencies map[string]map[string]struct{}
+	// bidirectional map of module->module
+	// moduleRelativeName <-> modulePublicName
+	var dependencyNames *bimap.BiMap[string, string]
 
 	// iterate through each module in the go.work file
 	// create a list of dependencies for each module
 	// and module names
-	dependencies, moduleNames, err = makeDepMaps(repoPath, parsedWorkFile.Use)
+	dependencies, dependencyNames, err = makeDepMaps(repoPath, parsedWorkFile.Use)
 	if err != nil {
 		return nil, fmt.Errorf("failed to create dependency maps: %w", err)
 	}
 
 	depGraph := depgraph.New()
 	// build the dependency graph
 	for _, module := range parsedWorkFile.Use {
-		moduleDeps := dependencies[module.Path]
-		for _, dep := range moduleDeps {
+		for dep := range dependencies[module.Path] {
 			// check if the full package name (e.g. github.com/myorg/myrepo/mymodule) is in the list of modules. If it is, add it as a dependency after renaming
-			renamedDep, hasDep := moduleNames.GetInverse(dep)
+			renamedDep, hasDep := dependencyNames.GetInverse(dep)
 			if hasDep {
 				err = depGraph.DependOn(module.Path, renamedDep)
 				if err != nil {
@@ -81,41 +81,45 @@ func getDependencyGraph(repoPath string) (moduleDeps map[string][]string, err er
 	return moduleDeps, nil
 }
 
-// makeDepMaps makes a dependency map and a bidirectional map of dep<->module.
-func makeDepMaps(repoPath string, uses []*modfile.Use) (dependencies map[string][]string, moduleNames *bimap.BiMap[string, string], err error) {
-	// map of module->dependencies + replaces
-	dependencies = make(map[string][]string)
+// makeDepMaps builds a
+// 1. module->dependency map
+// 2. bidirectional map of module<->module (moduleRelativeName to modulePublicName).
+func makeDepMaps(repoPath string, uses []*modfile.Use) (dependencies map[string]map[string]struct{}, dependencyNames *bimap.BiMap[string, string], err error) {
+	// map of module->depndencies
+	dependencies = make(map[string]map[string]struct{})
+
 	// bidirectional map of module->module name
-	moduleNames = bimap.NewBiMap[string, string]()
+	// Maps relative to public names, used to filer out all external libraries/packages.
+	dependencyNames = bimap.NewBiMap[string, string]()
 
 	// iterate through each module in the go.work file
-	// create a list of dependencies for each module
-	// and module names
+	// 1. Create a list of dependencies for each module
+	// 2. Map public names to private names for each module
+	//nolint: gosec
 	for _, module := range uses {
-		//nolint: gosec
 		modContents, err := os.ReadFile(filepath.Join(repoPath, module.Path, "go.mod"))
 		if err != nil {
-			return dependencies, moduleNames, fmt.Errorf("failed to read module file %s: %w", module.Path, err)
+			return dependencies, dependencyNames, fmt.Errorf("failed to read module file %s: %w", module.Path, err)
 		}
 
 		parsedModFile, err := modfile.Parse(module.Path, modContents, nil)
 		if err != nil {
-			return dependencies, moduleNames, fmt.Errorf("failed to parse module file %s: %w", module.Path, err)
+			return dependencies, dependencyNames, fmt.Errorf("failed to parse module file %s: %w", module.Path, err)
 		}
 
-		moduleNames.Insert(module.Path, parsedModFile.Module.Mod.Path)
+		dependencyNames.Insert(module.Path, parsedModFile.Module.Mod.Path)
+		dependencies[module.Path] = make(map[string]struct{})
 
-		dependencies[module.Path] = make([]string, 0)
 		// include all requires and replaces, as they are dependencies
 		for _, require := range parsedModFile.Require {
-			dependencies[module.Path] = append(dependencies[module.Path], convertRelPath(repoPath, module.Path, require.Mod.Path))
+			dependencies[module.Path][convertRelPath(repoPath, module.Path, require.Mod.Path)] = struct{}{}
 		}
 		for _, require := range parsedModFile.Replace {
-			dependencies[module.Path] = append(dependencies[module.Path], convertRelPath(repoPath, module.Path, require.New.Path))
+			dependencies[module.Path][convertRelPath(repoPath, module.Path, require.New.Path)] = struct{}{}
 		}
 	}
 
-	return dependencies, moduleNames, nil
+	return dependencies, dependencyNames, nil
 }
 
 // isRelativeDep returns true if the dependency is relative to the module (starts with ./ or ../).