Azure Pipelines on DevOps with certificate-based authentication and Graph v2.0 not working #3454
Comments
andikrueger
added
Enhancement
|
@ricmestre Did you have any luck with this? Mine is totally unrelated to M365DSC but my pipeline is broken because of this. |
|
@r4hulp No, this is not working yet for me. In terms of M365Dsc I need to keep using version 1.23.628.1 (which uses Graph v1.28.0) in my pipelines until Microsoft updates their containers images to have Graph v2.0 modules installed by default. |
|
Just ran a test pipeline 10 times in a row, in one of them I got a container which had Graph v1.28 installed, all other remaining jobs got a container with v2.0 installed so I guess they are gradually updating them right now. It's still not official though, link below shows v1.28.0 as the one installed by default. https://github.com/actions/runner-images/blob/main/images/win/Windows2022-Readme.md |
|
@peombwa do you have any insight on the rollout status of V2.0 in containers or are you aware of a functioning work around? |
|
Having the same issue here. My pipeline in Azure DevOps is executing a Powershell script that has Graph commands over a public DevOps agent and almost any Graph command ends up in this error message: ##[error]Method not found: 'Void Microsoft.Graph.Authentication.AzureIdentityAccessTokenProvider..ctor(Azure.Core.TokenCredential, System.String[], Microsoft.Kiota.Authentication.Azure.ObservabilityOptions, System.String[])' I've done various tests, including the ones mentioned: what I can tell you for sure is that as long as Microsoft.Graph v2.0.0 or v1.28.0 is installed, most commands end up with the same error. I've even tried uninstalling the current version that is pushed on the agent (whichever that might be) and installing v1.19.0 - same behavior. @andikrueger @peombwa do you guys have any additional insight on this? |
|
There seems to be a bigger issue now... I've run several more tests and Graph v2.0 now is always available, nevertheless just by calling Export-M365DSCConfiguration on Intune workload it fails with: ##[error]The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: Function Export-TargetResource cannot be created because function capacity 4096 has been exceeded for this scope.To "solve" this the value of $MaximumFunctionCount must be increased, I'm able to set it to 32768 (its max value) but for some reason the value gets lost when it starts the export and then I get that error message above which basically means that without solution I'm stuck with running M365Dsc v1.23.628.1, on Azure Pipelines, which is the last one to support Graph v1.0, this is not good and will have to report this to my management. This can be replicated locally with the following with Graph v2.0 installed: $MaximumFunctionCount = 4096
Get-Module -ListAvailable Microsoft.Graph.* | `
Select-Object -ExpandProperty name | `
Sort-Object -Unique | `
%{ Import-Module -Force $_ }@andikrueger Probably you should change the label of this issue to Bug instead of Enhancement. |
|
It seems that the first thing that Export-M365DSCConfiguration does is to call $Global:MaximumFunctionCount = 16000 (M365DSCUtil.psm1:L1158, also for New-M365DSCConnection in L1663) but the error message still says that I've set it to 4096? What is going on here? |
|
Please have a look at this preliminary solution: microsoftgraph/msgraph-sdk-powershell#2148 (comment) |
|
@andikrueger This I've just reported is another problem, the issue is still the same since I'm not able to run my pipelines with Graph v2.0 with certificates but I'm not getting the same original problem. |
|
Could you try to rollback to AZ.Accounts V2.12.1 within the container, if this would have any impact on these issues? |
|
@andikrueger Like I said, this is not the same problem as the original so there's no point in rolling back Az.Accounts, that won't solve my new problem. |
|
Additionally I also use AzurePowerShell@5 in my pipeline as @r4hulp so if that was the case I'd have to use PS7 which probably would fix my new problem BUT brings along an whole set of other problems, that's why it's recommended to use M365Dsc with PS5.1 in the first place. |
andikrueger
added
Bug
|
Just to make sure, I do get both issues correctly:
switching to PowerShell 7 would not work due to the other restrictions within M365DSC |
|
My initial problem was 2. when containers had Graph v1.28 installed, now they have Graph v2.0 installed which is causing me problem 1., but most likely it doesn't affect exports only but affects other use cases as well such as Assert-M365DSCBlueprint, which I'm testing right now in a pipeline, can't say if it affects deployments as well until a little bit later. |
|
@andikrueger As expected Assert-M365DSCBlueprint also fails, but this is getting ludicrous, now I've noticed that all the runs I tested which have containers with Graph v2.0 installed cannot install about half of M365Dsc's dependencies through Update-M365DSCDependencies. The first half of dependencies get installed correctly but when it reaches Microsoft.Graph.Beta.Identity.SignIns all others fail 2023-07-13T20:06:21.1349432Z [2023-07-13 20:06:21] - Updating Microsoft365Dsc dependencies
2023-07-13T20:06:21.1399749Z Installing DSCParser version {1.3.0.10}
2023-07-13T20:06:40.9734143Z Installing ExchangeOnlineManagement version {3.2.0}
2023-07-13T20:06:45.7641148Z Installing Microsoft.Graph.Beta.DeviceManagement version {2.0.0}
2023-07-13T20:07:14.3288893Z Installing Microsoft.Graph.Beta.Devices.CorporateManagement version {2.0.0}
2023-07-13T20:07:20.4391276Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after
2023-07-13T20:07:20.4401114Z closing the applications.
2023-07-13T20:07:26.1225205Z Installing Microsoft.Graph.Beta.DeviceManagement.Administration version {2.0.0}
2023-07-13T20:07:32.6872658Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after
2023-07-13T20:07:32.6886946Z closing the applications.
2023-07-13T20:07:37.1970608Z Installing Microsoft.Graph.Beta.DeviceManagement.Enrollment version {2.0.0}
2023-07-13T20:07:43.1460811Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after
2023-07-13T20:07:43.1468116Z closing the applications.
2023-07-13T20:07:45.4320831Z Installing Microsoft.Graph.Beta.Identity.DirectoryManagement version {2.0.0}
2023-07-13T20:08:20.1690475Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after
2023-07-13T20:08:20.1697318Z closing the applications.
2023-07-13T20:08:24.8400663Z Installing Microsoft.Graph.Beta.Identity.Governance version {2.0.0}
2023-07-13T20:08:33.0648725Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after
2023-07-13T20:08:33.0655369Z closing the applications.
2023-07-13T20:09:02.4197350Z Installing Microsoft.Graph.Beta.Identity.SignIns version {2.0.0}
2023-07-13T20:09:09.0260415Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after
2023-07-13T20:09:09.0266726Z closing the applications.
2023-07-13T20:09:15.6410743Z Could not update {Microsoft.Graph.Beta.Identity.SignIns}
2023-07-13T20:09:15.6965405Z Installing Microsoft.Graph.Beta.Teams version {2.0.0}
2023-07-13T20:09:23.3299717Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after
2023-07-13T20:09:23.3306468Z closing the applications.
2023-07-13T20:09:42.2736582Z Could not update {Microsoft.Graph.Beta.Teams}
2023-07-13T20:09:42.2737061Z Installing Microsoft.Graph.Beta.DirectoryObjects version {2.0.0}
2023-07-13T20:09:47.3763883Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after
2023-07-13T20:09:47.3770973Z closing the applications.
2023-07-13T20:09:48.9785769Z Could not update {Microsoft.Graph.Beta.DirectoryObjects}
2023-07-13T20:09:49.4238025Z Installing Microsoft.PowerApps.Administration.PowerShell version {2.0.167}
2023-07-13T20:09:54.4059971Z WARNING: The names of some imported commands from the module 'Microsoft.PowerApps.AuthModule' include unapproved verbs
2023-07-13T20:09:54.4070426Z that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again
2023-07-13T20:09:54.4073873Z with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2023-07-13T20:09:54.4238566Z Could not update {Microsoft.PowerApps.Administration.PowerShell}
2023-07-13T20:09:54.4806393Z Installing MicrosoftTeams version {5.3.0}
2023-07-13T20:10:17.4465965Z Could not update {MicrosoftTeams}
2023-07-13T20:10:17.4467837Z Installing MSCloudLoginAssistant version {1.0.116}
2023-07-13T20:10:21.9806395Z Could not update {MSCloudLoginAssistant}
2023-07-13T20:10:21.9806913Z Installing PnP.PowerShell version {1.12.0}
2023-07-13T20:10:33.6043906Z Installing ReverseDSC version {2.0.0.16}
2023-07-13T20:10:38.5298594Z Could not update {ReverseDSC} |
|
@andikrueger Deployments with Graph v2.0 for the Intune policies I have are not affected by "1. function capacity is exceeded", and the reason they didn't fail because of missing installed dependencies is because those policies are not using any of the affected modules that cannot be installed as shown above. This also means that even if function capacity is resolved somehow the export, assert, but also the code I have on my side, will fail because they all rely on calling Get-M365DSCExportContentForResource which calls Get-DSCBlock from ReverseDSC which cannot be installed (the last one from the list above). Everywhere I look there are new problems. |
|
The warning messages above should not be blocking. This happens, if the right module version is present within the system. The 2023-07-13T20:09:54.4806393Z Installing MicrosoftTeams version {5.3.0}
2023-07-13T20:10:17.4465965Z Could not update {MicrosoftTeams}
2023-07-13T20:10:17.4467837Z Installing MSCloudLoginAssistant version {1.0.116}
2023-07-13T20:10:21.9806395Z Could not update {MSCloudLoginAssistant}
2023-07-13T20:10:21.9806913Z Installing PnP.PowerShell version {1.12.0}
2023-07-13T20:10:33.6043906Z Installing ReverseDSC version {2.0.0.16}
2023-07-13T20:10:38.5298594Z Could not update {ReverseDSC}Could you add the following code: Update-M365DSCDependencies -Force
$dependencies = Update-M365DSCDependencies -ValidateOnly
foreach ($dependency in $dependencies){
Write-Host "Failed to find $($dependency.ModuleName)"
} |
|
@andikrueger It seems that all dependencies are installed after all even if it says "Could not update {...}" for some of them, which means it's failing here since it's the only thing that it's called after Install-Module and that can throw an exception: The catch block could use some love and actually show why it failed: Let's hope this specific problem doesn't cause any other issues, such as the ones I mentioned in previous comment, but in any case, I'm still affected with "function capacity is exceeded", how can I solve this if increasing the value doesn't do anything? |
|
This is the current state of affairs, I need some sleep before looking at this again in a few hours. Problem 1: Function capacity exceeded (4096 is default value for $global:MaximumFunctionCount in PS5.1, this doesn't apply to PS7.x) "Solution" to Problem 1: Copied M365DSCReverse.psm1 and M365DSCUtil.psm1 to my repo, changed them to display $global:MaximumFunctionCount after it's set to 16000 in all places (and it does display 16000 a couple of times in the logs). I'm not joking, this actually makes me go further with the export, but see below. Problem 2: With "solution" to Problem 1 in place I get again 'Void Microsoft.Graph.Authentication.AzureIdentityAccessTokenProvider..ctor(Azure.Core.TokenCredential, System.String[], Microsoft.Kiota.Authentication.Azure.ObservabilityOptions, System.String[])' before export begins. "Solution" to Problem 2: Setup pipeline task AzurePowerShell@5 with azurePowerShellVersion = 9.2.0 (Uses earlier version of Az.Accounts not affected by this problem). Problem 3 - Part 1: With "solution" to Problem 2 in place the export now runs but it fails to extract all of my policies and at the end shows another error You cannot call a method on a null-valued expression., but not sure exactly where this one fails. "Solution" to Problem 3 - Part 1: In order to see what gets written in the registry about the failed extraction of the policies I modified Start-M365DSCConfigurationExtract in M365DSCReverse.psm1 so that it returns early, before the function fails. Problem 3 - Part 2: Outputting 16000 in the screen (just by calling $global:MaximumFunctionCount) seems to be affecting Start-M365DSCConfigurationExtract since it cannot determine the ConnectionMode. { Cannot process argument transformation on parameter 'ConnectionMode'. Cannot convert value to type System.String. } \ at Export-TargetResource, C:\Program Files\WindowsPowerShell\Modules\Microsoft365Dsc\1.23.712.1\DSCResources\MSFT_IntuneAppProtectionPolicyAndroid\MSFT_IntuneAppProtectionPolicyAndroid.psm1: line 1097 "Solution" to Problem 3 - Part 2: Display $global:MaximumFunctionCount value but using Write-Host cmdlet, this makes the policies extract now but they all show 16000 at the end of the line before the green checkmark. [57/64] Extracting [IntuneWifiConfigurationPolicyIOS] using {CertificateThumbprint}...16000 |
|
@ricmestre Thank you so much for the detailed summary of the various issue within the pipeline. I just got word, there is a new Graph PS available: V 2.1.0 https://github.com/microsoftgraph/msgraph-sdk-powershell/releases/tag/2.1.0 I just updated the modules within the PR #3474 and added the suggestions changes of yours to the output of MaxFunction count and the Update-M365DSCDependencies function Any chance to test-drive this version? |
|
Just found the issue that was causing Error-Mesage: Function Get-MgUser cannot be created because function capacity 4096 has been exceeded for this scope. |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
And here's the output for counting the current exported commands of an empty session |
|
With this observation in mind, it looks like I'm eagerly looking forward to your test-results. |
|
Yes the culprit it's there, I'm running with the following code inside its loop just to confirm that if increasing $global:MaximumFunctionCount to 32768 instead of 16000 makes a difference. $global:MaximumFunctionCount = 32768
Write-Host "XXX - $($global:MaximumFunctionCount) - Importing $($resource.FullName)"
try {
Import-Module $resource.FullName -Force
}
catch {
Write-Host "YYY - $($global:MaximumFunctionCount) - Failed to import $($resource.FullName)"
}
$global:MaximumFunctionCount = 32768
Write-Host "ZZZ - $($global:MaximumFunctionCount) - Getting parameters for $($resource.FullName)"
try {
$parameters = (Get-Command 'Set-TargetResource').Parameters.Keys
}
catch {
Write-Host "YYY - $($global:MaximumFunctionCount) - Failed to get parameters for $($resource.FullName)"
$Components += @{
Resource = $resource.Name -replace 'MSFT_', '' -replace '.psm1', ''
AuthMethod = 'CertificateThumbprint'
}
continue
} |
|
Setting the value to 32768 also doesn't have any effect, both commands still fail unfortunately, the stick doesn't seem to be hitting it hard enough! |
|
One more idea: have you tried with 32767. https://learn.microsoft.com/en-us/dotnet/api/system.int16.maxvalue?view=net-7.0 |
|
$MaximumFunctionCount is actually an Int32 (max 2147483647), not an Int16, but the maximum value you can set to it is 32768 and that is being limited internally by PS5.1. Anything higher gets an error message, and setting it to anything lower in this case won't solve the issue. The variable cannot be validated because the value 32769 is not a valid value for the MaximumFunctionCount variable.
At line:1 char:1
+ $MaximumFunctionCount=32769
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : MetadataError: (:) [], ValidationMetadataException
+ FullyQualifiedErrorId : ValidateSetFailure |
|
So I think I'm getting there, it's still a big hammer thrown at the issue but it seems to solve it by unloading all loaded Graph modules at the start of Get-M365DSCComponentsWithMostSecureAuthenticationType and by doing that it makes everything work correctly. Not sure this patch will be accepted but by the numbers of some other dependencies of M365Dsc as well this will most likely happen in the future again in same place or even another one. See below the results for the 2 runs I tested with your code plus exported functions number as well (last column), first without patch and then with patch applied, and at the very bottom is the patch itself.
This is inside Get-M365DSCComponentsWithMostSecureAuthenticationType in M365DSCReverse.psm1 This is back in Start-M365DSCConfigurationExtract (M365DSCUtil.psm1) right after calling Get-M365DSCComponentsWithMostSecureAuthenticationType, so it seems to be unloading all M365DSC dependencies automatically (there is no code for it) when it moves from module M365DSCUtil.psm1 to module M365DSCReverse.psm1
This is inside Get-M365DSCComponentsWithMostSecureAuthenticationType in M365DSCReverse.psm1 This is back in Start-M365DSCConfigurationExtract (M365DSCUtil.psm1) right after calling Get-M365DSCComponentsWithMostSecureAuthenticationType This is the patch I used to make it work: diff --git a/Modules/Microsoft365DSC/Modules/M365DSCUtil.psm1 b/Modules/Microsoft365DSC/Modules/M365DSCUtil.psm1
index 7c1a43ca6..6bbad23de 100644
--- a/Modules/Microsoft365DSC/Modules/M365DSCUtil.psm1
+++ b/Modules/Microsoft365DSC/Modules/M365DSCUtil.psm1
@@ -3403,6 +3403,13 @@ function Get-M365DSCComponentsWithMostSecureAuthenticationType
$Resources
)
+ $GraphModules = @()
+ $GraphModules += Get-Module Microsoft.Graph* | ?{ $_.Name -ne "Microsoft.Graph.Authentication" }
+ $GraphModules += Get-Module Microsoft.Graph.Authentication
+ foreach ($GraphModule in $GraphModules) {
+ Remove-Module -Name $GraphModule.Name -Force -ErrorAction SilentlyContinue
+ }
+
$modules = Get-ChildItem -Path ($PSScriptRoot + '\..\DSCResources\') -Recurse -Filter '*.psm1'
$Components = @()
foreach ($resource in $modules) |
|
Given that all M365Dsc's dependencies are unloaded when returning from one module to the other anyway then unloading all of those dependencies upfront in Get-M365DSCComponentsWithMostSecureAuthenticationType also does the trick, this patch also works. I really need some sleep, we can check this further by tomorrow. diff --git a/Modules/Microsoft365DSC/Modules/M365DSCUtil.psm1 b/Modules/Microsoft365DSC/Modules/M365DSCUtil.psm1
index 7c1a43ca6..307b3f4e2 100644
--- a/Modules/Microsoft365DSC/Modules/M365DSCUtil.psm1
+++ b/Modules/Microsoft365DSC/Modules/M365DSCUtil.psm1
@@ -3403,6 +3403,13 @@ function Get-M365DSCComponentsWithMostSecureAuthenticationType
$Resources
)
+ $ManifestPath = Join-Path -Path $PSScriptRoot -ChildPath '..\Dependencies\Manifest.psd1'
+ $Manifest = Import-PowerShellDataFile -Path $ManifestPath
+ foreach ($Dependency in $Manifest.Dependencies) {
+ Remove-Module -Name $Dependency.ModuleName -Force `
+ -ErrorAction SilentlyContinue
+ }
+
$modules = Get-ChildItem -Path ($PSScriptRoot + '\..\DSCResources\') -Recurse -Filter '*.psm1'
$Components = @()
foreach ($resource in $modules) |
|
Wow - this was an all night job. Incredible work! I like the second solution more and will run some tests within my environment. |
|
so. finally I am able to run my own pipeline and it looks strange: The AZ module is 9.30 and I uploaded the dev branch of M365DSC into me repo to be able to run modifications. Right now I modified the $Global:MaximumFunctionCount to be 32767 and added some counting methodology to Update-M365DSCDependencies within a new finally block of the already present try-catch: This is part of the scripts output. [2023-07-18 12:54:34] - current session exports: 6204
[2023-07-18 12:54:34] -
Info-Mesage Function Count: 32767
Installing Microsoft.Graph.Beta.DirectoryObjects version {2.1.0}
WARNING: The version '2.1.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after
closing the applications.
Could not update or import {Microsoft.Graph.Beta.DirectoryObjects}
Error-Mesage: Function Confirm-MgBetaDirectoryObjectMemberGroup cannot be created because function capacity 4096 has been exceeded for this scope.
[2023-07-18 12:54:41] - Az.Accounts - 2.12.4 - 47
[2023-07-18 12:54:41] - DSCParser - 1.3.0.10 - 1
[2023-07-18 12:54:41] - ExchangeOnlineManagement - 3.2.0 - 29
[2023-07-18 12:54:41] - Microsoft.Graph.Applications - 2.1.0 - 265
[2023-07-18 12:54:41] - Microsoft.Graph.Authentication - 2.1.0 - 18
[2023-07-18 12:54:41] - Microsoft.Graph.Beta.DeviceManagement - 2.1.0 - 947
[2023-07-18 12:54:41] - Microsoft.Graph.Beta.DeviceManagement.Administration - 2.1.0 - 344
[2023-07-18 12:54:41] - Microsoft.Graph.Beta.DeviceManagement.Enrollment - 2.1.0 - 190
[2023-07-18 12:54:41] - Microsoft.Graph.Beta.Devices.CorporateManagement - 2.1.0 - 802
[2023-07-18 12:54:41] - Microsoft.Graph.Beta.DirectoryObjects - 2.1.0 - 13
[2023-07-18 12:54:41] - Microsoft.Graph.Beta.Identity.DirectoryManagement - 2.1.0 - 524
[2023-07-18 12:54:41] - Microsoft.Graph.Beta.Identity.Governance - 2.1.0 - 1223
[2023-07-18 12:54:41] - Microsoft.Graph.Beta.Identity.SignIns - 2.1.0 - 563
[2023-07-18 12:54:41] - Microsoft.Graph.Beta.Teams - 2.1.0 - 757
[2023-07-18 12:54:41] - Microsoft.Graph.DeviceManagement.Administration - 2.1.0 - 81
[2023-07-18 12:54:41] - Microsoft.PowerShell.Management - 3.1.0.0 - 94
[2023-07-18 12:54:41] - Microsoft.PowerShell.Security - 3.0.0.0 - 15
[2023-07-18 12:54:41] - Microsoft.PowerShell.Utility - 3.1.0.0 - 109
[2023-07-18 12:54:41] - Microsoft.WSMan.Management - 3.0.0.0 - 13
[2023-07-18 12:54:41] - Microsoft365DSC - 1.23.712.1 - 84
[2023-07-18 12:54:41] - PackageManagement - 1.4.8.1 - 13
[2023-07-18 12:54:41] - PowerShellGet - 2.2.5 - 30
[2023-07-18 12:54:41] - VstsAzureHelpers_ - 0.0 - 7
[2023-07-18 12:54:41] - VstsTaskSdk - 0.11.0 - 48
[2023-07-18 12:54:41] -
[2023-07-18 12:54:41] -
[2023-07-18 12:54:41] - current session exports: 6217
[2023-07-18 12:54:41] -
|
|
That's insanely huge! And the problem is that $Global:MaximumFunctionCount loses context when hopping between modules which means that you have to sprinkle it all over the place to a bumped value such as 16K or more (in modules/scripts you control), but for ones that you don't control such as PS built-in commands then it will fail and you don't have access to the source to fix it. An example I already gave was adding $Global:MaximumFunctionCount = 32768 but then calling Import-Module right afterwards fails because there is something inside that makes it trip and there's no way to workaround it so the only way to really fix it is to unload some modules, such as all M365Dsc's dependencies as I shown earlier. |
|
Additionally mind you that as I've also already stated before this might trigger on other places that we are not aware yet where all modules were forced imported and then between module hopping the issue appears again, because of the lost context, and the same type of fix will be required in that place. |
|
This is something I was wondering about. Somehow this Graph module has something else in mind. I output the MaxfunctionCount variable. The output value is the same even in cases that there is an exception. |
|
Having another thought right now. Is it really needed that all modules are loaded/imported during the setup? Shouldn't this be done by PowerShell to load a module on the fly if a command is needed? To answer my question: Within M365DSC we are preloading all modules so that this process is only run once and does not consume any more time... |
|
Knowing what I know now to be perfectly honest with you that was my thought, the problem is that in certain scenarios the modules might not be properly imported if not instructed to do so specifically and I really don't want it to make it work for "me" and then suddenly it breaks for everyone else. Let's have a middle ground right now to get this fix in as is and then we can test letting the session importing the modules on the fly on another release. |
|
Oh by the way, actually I use pipeline caching as a couple of days now and I just copy M365DSC and its dependencies over to Program Files without even importing and I didn't see any problems, whatever modules need to be imported they are imported on the fly. But like I said, let's not make it work for my scenario right now, let's make it work for everyone out there and we can look at this afterwards. |
|
The challenge right now, is to understand the difference between a locally run M365DSC session and the on within a Pipeline. It’s very important that any change we make does not conflict with the on-premises world. I’ve seen several reports of users that have troubles with the V2.+ graph modules over the last few days. V2.1 is an improvement and still there are some issues with this release. Microsoft365DSC relies on these modules and their availability. Also from a performance perspective we need to have these modules in the current session. Loading them individually is too time consuming. I’ll circle around these thoughts and hope to create a discussion or come up with a solution to these issues. |
|
The best way to get out of this is to get everyone on board and make M365dsc and all its modules work with PS7 which would solve a lot of problems |
|
For sure it would. Right now, the challenge lies in Microsoft Teams and Exchange. I was able to successfully run $allResources = Get-M365DSCAllResources
$authType = Get-M365DSCComponentsWithMostSecureAuthenticationType -AuthenticationMethod CertificateThumbprint -Resources $allResourcesI did modify the Update-M365DSCDependency script to not load any module after the installation. All test are currently run within an AZ 9.3.0 environment and $MaximumFunctionCount = 32767 My minified script looks like this at the moment: $MaximumFunctionCount = 32767
Update-M365DSCDependencies
Uninstall-M365DSCOutdatedDependencies
$allResources = Get-M365DSCAllResources
$authType = Get-M365DSCComponentsWithMostSecureAuthenticationType -AuthenticationMethod CertificateThumbprint -Resources $allResourcesUpdate-M365DSCDependencies did get an update as well:
I'll do some exports next. |
|
Either something got fixed on the Azure Pipeline side or the changes for Update-M365DSCDependencies were the game changer. I just added the following lines to my script and am now able to export my tenant without any issues so far. Will try some other workloads as well. My Pipeline: trigger:
- "*"
pool:
vmImage: windows-latest
steps:
- task: AzureCLI@2
displayName: "Install Certificate"
inputs:
azureSubscription: 'KeyVaultConnection'
scriptType: 'ps'
scriptLocation: 'inlineScript'
addSpnToEnvironment: true
inlineScript: |
Set-PSDebug -Trace 1
$vaultName="Graph"
$certName="Graph"
az keyvault secret show --vault-name $vaultName --name $certName | ConvertFrom-Json | Select-Object -Expand value > certName.pfx
Import-PfxCertificate -FilePath certName.pfx Cert:\LocalMachine\My
- task: AzurePowerShell@5
displayName: "Run Exporter script"
inputs:
azurePowerShellVersion: latestVersion
azureSubscription: KeyVaultConnection
scriptType: "filePath"
scriptPath: "Exporter.ps1"
pwsh: false
failOnStandardError: true
errorActionPreference: "stop"
My export script: $Global:MaximumFunctionCount = 32767
$MaximumFunctionCount = 32767
$workingDirectory = $PSScriptRoot
Set-Location -Path $workingDirectory
Write-Log -Message "Switching to path: $workingDirectory"
Write-Log -Message ' '
if (($env:PSModulePath -like "*$($workingDirectory)*") -eq $false)
{
Write-Log -Message 'Adding current folder to PSModulePath'
$env:PSModulePath = $env:PSModulePath.TrimEnd(';') + "$([IO.Path]::PathSeparator)$($workingDirectory)"
}
Update-M365DSCDependencies
Uninstall-M365DSCOutdatedDependencies
$WorkloadName = 'AAD'
$ExportParameters = @{
Path = $env:TEMP
ConfigurationName = 'ConfigureMicrosoft365'
TenantId = '.onmicrosoft.com'
Validate = [Switch]$true
}
$ExportParameters.FileName = 'M365Configuration-{0}.ps1' -f $WorkloadName
$ExportParameters.Workloads = $WorkloadName
$ExportParameters.ApplicationId = 'e8b546df-24e3--b31f-'
$ExportParameters.CertificateThumbprint = 'AABF7B9DB9CF5'
try
{
Export-M365DSCConfiguration @ExportParameters
}
catch
{
Write-Log -Message $_.Exception.Message
}
One thing left: Validation fails with this error ⌛ Export took {216 seconds}
🔍 Starting configuration validation...Results:
[2023-07-18 19:57:46] - Exception calling "Parse" with "1" argument(s): "Guid string should only contain hexadecimal characters." |
|
@ricmestre any chance you could test my change within your environment? Shall I provide my modified module somewhere? |
|
@andikrueger Unfortunately I don't have time today, I'll have a look into it tomorrow. I can write down the changes to the module myself, but it seems you really went brute force there in removing each dependency twice, plus Microsoft.Graph.Authentication per each dependency called Microsoft.Graph*. As they say better safe than sorry! I'll get back to you tomorrow with my results, have a nice one! |
|
@andikrueger All tests done with Az 9.3.0, M365Dsc 1.23.719.1, Graph v2.1, my test script is not exactly like that but the important thing is that steps below are run in order. Defining a new path from where modules can be imported won't help since they are not being installed into $PSScriptRoot. M365Dsc is installed As mentioned before I'm using pipeline caching which installs them in a pipeline task with Save-Module (not for these tests though), next time it runs (when pipeline doesn't fail) if it detects that they are cached then I'm not even installing them, I just copy them into the current session and don't force their import, and even then I think that is still fails if I don't unload all dependencies or at least all Graph modules so I'm carrying a local patched M365DSCUtil.psm1 just like yourself. Here's my results today. EDIT: I have the local patched module but installing the modules like I've just described makes it work, so I don't even need the patched module.
So I already said earlier, if importing the modules implicitly when they are being used instead of forcing them works and doesn't break anyone else's environment then removing Import-Module from Update-M365DSCDependencies is probably the shortest and cleanest solution. You'll also need to change this chunk, just say "Could not update {$($dependency.ModuleName)}" and 's/Mesage/Message'. catch { |
|
Thanks for the update. I’m with a customer right now and we see the same issue about max function count in ISE 🙃 |
|
Totally bonkers! But expected, 4096 is really the default for PS5.1, so not so surprisingly Graph modules make it explode. $MaximumFunctionCount = 4096
Get-Module -ListAvailable Microsoft.Graph.* | `
Select-Object -ExpandProperty name | `
Sort-Object -Unique | `
%{ Import-Module -Force $_ } |
#### Pull Request (PR) description Fixes issues due to MaxFunctionCount in PS5.1 #### This Pull Request (PR) fixes the following issues - Fixes microsoft#3454
Details of the scenario you tried and the problem that is occurring
Note: I know this is not a problem with M365Dsc itself, but that's what I'm using in the pipelines and I don't know where else to report the problem.
Currently Windows containers running on Azure Pipelines on DevOps have Graph v1.28 pre-installed, when installing the latest M365Dsc version v1.23.705.1 they get updated to v2.0 (Update-M365DSCDependencies) and whatever is outdated is uninstalled (Uninstall-M365DSCOutdatedDependencies).
Nevertheless calling simple cmdlets such as Get-MgUser or Get-MgGroup ends up in error, see below, I cannot test every single resource to check which ones are affected by this but my solution relies on calling Get-MgGroup to translate GroupIds to DisplayNames and vice-versa due to the way Assignments work on Intune resources (only by GroupId) which makes upgrading not an option, which at the same time makes me also not able to test the other issues I reported with the latest version.
I already tested this unsuccessfully:
- Removing all Microsoft.Graph.* v1.28 from the session
- Removing all their associated folders from "C:\Program Files\WindowsPowerShell\Modules"
- Install Microsoft.Graph.Authentication v2.0.0
- Install Microsoft.Graph.Groups v2.0.0
- Install MSCloudLoginAssistant v1.0.116
- Connect to Graph using Connect-M365Tenant with certificate
- Calling Get-MgGroup, error happens here
Verbose logs showing the problem
##[error]Method not found: 'Void Microsoft.Graph.Authentication.AzureIdentityAccessTokenProvider..ctor(Azure.Core.TokenCredential, System.String[], Microsoft.Kiota.Authentication.Azure.ObservabilityOptions, System.String[])'Suggested solution to the issue
Update Windows container image to have Graph v2.0 by default instead of v1.28.0.
The DSC configuration that is used to reproduce the issue (as detailed as possible)
N/A
The operating system the target node is running
OsName : Microsoft Windows Server 2022 Datacenter
OsOperatingSystemSKU : DatacenterServerEdition
OsArchitecture : 64-bit
WindowsVersion : 2009
WindowsBuildLabEx : 20348.1.amd64fre.fe_release.210507-1500
OsLanguage : en-US
OsMuiLanguages : {en-US}
Version of the DSC module that was used ('dev' if using current dev branch)
1.23.705.1
The text was updated successfully, but these errors were encountered: