-
Notifications
You must be signed in to change notification settings - Fork 477
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Chime chat(messaging) messagingSessionDidStop did not fire on disconnect #2372
Comments
May be this was fixed in Could you please check upgrading to |
I will: While I'm doing this, can you answer the following: From AWS documentation: When the application uses a close code to reconnect, the application should:
What event is supposed to through the close code and how are we supposed to listen for it? |
When you are using |
We updated amazon-chime-sdk-js to 3.6.0 I'm attaching an example log file: Any help would be greatly appreciated. |
The fix has not been made in the 3.x line yet. It is present in the latest 2.30.1 version We are working on the 3.x line, an initial attempt at that was made here: #2180. This had to be reverted because amazon-chime-sdk-js 3.x uses the new AWS javascript v3 client. The v3 aws client is modular and has (1) a new Command based API and (2) a backwards compatible based API. We are still figuring out how to make the fix in a way that will work regardless of which one of those 2 APIs is supplied to the amazon-chime-sdk-js. The initial fix only worked when using the Command based API. |
Updated 3.x line PR for fix on reconnect with 4401: https://github.com/aws/amazon-chime-sdk-js/pull/2400/files |
This is great news. |
Fix was was released in 3.7 here: https://github.com/aws/amazon-chime-sdk-js/tree/v3.7.0. |
We applied the fix, we are not getting tons of error messages intermittently This is a huge deal for us. |
Can you show how you are initiating the Messaging Session Configuration and how you are importing the AWS client |
{
"sequenceNumber": 0,
"message": "adding messaging observer",
"timestampMs": 1661348491672,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 1,
"message": "opening connection to wss://node001.ue1.ws-messaging.chime.aws/connect?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA23LK62QLMEZNOYUB%2F20220824%2Fus-east-1%2Fchime%2Faws4_request&X-Amz-Date=20220824T134131Z&X-Amz-Expires=10&X-Amz-Security-Token=FwoGZXIvYXdzED8aDCJvwZdV3PA%2B%2FKi56iKVApqRTqNovIv4wRar2BAmOpUY5uB0Om6y9TqDnFJIS7IRLA6SM2KrKKJfdFRUtYZONO%2B20dfp6joVqKSDQX5hUJYZmRv3TjHorYh3BJ5k3ZWAaA5CuFkSgB1uxgrx2zuDboZuUYV8e3ZA53OduVPbYmrpTy7hzGDo7j67f%2F75LGdwUk%2BDckfRQP4N4V%2BUU6ON0JyoV8vhcy%2FxV8hh0iK9K5twOfS4%2FoAKUmBHyuDQnISqwpz%2BJuYISCP0zMShh5JJ2tFduWFuKFxt%2BfZ7zKGNyJ8HGaz2Y7jeWClBQCUxT4pCCHwidsdRMYTNNEJxAXntmUxTmg%2BRRESOWegga1CvWSRsisXX7FKfaIU3QiKMcLe3CadU6Tkog9WYmAYyLe%2BMAQ4R0uImKQX1x4apMuze44atZ%2FVw2vE%2FtrP3qbHe%2FuLomzTi6VhACBoHRA%3D%3D&X-Amz-SignedHeaders=host&X-Amzn-User-Agent=chrome-104&X-Amzn-Version=3.7.0&sessionId=c4c6041f-da73-4d5e-a45d-d88037b8ebae&userArn=arn%3Aaws%3Achime%3Aus-east-1%3A745938015254%3Aapp-instance%2F1aefb1f1-34c1-44dc-9b91-9981f848e5b9%2Fuser%2F27-5492-21f34b2f-0fe0-4baa-8a1a-cabe570923c5&X-Amz-Signature=6e0a74f401dba141c0df7aceae6ded37acfc020a6a54ec66843a215464b2e320",
"timestampMs": 1661348491762,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 2,
"message": "WebSocket close: 1006 ",
"timestampMs": 1661352094343,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 3,
"message": "Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url')",
"timestampMs": 1661352094853,
"logLevel": "ERROR",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 4,
"message": "WebSocket close: 4999 Failed to get messaging session endpoint URL",
"timestampMs": 1661352094855,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 5,
"message": "Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url')",
"timestampMs": 1661352096833,
"logLevel": "ERROR",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 6,
"message": "WebSocket close: 4999 Failed to get messaging session endpoint URL",
"timestampMs": 1661352096833,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 7,
"message": "Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url')",
"timestampMs": 1661352100548,
"logLevel": "ERROR",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 8,
"message": "WebSocket close: 4999 Failed to get messaging session endpoint URL",
"timestampMs": 1661352100548,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 9,
"message": "Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url')",
"timestampMs": 1661352104907,
"logLevel": "ERROR",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 10,
"message": "WebSocket close: 4999 Failed to get messaging session endpoint URL",
"timestampMs": 1661352104907,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 11,
"message": "Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url')",
"timestampMs": 1661352107457,
"logLevel": "ERROR",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 12,
"message": "WebSocket close: 4999 Failed to get messaging session endpoint URL",
"timestampMs": 1661352107457,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 13,
"message": "Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url')",
"timestampMs": 1661352108799,
"logLevel": "ERROR",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 14,
"message": "WebSocket close: 4999 Failed to get messaging session endpoint URL",
"timestampMs": 1661352108799,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 15,
"message": "Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url')",
"timestampMs": 1661352111128,
"logLevel": "ERROR",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 16,
"message": "WebSocket close: 4999 Failed to get messaging session endpoint URL",
"timestampMs": 1661352111129,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 17,
"message": "Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url')",
"timestampMs": 1661352112263,
"logLevel": "ERROR",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 18,
"message": "WebSocket close: 4999 Failed to get messaging session endpoint URL",
"timestampMs": 1661352112263,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 19,
"message": "Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url')",
"timestampMs": 1661352113186,
"logLevel": "ERROR",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 20,
"message": "WebSocket close: 4999 Failed to get messaging session endpoint URL",
"timestampMs": 1661352113187,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 21,
"message": "Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url')",
"timestampMs": 1661352118127,
"logLevel": "ERROR",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 22,
"message": "WebSocket close: 4999 Failed to get messaging session endpoint URL",
"timestampMs": 1661352118127,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 23,
"message": "Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url')",
"timestampMs": 1661352118456,
"logLevel": "ERROR",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
{
"sequenceNumber": 24,
"message": "WebSocket close: 4999 Failed to get messaging session endpoint URL",
"timestampMs": 1661352118456,
"logLevel": "INFO",
"tenantID": "27",
"sessionId": "c4c6041f-da73-4d5e-a45d-d88037b8ebae",
"userName": "MSnookFehr"
}
|
Message Session Connection: async connect(currentUser: User, currentTenant: Tenant) {
} |
imports import { v4 as uuid } from 'uuid'; import { } from 'amazon-chime-sdk-js'; |
If we revert to 2.31.0 will we have the same issue? |
What version of the AWS client are you using?
Can you debug where The Demo here shows an example of how it is working with 3.x AWS SDK: https://github.com/aws/amazon-chime-sdk-js/blob/v3.7.0/demos/browser/app/messagingSession/messagingSession.ts#L64, but would like to dive deeper into what you are seeing as well |
package.json: "aws-sdk": "^2.1148.0", npm list aws-sdk |
These logs are coming from the server in our production environment - and they are sporadic, not everyone is having the issue. So the question is how to recreate to debug. Throttling or taking the browser off line, then back on may do it, but I have other processes running in my app that will throw errors so hard to capture just for messaging. Can you simulate the WebSocket close: 1006 error when you test? Unreliable connection when the amazon-chime-sdk-js ping does not get a response? When you do, what happens when the sdk tries to reconnect? |
1006 means closed abnormally and is normally due to a network disconnect. You can likely simulate by running the app on wifi and then turning off the wifi connection. If this was the case this would match the "WebSocket close: 4999 Failed to get messaging session endpoint URL" that is retrying to reconnect. The expectation is that those will repeat while network connectivity is down. When network connectivity is restored the WebSocket will automatically reconnect (with the latest fix) |
In the 3x code (The Demo here shows an example of how it is working with 3.x AWS SDK) You have the line: this.configuration = new MessagingSessionConfiguration(this.userArn, this.sessionId, undefined, chime); |
In the newer versions the chime client is used internally to resolve the session endpoint. This is done because on reconnect the endpoint might change. You can still pass it in, but it will only be used for the initial connect. |
If I pass in the endpoint as undefined in the MessagingSessionConfiguration
I get 2022-08-24T17:23:32.041Z [ERROR] SDK Chat - Messaging Session failed to resolve endpoint: TypeError: Cannot read properties of undefined (reading 'Url') This is not what I should be getting, correct? |
Do I need to install AWS SDK version 3.x? as well as the amazon-chime-sdk-js? |
That is correct, will look into getting this fixed. The PR here has both tests and manual test instructions for the AWS SDK version 3.x client, for both the 3.x clients APIs; the new command based way, and the backwards compatible API. Based on what you are saying the test for the backwards compatible API using 3.x AWS does not seem to be working when AWS SDK 2.x client is used. Will see if I can reproduce and get a fix that will make AWS SDK 2.x work with the chime sdk 3.x I do see however that there appears to be a slight different on the chime aws client sdk 2.x (only supports AWS SDK 2.x) and 3.x on the implementation of backwards compatible API.
That is likely your quickest path forward OR trying the amazon-chime-sdk-js 2.x with AWS SDK 2.x Thanks for pointing this out |
Updated to AWS SDK 3.x using "@aws-sdk/client-chime-sdk-messaging": "^3.154.0" and moved to prod last night. We're watching the logs. We were getting some looping when the credentials expire - but this may be due problems with our shutdown logic. However, you may want to check what happens in the reconnect logic when credentials expire. You may want to end the reconnect attempt loop if expired credentials. |
The AWS client that you pass in can be configured with logic to refresh the credentials when expired. Depending on how credentials are initialized this can be auto configured but sometimes requires manual configuration. The retry logic should be somewhat configurable here:
Would be interested if there are different options that would be of use? Perhaps a max retry before giving up, or giving up if we can detect if credentials are not configured to refresh |
Is there a reason you would want to continue retrying if credentials are expired (receive an ExpiredTokenException)? I'm not sure off hand. Also, do you have any example code for refreshing credentials in version 3.x - we destroy and recreate ChimeSDKMessagingClient, ChimeSDKMessaging, and DefaultMessagingSession with new credentials 5 min before they expire but was wondering if there is a cleaner way... |
Looks like we've captured the looping scenario in chatlog5.txt attached above.
What should be happening in this scenario with 3.x? Thanks again for all your help. |
any updates? i have the same problem. FYI: The SDK version i used.
|
We have been able to identify the root cause of the infinite reconnect loop on credential expiry and are working on a fix. In the meantime, if credentials are refreshed periodically prior to expiry, the above scenario will not be hit. How are you refreshing credentials with the aws sdk v3 upgrade? Are you using aws Cognito identity? |
Approach to refreshing credentials in 3.x Currently, we have custom timer that runs every minute and checks to see if the credentials are expired. If they are getting close to expiration, we
Is there a more elegant way to refresh the credentials in 3.x? |
I believe works as follows. On AWS 3.x you can provide the client with a credential provider. The credential provider is a function that returns a promise of credentials. The credentials can have an I used the following code snip it to test/demonstrate: let firstCall = true;
const credentialsThatAreActuallyValidFor1Hour = await this.fetchCredentials();
// A credential provider just is a function that returns a promise with credentials
const credentialProviderExample = function() {
return new Promise((resolve, reject)=> {
console.log("Credential Provider called to resolve credentials, is first call " + firstCall);
// credential expiration is a Date object (getTime() is called internally)
const credentialsExpireAt = new Date();
credentialsExpireAt.setSeconds(credentialsExpireAt.getSeconds() + 1)
credentialsThatAreActuallyValidFor1Hour.expiration = credentialsExpireAt;
// To demonstrate refreshing credentials, always return expired credentials on first call
if (firstCall) {
firstCall = false;
setTimeout(()=> {
resolve(credentialsThatAreActuallyValidFor1Hour)
}, 2000)
} else {
resolve(credentialsThatAreActuallyValidFor1Hour);
}
});
}
// Initialize client with credential provider
const chime = new ChimeSDKMessagingClient({ region: 'us-east-1', credentials: credentialProviderExample });
// not shown, make an API call and got on the console:
|
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.pl-k
{mso-style-name:pl-k;}
span.pl-s1
{mso-style-name:pl-s1;}
span.pl-c1
{mso-style-name:pl-c1;}
span.pl-kos
{mso-style-name:pl-kos;}
span.pl-smi
{mso-style-name:pl-smi;}
span.pl-en
{mso-style-name:pl-en;}
span.pl-c
{mso-style-name:pl-c;}
span.pl-s
{mso-style-name:pl-s;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
-->Thank you very much. This was very helpful 😊 Scott IngulliCEO, ***@***.*** From: David WitherspoonSent: Friday, September 9, 2022 3:50 PMTo: aws/amazon-chime-sdk-jsCc: singulli1; AuthorSubject: Re: [aws/amazon-chime-sdk-js] Chime chat(messaging) messagingSessionDidStop did not fire on disconnect (Issue #2372) I believe works as follows. On AWS 3.x you can provide the client with a credential provider. The credential provider is a function that returns a promise of credentials. The credentials can have an expiration set. When the client calls the APIs it will get/store credentials from the credential provider. When those credentials expire (based on expiration) the client will get new credentials from the provider.I used the following code snip it to test/demonstrate:let firstCall = true;const credentialsThatAreActuallyValidFor1Hour = await this.fetchCredentials(); // A credential provider just is a function that returns a promise with credentialsconst credentialProviderExample = function() { return new Promise((resolve, reject)=> { console.log("Credential Provider called to resolve credentials, is first call " + firstCall); // credential expiration is a Date object (getTime() is called internally) const credentialsExpireAt = new Date(); credentialsExpireAt.setSeconds(credentialsExpireAt.getSeconds() + 1) credentialsThatAreActuallyValidFor1Hour.expiration = credentialsExpireAt; // To demonstrate refreshing credentials, always return expired credentials on first call if (firstCall) { firstCall = false; setTimeout(()=> { resolve(credentialsThatAreActuallyValidFor1Hour) }, 2000) } else { resolve(credentialsThatAreActuallyValidFor1Hour); } });} // Initialize client with credential providerconst chime = new ChimeSDKMessagingClient({ region: 'us-east-1', credentials: credentialProviderExample }); // not shown, make an API calland got on the console:Calling APICredential Provider called to resolve credentials, is first call trueCredential Provider called to resolve credentials, is first call falseAPI Called successfully—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Above from dpwspoon shows how to refresh credentials, we don't expect any of the issues would have been hit if done that way. The release 3.8.0 fixes the following :
Feel free to reopen. |
What happened and what did you expect to happen?
From time to time the chat websocket will disconnect.
We were able to capture a disconnect in the below log file:
The log shows a webSocket close 1006, opening a connection , and another webSocket close of 4401.
The documentation : https://docs.aws.amazon.com/chime-sdk/latest/dg/websockets.html
mentions handling disconnects.
I'm assuming that a disconnect will fire the messagingSessionDidStop event of the MessagingSessionObserver. However, in the log file, I didn't see this happening.
What event should we use to initiate a reconnect? I'm assuming its the messagingSessionDidStop event but this isn't explicitly stated in the documentation.
Have you reviewed our existing documentation?
Reproduction steps
See above
Amazon Chime SDK for JavaScript version
2.24.0
What browsers are you seeing the problem on?
chrome
Browser version
Version 103.0.5060.114
Meeting and Attendee ID Information.
No response
Browser console logs
see above
The text was updated successfully, but these errors were encountered: