[front] - feat(connections): datasource request flow #6684

JulesBelveze · 2024-08-06T13:39:57Z

Description

This PR implements a new data source request flow, introducing a RequestDataSourcesModal component.

The main changes include:

A new RequestDataSourcesModal component that allows users to request access to data sources they don't currently have permissions for.
Enhancement of the sendEmail function to support cc recipients, enabling the inclusion of data source administrators in request emails.
A new backend endpoint
Integration of the new request flow in the data sources management page.
Updates to the data source selection UI in the assistant builder to accommodate the new request flow.

New look:

References:

[Groups UI] Request data source flow #6630

Risk

Breaking UX changes

Deploy Plan

Deploy tofront-edge
Deploy front

flvndvd · 2024-08-06T14:52:48Z

front/components/data_source/RequestDataSourcesModal.tsx

+  emailRequester?: string;
+  owner: WorkspaceType;
+}) {
+  const res = await fetch(`/api/w/${owner.sId}/data_sources/request-email`, {


Let's simply do request-access as it might not be an email anymore in the future.

flvndvd · 2024-08-06T14:53:18Z

front/pages/api/w/[wId]/data_sources/request-email.ts

+    });
+  }
+
+  const { email, emailMessage, emailRequester, dataSourceName } = req.body;


We should use io-ts here.

fontanierh · 2024-08-06T15:00:28Z

front/pages/api/w/[wId]/data_sources/request-email.ts

+  res: NextApiResponse,
+  auth: Authenticator
+) {
+  const owner = auth.getNonNullableWorkspace();


We can't use nonNullableWorkspace because it throws an error. Here we want to 404 if workspace is null

fontanierh · 2024-08-06T15:00:53Z

front/pages/api/w/[wId]/data_sources/request-email.ts

+  const owner = auth.getNonNullableWorkspace();
+  const user = auth.user();
+
+  if (!user || !owner) {


You already check it here, so as simple as changing to .workspace

fontanierh · 2024-08-06T15:01:43Z

front/pages/w/[wId]/builder/data-sources/managed.tsx

@@ -112,6 +116,16 @@ type GetTableRowParams = {
  setShowPreviewPopupForProvider: (provider: ConnectorProvider | null) => void;
 };

+type ManagedConnector = {


naming is weird to me, since connector is always managed

flvndvd · 2024-08-07T09:01:21Z

front/components/data_source/RequestDataSourcesModal.tsx

+import type { PostRequestAccessBody } from "@app/pages/api/w/[wId]/data_sources/request-access";
+import type { DataSourceIntegration } from "@app/pages/w/[wId]/builder/data-sources/managed";
+
+type RequestDataSourceProps = {


nit

Suggested change

type RequestDataSourceProps = {

interface RequestDataSourceProps {

flvndvd · 2024-08-07T09:03:58Z

front/pages/api/w/[wId]/data_sources/request-access.ts

+export const PostRequestAccessBodySchema = t.type({
+  email: t.string,
+  emailMessage: t.string,
+  emailRequester: t.string,


Should we expose this endpoint behind /data_sources/[name]/request-access so we can infer here the email address of the owner of the data source/data source view?

I think leaving it here would enable us to have a general request-access endpoint. There could be a world in which users ask to an admin to set up a new connector 🤷🏼‍♂️

flvndvd · 2024-08-07T09:04:34Z

front/pages/api/w/[wId]/data_sources/request-access.ts

@@ -0,0 +1,96 @@
+import { isLeft } from "fp-ts/Either";


nit but given that we do data_source we should do request_access.

flvndvd · 2024-08-07T09:06:52Z

front/pages/api/w/[wId]/data_sources/request-access.ts

+  const owner = auth.workspace();
+  const user = auth.user();
+
+  if (!user || !owner) {


We need to check for auth.isUser() to ensure that the user has access to the workspace.

flvndvd · 2024-08-07T09:07:21Z

front/pages/api/w/[wId]/data_sources/request-access.ts

+      subject: `[Dust] Request Data source from ${emailRequester}`,
+      html: sanitizeHtml(html),
+    };
+    await sendEmail(email, message);


We should use sendgrid templates here. Happy to hope on a call to show you.

front/pages/w/[wId]/builder/data-sources/managed.tsx

flvndvd · 2024-08-07T13:33:06Z

front/components/data_source/RequestDataSourcesModal.tsx

+                      email: userEmail,
+                      emailMessage: message,
+                      dataSourceName: selectedDataSourceIntegration.name,
+                      emailRequester: currentUserEmail,


IMHO we should infer this in the endpoint using the auth.

flvndvd · 2024-08-07T13:34:37Z

front/pages/api/w/[wId]/data_sources/request_access.ts

+      dynamic_template_data: {
+        subject: `[Dust] Request Data source from ${emailRequester}`,
+        body,
+      },
+    };


We could have a small helper like sendEmailWithTemplate() that abstract the dynamic_template_data that returns a result so we don't need a try/catch block in the endpoint.

flvndvd

Outside of the two last nits

flvndvd · 2024-08-07T16:23:10Z

front/pages/api/w/[wId]/data_sources/request_access.ts

+async function sendEmailWithTemplate(
+  to: string,
+  from: { name: string; email: string },
+  subject: string,
+  body: string
+): Promise<Result<void, Error>> {


When we only deal with native types for params we should try to use object instead.

flvndvd · 2024-08-07T16:23:35Z

front/pages/api/w/[wId]/data_sources/request_access.ts

+  typeof PostRequestAccessBodySchema
+>;
+
+async function sendEmailWithTemplate(


Let's move this to front/lib/email.ts so we can re-use it 😛

flvndvd · 2024-08-07T16:27:08Z

front/pages/api/w/[wId]/data_sources/request_access.ts

+
+export const PostRequestAccessBodySchema = t.type({
+  emailMessage: t.string,
+  email: t.string,


Reading this one more time, we should make sure to include the user's sId here instead of email. This way, we can prevent any misuse of the endpoint, like sending emails to random addresses. Just a little extra precaution to keep things secure!

- Include the email address of the user who last edited a data source for better tracking [front] - feature: introduce RequestDataSourcesModal component - Implement a new modal allowing non-admin users to request new data sources - Add a button to open the RequestDataSourcesModal in the data sources view - Adjust managed connectors to include the full EditedByUser object [types] - refactor: update EditedByUser type to include email field - Enhance the EditedByUser type with an optional email field for additional user context

…tion - Implement a new modal interface for users to request integration with a specific data source - Add functionality to send an email to the responsible administrator, detailing the integration request - Include appropriate form elements for message composition and data source selection

…nality for data sources - Add a new POST API endpoint to handle email requests for data source connections - Ensure input validation and sanitation for email-related fields before sending the email

… requests - Replace direct sendEmail call with API endpoint for sending request data source emails - Include WorkspaceType owner in the email sending payload to scope the request properly - Change parameter names to better reflect their purpose in the email sending context - Improve error handling for the email sending process by checking response status and providing error messages - Pass owner information when opening RequestDataSourcesModal for proper email request scoping

- Changed the API endpoint to better reflect the action of requesting access rather than sending an email - Renamed the file to match the updated API endpoint path for consistency

…request-access - Change the method used to fetch a non-nullable workspace to a more generic workspace fetching method [pages/w/[wId]/builder/data-sources] - refactor: rename ManagedConnector to ManagedSourceType - Standardize terminology by renaming ManagedConnector type to ManagedSourceType - Adjust variables and mappings to reflect new ManagedSourceType name throughout data-sources managed page

…ludes requester info - Modify sendRequestDataSourceEmail function to guarantee emailRequester is always passed as a non-optional string - Implement type validation using io-ts for request access API endpoint to enhance data integrity

…PI endpoint - Change RequestDataSourceProps from type to interface for consistency - Correct the API endpoint for sending data source requests to use underscore [data_source] - fix: restrict data source request to workspace users - Add check to ensure only authenticated workspace users can send data source request emails

…te ID - Provide a method to fetch `SENDGRID_GENERIC_EMAIL_TEMPLATE_ID` from environment variables [api/data_sources/request_access] - refactor: use dynamic template for sending request access emails - Replace inline email HTML with SendGrid dynamic template for better maintenance and consistency - Include `subject` and sanitized `body` in the dynamic template data for the email - Remove unused import of `sanitize_html` as sanitization is now handled by the template system

- Simplified the email body format for data source access requests by removing HTML tags and fixing a typo

…emplate function - Extract the email sending logic into a dedicated `sendEmailWithTemplate` function - Employ `Result` type from `@dust-tt/types` for error handling in the new function - Replace direct email sending with the new function in the request access handler - Add logging for both successful and failed email sending attempts within the new function

…st access function - The email parameter for sendRequestDataSourceEmail is no longer required and has been removed - The email is now directly obtained from the user object within the API handler logic

- Remove `currentUserEmail` param and use `userEmail` consistently for request access - Update `sendRequestDataSourceEmail` function to accept a single email parameter - Adjust API handler to reflect parameter changes for clarity and consistency - Ensure that email sender is correctly assigned in the request access email logic

…e templates - Added a new function to support sending emails using templates from configuration - Moved `sendEmailWithTemplate` function to the centralized email module for reusability [front/pages/api/w/[wId]/data_sources/request_access] - refactor: update request access to use centralized email sending function - Replaced the local `sendEmailWithTemplate` implementation with the one from the `email` module - Cleaned up the imports and dependencies related to email sending to use the new function

…ntifier to user ID - Replace email with userId to identify the recipient of data source request emails - Modify sendRequestDataSourceEmail and RequestDataSourcesModal to work with userId instead of userEmail [front/lib/email] - refactor: structure sendEmailWithTemplate function to use params object - Refactor sendEmailWithTemplate to take an object with named properties as parameters for better readability [front/lib/resources] - feature: add userId to DataSource editor details - Extend EditedByUser type to include userId property in data source information [front/pages/api] - fix: ensure user existence before sending email - Check whether the recipient user exists and return a 404 error if not found when requesting data source access - Pass structured object to sendEmailWithTemplate for clarity and consistency [types] - feature: introduce userId property to EditedByUser type - Add userId property to EditedByUser type to support user identification changes in front-end components

…fication logic - Move success notification inside try block to ensure it only displays after email is sent - Add error notification to catch block to alert user when email sending fails - Include the error details in logging for better error tracking [front/pages/api/w/[wId]/data_sources] - feature: implement rate limiting for access requests - Add rate limiter to restrict users to a maximum number of access requests per day - Return a 429 error when the rate limit is reached, indicating the user should try again tomorrow - Fix typo in email body content for clarity in access request emails

- Daily access request limit per user has been updated from 1 to 30

flvndvd

LGTM, left one last comment but feel free to merge afterward.

flvndvd · 2024-08-08T11:40:09Z

front/lib/email.ts

+  body: string;
+};
+
+export async function sendEmailWithTemplate({


flvndvd · 2024-08-08T11:41:42Z

front/pages/api/w/[wId]/data_sources/request_access.ts

+  }
+
+  const rateLimitKey = `access_requests:${user.sId}`;
+  const remaining = await rateLimiter({


We should probably do this once we know that the body is good and that the userTo exist. Otherwise, what might happen is that we will remove access request availability even though it did not work.

- Relocated rate limiting code to perform the check after initial request body validation - Ensures body validation is prioritized before rate limiting logic is applied

* [front] - feature: add user email to data source edit history - Include the email address of the user who last edited a data source for better tracking [front] - feature: introduce RequestDataSourcesModal component - Implement a new modal allowing non-admin users to request new data sources - Add a button to open the RequestDataSourcesModal in the data sources view - Adjust managed connectors to include the full EditedByUser object [types] - refactor: update EditedByUser type to include email field - Enhance the EditedByUser type with an optional email field for additional user context * [data_source] - feature: add modal for requesting data source integration - Implement a new modal interface for users to request integration with a specific data source - Add functionality to send an email to the responsible administrator, detailing the integration request - Include appropriate form elements for message composition and data source selection * [api/w/[wId]/data_sources] - feature: implement email request functionality for data sources - Add a new POST API endpoint to handle email requests for data source connections - Ensure input validation and sanitation for email-related fields before sending the email * [data_source] - refactor: update email sending method for data source requests - Replace direct sendEmail call with API endpoint for sending request data source emails - Include WorkspaceType owner in the email sending payload to scope the request properly - Change parameter names to better reflect their purpose in the email sending context - Improve error handling for the email sending process by checking response status and providing error messages - Pass owner information when opening RequestDataSourcesModal for proper email request scoping * [data_source] - refactor: update endpoint for data source access request - Changed the API endpoint to better reflect the action of requesting access rather than sending an email - Renamed the file to match the updated API endpoint path for consistency * [api/wId/data_sources] - refactor: update method to get workspace in request-access - Change the method used to fetch a non-nullable workspace to a more generic workspace fetching method [pages/w/[wId]/builder/data-sources] - refactor: rename ManagedConnector to ManagedSourceType - Standardize terminology by renaming ManagedConnector type to ManagedSourceType - Adjust variables and mappings to reflect new ManagedSourceType name throughout data-sources managed page * [front/components/data_source] - fix: ensure request access email includes requester info - Modify sendRequestDataSourceEmail function to guarantee emailRequester is always passed as a non-optional string - Implement type validation using io-ts for request access API endpoint to enhance data integrity * fix: lint/format * [data_source] - refactor: update RequestDataSourcesModal typing and API endpoint - Change RequestDataSourceProps from type to interface for consistency - Correct the API endpoint for sending data source requests to use underscore [data_source] - fix: restrict data source request to workspace users - Add check to ensure only authenticated workspace users can send data source request emails * [api/config] - feature: add function to retrieve generic email template ID - Provide a method to fetch `SENDGRID_GENERIC_EMAIL_TEMPLATE_ID` from environment variables [api/data_sources/request_access] - refactor: use dynamic template for sending request access emails - Replace inline email HTML with SendGrid dynamic template for better maintenance and consistency - Include `subject` and sanitized `body` in the dynamic template data for the email - Remove unused import of `sanitize_html` as sanitization is now handled by the template system * [front] - fix: streamline email notification format - Simplified the email body format for data source access requests by removing HTML tags and fixing a typo * [api/w/[wId]/data_sources] - refactor: implement email sending with template function - Extract the email sending logic into a dedicated `sendEmailWithTemplate` function - Employ `Result` type from `@dust-tt/types` for error handling in the new function - Replace direct email sending with the new function in the request access handler - Add logging for both successful and failed email sending attempts within the new function * [data_source] - refactor: remove redundant email parameter from request access function - The email parameter for sendRequestDataSourceEmail is no longer required and has been removed - The email is now directly obtained from the user object within the API handler logic * [data_source] - refactor: streamline request access email parameters - Remove `currentUserEmail` param and use `userEmail` consistently for request access - Update `sendRequestDataSourceEmail` function to accept a single email parameter - Adjust API handler to reflect parameter changes for clarity and consistency - Ensure that email sender is correctly assigned in the request access email logic * [front/lib/email] - feature: implement email sending with configurable templates - Added a new function to support sending emails using templates from configuration - Moved `sendEmailWithTemplate` function to the centralized email module for reusability [front/pages/api/w/[wId]/data_sources/request_access] - refactor: update request access to use centralized email sending function - Replaced the local `sendEmailWithTemplate` implementation with the one from the `email` module - Cleaned up the imports and dependencies related to email sending to use the new function * [front/components/data_source] - refactor: change email recipient identifier to user ID - Replace email with userId to identify the recipient of data source request emails - Modify sendRequestDataSourceEmail and RequestDataSourcesModal to work with userId instead of userEmail [front/lib/email] - refactor: structure sendEmailWithTemplate function to use params object - Refactor sendEmailWithTemplate to take an object with named properties as parameters for better readability [front/lib/resources] - feature: add userId to DataSource editor details - Extend EditedByUser type to include userId property in data source information [front/pages/api] - fix: ensure user existence before sending email - Check whether the recipient user exists and return a 404 error if not found when requesting data source access - Pass structured object to sendEmailWithTemplate for clarity and consistency [types] - feature: introduce userId property to EditedByUser type - Add userId property to EditedByUser type to support user identification changes in front-end components * fix: lint/format * [front/components/data_source] - refactor: improve email request notification logic - Move success notification inside try block to ensure it only displays after email is sent - Add error notification to catch block to alert user when email sending fails - Include the error details in logging for better error tracking [front/pages/api/w/[wId]/data_sources] - feature: implement rate limiting for access requests - Add rate limiter to restrict users to a maximum number of access requests per day - Return a 429 error when the rate limit is reached, indicating the user should try again tomorrow - Fix typo in email body content for clarity in access request emails * [api] - feature: increase daily limit for access requests - Daily access request limit per user has been updated from 1 to 30 * [front] - refactor: move rate limiter logic in request_access - Relocated rate limiting code to perform the check after initial request body validation - Ensures body validation is prioritized before rate limiting logic is applied --------- Co-authored-by: Jules <jules@MacBook-Pro.local> Co-authored-by: Jules <jules@MacBook-Pro.lan>

JulesBelveze marked this pull request as draft August 6, 2024 13:40

JulesBelveze marked this pull request as ready for review August 6, 2024 14:50

JulesBelveze requested a review from fontanierh August 6, 2024 14:51

flvndvd reviewed Aug 6, 2024

View reviewed changes

fontanierh reviewed Aug 6, 2024

View reviewed changes

JulesBelveze requested review from fontanierh and flvndvd August 6, 2024 16:20

flvndvd reviewed Aug 7, 2024

View reviewed changes

Jules and others added 17 commits August 8, 2024 09:49

[api/w/[wId]/data_sources] - feature: implement email request functio…

0e39d07

…nality for data sources - Add a new POST API endpoint to handle email requests for data source connections - Ensure input validation and sanitation for email-related fields before sending the email

[data_source] - refactor: update endpoint for data source access request

66f2acf

- Changed the API endpoint to better reflect the action of requesting access rather than sending an email - Renamed the file to match the updated API endpoint path for consistency

fix: lint/format

19b8882

[front] - fix: streamline email notification format

9665d34

- Simplified the email body format for data source access requests by removing HTML tags and fixing a typo

[data_source] - refactor: remove redundant email parameter from reque…

8138155

…st access function - The email parameter for sendRequestDataSourceEmail is no longer required and has been removed - The email is now directly obtained from the user object within the API handler logic

fix: lint/format

a946644

JulesBelveze force-pushed the feat/datasource-request-flow branch from e6bfaea to a946644 Compare August 8, 2024 07:52

Jules added 2 commits August 8, 2024 11:18

[api] - feature: increase daily limit for access requests

b4f099f

- Daily access request limit per user has been updated from 1 to 30

JulesBelveze requested a review from flvndvd August 8, 2024 10:29

flvndvd approved these changes Aug 8, 2024

View reviewed changes

[front] - refactor: move rate limiter logic in request_access

f9b4f2e

- Relocated rate limiting code to perform the check after initial request body validation - Ensures body validation is prioritized before rate limiting logic is applied

JulesBelveze merged commit b8166ef into main Aug 8, 2024
3 checks passed

JulesBelveze deleted the feat/datasource-request-flow branch August 8, 2024 11:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[front] - feat(connections): datasource request flow #6684

[front] - feat(connections): datasource request flow #6684

JulesBelveze commented Aug 6, 2024 •

edited

Loading

flvndvd Aug 6, 2024

flvndvd Aug 6, 2024

fontanierh Aug 6, 2024

fontanierh Aug 6, 2024

fontanierh Aug 6, 2024

flvndvd Aug 7, 2024

flvndvd Aug 7, 2024

JulesBelveze Aug 7, 2024

flvndvd Aug 7, 2024

flvndvd Aug 7, 2024

flvndvd Aug 7, 2024

flvndvd Aug 7, 2024

flvndvd Aug 7, 2024

JulesBelveze Aug 7, 2024

flvndvd Aug 7, 2024

JulesBelveze Aug 7, 2024

flvndvd left a comment

flvndvd Aug 7, 2024

flvndvd Aug 7, 2024

flvndvd Aug 7, 2024

flvndvd left a comment

flvndvd Aug 8, 2024

flvndvd Aug 8, 2024

	type RequestDataSourceProps = {
	interface RequestDataSourceProps {

[front] - feat(connections): datasource request flow #6684

[front] - feat(connections): datasource request flow #6684

Conversation

JulesBelveze commented Aug 6, 2024 • edited Loading

Description

Risk

Deploy Plan

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flvndvd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flvndvd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JulesBelveze commented Aug 6, 2024 •

edited

Loading