Updates 2.x branch with opensearch-spark changes (#139)

* Add table acceleration flyout (#128) * Add table acceleration flyout Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * comment on hardcoded elements Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * additional comment on hardcoded Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * remove console logs Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * review fixes Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * revert version changes, inline type declare Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> --------- Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * Increment version to 3.0.0.0 (#121) Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> * Add materialized view visual builder and query builders (#129) * add materialized view visual builder and query builders Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * organize header and PR comments Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> --------- Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * Define tables (#126) * added changes for sidebar Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added side bar with tables indexed Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added dynamic element for tree view, addressed pr comments Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * made a few design and panel changes, addressed pr comments Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * changed the constants file Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added changes for search bar removal, pr comments Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * readded console.log Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * updated test suites, removed search field Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added changes for Data connection and combo box Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * updated mock for http query Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * updated mock template name in tests Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * addressed pr comments Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * addressed more comments Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added error message for sidebar Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * updated snapshots Signed-off-by: sumukhswamy <sumukhhs@amazon.com> --------- Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * update packages and tsconfig (#130) Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * Update form validations and tests for acceleration (#133) * update form validaitons Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update snapshots, add skipping validator Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update sqlpage snapshot Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add tests for acceleration create Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update acceleration builder tests Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update PR comments Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * adding fix for windows snapshots Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update snapshot files Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> --------- Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * implementing async query support with polling (#131) * successfully getting async post requests Signed-off-by: Paul Sebastian <paulstn@amazon.com> * get query with hardcoded job id working Signed-off-by: Paul Sebastian <paulstn@amazon.com> * working spark query after constant time wait Signed-off-by: Paul Sebastian <paulstn@amazon.com> * added polling with some hardcoded values Signed-off-by: Paul Sebastian <paulstn@amazon.com> * ability to switch between sources implemented Signed-off-by: Paul Sebastian <paulstn@amazon.com> * implemented basic spinner Signed-off-by: Paul Sebastian <paulstn@amazon.com> * small pr asks and cancellation implemented Signed-off-by: Paul Sebastian <paulstn@amazon.com> * fixing small clear state issues Signed-off-by: Paul Sebastian <paulstn@amazon.com> * reduce route name redundancy Signed-off-by: Paul Sebastian <paulstn@amazon.com> * remove multiple query implementation for async Signed-off-by: Paul Sebastian <paulstn@amazon.com> * needed to modify the endpoint Signed-off-by: Paul Sebastian <paulstn@amazon.com> * default data source being Opensearch and updated snapshots Signed-off-by: Paul Sebastian <paulstn@amazon.com> --------- Signed-off-by: Paul Sebastian <paulstn@amazon.com> * added the skipping index queries, covering index queries,updates the … (#134) * added the skipping index queries, covering index queries,updates the data picker Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added the skipping index queries, covering index queries,updates the data picker Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * reverted updates to yarn Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * updated tests, snapshots Signed-off-by: sumukhswamy <sumukhhs@amazon.com> --------- Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * Acceleration related changes and minor fixes (#135) * add acc index flyout Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * remove [if not exists] from acc creation Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * merge tableview from main Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * adding acc index flyout Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add hash router Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * hide materialized view index type Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update snapshots Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * loading combo boxes for acc flyout Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * adding acceleration backend integ Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update jest tests for acc flyout Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add redirection support for home page Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * fix primary shards count and replica validation Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * remove the caution callout for acc flyout Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * support acc flyout redirection from data sources Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> --------- Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * disable buttons while running async query (#136) * disable most buttons Signed-off-by: Paul Sebastian <paulstn@amazon.com> * code editor read only Signed-off-by: Paul Sebastian <paulstn@amazon.com> * disabled accelerate button Signed-off-by: Paul Sebastian <paulstn@amazon.com> --------- Signed-off-by: Paul Sebastian <paulstn@amazon.com> * Design changes for the sidebar and update to create button (#138) * added changes for loading, create button Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * removed comments, updated snapshots Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * addressed pr comments Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * addressed pr comments Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * addressed pr comments Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * updated snapshots, constant for skipping index Signed-off-by: sumukhswamy <sumukhhs@amazon.com> --------- Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * Load table fields for acceleration index flyout (#137) * load table fields after table is selected Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * simplify the map function Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> --------- Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> --------- Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Signed-off-by: sumukhswamy <sumukhhs@amazon.com> Signed-off-by: Paul Sebastian <paulstn@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: sumukhswamy <sumukhhs@amazon.com> Co-authored-by: Paul Sebastian <paul_sebastian@live.com> Co-authored-by: Paul Sebastian <paulstn@amazon.com>
opensearch-project · Oct 4, 2023 · fae31c9 · fae31c9
1 parent 3f5817f
commit fae31c9
Show file tree

Hide file tree

Showing 82 changed files with 21,515 additions and 3,414 deletions.
diff --git a/common/constants/index.ts b/common/constants/index.ts
@@ -0,0 +1,82 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+export const PLUGIN_ID = 'queryWorkbenchDashboards';
+export const PLUGIN_NAME = 'Query Workbench';
+export const OPENSEARCH_ACC_DOCUMENTATION_URL = 'https://opensearch.org/docs/latest';
+export const ACC_INDEX_TYPE_DOCUMENTATION_URL = 'https://opensearch.org/docs/latest';
+
+export const SKIPPING_INDEX = `skipping_index`
+export const ON_LOAD_QUERY = `SHOW tables LIKE '%';`;
+export const SKIPPING_INDEX_QUERY = `CREATE SKIPPING INDEX ON myS3.logs_db.http_logs 
+(status VALUE_SET) 
+WITH (
+  auto_refresh = true
+  )`
+export const COVERING_INDEX_QUERY =`CREATE INDEX covering_idx ON myS3.logs_db.http_logs
+ (status) 
+ WITH (
+  auto_refresh = true
+  )`
+export const CREATE_DATABASE_QUERY =`CREATE DATABASE myS3.logs_db`
+export const CREATE_TABLE_QUERY =`CREATE EXTERNAL TABLE logs (
+  key BIGINT,
+  status INTEGER,
+  size FLOAT,
+  agent STRING,
+  timestamp DATE
+)
+USING JSON
+OPTIONS (
+  path 's3://test/path',
+  compression 'gzip'
+);`
+
+export const ACCELERATION_INDEX_TYPES = [
+  { label: 'Skipping Index', value: 'skipping' },
+  { label: 'Covering Index', value: 'covering' },
+  // { label: 'Materialized View', value: 'materialized' }, Hidden Option -> Until opensearch-spark feature is ready
+];
+
+export const ACCELERATION_AGGREGRATION_FUNCTIONS = [
+  { label: 'count' },
+  { label: 'sum' },
+  { label: 'avg' },
+  { label: 'max' },
+  { label: 'min' },
+];
+
+export const ACCELERATION_TIME_INTERVAL = [
+  { text: 'millisecond(s)', value: 'millisecond' },
+  { text: 'second(s)', value: 'second' },
+  { text: 'hour(s)', value: 'hour' },
+  { text: 'day(s)', value: 'day' },
+  { text: 'week(s)', value: 'week' },
+];
+
+export const SKIPPING_INDEX_ACCELERATION_METHODS = [
+  { value: 'PARTITION', text: 'Partition' },
+  { value: 'VALUE_SET', text: 'Value Set' },
+  { value: 'MIN_MAX', text: 'Min Max' },
+];
+
+export const ACCELERATION_ADD_FIELDS_TEXT = '(add fields here)';
+export const ACCELERATION_INDEX_NAME_REGEX = /^[a-z][a-z_\-]*$/;
+export const ACCELERATION_S3_URL_REGEX = /^(s3|s3a):\/\/[a-zA-Z0-9.\-]+\/.*/;
+export const ACCELERATION_DEFUALT_SKIPPING_INDEX_NAME = 'skipping';
+
+export const ACCELERATION_INDEX_NAME_INFO = `All OpenSearch acceleration indices have a naming format of pattern: \`prefix_<index name>_suffix\`. They share a common prefix structure, which is \`flint_<data source name>_<database name>_<table name>_\`. Additionally, they may have a suffix that varies based on the index type. 
+##### Skipping Index
+- For 'Skipping' indices, a fixed index name 'skipping' is used, and this name cannot be modified by the user. The suffix added to this type is \`_index\`.
+  - An example of a 'Skipping' index name would be: \`flint_mydatasource_mydb_mytable_skipping_index\`.
+##### Covering Index
+- 'Covering' indices allow users to specify their index name. The suffix added to this type is \`_index\`.
+  - For instance, a 'Covering' index name could be: \`flint_mydatasource_mydb_mytable_myindexname_index\`.
+##### Materialized View Index
+- 'Materialized View' indices also enable users to define their index name, but they do not have a suffix.
+  - An example of a 'Materialized View' index name might look like: \`flint_mydatasource_mydb_mytable_myindexname\`.
+##### Note:
+- All user given index names must be in lowercase letters. Cannot begin with underscores or hyphens. Spaces, commas, and characters :, ", *, +, /, \, |, ?, #, >, or < are not allowed.  
+  `;
diff --git a/common/index.ts b/common/index.ts
diff --git a/common/types/index.ts b/common/types/index.ts
@@ -0,0 +1,80 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+export type AggregationFunctionType = 'count' | 'sum' | 'avg' | 'max' | 'min';
+
+export interface MaterializedViewColumn {
+  id: string;
+  functionName: AggregationFunctionType;
+  functionParam: string;
+  fieldAlias?: string;
+}
+
+export type SkippingIndexAccMethodType = 'PARTITION' | 'VALUE_SET' | 'MIN_MAX';
+
+export interface SkippingIndexRowType {
+  id: string;
+  fieldName: string;
+  dataType: string;
+  accelerationMethod: SkippingIndexAccMethodType;
+}
+
+export interface DataTableFieldsType {
+  id: string;
+  fieldName: string;
+  dataType: string;
+}
+
+export interface RefreshIntervalType {
+  refreshWindow: number;
+  refreshInterval: string;
+}
+
+export type AccelerationIndexType = 'skipping' | 'covering' | 'materialized';
+
+export interface GroupByTumbleType {
+  timeField: string;
+  tumbleWindow: number;
+  tumbleInterval: string;
+}
+
+export interface materializedViewQueryType {
+  columnsValues: MaterializedViewColumn[];
+  groupByTumbleValue: GroupByTumbleType;
+}
+
+export interface FormErrorsType {
+  dataSourceError: string[];
+  databaseError: string[];
+  dataTableError: string[];
+  skippingIndexError: string[];
+  coveringIndexError: string[];
+  materializedViewError: string[];
+  indexNameError: string[];
+  primaryShardsError: string[];
+  replicaShardsError: string[];
+  refreshIntervalError: string[];
+  checkpointLocationError: string[];
+}
+
+export interface CreateAccelerationForm {
+  dataSource: string;
+  database: string;
+  dataTable: string;
+  dataTableFields: DataTableFieldsType[];
+  accelerationIndexType: AccelerationIndexType;
+  skippingIndexQueryData: SkippingIndexRowType[];
+  coveringIndexQueryData: string[];
+  materializedViewQueryData: materializedViewQueryType;
+  accelerationIndexName: string;
+  primaryShardsCount: number;
+  replicaShardsCount: number;
+  refreshType: 'interval' | 'auto';
+  checkpointLocation: string | undefined;
+  refreshIntervalOptions: RefreshIntervalType;
+  formErrors: FormErrorsType;
+}
+
+export type AsyncQueryLoadingStatus = "SUCCESS" | "FAILED" | "RUNNING" | "SCHEDULED" | "CANCELED"
diff --git a/package.json b/package.json
@@ -27,35 +27,23 @@
   },
   "devDependencies": {
     "@testing-library/user-event": "^13.1.9",
-    "@types/hapi-latest": "npm:@types/hapi@18.0.7",
-    "@types/react-router-dom": "^5.3.2",
+    "@types/enzyme-adapter-react-16": "^1.0.6",
+    "@types/react-test-renderer": "^16.9.1",
     "cypress": "^5.0.0",
     "eslint": "^6.8.0",
-    "eslint-plugin-no-unsanitized": "^3.0.2",
-    "eslint-plugin-prefer-object-spread": "^1.2.1",
     "husky": "^4.2.5",
     "jest-raw-loader": "^1.0.1",
     "lint-staged": "^10.2.0",
     "mutationobserver-shim": "^0.3.3",
-    "ts-jest": "^26.1.0",
-    "ts-node": "^8.9.1",
-    "tslint": "^6.1.2",
-    "tslint-config-prettier": "^1.18.0",
-    "tslint-plugin-prettier": "^2.0.1"
+    "jest-dom": "^4.0.0",
+    "ts-jest": "^29.1.0"
   },
   "resolutions": {
-    "**/@types/node": "^10.12.27",
-    "@types/react": "^16.3.14",
-    "**/@types/angular": "^1.6.50",
-    "**/@types/jest": "^24.0.9",
-    "**/@types/react-dom": "^16.0.5",
-    "**/@types/react-router-dom": "^4.3.1",
+    "ansi-regex": "^5.0.1",
     "eslint-utils": "^2.0.0",
     "json-schema": "^0.4.0",
-    "**/@types/react": "^16.3.14",
-    "ansi-regex": "^5.0.1",
     "yaml": "^2.2.2",
     "tough-cookie": "^4.1.3",
     "semver": "^7.5.2"
   }
-}
+}
diff --git a/public/ace-themes/sql_console.js b/public/ace-themes/sql_console.js
@@ -3,10 +3,13 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-
 import * as ace from 'brace';
 
-ace.define('ace/theme/sql_console', ['require', 'exports', 'module', 'ace/lib/dom'], function (acequire, exports, module) {
+ace.define('ace/theme/sql_console', ['require', 'exports', 'module', 'ace/lib/dom'], function (
+  acequire,
+  exports,
+  module
+) {
   exports.isDark = false;
   exports.cssClass = 'ace-sql-console';
   exports.cssText = require('../index.scss');