v0.3.6

0.3.6

Released on June 6, 2023.

Key Features

• Enables users to selectively or completely disable Aqueduct's artifact snapshotting functionality. When defining a workflow, users can now decide to disable snapshotting for an individual artifact or for all artifacts in a workflow; if selected, these artifacts will only be generated during computation time and then garbage collected. No data will be persisted beyond any data that is explicitly saved. See our documentation for more details.

Enhancements

• Enables on-demand Kubernetes clusters on Google Cloud to use GPU nodes in an autoscaling capacity.

Bugfixes

• Fixes bug where resources could not be deleted even when unused.
• Fixes bug where workflows that succeeded with a warning check were shown in a regular success state instead of a warning state.
• Fixes page load jitter when a large number of workflows are present.

All Changes

v0.3.5...v0.3.6

All commits

• Fix integration - resource rename for doc gen by @likawind in #1395
• Update release note to include links to GKE and GAR docs by @likawind in #1397
• Update doc generation script to account for GAR by @cw75 in #1396
• Many Small Workflows system test by @hsubbaraj-spiral in #1398
• Added notebook for running Simple Model with large data by @hsubbaraj-spiral in #1368
• ENG 3056 Cannot delete resource when no workflow is using it by @eunice-chan in #1382
• ENG 3042 Use checks status in workflow status by @eunice-chan in #1376
• Adds sytem test 2 notebook by @saurav-c in #1402
• ENG-3070 Disable workflows list sorting by @eunice-chan in #1408
• ENG-3034 Frontend Linter Fixes by @agiron123 in #1405
• ENG-3095: Fixes console warning related to @parcel-resolver/buffer by @agiron123 in #1406
• Enable on demand GPU scheduling support for GKE by @jpurusho65 in #1407
• [1/n] Add should persist column to artf table by @likawind in #1390
• bumps version number to 0.3.6 by @hsubbaraj-spiral in #1409

v0.3.5

Released on May 31, 2023.

Key Features

• Adds support for on-demand Kubernetes clusters on Google Cloud; you can now select the Google Cloud option when asking Aqueduct to create a Kubernetes cluster, and Aqueduct will automatically use the Google Kubernetes Engine. See our documentation here.
• Adds support for Google Artifact Registry as a conatiner registry in Aqueduct. You can now use custom Docker images that are pushed to GAR. See our documentation here.

Bugfixes

• Fixes bug where editing certain resource configurations from the UI would cause the resource edit dialog to error out.
• Fixes bug where updating a workflow that has been previously registered but not run would cause the workflow to be duplicated.
• Fixes bug where certain resources could be registered without specifying a name for the resource.

All Changes

v0.3.4...v0.3.5

All commits

• Add support for Google Artifact Registry by @cw75 in #1377
• Fix broken engine_configs due to integration/resource naming inconsistency by @kenxu95 in #1379
• Update Manual UI QA checklist by @eunice-chan in #1373
• Fix UI build on main by @likawind in #1383
• Fix resource editing dialog crashing by @likawind in #1380
• Update EKS VPC Terraform module version number by @cw75 in #1385
• ENG 2714 Migrate get_workflow to use V2 routes by @eunice-chan in #1349
• ENG 3008 Fix registered with no run workflows by @eunice-chan in #1374
• Update workflow links after UI refactor by @likawind in #1387
• ENG-3027: Require Integration Name by @agiron123 in #1360
• Fix default GCS store metadata value by @likawind in #1389
• Fix ondemand k8s showing aws icon rather than k8s by @likawind in #1391
• Rebase code with latest main after the refactor and implement UI for ondemand GKE by @cw75 in #1384
• Fix all lint warnings in UI by @likawind in #1388
• Update manual QA guide with gke by @likawind in #1394
• Rc v035 by @likawind in #1393

v0.3.4

Released on May 24, 2023.

Enhancements

• Improves consistency of icon sizing on resources page.
• Merges flow_id and name into a single argument when retrieving workflows
  from the SDK.
• Adds ability to parametrize save operators when using AWS S3.

Bugfixes

• Fixes bug where race condition could occur when syncing Airflow workflows to
  Aqueduct.
• Fixes bug where listing Snowflake data objects would return an error.

All Changes

v0.3.3...v0.3.4

All commits

• Make resource logo sizing more consistent by @kenxu95 in #1329
• Implement workflow dags endpoint by @likawind in #1330
• Eng 2736 Use V2 Routes for Workflows Page by @eunice-chan in #1345
• Merge id and name arguments to one field by @jpurusho65 in #1325
• Fixes Airflow to Aqueduct syncing bug by @saurav-c in #1347
• ENG-2964 GetEnvironment RTKQuery by @agiron123 in #1343
• Adds Github Action for Airflow compute test by @saurav-c in #1348
• ENG-3010: S3 Dialog Use as Metadata Checkbox fix by @agiron123 in #1352
• ENG-2812: Announcement Banner Error Fixes by @agiron123 in #1351
• Fix missing imports in UI build by @likawind in #1353
• Add back optional flow_id and flow_name arguments by @jpurusho65 in #1350
• ENG-3011 and ENG-3012 Kubernetes/OnDemandKubernetes Dialog Fixes by @agiron123 in #1355
• Add annotation to gate a test case only for relational data integration by @jpurusho65 in #1356
• Add ability to parameterize S3 Saves by @kenxu95 in #1354
• Add wildcard prefix to test-config file in gitignore by @kenxu95 in #1358
• Fix regression of fetching flow by name by @jpurusho65 in #1357
• Fix bug where bigquery extracts fail due to missing import by @kenxu95 in #1359
• Adjust padding around icons in resources/ and dag view by @kenxu95 in #1361
• ENG-3030: fix broken GCS documentation link in GCS dialog by @agiron123 in #1364
• Padding Tweak by @kenxu95 in #1363
• Rename integration to resources in our codebase by @kenxu95 in #1370
• [1/n] Refactor workflow details page (headers, version selection) by @likawind in #1308
• Rc v034 by @eunice-chan in #1367

v0.3.3

Released on May 17, 2023.

Enhancements

• Adds ability to use a parameter in save operators to dynamically name saves.
• Updates the artifact storage card on the resources page to be consistent with
  other resource cards.
• Shows how many workflows are using notification resources.
• Improves form validation with react-hook-forms.
• Replaces the icon that signifies a workflow that has been registered but not
  executed.
• On the resouces page, displays the Conda integration as modifying the Aqueduct
  server execution layer rather than as a separate execution layer.
• Improves BigQuery error checking to ensure that a dataset exists before
  accessing it.

Bugfixes

• Fixes bug where documentation links on the home page were broken.
• Fixes bug where the SDK's generated docstrings were broken.

All Changes

v0.3.2...v0.3.3

All commits

• ENG_2960: Fix broken documentation link for Aqueduct demo resource. by @agiron123 in #1306
• Update the artifact storage card presentation to be consistent with the other data integrations by @kenxu95 in #1297
• Add appropriate workflow count to our connected notification cards and details page by @kenxu95 in #1302
• ENG-2767 react hook form submission by @agiron123 in #1232
• README typo fix by @cw75 in #1313
• Update the workflow "No Run" logo by @kenxu95 in #1312
• Fixes broken storybook build by @agiron123 in #1317
• Eng 2884 test node routes by @eunice-chan in #1299
• Fix generated docstrings when viewing resource methods in Jupyter by @kenxu95 in #1315
• Fix bug where duplicate fields showing in resource details dropdown by @kenxu95 in #1321
• [2/n] Adds gh actions to publish test pypi packages by @likawind in #1262
• [UI] Merge the Conda resource into the Aqueduct Server by @kenxu95 in #1311
• Add ability to parameterize the save operator by @kenxu95 in #1320
• Adds error checking for whether big query dataset exists by @saurav-c in #1319
• Fixes cloudpickle serialization for Python 3.7 by @saurav-c in #1324
• Fix Snowflake save missing schema bug by @kenxu95 in #1331
• ENG-2979 S3 Dialog Validation Fixes by @agiron123 in #1328
• ENG-2979: Fixes bigquery and gcs dialogs that were crashing after file uploads. by @agiron123 in #1326
• Move any overflow rows on the details header fields into an additional column by @kenxu95 in #1322
• Eng 2885 create and test metric and check routes by @eunice-chan in #1323
• Changes schedule for spark periodic tests by @hsubbaraj-spiral in #1336
• Rc v033 by @jpurusho65 in #1334

v0.3.2

Released on May 10, 2023.

Enhancements

• Improves error handling when the user attempts to take an action without
  connecting to the Aqueduct server.
• When connecting an S3 resource to Aqueduct, if the specified subdirectory
  does not exist, it is now automatically created.

Bugfixes

• Replaces outdated integration terminology on the UI.
• Fixes bug where previously created resource names were not editable.

All Changes

v0.3.1...v0.3.2

All commits

• Fix leftover integration renaming on the UI by @cw75 in #1291
• Make the error message of initializing Aqueduct client more informative by @cw75 in #1288
• Fix nil pointer bug for k8s by @cw75 in #1293
• eng-2888-investigate-resource-edit-bug by @eunice-chan in #1273
• Change aqueduct_demo name to Demo by @kenxu95 in #1289
• Add Databricks Periodic Integration tests by @hsubbaraj-spiral in #1270
• Fixes doc generation script by @saurav-c in #1285
• Add Spark EMR periodic tests by @hsubbaraj-spiral in #1294
• Adds guardrails for checks on config by @hsubbaraj-spiral in #1298
• Bumps up timeout for conda integration tests by @saurav-c in #1301
• Increases timeout for Redshift integration test cluster start and stop by @saurav-c in #1300
• Create user supplied root directory on s3 bucket if it does not exist by @jpurusho65 in #1269
• Update README.md by @vsreekanti in #1304
• Release version 0.3.2 by @agiron123 in #1307

v0.3.1

Key Features

• Introduces the aqueduct.llm_op API and the aqueduct-llm package. Aqueduct
  now has support for invoking LLMs with a single API call and comes with
  pre-built Docker images optimized for executing LLMs on Kubernetes. The
  llm_op API supports both ad hoc execution, as pictured below, as well as
  batch execution over a list of inputs or a Pandas Series. See our
  documentation for more details.

  from aqueduct import Client, llm_op
  
  client = Client() # initialize Aqueduct client so we can check if the engine name below is valid
  vicuna = llm_op('vicuna_7b', engine='my_k8s_engine')
  vicuna('What is the best LLM?')

• Reorganizes integrations around the concept of resources. Resources are any
  external tool, system, or API that Aqueduct can connect to; existing data
  and compute integrations are automatically converted into resources. A
  container registry resource is added in this release, and future releases
  will introduce new resource types. The recommended SDK API for accessing
  resources is now client.resource, with client.integration slated to
  deprecated in a future release.
• Allows users to specify a custom Docker image when running an Aqueduct
  operator on Kubernetes. The Docker image is required to have the Aqueduct
  executor scaffolding installed; for more details, please see our
  documentation here.

Enhancements

• Improves logging and error handling when an operator fails because it's able
  to successfully generate a result, typically in the setup phase.
• Enables connecting a Databricks cluster to Aqueduct via the Python SDK.

Bugfixes

• Fixes bug where installing pre-requisites for using Aqueduct-managed
  Kubernetes clusters would fail on an M1 Mac with certain configurations.

All Changes

v0.2.12...v0.3.1

All commits

• Fix spark dkr build by @likawind in #1250
• Eng 2142 add ability to run the conda integration test by @likawind in #1216
• Fix bug where on-demand k8s integration registration fails on Mac by @cw75 in #1242
• [1/n] Allow pypi pkg version injection by splitting version into a separate file by @likawind in #1259
• Updates README.md by @vsreekanti in #1263
• Improve logging when operator failed before generating artifact by @likawind in #1252
• Adds DatabricksConfig to connect_config by @hsubbaraj-spiral in #1256
• Adds support for running integration tests against Airflow by @saurav-c in #1204
• Adds status checks for integration test Redshift cluster by @saurav-c in #1255
• Allow users to specify custom Docker image when running operators on K8s by @cw75 in #1264
• Increase server setup timeouts in integration tests by @likawind in #1254
• Implement use argument for LLM for decorators by @cw75 in #1234
• Add Docker images that includes the LLM library and dependencies by @cw75 in #1235
• Do not schedule execution of save operator if other computer operator… by @jpurusho65 in #1253
• Small improvement to ECR error surfacing by @cw75 in #1272
• Improves error message returned to user when Airflow creds are incorrect by @saurav-c in #1275
• Add MANIFEST file to LLM python package by @cw75 in #1278
• This PR contains everything that's part of resource reframing milestone 1 by @cw75 in #1277
• Rename integrations to resources on the SDK and example notebooks by @kenxu95 in #1276
• Add more white space to the bottom of the integrations page by @cw75 in #1279
• ENG-2895 Updates link for GCS storage documentation in GCS dialog. by @agiron123 in #1281
• Remove additional log line by @jpurusho65 in #1282

v0.2.12

Released on April 25, 2023.

Enhancements

• Improves the describe method on integration objects in the SDK; if there
  was an error connecting or validating the integration, the error will be
  shown.
• Eagerly validates integration connections, so users will be notified
  immediately if they are using a mis-configured integration when, for
  example, attempting to access data.
• Replaces placeholder values in integration connection forms with more
  realistic values.
• Adds support for the most recent version of numpy.

All Changes

v0.2.11...v0.2.12

All commits

* Fix isort in our code and automated linter by @kenxu95 in https://github.com//pull/1227 * Bump numpy version to 1.24.2 by @kenxu95 in https://github.com//pull/1228 * Eng 2635 Add automated periodic testing coverage for MariaDB by @eunice-chan in https://github.com//pull/1225 * Track the execution state of every integration connection on the backend by @kenxu95 in https://github.com//pull/1220 * [SDK] Fail when using any integrations that aren't connected yet by @kenxu95 in https://github.com//pull/1223 * Improve placeholder values for integrations by @hsubbaraj-spiral in https://github.com//pull/1233 * Refactor Spark Python Executor by @hsubbaraj-spiral in https://github.com//pull/1231 * Fix documentation bug for missing local file by @Fanjia-Yan in https://github.com//pull/1230 * Eng 2735 Add order by and limit parameters for V2 workflow results by @eunice-chan in https://github.com//pull/1240 * Rc v0212 by @likawind in https://github.com//pull/1249

v0.2.11

Released on April 18, 2023.

Enhancements

• Extends the Aqueduct-Databricks integration to optionally support long-standing instance pools. If specified, Aqueduct will automatically use nodes in the instance pool, and otherwise, Aqueduct will create a cluster with new instances.
• Improves form validation during integration connection to ensure that valid and invalid form fields are correctly detected and visualized.

Bugfixes

• Fixes bug where MySQL dependency installation could fail on an M-series Mac.
• Fixes bug where integration connections dialogs could re-render repeatedly, causing slight performance degradation.
• Fixes bug where loading a Tensorflow object from a file could fail due to library version mismatches.
• Fixes concurrency bug when multiple processes attempt to simultaneously create the same Kubernetes secret.
• Fixes bug where the release notification banner introduced in the last release was not picking up new Aqueduct versions from PyPI.
• Fixes bug where CUDA versions specified via the SDK may not have been faithfully installed at execution time.

All Changes

v0.2.10...v0.2.11

All commits

• Adds instance pool option to Databricks integration by @hsubbaraj-spiral in #1201
• Use 4 core instead of 8 core machines in GH actions by @kenxu95 in #1196
• Adds Roadmap, updates format of README by @vsreekanti in #1208
• ENG-2759 Installs dependencies for mac mysql connection by @agiron123 in #1202
• ENG-2790 Set Default Tabs for Integrations Dialog by @agiron123 in #1207
• ENG-2789 Fix for Airflow Integration Dialog Rendering Infinitely by @agiron123 in #1206
• Fix creating local tensorflow model data fails with module import and tensorflow versioning by @Fanjia-Yan in #1205
• [1/4] Eng 2711 Implement operator and artifact node views to capture structure information by @likawind in #1164
• ENG-2640: MySQL periodic integration test by @agiron123 in #1179
• Fix bug where race condition may occur when we create k8s secret for AwsCredentialsSecretName by @cw75 in #1214
• Adds support for running BigQuery tests periodically by @saurav-c in #1195
• Eng 2589 add documentation for the local data by @Fanjia-Yan in #1178
• ENG-2804 Check PyPI to determine whether to show Announcement Banner by @agiron123 in #1215
• Fix bug with CUDA versioning on GPUs by @hsubbaraj-spiral in #1212
• [1/2]: ENG-2805: Integration Form Empty State Console Error Fixes by @agiron123 in #1218
• [2/2]: ENG-2806 Ensure Integration Form Validation Works by @agiron123 in #1219
• Adds v2 route to retrieve workflow historical results by @likawind in #1221
• Eng 2733 implement v2 list workflow route by @eunice-chan in #1213
• Reduce concurrency of k8s tests to 1 by @hsubbaraj-spiral in #1222
• Release 0.2.11 by @hsubbaraj-spiral in #1226

v0.2.10

Released on April 11, 2023.

Enhancements

• Enables subdirectory support when configuring the AWS S3 integration. When
  a subdirectory is specified, everything outside of that directory will be
  ignored by Aqueduct.
• Extends support for creating artifact from the local file system to all
  types supported by Aqueduct.
• If executing code on the Aqueduct server without Conda, the server will now
  proactively ensure that the Python version from your environment matches
  the server's Python environment. If there's a mismatch, an error will be
  thrown.
• Adds a dismissable banner to the Aqueduct UI notifying users of new releases.

Bugfixes

• Fixes bug where errors occurring during the execution of a metric were not
  being properly surfaced.
• Fixes bug where checks with a severity level of warning were being shown as
  errors in check history.
• Fixes bug where creating a schedule with helper functions on the SDK would
  fail for monthly schedules.

All Changes

v0.2.9...v0.2.10

All commits

• Fix broken run_notebooks GH action by @kenxu95 in #1176
• Adds automated periodic testing coverage for Redshift by @saurav-c in #1168
• ENG-2639 Postgres Periodic Integration Tests by @agiron123 in #1135
• Parallelize manual_aq_tests/initialize.py by @kenxu95 in #1180
• ENG-2753: Fix issue where warning level checks show as failed in history list. by @agiron123 in #1175
• Enable previews in our integration tests for compute engines by @hsubbaraj-spiral in #1177
• ENG-2757 Fix SDK function arg validation failures by @eunice-chan in #1183
• ENG-2761 Fix issue where errors were not being shown in Metric details pages by @agiron123 in #1187
• Eng 2762 add aqueduct ml version to spark docker by @hsubbaraj-spiral in #1188
• Update CONTRIBUTING.md with some advice about install_local.py by @kenxu95 in #1184
• Made various fixes to our linting process by @kenxu95 in #1189
• Eng 1986 warn user when there is a python version mismatch between server and client by @eunice-chan in #1182
• ENG-2486 Announcement Banner to announce new releases by @agiron123 in #1181
• Allow users to create all types of Artifact from Local Data by @Fanjia-Yan in #1153
• Add support for subfolders for aws s3 by @kenxu95 in #1191
• Eng 1187 Script to automatically run gitbook snippets by @eunice-chan in #1172
• Get sdk integration tests working with Spark by @hsubbaraj-spiral in #1190
• Eng 1187 investigate an automated way to keep all by @eunice-chan in #1194
• Allow more flexibility in the run_tests.py script by @kenxu95 in #1192
• Add flags for finer control of compute engines in testing. by @hsubbaraj-spiral in #1198
• Ready for release 0.2.10 by @cw75 in #1197

v0.2.9

Released on April 5, 2023.

Key Features

• [Beta] Aqueduct now has support for loading dataframe and image data from
  the local filesystem as parameters. See client.create_param() for details.

Enhancements

• Improves the artifact storage management process. The integrations page now
  shows which system is being used for artifact storage, and if a migration
  between artifact stores fails, the error will be surfaced on the
  integration details page until the user triggers a new artifact migration.
• Adds metadata to read and write operators' sidesheets. Both sidesheets now
  show which system is being used for the IO operation, and the save operator
  now shows to what location and (if relevant) in what format the data is
  being saved.

Bugfixes

• Fixes typos in MongoDB connection dialog.
• Fixes bug where workflow status at the top of the workflow details page would
  not update in sync with other parts of the page.
• Fixes a bug where stopping the Aqueduct server immediately after starting it
  could potentially trigger and then kill a workflow run that would then
  stay in a pending state permanently.

All Changes

v0.2.8...v0.2.9

All commits

• Eng 2698 Implement v2 api framework by @likawind in #1148
• ENG 2707 MongoDB text consistency in Integration Connection Card by @eunice-chan in #1157
• [ENG-2621] Fix bug where status in workflow header is not correctly updated by @vsreekanti in #1156
• Customize log message in UpdateOnFailure() by @Fanjia-Yan in #1078
• Allow users to create table and image parameters from local data by @Fanjia-Yan in #1077
• [K8s] Handle case where multiple tests race to create the aqueduct namespace by @kenxu95 in #1159
• [Backend] Improve lifecycle management of the artifact storage migration by @kenxu95 in #1146
• Only upload terraform directory when the K8s cleanup fails by @kenxu95 in #1145
• Update the jupyiter notebook links in example workflows section by @jpurusho65 in #1160
• ENG 2154 Add parameter information to save operator by @eunice-chan in #1137
• Eng 2724 enable usage tracking for code spaces by @likawind in #1154
• Eng 2331 Fix bug where workflow run can stay in pending states forever by @likawind in #1144
• Fix a bug where UI builds works in dev mode but not production by @likawind in #1162
• Eng 2702 Implement v2 workflow metadata routes by @likawind in #1151
• Added documentation links to Databricks Dialog by @hsubbaraj-spiral in #1161
• Adds new example notebooks by @vsreekanti in #1158
• Check for data in response payload for failed execution by @jpurusho65 in #1152
• Update README.md by @vsreekanti in #1169
• Enhance flow_run.describe() to also print failed execution state by @kenxu95 in #1163
• Add market landscape diagram to README by @vsreekanti in #1174
• Release 0.2.9 by @kenxu95 in #1170