Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metastore API spec #7

Open
wants to merge 60 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
bf00994
[SPARK] gitignore signing keys and access tokens (#396)
vksx Sep 11, 2024
8599fd2
enable Delta tests and fix gs/abfs credential (#468)
cloud-fan Sep 11, 2024
8bdbb99
Fix behavior of creation of managed tables (#449)
cloud-fan Sep 11, 2024
6b4be80
Expose and Publish default port 8080 for start-uc-server-docker. (#440)
yaoderek Sep 12, 2024
19c8dd5
update docs by running `sbt generate` (#474)
vikrantpuppala Sep 12, 2024
7ddb4d5
Feat/login UI (#450)
jamieknight-db Sep 12, 2024
9ff56d4
Add owner, createdBy, updatedBy to all securables and regen openapi d…
kriscon-db Sep 12, 2024
41f5e11
Run `javafmtAll` on main repo (#479)
vikrantpuppala Sep 13, 2024
f884816
Enable identity logging on create/update paths (#483)
kriscon-db Sep 13, 2024
842ee33
Specify release settings for spark connector (#484)
bijanhoule Sep 13, 2024
6ab4a64
[SPARK] Test parquet table creation by Spark 3.5.3 (#357)
amaliujia Sep 14, 2024
1b73ebb
Build/release fixes (#489)
bijanhoule Sep 16, 2024
f6f4f70
UI READ support for models and versions (#460)
jamieknight-db Sep 17, 2024
a4e7e4e
Add temporary path credentials api (#478)
vikrantpuppala Sep 17, 2024
4a27bf8
Fixed UC Cli NullPointerException for invalid entity types (#445)
yaoderek Sep 17, 2024
ae5bff2
Update UI Readme (#459)
JuanPabloDiaz Sep 17, 2024
67afede
Mark spark-sql dependency as Provided (#490)
bijanhoule Sep 17, 2024
26d2303
add new acl fields to all entities (#491)
jamieknight-db Sep 17, 2024
d7f05fa
Added a python script explaining how to integration test mlflow with …
kriscon-db Sep 18, 2024
6a214a8
Fix feature warning by adding import clause in UCSingleCatalog.scala …
vksx Sep 18, 2024
4e5d6a5
docs: add docs for ORC tables (#388)
avriiil Sep 20, 2024
c70d837
Docs: Managed vs External tables (#471)
avriiil Sep 20, 2024
123e151
UC should work without Delta Lake (#467)
cloud-fan Sep 21, 2024
a7cfdc9
docs: fix Daft integration guide (#498)
peter-gy Sep 21, 2024
62d819d
Added initial documentation for models (#447)
kriscon-db Sep 23, 2024
9af41c6
fix: show detail message while create catalog (#497)
foxty Sep 24, 2024
aee192c
Updated Delta 3.2.1 to new RC (#505)
tdas Sep 24, 2024
f9a9bf1
Basic server access control (#378)
creechy Sep 25, 2024
5d7ba0d
Fix duplicate index name. (#492)
creechy Sep 25, 2024
b550971
Change wording on successful identity provider validation. (#509)
creechy Sep 25, 2024
ddf5496
support create external delta table (#473)
cloud-fan Sep 26, 2024
caebaa7
Include Rapidstart.md to run Spark, MLflow, and Auth integration end-…
dennyglee Sep 26, 2024
50803bd
fix ui readme image link (#514)
jamieknight-db Sep 26, 2024
7e6e7f4
Mlflow docs patch (#515)
dennyglee Sep 26, 2024
af6365a
Add access control to temporary credentials endpoints. (#511)
creechy Sep 26, 2024
d79a9f5
Remove delta 3.2.1 prerelease resolver, official release is out (#518)
bijanhoule Sep 26, 2024
706f1a5
Remove PATH_REFRESH as it is an unused path operation (#520)
rameshchandra Sep 27, 2024
c5b687d
Update UC docs (#526)
dennyglee Sep 28, 2024
1419360
Fix javadoc generation issue (#519)
bijanhoule Sep 28, 2024
3e428c8
Include UI and auth docs (#528)
dennyglee Sep 29, 2024
a4382c6
Feat/delete model version (#517)
jamieknight-db Sep 30, 2024
d0d1bb8
Update docs to include CLI managing users (#530)
dennyglee Sep 30, 2024
e86b836
Update auth.md (#531)
dennyglee Sep 30, 2024
44ee610
Update docs by running `build/sbt generate` (#534)
vikrantpuppala Oct 1, 2024
f9e2d3c
parent f58216efe8b13e487e95a058cd20d84690125755
vikrantpuppala Sep 11, 2024
a12f45e
Merge remote-tracking branch 'unitycatalog/main' into metastore-api
vikrantpuppala Oct 1, 2024
f81126b
revert unknown changes
vikrantpuppala Oct 1, 2024
45001a6
[UI] Feat/delete model (#532)
jamieknight-db Oct 1, 2024
fb9267b
Fix UpdateRegisteredModel and UpdateModelVersion to honor path variab…
kriscon-db Oct 2, 2024
0321c68
Format files instead of blocking compile on format (#495)
vikrantpuppala Oct 3, 2024
274cd30
Merge remote-tracking branch 'unitycatalog/main' into metastore-api
vikrantpuppala Oct 3, 2024
549e9ba
clean
vikrantpuppala Oct 3, 2024
d735df8
update description
vikrantpuppala Oct 4, 2024
966b93a
fix
vikrantpuppala Oct 7, 2024
79e68e5
only keep metastore id in resp
vikrantpuppala Oct 8, 2024
111b3fd
update docs
vikrantpuppala Oct 8, 2024
3b025cd
Update api/all.yaml
vikrantpuppala Oct 11, 2024
2154fb0
Update api/all.yaml
vikrantpuppala Oct 11, 2024
041c2ce
Update api/all.yaml
vikrantpuppala Oct 11, 2024
6b354d9
update docs
vikrantpuppala Oct 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,7 @@ python_engine.log
# Signing keys and access tokens
etc/conf/certs.json
etc/conf/*.txt
etc/conf/*.der
etc/conf/*.der
connectors/spark/etc/conf/certs.json
connectors/spark/etc/conf/*.txt
connectors/spark/etc/conf/*.der
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ ARG sbt_args
ARG unitycatalog_version
ARG jars_directory

EXPOSE 8081
EXPOSE 8080 8081

RUN <<EOF
set -ex;
Expand Down
9 changes: 6 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,15 +151,18 @@ To quit DuckDB, press `Ctrl`+`D` (if your platform supports it), press `Ctrl`+`C

To use the Unity Catalog UI, start a new terminal and ensure you have already started the UC server (e.g., `./bin/start-uc-server`)

**Prerequisites**
* Node: https://nodejs.org/en/download/package-manager
* Yarn: https://classic.yarnpkg.com/lang/en/docs/install

**How to start the UI through yarn**
```
cd /ui
yarn install
yarn start
open localhost:3000
```



## CLI tutorial

You can interact with a Unity Catalog server to create and manage catalogs, schemas and tables,
Expand All @@ -168,7 +171,7 @@ See the [cli usage](docs/usage/cli.md) for more details.

## APIs and Compatibility

- Open API specification: The Unity Catalog Rest API is documented [here](api).
- Open API specification: See the [Unity Catalog Rest API](https://docs.unitycatalog.io/swagger-docs/).
- Compatibility and stability: The APIs are currently evolving and should not be assumed to be stable.

## Building Unity Catalog
Expand Down
68 changes: 68 additions & 0 deletions api/Apis/GrantsApi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# GrantsApi

All URIs are relative to *http://localhost:8080/api/2.1/unity-catalog*

| Method | HTTP request | Description |
|------------- | ------------- | -------------|
| [**get**](GrantsApi.md#get) | **GET** /permissions/{securable_type}/{full_name} | Get permissions |
| [**update**](GrantsApi.md#update) | **PATCH** /permissions/{securable_type}/{full_name} | Update a permission |


<a name="get"></a>
# **get**
> PermissionsList get(securable\_type, full\_name, principal)
Get permissions

Gets the permissions for a securable.

### Parameters

|Name | Type | Description | Notes |
|------------- | ------------- | ------------- | -------------|
| **securable\_type** | [**SecurableType**](../Models/.md)| Type of securable. | [default to null] [enum: metastore, catalog, schema, table, function, volume, registered_model] |
| **full\_name** | **String**| Full name of securable. | [default to null] |
| **principal** | **String**| If provided, only the permissions for the specified principal (user or group) are returned. | [optional] [default to null] |

### Return type

[**PermissionsList**](../Models/PermissionsList.md)

### Authorization

No authorization required

### HTTP request headers

- **Content-Type**: Not defined
- **Accept**: application/json

<a name="update"></a>
# **update**
> PermissionsList update(securable\_type, full\_name, UpdatePermissions)
Update a permission

Updates the permissions for a securable.

### Parameters

|Name | Type | Description | Notes |
|------------- | ------------- | ------------- | -------------|
| **securable\_type** | [**SecurableType**](../Models/.md)| Type of securable. | [default to null] [enum: metastore, catalog, schema, table, function, volume, registered_model] |
| **full\_name** | **String**| Full name of securable. | [default to null] |
| **UpdatePermissions** | [**UpdatePermissions**](../Models/UpdatePermissions.md)| | [optional] |

### Return type

[**PermissionsList**](../Models/PermissionsList.md)

### Authorization

No authorization required

### HTTP request headers

- **Content-Type**: application/json
- **Accept**: application/json

33 changes: 33 additions & 0 deletions api/Apis/MetastoresApi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# MetastoresApi

All URIs are relative to *http://localhost:8080/api/2.1/unity-catalog*

| Method | HTTP request | Description |
|------------- | ------------- | -------------|
| [**summary**](MetastoresApi.md#summary) | **GET** /metastore_summary | Get metastore summary |


<a name="summary"></a>
# **summary**
> GetMetastoreSummaryResponse summary()

Get metastore summary

Gets information about the metastore hosted by this Unity Catalog service (currently the service hosts only one metastore)

### Parameters
This endpoint does not need any parameter.

### Return type

[**GetMetastoreSummaryResponse**](../Models/GetMetastoreSummaryResponse.md)

### Authorization

No authorization required

### HTTP request headers

- **Content-Type**: Not defined
- **Accept**: application/json

184 changes: 184 additions & 0 deletions api/Apis/ModelVersionsApi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
# ModelVersionsApi

All URIs are relative to *http://localhost:8080/api/2.1/unity-catalog*

| Method | HTTP request | Description |
|------------- | ------------- | -------------|
| [**createModelVersion**](ModelVersionsApi.md#createModelVersion) | **POST** /models/versions | Create a model version. |
| [**deleteModelVersion**](ModelVersionsApi.md#deleteModelVersion) | **DELETE** /models/{full_name}/versions/{version} | Delete a model version |
| [**finalizeModelVersion**](ModelVersionsApi.md#finalizeModelVersion) | **PATCH** /models/{full_name}/versions/{version}/finalize | Finalize a model version |
| [**getModelVersion**](ModelVersionsApi.md#getModelVersion) | **GET** /models/{full_name}/versions/{version} | Get a model version |
| [**listModelVersions**](ModelVersionsApi.md#listModelVersions) | **GET** /models/{full_name}/versions | List model versions of the specified registered model. |
| [**updateModelVersion**](ModelVersionsApi.md#updateModelVersion) | **PATCH** /models/{full_name}/versions/{version} | Update a model version |


<a name="createModelVersion"></a>
# **createModelVersion**
> ModelVersionInfo createModelVersion(CreateModelVersion)

Create a model version.

Creates a new model version instance.

### Parameters

|Name | Type | Description | Notes |
|------------- | ------------- | ------------- | -------------|
| **CreateModelVersion** | [**CreateModelVersion**](../Models/CreateModelVersion.md)| | [optional] |

### Return type

[**ModelVersionInfo**](../Models/ModelVersionInfo.md)

### Authorization

No authorization required

### HTTP request headers

- **Content-Type**: application/json
- **Accept**: application/json

<a name="deleteModelVersion"></a>
# **deleteModelVersion**
> oas_any_type_not_mapped deleteModelVersion(full\_name, version)

Delete a model version

Deletes the specified model version.

### Parameters

|Name | Type | Description | Notes |
|------------- | ------------- | ------------- | -------------|
| **full\_name** | **String**| Full name of the model. | [default to null] |
| **version** | **Long**| Version number of the model version. | [default to null] |

### Return type

[**oas_any_type_not_mapped**](../Models/AnyType.md)

### Authorization

No authorization required

### HTTP request headers

- **Content-Type**: Not defined
- **Accept**: application/json

<a name="finalizeModelVersion"></a>
# **finalizeModelVersion**
> ModelVersionInfo finalizeModelVersion(full\_name, version, FinalizeModelVersion)

Finalize a model version

Finalizes the status of the specified model version.

### Parameters

|Name | Type | Description | Notes |
|------------- | ------------- | ------------- | -------------|
| **full\_name** | **String**| Full name of the model. | [default to null] |
| **version** | **Long**| Version number of the model version. | [default to null] |
| **FinalizeModelVersion** | [**FinalizeModelVersion**](../Models/FinalizeModelVersion.md)| | [optional] |

### Return type

[**ModelVersionInfo**](../Models/ModelVersionInfo.md)

### Authorization

No authorization required

### HTTP request headers

- **Content-Type**: application/json
- **Accept**: application/json

<a name="getModelVersion"></a>
# **getModelVersion**
> ModelVersionInfo getModelVersion(full\_name, version)

Get a model version

Gets a specific model version for a specific model.

### Parameters

|Name | Type | Description | Notes |
|------------- | ------------- | ------------- | -------------|
| **full\_name** | **String**| Full name of the model. | [default to null] |
| **version** | **Long**| Version number of the model version. | [default to null] |

### Return type

[**ModelVersionInfo**](../Models/ModelVersionInfo.md)

### Authorization

No authorization required

### HTTP request headers

- **Content-Type**: Not defined
- **Accept**: application/json

<a name="listModelVersions"></a>
# **listModelVersions**
> ListModelVersionsResponse listModelVersions(full\_name, max\_results, page\_token)

List model versions of the specified registered model.

Gets the paginated list of all available model versions under the specified registered model. There is no guarantee of a specific ordering of the elements in the array.

### Parameters

|Name | Type | Description | Notes |
|------------- | ------------- | ------------- | -------------|
| **full\_name** | **String**| Full name of the registered model. | [default to null] |
| **max\_results** | **Integer**| Maximum number of model versions to return. - when set to a value greater than 0, the page length is the minimum of this value and a server configured value; - when set to 0, the page length is set to a server configured value; - when set to a value less than 0, an invalid parameter error is returned; | [optional] [default to null] |
| **page\_token** | **String**| Opaque token to send for the next page of results (pagination). | [optional] [default to null] |

### Return type

[**ListModelVersionsResponse**](../Models/ListModelVersionsResponse.md)

### Authorization

No authorization required

### HTTP request headers

- **Content-Type**: Not defined
- **Accept**: application/json

<a name="updateModelVersion"></a>
# **updateModelVersion**
> ModelVersionInfo updateModelVersion(full\_name, version, UpdateModelVersion)

Update a model version

Updates the specified model version.

### Parameters

|Name | Type | Description | Notes |
|------------- | ------------- | ------------- | -------------|
| **full\_name** | **String**| Full name of the model. | [default to null] |
| **version** | **Long**| Version number of the model version. | [default to null] |
| **UpdateModelVersion** | [**UpdateModelVersion**](../Models/UpdateModelVersion.md)| | [optional] |

### Return type

[**ModelVersionInfo**](../Models/ModelVersionInfo.md)

### Authorization

No authorization required

### HTTP request headers

- **Content-Type**: application/json
- **Accept**: application/json

Loading
Loading