Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V2.0 Definition for snowpark #1402

Merged
merged 39 commits into from
Aug 10, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
8f9b89a
Support project definition V2 in streamlit deploy command (#1369)
sfc-gh-turbaszek Jul 26, 2024
1603dd8
Init
sfc-gh-jsikorski Jul 30, 2024
06c6d63
Solution outline
sfc-gh-jsikorski Jul 31, 2024
45703d9
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Jul 31, 2024
0884c45
Fixing paths
sfc-gh-jsikorski Jul 31, 2024
55761e9
build fix
sfc-gh-jsikorski Aug 1, 2024
f226451
build fix
sfc-gh-jsikorski Aug 1, 2024
e4ef4be
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Aug 1, 2024
ac53ecc
Update src/snowflake/cli/api/project/schemas/entities/snowpark_entity.py
sfc-gh-jsikorski Aug 1, 2024
10fc53f
Update src/snowflake/cli/api/project/schemas/entities/snowpark_entity.py
sfc-gh-jsikorski Aug 1, 2024
a3db93b
query problem
sfc-gh-jsikorski Aug 1, 2024
cf933f2
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Aug 1, 2024
563886d
Fix for zipper
sfc-gh-jsikorski Aug 1, 2024
5a4a7a0
Test fix
sfc-gh-jsikorski Aug 1, 2024
403e3c7
Test fix
sfc-gh-jsikorski Aug 1, 2024
67710b8
Update src/snowflake/cli/plugins/snowpark/commands.py
sfc-gh-jsikorski Aug 2, 2024
01c284e
Update src/snowflake/cli/plugins/snowpark/commands.py
sfc-gh-jsikorski Aug 2, 2024
8553cf7
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Aug 2, 2024
83e5012
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Aug 6, 2024
6a4cc33
Fixes
sfc-gh-jsikorski Aug 6, 2024
aca0d21
Test fix
sfc-gh-jsikorski Aug 6, 2024
4c5ab82
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Aug 6, 2024
2cc7c4f
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Aug 6, 2024
9128470
Merge fixes
sfc-gh-jsikorski Aug 6, 2024
73e29a6
Test fix
sfc-gh-jsikorski Aug 6, 2024
3aad2eb
Reformat
sfc-gh-jsikorski Aug 6, 2024
0747807
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Aug 6, 2024
a7d449a
Merge cleanup
sfc-gh-jsikorski Aug 6, 2024
2766b72
Fixup
sfc-gh-jsikorski Aug 7, 2024
659636f
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Aug 7, 2024
be0d58f
Changed to artifacts
sfc-gh-jsikorski Aug 8, 2024
221d5e5
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Aug 8, 2024
4176171
Fixup
sfc-gh-jsikorski Aug 8, 2024
bff4b4d
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Aug 8, 2024
33ac466
Fixup
sfc-gh-jsikorski Aug 8, 2024
1f6d50c
After review fixes
sfc-gh-jsikorski Aug 9, 2024
ccdea97
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-jsikorski Aug 9, 2024
bd98653
Post merge cleanup
sfc-gh-jsikorski Aug 9, 2024
91b7a5f
Merge branch 'main' into jsikorski/snowpark-v2
sfc-gh-turbaszek Aug 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions src/snowflake/cli/api/project/schemas/entities/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@ class DefaultsField(UpdatableModel):
default=None,
)

project_name: Optional[str] = Field(
title="Name of the project.",
default=None,
)


class EntityBase(ABC, UpdatableModel):
@classmethod
Expand Down
12 changes: 11 additions & 1 deletion src/snowflake/cli/api/project/schemas/entities/entities.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,19 @@
from snowflake.cli.api.project.schemas.entities.application_package_entity import (
ApplicationPackageEntity,
)
from snowflake.cli.api.project.schemas.entities.snowpark_entity import (
FunctionEntity,
ProcedureEntity,
)
from snowflake.cli.api.project.schemas.entities.streamlit_entity import StreamlitEntity

Entity = Union[ApplicationEntity, ApplicationPackageEntity, StreamlitEntity]
Entity = Union[
ApplicationEntity,
ApplicationPackageEntity,
StreamlitEntity,
ProcedureEntity,
FunctionEntity,
]
sfc-gh-gbloom marked this conversation as resolved.
Show resolved Hide resolved

ALL_ENTITIES = [*get_args(Entity)]

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Copyright (c) 2024 Snowflake Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from __future__ import annotations

from typing import Dict, List, Literal, Optional, Union

from pydantic import Field, field_validator
from snowflake.cli.api.project.schemas.entities.common import EntityBase
from snowflake.cli.api.project.schemas.identifier_model import ObjectIdentifierModel
from snowflake.cli.api.project.schemas.snowpark.argument import Argument
from snowflake.cli.api.project.schemas.updatable_model import DiscriminatorField


class SnowparkEntity(EntityBase):
handler: str = Field(
title="Function’s or procedure’s implementation of the object inside source module",
examples=["functions.hello_function"],
)
returns: str = Field(
title="Type of the result"
) # TODO: again, consider Literal/Enum
signature: Union[str, List[Argument]] = Field(
title="The signature parameter describes consecutive arguments passed to the object"
)
runtime: Optional[Union[str, float]] = Field(
title="Python version to use when executing ", default=None
)
external_access_integrations: Optional[List[str]] = Field(
title="Names of external access integrations needed for this procedure’s handler code to access external networks",
default=[],
)
secrets: Optional[Dict[str, str]] = Field(
title="Assigns the names of secrets to variables so that you can use the variables to reference the secrets",
default={},
)
imports: Optional[List[str]] = Field(
title="Stage and path to previously uploaded files you want to import",
default=[],
)
stage: str = Field(title="Stage in which artifacts will be stored")
src: str = Field(title="Folder where your code should be located")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

source and imports should be replaced with artifacts as per design

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it seems, that artifacts mean local files that will be uploaded - so imports should be separated, as those are the files that already exist server-side, and just need to be included in the SQL declaration


@field_validator("runtime")
@classmethod
def convert_runtime(cls, runtime_input: Union[str, float]) -> str:
if isinstance(runtime_input, float):
return str(runtime_input)
return runtime_input


class ProcedureEntity(SnowparkEntity, ObjectIdentifierModel("procedure")): # type: ignore
type: Literal["procedure"] = DiscriminatorField() # noqa: A003
execute_as_caller: Optional[bool] = Field(
title="Determine whether the procedure is executed with the privileges of "
"the owner (you) or with the privileges of the caller",
default=False,
)


class FunctionEntity(SnowparkEntity, ObjectIdentifierModel("function")): # type: ignore
type: Literal["function"] = DiscriminatorField() # noqa: A003
50 changes: 31 additions & 19 deletions src/snowflake/cli/api/project/schemas/project_definition.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@
from snowflake.cli.api.utils.types import Context
from typing_extensions import Annotated

AnnotatedEntity = Annotated[Entity, Field(discriminator="type")]

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change neccessary? Are you building on current main?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just for simplicity - same thing is used 4 times, so seems reasonable to put it into variable


@dataclass
class ProjectProperties:
Expand Down Expand Up @@ -111,8 +113,16 @@ class DefinitionV11(DefinitionV10):


class DefinitionV20(_ProjectDefinitionBase):
entities: Dict[str, Annotated[Entity, Field(discriminator="type")]] = Field(
title="Entity definitions."
entities: Dict[str, AnnotatedEntity] = Field(title="Entity definitions.")

defaults: Optional[DefaultsField] = Field(
title="Default key/value entity values that are merged recursively for each entity.",
default=None,
)

env: Optional[Dict[str, Union[str, int, bool]]] = Field(
title="Default environment specification for this project.",
default=None,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really being added in this PR? How does it releat to Snowpark?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was moved from line 165. It seems a bit more consistent to have attributes first, then methods - not attribute, method, attribute, method etc.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's limit the changed to snowpark only. It's easier to revert or review when changes has well defined scope.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

)

@model_validator(mode="before")
Expand All @@ -137,17 +147,29 @@ def apply_defaults(cls, data: Dict) -> Dict:

@field_validator("entities", mode="after")
@classmethod
def validate_entities(cls, entities: Dict[str, Entity]) -> Dict[str, Entity]:
def validate_entities(
cls, entities: Dict[str, AnnotatedEntity]
) -> Dict[str, AnnotatedEntity]:
for key, entity in entities.items():
# TODO Automatically detect TargetFields to validate
if entity.type == ApplicationEntity.get_type():
if isinstance(entity.from_, TargetField):
target_key = entity.from_.target
target_object = entity.from_
target_type = target_object.get_type()
cls._validate_target_field(target_key, target_type, entities)
if isinstance(entity, list):
for e in entity:
cls._validate_single_entity(e, entities)
else:
cls._validate_single_entity(entity, entities)
return entities

@classmethod
def _validate_single_entity(
cls, entity: Entity, entities: Dict[str, AnnotatedEntity]
):
if entity.type == ApplicationEntity.get_type():
if isinstance(entity.from_, TargetField):
target_key = entity.from_.target
target_object = entity.from_
target_type = target_object.get_type()
cls._validate_target_field(target_key, target_type, entities)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How this change relates to snowpark?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again - it was moved in file, to make the code more readable

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@classmethod
def _validate_target_field(
cls, target_key: str, target_type: Entity, entities: Dict[str, Entity]
Expand All @@ -162,16 +184,6 @@ def _validate_target_field(
f"Target type mismatch. Expected {target_type.__name__}, got {actual_target_type.__name__}"
)

defaults: Optional[DefaultsField] = Field(
title="Default key/value entity values that are merged recursively for each entity.",
default=None,
)

env: Optional[Dict[str, Union[str, int, bool]]] = Field(
title="Default environment specification for this project.",
default=None,
)

def get_entities_by_type(self, entity_type: str):
return {i: e for i, e in self.entities.items() if e.get_type() == entity_type}

Expand Down
Loading
Loading