[DATASOURCE]: Persistent data handler update #478

amadolid · 2024-06-22T07:57:04Z

CHANGES:

!!! IMPORTANT !!!

As much as possible, Anchor classes should only communicate with Anchor classes too
Architype classes will act as view of Anchor on jaclang perspective
JacFeature implementations should be the bridge between Architype and Anchors

!!! --------- !!!

ObjectAnchor is now the description of the architype in datasource perspective.
datasource uses python's shelve.Shelf if specified. Backed up by memory.
Anchors now have synced and unsynced state.
Synced Anchors are either retrieve from the datasource or newly created from jaclang
Unsynced Anchors are either from reference id or from lazy loaded anchors from existing Synced Anchors
Saving Anchors will unsync non self reference to avoid recurssion
Access validation now support blacklist and whitelist approach
Access can be associated with read or write permission and also with root or node

!!! TO FOLLOW !!!

common operation to manipulate access

!!! --------- !!!

jaclang/cli/cli.py

jaclang/core/architype.py

jaclang/core/memory.py

jaclang/core/architype.py

jaclang/plugin/tests/test_jaseci.py

jaclang/core/architype.py

jaclang/cli/cli.py

jaclang/core/architype.py

jaclang/core/memory.py

jaclang/core/architype.py

jaclang/core/memory.py

ypkang

I am okay with merging this. @marsninja if you sign off, we can 🚢 this.

marsninja · 2024-07-18T12:58:04Z

jaclang/cli/cli.py

-        id_uuid = UUID(int=0)
-    else:
-        id_uuid = UUID(id)
+        super_root = jctx.super_root


perhaps we can call this system_root

I can rename it, tho my initial idea is super_root can be attached to a user. If we rename it to system_root, the implication is it's a non user type root

yeah we should change this to system_root

jaclang/cli/cli.py

jaclang/core/context.py

marsninja · 2024-07-18T13:05:01Z

jaclang/cli/cli.py

+        {
+            "session": session,
+            "root": NodeAnchor.ref(root),
+            "entry": NodeAnchor.ref(node),


not sure we should be using a free form dictionary here, should perhaps be a type ExecCtxOptions or something

the initial code here has fixed parameters before. However, as I integrate it, the fields there are too dynamic.
If I set it on Jac specs, will I able to override it on plugin ?

on jaclang args are session, root, entry
jaclang-jaseci request, root, entry (different classes)

marsninja · 2024-07-18T13:16:39Z

jaclang/compiler/constant.py

+    def __eq__(self, other: object) -> bool:
+        """Support string comparator."""
+        if isinstance(other, str):
+            return self.name == other
+        return super().__eq__(other)


Why would we need this?

there somethign fishy here 😜

ow actually, this is just a workaround since I don't know how the parser works before

but the issue is, disconnect uses string EdgeDir while edge_ref uses the Enum EdgeDir

I realign both on Jac Implementation side to use the Enum
and adding this changes allow it to still support the parser on disconnect using string edgedir

Discuss with @marsninja

marsninja · 2024-07-18T13:19:09Z

jaclang/plugin/default.py

+        def new_init(
+            self: Architype,
+            *args: object,
+            __jac__: Optional[Anchor] = None,


how is this used?

if you already have the jac, let say anchor is from the db, it will reuse it instead of creation a new one

jaclang/vendor/orjson/orjson.cpython-312-x86_64-linux-gnu.so

jaclang/cli/cli.py

marsninja · 2024-07-18T13:52:01Z

jaclang/compiler/constant.py

+    def __eq__(self, other: object) -> bool:
+        """Support string comparator."""
+        if isinstance(other, str):
+            return self.name == other
+        return super().__eq__(other)


there somethign fishy here 😜

amadolid · 2024-07-23T00:09:36Z

backup https://github.com/Jaseci-Labs/jaclang/tree/enhancements/persistent-v2-unrebased
backup https://github.com/Jaseci-Labs/jaclang/tree/enhancements/persistent-v2-backup4

amadolid · 2024-08-06T09:41:16Z

Auto Await reference
#514

ypkang · 2024-08-06T14:50:24Z

jaclang/runtimelib/architype.py

+        if self.whitelist:
+            return self.whitelist, self.anchors.get(anchor, -1)
+        else:
+            return self.whitelist, self.anchors.get(anchor, 2)


Can we use Enum for the access level instead of integer? It will be much more readable.

will check it

jaclang/runtimelib/architype.py

ypkang · 2024-08-06T14:58:04Z

jaclang/runtimelib/architype.py

+
+def populate_dataclasses(cls: type, attributes: dict[str, Any]) -> dict[str, Any]:
+    """Populate nested dataclasses."""
+    from dacite import from_dict


This does introduce a new core dependency dacite for jaclang so i wonder if @marsninja is okay with it.

@amadolid do we have to use this?

Yes, although we can create our own. This is to support edge case where attributes are nested dataclasses

ypkang · 2024-08-06T14:59:43Z

jaclang/runtimelib/architype.py

+            #     ): Access(**value)
+            #     for key, value in types.items()
+            # },
+            # nodes=Access(**data.get("nodes", {})),


Can we remove the code for the types-based permission for now? it will probably take more than just uncomment these to get it to work when we do want to introduce this later. So for now, I'd like to keep the code cleaner.

will remove it

ypkang · 2024-08-06T15:01:54Z

jaclang/runtimelib/architype.py

+
+    @staticmethod
+    def ref(ref_id: str) -> Optional[Anchor]:
+        """Return ObjectAnchor instance if ."""


Finish this docstring.

will update 😅

ypkang · 2024-08-06T15:40:05Z

jaclang/runtimelib/architype.py

+                and self.state.connected == other.state.connected
+            )
+        elif isinstance(other, Architype):
+            return self == other.__jac__


I think this is the same comment @marsninja had before -- why do we need to create custom eq for anchor? When do we need to compare anchor with architype?

in context, anchor act as description of a architype. Comparing with anchor is like checking if it only has the same description as it can resync anytime with the latest values (if its on "unloaded" state)

ypkang · 2024-08-06T15:41:50Z

jaclang/runtimelib/architype.py

+        return f"{self.type.value}:{self.name}:{self.id}"
+
+    @staticmethod
+    def ref(ref_id: str) -> Optional[Anchor]:


So it looks like this ref is overriden for each specific anchor type. So do we still need this one here?

Anchor.ref is like a non restricted way of referencing. NodeAnchor.ref will only allow node reference id
(same mechanism with Edge and Walker)

This is to support "blind" referencing

What is an example scenario where this is usedful?

ypkang · 2024-08-06T15:46:10Z

jaclang/runtimelib/architype.py

+
+    def sync(self, node: Optional["NodeAnchor"] = None) -> Optional[NodeArchitype]:
+        """Retrieve the Architype from db and return."""
+        return cast(Optional[NodeArchitype], super().sync(node))


Why do we need to cast to the parent class? instead of just using the inherited function?

I haven't revisit it again.
But initially, this is only for forcing the type hinting base on their current class

Yeah take a look. We might not need this i feel like.

ypkang · 2024-08-06T15:46:53Z

jaclang/runtimelib/architype.py

+        """Retrieve the Architype from db and return."""
+        return cast(Optional[NodeArchitype], super().sync(node))
+
+    def connect_node(self, nd: NodeAnchor, edg: EdgeAnchor) -> None:


edg --> edge?

Also, it looks like we are using nd in some places and node in others. We should unify them.

yup, will update it

ypkang · 2024-08-06T16:00:10Z

jaclang/runtimelib/architype.py



 class Architype:
    """Architype Protocol."""

    _jac_entry_funcs_: list[DSFunc]
    _jac_exit_funcs_: list[DSFunc]
+    __jac_classes__: dict[str, type[Architype]]
+    __jac_hintings__: dict[str, type]


hmm what are these for? 🤔

jac_classes is similar to subclasses but converted to dict and cached
applicable only on NodeArchitype, EdgeArchitype and WalkerArchitype

jac_hintings is a cached typing.get_type_hints used for walker api generations

But what purpose do they server?

ypkang · 2024-08-06T16:06:22Z

jaclang/runtimelib/context.py

        self.jac_machine = JacMachine(base_path)
+        self.datasource: ShelfStorage = ShelfStorage(session)
+        self.reports: list[Any] = []
+        self.super_root = self.load(


can we change this to system_root? @marsninja left this comment before. I like system_root better too :)

I can change it to system root tho my concern is, will that imply that its a non user type root?

I think its okay. It will be like a user is assuming the role of the system and this is their root. I think that still works.

ypkang · 2024-08-06T16:18:45Z

jaclang/cli/cli.py

-        id_uuid = UUID(int=0)
-    else:
-        id_uuid = UUID(id)
+        super_root = jctx.super_root


yeah we should change this to system_root

ypkang · 2024-08-06T16:20:55Z

jaclang/cli/cli.py

+            "root": NodeAnchor.ref(root),
+            "entry": NodeAnchor.ref(node),
+        }
+    )


Carrying over this comment from the last review -- there was concern of using free form dictionary here. we should discuss

ypkang · 2024-08-06T16:24:20Z

jaclang/plugin/tests/fixtures/other_root_access.jac

@@ -0,0 +1,31 @@
+node A {


This test is great. But, can we add some more test cases to increase coverage?

Update nodes

Create connection for nodes (test the connect permission)

some common sharing permission scenarios. Are sharing part of jaclang? Or its going to be in jaseci only?

its partially supported.
its supposed to be with plugin function but we can add unit test with manual trigger for now

ypkang · 2024-08-06T20:02:42Z

jaclang/runtimelib/architype.py



 class Architype:
    """Architype Protocol."""

    _jac_entry_funcs_: list[DSFunc]
    _jac_exit_funcs_: list[DSFunc]
+    __jac_classes__: dict[str, type[Architype]]
+    __jac_hintings__: dict[str, type]


But what purpose do they server?

ypkang · 2024-08-06T20:03:24Z

jaclang/runtimelib/context.py

        self.jac_machine = JacMachine(base_path)
+        self.datasource: ShelfStorage = ShelfStorage(session)
+        self.reports: list[Any] = []
+        self.super_root = self.load(


I think its okay. It will be like a user is assuming the role of the system and this is their root. I think that still works.

amadolid mentioned this pull request Jun 22, 2024

[INPROGRESS] #429

Closed

ypkang requested changes Jun 26, 2024

View reviewed changes

jaclang/core/architype.py Outdated Show resolved Hide resolved

jaclang/core/architype.py Outdated Show resolved Hide resolved

amadolid force-pushed the enhancements/persistent-v2 branch from 715c00a to d677938 Compare June 26, 2024 18:14

amadolid requested a review from ypkang June 27, 2024 14:38

ypkang requested changes Jun 27, 2024

View reviewed changes

amadolid force-pushed the enhancements/persistent-v2 branch from 0511d22 to 770d8b5 Compare June 27, 2024 18:11

amadolid requested a review from ypkang June 27, 2024 18:11

amadolid force-pushed the enhancements/persistent-v2 branch 5 times, most recently from ab10a4e to a70412a Compare July 3, 2024 20:02

ypkang requested changes Jul 4, 2024

View reviewed changes

jaclang/core/architype.py Outdated Show resolved Hide resolved

jaclang/core/architype.py Outdated Show resolved Hide resolved

jaclang/core/memory.py Outdated Show resolved Hide resolved

jaclang/core/memory.py Outdated Show resolved Hide resolved

amadolid requested a review from ypkang July 8, 2024 07:14

amadolid force-pushed the enhancements/persistent-v2 branch 2 times, most recently from bad20f1 to 30232ce Compare July 12, 2024 12:30

ypkang previously approved these changes Jul 12, 2024

View reviewed changes

amadolid dismissed ypkang’s stale review via cd22d37 July 15, 2024 06:13

amadolid force-pushed the enhancements/persistent-v2 branch 4 times, most recently from f611dbc to 0e8e210 Compare July 15, 2024 15:37

marsninja reviewed Jul 18, 2024

View reviewed changes

marsninja requested changes Jul 18, 2024

View reviewed changes

amadolid force-pushed the enhancements/persistent-v2 branch from 909c47c to a1f032b Compare July 18, 2024 15:07

amadolid requested review from marsninja and ypkang July 18, 2024 15:08

amadolid force-pushed the enhancements/persistent-v2 branch from e988be6 to 410bebb Compare July 18, 2024 15:49

amadolid force-pushed the enhancements/persistent-v2 branch 5 times, most recently from 45b082f to b9cc829 Compare July 23, 2024 00:08

amadolid force-pushed the enhancements/persistent-v2 branch 7 times, most recently from 6d4c897 to 6b2ced7 Compare July 30, 2024 02:12

amadolid added 8 commits August 6, 2024 15:29

[PERSISTENCE-PR-SERIES]: Initial Commit

a1d3753

[PERSISTENCE-PR-SERIES]: Architype Refactor

24eb693

[PERSISTENCE-PR-SERIES]: Rename _jac_ and target_obj

e19aa4f

[PERSISTENCE-PR-SERIES]: Context Refactor

6ed045a

[PERSISTENCE-PR-SERIES]: Memory Refactor

e6f6991

[PERSISTENCE-PR-SERIES]: Plugin Refactor

8c83dde

[PERSISTENCE-PR-SERIES]: Align cli, constructs and jac_streamlit

2f6cad2

[PERSISTENCE-PR-SERIES]: Unit Tests

5d68ea5

amadolid force-pushed the enhancements/persistent-v2 branch from 1667520 to 5d68ea5 Compare August 6, 2024 09:32

[PERSISTENCE-PR-SERIES]: Auto await if necessary

36c4b6b

amadolid mentioned this pull request Aug 6, 2024

[AUTO-AWAIT]: Auto await if necessary #514

Closed

ypkang requested changes Aug 6, 2024

View reviewed changes

[DATASOURCE]: Persistent data handler update #478

Are you sure you want to change the base?

[DATASOURCE]: Persistent data handler update #478

Conversation

amadolid commented Jun 22, 2024 • edited Loading

ypkang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amadolid Jul 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amadolid Jul 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amadolid commented Jul 23, 2024 • edited Loading

amadolid commented Aug 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amadolid commented Jun 22, 2024 •

edited

Loading

amadolid Jul 18, 2024 •

edited

Loading

amadolid Jul 18, 2024 •

edited

Loading

amadolid commented Jul 23, 2024 •

edited

Loading