From 02befd2e597b9848b0a1d7dce5481da728636626 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Mon, 25 Sep 2023 16:14:11 -0700 Subject: [PATCH] Replace Dataset with Entity List (#1651) --- docs/central-entities.rst | 135 ++++++++++---------- docs/central-intro.rst | 2 +- docs/img/central-entities/intro-diagram.png | 4 +- 3 files changed, 73 insertions(+), 68 deletions(-) diff --git a/docs/central-entities.rst b/docs/central-entities.rst index 699976bb2..271f2dd60 100644 --- a/docs/central-entities.rst +++ b/docs/central-entities.rst @@ -1,11 +1,13 @@ Managing Entities in Central ================================ -New in Central as of version 2022.3, **Entities** are a powerful new way to automatically bring the information you are collecting right back to your devices and Forms in the field. +.. versionadded:: v2022.3 -In ODK today, you can attach existing data during the Form creation process, often through ``choices`` sheets or ``.csv`` files. Once this data is loaded into the Form, you can use it as the source for selection choices, to prepopulate data fields in the Form, or validate new input. A list of districts, for example, can be used as choices for selection, and information about each district can then be shown to the user or checked against. If you are new to these techniques, you can learn more about them on the :doc:`Form Datasets ` page. +**Entities** are a powerful way to automatically bring the information you collect right back to your devices and Forms in the field. -This is wonderful if this data is already known when you are publishing your Form, and if it doesn't change over the course of your project. In many cases, however, the data being collected today for the project *is* the data you need tomorrow for the same project. Think about seeing a patient for the first and then second time, or registering and then revisiting a survey site for a follow-up. These kinds of workflows are called :ref:`Multiple Encounter ` or longitudinal workflows. +Without Entities, you can attach existing data during the Form creation process, often through ``choices`` sheets or ``.csv`` files. Once this data is loaded into the Form, you can use it as the source for selection choices, to prepopulate data fields in the Form, or validate new input. A list of districts, for example, can be used as choices for selection, and information about each district can then be shown to the user or checked against. If you are new to these techniques, you can learn more about them on the :doc:`Form Datasets ` page. + +This is wonderful if the data you want to use is already known when you are publishing your Form, and if it doesn't change over the course of your project. In many cases, however, the data being collected today for the project *is* the data you need tomorrow for the same project. Think about seeing a patient for the first and then second time, or registering and then revisiting a survey site for a follow-up. These kinds of workflows are called :ref:`multiple encounter ` or longitudinal workflows. If you have this kind of workflow, you can use the registered data items in a follow-up Form without using Entities, but it requires some work. You can download the Submission data from the registration Form, and then you can attach that data to a follow-up Form, as described above. However, the data available to devices in the field is only as current as your latest follow-up Form update. Some people write programs that automate this process over the API but this requires technical skills that many projects don't have. @@ -16,13 +18,17 @@ Now there is a way to automate this process from within Central itself. Introducing Entities --------------------- -First, some definitions: each item that gets managed by an ODK workflow is called an **Entity**. Entities can be physical (e.g., a tree) or abstract (e.g., a site visit). A collection of Entities is called a **Dataset**. +First, some definitions: each item that gets managed by an ODK workflow is called an **Entity**. Entities can be physical (e.g., a tree) or abstract (e.g., a site visit). Entities of the same type are organized in **Entity Lists**. + +.. note:: + + In Central versions prior to v2023.4, Entity Lists were called Datasets. -Next, a new feature. You can now tell Central that some of your incoming Submission data should create Entities in a Dataset created and managed for you by Central. This Dataset can then be attached by that name to your follow-up Forms, exactly like you would manually attach a :doc:`CSV Dataset `. You can attach a Dataset to any number of Forms, even to a registration Form. +Next, a new feature. You can now tell Central that some of your incoming Submission data should create Entities in an Entity List created and managed for you by Central. This Entity List can then be attached by that name to your follow-up Forms, exactly like you would manually attach a :doc:`CSV document `. You can attach an Entity List to any number of Forms, even to a registration Form. .. image:: /img/central-entities/intro-diagram.png -Using this functionality, you don't have to create and upload Datasets yourself. When Submissions arrive and are approved, Central will create new Entities that will appear in follow-up Forms as soon as those Form updates are received. +Using this functionality, you don't have to create and upload Entity Lists yourself. When Central receives submissions (or they are approved, see :ref:`the settings section `), new Entities are automatically created. As soon as users of the Follow-up Form(s) get Form updates, they can interact with the new Entities. If you are interested in seeing how Entities can fit into your workflow right away, we recommend following the Quick Start guide below, where you will upload a tree registration Form and a tree follow-up Form we have created already and see how trees are created by one Form and appear in another. @@ -31,20 +37,20 @@ If you are interested in seeing how Entities can fit into your workflow right aw Roadmap and limitations ----------------------- -Below is what's available now. +Entities are a big new concept that open up a lot of new possibilities. While we think many workflows can benefit from Entities today, they have some limitations that you should be aware of. + +What's available now: - Create an Entity with a registration form (automatically or after project manager approval) -- Create an Entity with different registration forms (e.g., child vs adult registration) -- Use Entities in a follow-up form -- Use Entities in different follow-up forms (e.g., weekly vs monthly follow-up) -- Use different Datasets in different registration and follow-up forms -- Download Datasets into Power BI, Excel, Python, and R +- Use multiple different registration forms targeting the same Entity List (e.g., registration at school vs. registration at home) +- Use Entities in one or more follow-up form +- Download Entities into Power BI, Excel, Python, and R -The current limitations are: +Important limitations: - Entity create and update requires Internet access - Entity delete is only available via API -- Performance suffers when managing more than 10,000 Entities +- Client performance (ODK Collect or Enketo web forms) suffers when managing more than 10,000 Entities - All devices will always download all Entities which may be a privacy concern - The Form specification and API may change @@ -57,6 +63,10 @@ Quick Start We recommend watching the video below once or twice to get an overview of how Entities work in Central. Once you have that context, proceed with the steps below to try it yourself! +.. note:: + + In Central versions prior to v2023.4, Entity Lists were called Datasets. The video below was recorded with Central v2022.3 so many small improvements have been made since. + .. youtube:: hbff-oaI8yg :width: 100% @@ -65,67 +75,64 @@ In this Quick Start, you will upload two Forms we have already authored for you: First, let's prepare the Forms for use with Central. 1. Create a new Project, if you can. It can be easier to see what's going on without other Forms in the way. -2. Upload the `Tree registration Form `_. Notice how it has seen from the Form definition that the Form creates Entities in a Dataset, and populates three Entity Properties. -3. Publish the Form. As the dialog says, this action creates the ``trees`` Dataset within the Project you created. +2. Upload the `Tree registration Form `_. Notice how Central knows from the Form definition that the Form creates Entities in an Entity List, and populates three Entity Properties. +3. Publish the Form. As the dialog says, this action creates the ``trees`` Entity List within the Project you created. 4. Now, upload the `Tree follow-up Form `_. -5. Switch over to the Form Attachments tab before you publish this Form, and notice how the Form wants a data file called ``trees.csv``, but that Central has already linked the Dataset ``trees`` (created when you published the registration Form in step 3) and you don't need to upload any data here. +5. Switch over to the Form Attachments tab before you publish this Form, and notice how the Form wants a data file called ``trees.csv``, but that Central has already linked the Entity List ``trees`` (created when you published the registration Form in step 3) and you don't need to upload any data here. 6. Go back to the Draft Status page and publish this Form. Because this Form only *consumes* ``trees``, it does not make any changes to any Entity Properties. 7. Now, create an App User within this Project, allow it to access both of these Forms, and load them up into Collect or some other ODK-compatible client. Next, let's see these Forms working together. -1. First, fill out and submit the Tree registration Form. Be sure to choose a species and specify a Tree circumference. Use 100 if you are not sure what to fill in. -2. Go back to Central. You can download the ``trees`` Dataset under the Datasets tab in your Project. -3. But your tree isn't there. It does take a moment sometimes to create an Entity from a Submission, but in this case it's because we're not done yet. An Entity will not be created until you *approve* the submission. -4. Go to the Trees registration Form Submissions page, and approve your tree. -5. Open the Submission details page for that Submission by putting your mouse on its row and clicking More. -6. You should now see a record of your approval, as well as of the creation of a new Entity based on the Submission. If you don't see the Entity yet, wait a second and refresh. -7. You can try downloading the ``trees`` Dataset again if you want to see your tree there. -8. Now go back to Collect and update your Forms to fetch the new data. -9. Next, fill out the Tree follow-up Form. -10. Your tree is here! Choose it. -11. Report a new circumference that is smaller than the old one. This is probably not a good idea for a tree. See how the Form warns you about this problem, based on the data you'd put into the registration Form. -12. Correct the circumference to a larger number, and submit the Form. - -That's it! The follow-up Form only creates normal Submissions, so you can access the data it collects like any Form. +#. First, fill out and submit the Tree registration Form. Be sure to choose a species and specify a Tree circumference. Use 100 if you are not sure what to fill in. +#. Go back to Central and click on ``trees`` in the :guilabel:`Entities` tab in your Project. +#. Click on the :guilabel:`Data` tab and you should see your new tree! If you don't, you can wait a second or two and refresh. +#. Now go back to Collect and update your Forms to fetch the new data. +#. Next, fill out the Tree follow-up Form. +#. Your tree is here! Choose it. +#. Report a new circumference that is smaller than the old one. This is probably not a good idea for a tree. See how the Form warns you about this problem, based on the data you'd put into the registration Form. +#. Correct the circumference to a larger number, and submit the Form. + +That's it! The follow-up Form creates normal Submissions, so you can access the data it collects like any Form. .. _central-entities-overview: -Entities in v2022.3 +Creating Entity Lists --------------------- -If you skipped the Quick Start above, we suggest you go back and give it a try. You will learn hands on with Central a lot of what will be described below. +.. note:: + If you skipped the Quick Start above, we suggest you go back and give it a try. You will learn hands on with Central a lot of what will be described below. -In these early versions of Entities, you cannot create a Dataset directly through the Central website. To begin using Entities, you will need to author a Form which defines them. By uploading a Form that specifies the fields in a Submission that should be used to create a new Entity, and the name of the Dataset these new Entities should go to, you will prompt Central to create the Dataset. You'll be able to see the Dataset information Central recognized in your Form once you upload it. +In these early versions of Entities, you cannot create an Entity List directly through the Central website. To begin using Entities, you will need to author a Form which defines them. By uploading a Form that specifies the fields in a Submission that should be used to create a new Entity, and the name of the Entity List these new Entities should go to, you will prompt Central to create the Entity List. You'll be able to see the Entity information that Central recognized in your Form once you upload it. -When you publish this Form, the new Dataset and/or new Entity Properties will be created for you automatically within the Project. You can learn more about authoring these kinds of Forms :ref:`in the sections below `. +When you publish this Form, the new Entity List and new Entity Properties will be created for you automatically within the Project. You can learn more about authoring these kinds of Forms :ref:`in the sections below `. .. note:: - In this version of Entities, a Submission must be approved before an Entity will be created from it. In future versions, you will be able to choose to create the Entity immediately when the server receives the Submission. + By default, Entities are created immediately when the server receives the Submission. If you want to require a review step before Entities are available for follow-up, you can configure this in each Entity List's :ref:`settings `. -To see this new Dataset and download data from it, visit the :guilabel:`Datasets` tab on the Project page. In future versions, you will see many more controls and more helpful information than you do now. +To see this new Entity List and download data from it, visit the :guilabel:`Entities` tab on the Project page. -To use data from a Dataset in another Form, you can refer to it by ``NAME.csv`` where ``NAME`` is the name of your Dataset. When you upload that Form, you should see on the Form Attachments tab that the file has been automatically linked to the Dataset. You can always override this connection by uploading your own data file to use instead. This does not affect the Dataset itself, your file is used *instead* of the Dataset for that Form only. +To use data from a Entity List in a Form, you can refer to it in a :ref:`select_one_from_file ` question as ``NAME.csv`` where ``NAME`` is the name of your Entity List. When you upload that Form, you will see on the Form Attachments tab that the file has been automatically linked to the Entity List. .. _central-entities-testing: Testing Forms with Entities ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Checking that your Forms are working together the way you expect is challenging with Entities. You can create a Draft of a Form and use it as a safe space to try out Form definitions and see resulting Submissions. But Datasets reach *across* Forms. They live alongside Forms within the Project. How do Form Drafts related to the same Entities connect together? +Checking that your Forms are working together the way you expect is challenging with Entities. You can create a Draft of a Form and use it as a safe space to try out Form definitions and see resulting Submissions. But Entity Lists reach *across* Forms. They live alongside Forms within the Project. How do Form Drafts related to the same Entities connect together? -It's very confusing, and Drafts currently don't handle this question very well. A future version of Central will provide better answers. +Drafts currently don't handle this question very well and a future version of Central will provide better answers. -For now, *Datasets work on published Forms and Submissions only*. The creation of Datasets or new Entity Properties only occurs at the moment you publish the Form. Only real form Submissions create Entities. This means that you can't test the usage of Datasets in follow-up Forms until real Entities have been created. +For now, *you can only create Entity Lists and Entities from published Forms and Submissions*. The creation of Entity Lists or new Entity Properties only occurs at the moment you publish the Form. Only real form Submissions create Entities. This means that you can't test the usage of Entities in follow-up Forms until real Entities have been created. -To try the end-to-end workflow across multiple Forms, we recommend creating a temporary project just for testing. You can publish all the Forms, create Entities for testing, and make sure that all the Forms work well together. +To try the end-to-end workflow across multiple Forms, we recommend creating a temporary Project just for testing. You can publish all the Forms, create Entities for testing, and make sure that all the Forms work well together. -You can also try a follow-up Form Draft by manually creating a CSV of sample Entities and then attaching it to your Draft, as described in :ref:`central-forms-attachments`. When you have verified the logic of the follow-up Form and are ready to publish it, you can change the link from the CSV to the desired Dataset. +You can also try a follow-up Form Draft by manually creating a CSV of sample Entities and then attaching it to your Draft, as described in :ref:`central-forms-attachments`. When you have verified the logic of the follow-up Form and are ready to publish it, you can change the link from the CSV to the desired Entity List. .. _central-entities-authoring: Creating Forms to use Entities -------------------------------------------- +-------------------------------- Central does a lot of work to help you manage Entities, but at least for now the only way to ask it to do so is to create a Form that describes how. @@ -136,7 +143,7 @@ In the following section, we describe how to author Forms that create new Entiti Build a Form that creates Entities ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -You’ll start by building a Form that creates new Entities in a Dataset called ``trees``. When you publish this Form, a ``trees`` Dataset will be created for you. When a Submission to this Form is approved, an Entity will be created in the ``trees`` Dataset from data in the Submission. These types of Forms are often referred to as registration, enrollment, intake or discovery Forms. +You’ll start by building a Form that creates new Entities in an Entity List called ``trees``. When you publish this Form, a ``trees`` Entity List will be created for you. When a Submission to this Form is received, an Entity will be created in the ``trees`` Entity List from data in the Submission. These types of Forms are often referred to as registration, enrollment, intake or discovery Forms. .. _central-entities-registration-forms-structure: @@ -159,21 +166,19 @@ Test your Form to make sure it works and collects the data that you need. .. _central-entities-registration-forms-destination: -Specify the Dataset the Form should save Entities to -"""""""""""""""""""""""""""""""""""""""""""""""""""" +Specify the Entity List the Form should save Entities to +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Add a new ``entities`` sheet to your XLSForm. This is where you will specify your Dataset’s name, under ``dataset``. +Add a new ``entities`` sheet to your XLSForm. This is where you will specify your Entity List name, under ``list_name``. -Each Entity will be automatically assigned a unique name based on one or more Properties from the Entity data, just like the ``instanceName`` on submissions. The ``label`` field here is where you provide the expression to use as a label for each Entity in the Dataset. +The Entity List name will be used by Central to uniquely identify that Entity List. If an Entity List with the name you specify already exists in Central, this Form will create Entities in that existing Entity List. If Central doesn't yet have a Entity List with the specified name, it will be created. -The Dataset name will be used by Central to uniquely identify that Dataset. If a Dataset with the name you specify already exists in Central, this Form will create Entities in that existing Dataset. If Central doesn't yet have a Dataset with the specified name, it will be created. - -The label expression can use any field in the Form. +Each Entity must have a label to identify it on Central and for use in follow-up Forms. The ``label`` field on the ``entities`` sheet is where you provide the expression to define the label for each Entity. This is very similar to the `instance name specified for a Submission `_. The label expression can use any field in the Form, including ones that aren't saved to Entity Properties. .. rubric:: XLSForm .. csv-table:: entities - :header: dataset, label + :header: list_name, label trees,"concat(${circumference}, ""cm "", ${species})" @@ -182,7 +187,7 @@ The label expression can use any field in the Form. Specify the Form fields that are saved to Entities """""""""""""""""""""""""""""""""""""""""""""""""" -If you think of your Dataset as a spreadsheet, each row represents an individual Entity and each column specifies an Entity Property. +If you think of your Entity List as a spreadsheet, each row represents an individual Entity and each column specifies an Entity Property. You define Entity Properties by adding a ``save_to`` column to your XLSForm. You then put an Entity Property name in the ``save_to`` column for each Form field that you would like to save for use in follow-up Forms. @@ -198,14 +203,14 @@ You define Entity Properties by adding a ``save_to`` column to your XLSForm. You If you'd like to check your work, you can compare with `this example form `_, with the ``entities`` sheet and ``save_to`` information. -When you publish this Form on Central, the ``trees`` Dataset will be created for you. +When you publish this Form on Central, the ``trees`` Entity List will be created for you. .. _central-entities-follow-up-forms: Build a Form that uses Entities ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Your ``trees`` Dataset can now be attached to any Form using ``select_one_from_file`` or ``csv-external``. +Your ``trees`` Entity List can now be attached to any Form using ``select_one_from_file`` or ``csv-external``. .. rubric:: XLSForm @@ -218,25 +223,25 @@ Your ``trees`` Dataset can now be attached to any Form using ``select_one_from_f You can see the full XLSForm `here `_. -The same Dataset can be used in many different Forms. The concepts and patterns described in the :doc:`data collector workflows ` and the :doc:`Form Datasets ` sections apply to server-managed Datasets as well. +The same Entity List can be used in many different Forms. The concepts and patterns described in the :doc:`data collector workflows ` and the :doc:`Form Datasets ` sections apply to Entity Lists as well. .. _central-entities-managing: -Managing Datasets and Entities ------------------------------- +Managing Entities +------------------- -To browse all Datasets in a Project, go to the :guilabel:`Datasets` tab within the Project. You will see a list of all Datasets that have been created by Forms in this Project. Click on any Dataset to see basic details about it. +To browse all Entity Lists in a Project, go to the :guilabel:`Entities` tab within the Project. You will see a list of all Entity Lists that have been created by Forms in this Project. Click on any Entity List to see basic details about it. .. image:: /img/central-entities/entity-landing.png -On this page, you can see how this Dataset relates to other incoming data in your Project: which Forms contribute to the Dataset, which ones read data from it, and which fields are being read or written. To see the actual data in your Dataset, click on the :guilabel:`Data` tab at the top. +On this page, you can see how this Entity List relates to other incoming data in your Project: which Forms contribute to the Entity List, which ones read data from it, and which fields are being read or written. To see the actual data in your Entity List, click on the :guilabel:`Data` tab at the top. .. _central-entities-data: Managing Entity Data -------------------- -You can preview or download Entity data from Central from the :guilabel:`Data` tab on the Dataset's page. +You can preview or download Entity data from Central from the :guilabel:`Data` tab on the Entity List's page. .. image:: /img/central-entities/entity-table.png @@ -276,10 +281,10 @@ To complete the process press the :guilabel:`Update` button to save your changes .. _central-entities-settings: -Changing Dataset Settings -------------------------- +Changing Entity List Settings +------------------------------ -Right now, only one setting is available for Datasets in Central. To reach it, click on the :guilabel:`Settings` tab on the Dataset page. +Right now, only one setting is available for Entity Lists in Central. To reach it, click on the :guilabel:`Settings` tab on the Entity List page. .. image:: /img/central-entities/entity-settings.png diff --git a/docs/central-intro.rst b/docs/central-intro.rst index 93603bd0e..2a58b755b 100644 --- a/docs/central-intro.rst +++ b/docs/central-intro.rst @@ -31,7 +31,7 @@ Here are some of the major features we support today: - With an interactive table preview of submission data - Support for reviewing, commenting on, and editing submissions after upload - Connectable to analysis and dashboard applications like Excel, Power BI, or R over OData - - Server-managed datasets for registration/follow-up workflows + - Entities for registration/follow-up workflows - Integrated checklist-based help system - Clean and modern REST API for integration and extensibility - High performance on low-cost hardware or cloud providers diff --git a/docs/img/central-entities/intro-diagram.png b/docs/img/central-entities/intro-diagram.png index bf8967dc5..698eb446c 100644 --- a/docs/img/central-entities/intro-diagram.png +++ b/docs/img/central-entities/intro-diagram.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:80acf557aed915f5c21b5530c0619c50916f88d952ac73e9a399ed449bc4f872 -size 8509 +oid sha256:59df2fde65fed2418c088601642765abfe93591e9abd56109db0eb1b5c5572b9 +size 8199