From 1c58d1e13ebd1d5acfed6e396a4e9cfb1d5dd48d Mon Sep 17 00:00:00 2001 From: Timothy Jennison Date: Mon, 19 Aug 2024 14:03:42 +0000 Subject: [PATCH] Add survey criteria * Loads all survey data for real time filtering. * Improves question/answer name handling, though indexing changes will be needed to handle global search correctly. * Multi-select plus instance level data still to be dealt with. * Config changes will come separately since they'll break stored data. * Filter generation shares code with existing entity group criteria. * Copies some filter generation code from entity group criteria but this allows the config and selection data to be distinct from entity group criteria. There is already some divergence and this allows them to evolve separately. This wasn't originally planned but handling backwards compatability while switching to a new plugin/config/selection data seemed pretty sketchy. * Fixes a bug where entity group criteria with modifiers but no selection weren't handled correctly. * Testing is weird because we don't currently have survey data. The basics are the same as entity group criteria so using the same data for now but TBD how we want to handle this. --- docs/generated/PROTOCOL_BUFFERS.md | 174 +++- docs/generated/UNDERLAY_CONFIG.md | 5 + ui/src/components/treegrid.tsx | 25 +- ui/src/criteria/classification.tsx | 13 +- ui/src/criteria/survey.tsx | 860 ++++++++++++++++++ ui/src/data/source.tsx | 55 +- ui/src/plugins.ts | 1 + ui/src/tanagra-underlay/underlayConfig.ts | 1 + .../bio/terra/tanagra/api/shared/Literal.java | 2 +- .../impl/core/EntityGroupFilterBuilder.java | 343 +------ .../core/EntityGroupFilterBuilderBase.java | 368 ++++++++ .../impl/core/SurveyFilterBuilder.java | 62 ++ .../core/utils/EntityGroupFilterUtils.java | 13 +- .../underlay/serialization/SZCorePlugin.java | 4 +- underlay/src/main/proto/column.proto | 29 + .../configschema/entity_group.proto | 27 +- .../configschema/survey.proto | 44 + .../criteriaselector/dataschema/survey.proto | 39 + ...ilterBuilderForCriteriaOccurrenceTest.java | 4 +- .../EntityGroupFilterBuilderForGroupTest.java | 4 +- .../EntityGroupFilterBuilderForItemsTest.java | 122 +-- 21 files changed, 1734 insertions(+), 461 deletions(-) create mode 100644 ui/src/criteria/survey.tsx create mode 100644 underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/EntityGroupFilterBuilderBase.java create mode 100644 underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/SurveyFilterBuilder.java create mode 100644 underlay/src/main/proto/column.proto create mode 100644 underlay/src/main/proto/criteriaselector/configschema/survey.proto create mode 100644 underlay/src/main/proto/criteriaselector/dataschema/survey.proto diff --git a/docs/generated/PROTOCOL_BUFFERS.md b/docs/generated/PROTOCOL_BUFFERS.md index 4639dc28c..8cc0e1061 100644 --- a/docs/generated/PROTOCOL_BUFFERS.md +++ b/docs/generated/PROTOCOL_BUFFERS.md @@ -3,6 +3,9 @@ ## Table of Contents +- [column.proto](#column-proto) + - [Column](#tanagra-Column) + - [criteriaselector/configschema/attribute.proto](#criteriaselector_configschema_attribute-proto) - [Attribute](#tanagra-configschema-Attribute) @@ -11,7 +14,6 @@ - [criteriaselector/configschema/entity_group.proto](#criteriaselector_configschema_entity_group-proto) - [EntityGroup](#tanagra-configschema-EntityGroup) - - [EntityGroup.Column](#tanagra-configschema-EntityGroup-Column) - [EntityGroup.EntityGroupConfig](#tanagra-configschema-EntityGroup-EntityGroupConfig) - [criteriaselector/configschema/multi_attribute.proto](#criteriaselector_configschema_multi_attribute-proto) @@ -20,6 +22,10 @@ - [criteriaselector/configschema/output_unfiltered.proto](#criteriaselector_configschema_output_unfiltered-proto) - [OutputUnfiltered](#tanagra-configschema-OutputUnfiltered) +- [criteriaselector/configschema/survey.proto](#criteriaselector_configschema_survey-proto) + - [Survey](#tanagra-configschema-Survey) + - [Survey.EntityGroupConfig](#tanagra-configschema-Survey-EntityGroupConfig) + - [criteriaselector/configschema/text_search.proto](#criteriaselector_configschema_text_search-proto) - [TextSearch](#tanagra-configschema-TextSearch) @@ -50,6 +56,10 @@ - [criteriaselector/dataschema/output_unfiltered.proto](#criteriaselector_dataschema_output_unfiltered-proto) - [OutputUnfiltered](#tanagra-dataschema-OutputUnfiltered) +- [criteriaselector/dataschema/survey.proto](#criteriaselector_dataschema_survey-proto) + - [Survey](#tanagra-dataschema-Survey) + - [Survey.Selection](#tanagra-dataschema-Survey-Selection) + - [criteriaselector/dataschema/text_search.proto](#criteriaselector_dataschema_text_search-proto) - [TextSearch](#tanagra-dataschema-TextSearch) - [TextSearch.Selection](#tanagra-dataschema-TextSearch-Selection) @@ -93,6 +103,42 @@ + +

Top

+ +## column.proto + + + + + +### Column +Defines a column in the UI. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| key | [string](#string) | | A unique key for the column. By default, used to look up attributes in the displayed data. | +| width_string | [string](#string) | | Passed directly to the style of the column. "100%" can be used to take up space remaining after laying out fixed columns. | +| width_double | [double](#double) | | Units used by the UI library to standardize dimensions. | +| title | [string](#string) | | The visible title of the column. | +| sortable | [bool](#bool) | | Whether the column supports sorting. | +| filterable | [bool](#bool) | | Whether the column supports filtering. | + + + + + + + + + + + + + + +

Top

@@ -176,8 +222,8 @@ which have condition_name of "Diabetes"). | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | -| columns | [EntityGroup.Column](#tanagra-configschema-EntityGroup-Column) | repeated | Columns displayed in the list view. | -| hierarchy_columns | [EntityGroup.Column](#tanagra-configschema-EntityGroup-Column) | repeated | Columns displayed in the hierarchy view. | +| columns | [tanagra.Column](#tanagra-Column) | repeated | Columns displayed in the list view. | +| hierarchy_columns | [tanagra.Column](#tanagra-Column) | repeated | Columns displayed in the hierarchy view. | | name_column_index | [int32](#int32) | | This has been replaced by nameAttribute for determining stored names. Now this only determines which is the primary column for checkboxes, etc. | | classification_entity_groups | [EntityGroup.EntityGroupConfig](#tanagra-configschema-EntityGroup-EntityGroupConfig) | repeated | Entity groups where the related entity is what is selected (e.g. condition when filtering condition_occurrences). | | grouping_entity_groups | [EntityGroup.EntityGroupConfig](#tanagra-configschema-EntityGroup-EntityGroupConfig) | repeated | Entity groups where the related entity is not what is selected (e.g. brands when filtering ingredients or genotyping platforms when filtering people). | @@ -192,26 +238,6 @@ which have condition_name of "Diabetes"). - - -### EntityGroup.Column -Defines a column in the UI. - - -| Field | Type | Label | Description | -| ----- | ---- | ----- | ----------- | -| key | [string](#string) | | A unique key for the column. By default, used to look up attributes in the displayed data. | -| width_string | [string](#string) | | Passed directly to the style of the column. "100%" can be used to take up space remaining after laying out fixed columns. | -| width_double | [double](#double) | | Units used by the UI library to standardize dimensions. | -| title | [string](#string) | | The visible title of the column. | -| sortable | [bool](#bool) | | Whether the column supports sorting. | -| filterable | [bool](#bool) | | Whether the column supports filtering. | - - - - - - ### EntityGroup.EntityGroupConfig @@ -307,6 +333,57 @@ include entire entities (e.g. demographics). + +

Top

+ +## criteriaselector/configschema/survey.proto + + + + + +### Survey + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| columns | [tanagra.Column](#tanagra-Column) | repeated | Columns displayed in the list view. | +| entity_groups | [Survey.EntityGroupConfig](#tanagra-configschema-Survey-EntityGroupConfig) | repeated | Entity groups where the related entity is what is selected (e.g. surveyBasics when filtering surveyOccurrence). | +| value_configs | [tanagra.ValueConfig](#tanagra-ValueConfig) | repeated | Optional configuration of a categorical or numeric value associated with the selection (e.g. a numeric answer). Applied to the entire selection so generally not compatible with multi_select. Currently only one is supported. | +| default_sort | [tanagra.SortOrder](#tanagra-SortOrder) | | The sort order to use in the list view, or in hierarchies where no sort order has been specified. | +| nameAttribute | [string](#string) | optional | The attribute used to name selections if not the first column. This can be used to include extra context with the selected values that's not visible in the table view. | + + + + + + + + +### Survey.EntityGroupConfig + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| id | [string](#string) | | The id of the entity group. | +| sort_order | [tanagra.SortOrder](#tanagra-SortOrder) | | The sort order applied to this entity group when displayed in the hierarchy view. | + + + + + + + + + + + + + + +

Top

@@ -644,6 +721,57 @@ values. + +

Top

+ +## criteriaselector/dataschema/survey.proto + + + + + +### Survey +Data for an entity group criteria is a list of selected values. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| selected | [Survey.Selection](#tanagra-dataschema-Survey-Selection) | repeated | | +| value_data | [tanagra.ValueData](#tanagra-ValueData) | | Data for an additional categorical or numeric value associated with the selection (e.g. a numeric answer). | + + + + + + + + +### Survey.Selection + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| key | [tanagra.Key](#tanagra-Key) | | The key of the selected value, which references a related entity (e.g. surveyBasics when filtering surveyOccurrence). | +| name | [string](#string) | | The visible name for the selection. This is stored to avoid extra lookups when rendering. | +| entityGroup | [string](#string) | | The entity group is stored to differentiate between them when multiple are configured within a single criteria. | +| question_key | [tanagra.Key](#tanagra-Key) | | If the selected item is an answer, the key of the question it belongs to. | +| question_name | [string](#string) | | If the selected item is an answer, the visible name of the question it belongs to. | + + + + + + + + + + + + + + +

Top

diff --git a/docs/generated/UNDERLAY_CONFIG.md b/docs/generated/UNDERLAY_CONFIG.md index d700d3b88..2c48725ae 100644 --- a/docs/generated/UNDERLAY_CONFIG.md +++ b/docs/generated/UNDERLAY_CONFIG.md @@ -193,6 +193,11 @@ Use `plugin: "multiAttribute"`. Use `plugin: "outputUnfiltered"`. +### SZCorePlugin.SURVEY +**required** [SZCorePlugin](#szcoreplugin) + +Use `plugin: "survey"`. + ### SZCorePlugin.TEXT_SEARCH **required** [SZCorePlugin](#szcoreplugin) diff --git a/ui/src/components/treegrid.tsx b/ui/src/components/treegrid.tsx index 7c833f28b..e5a988137 100644 --- a/ui/src/components/treegrid.tsx +++ b/ui/src/components/treegrid.tsx @@ -15,6 +15,7 @@ import Typography from "@mui/material/Typography"; import produce from "immer"; import { GridBox } from "layout/gridBox"; import GridLayout from "layout/gridLayout"; +import * as columnProto from "proto/column"; import { ChangeEvent, MutableRefObject, @@ -49,8 +50,8 @@ export type TreeGridItem = { data: TreeGridRowData; }; -export type TreeGridData = { - [key: TreeGridId]: TreeGridItem; +export type TreeGridData = { + [key: TreeGridId]: ItemType; }; export type TreeGridColumn = { @@ -84,9 +85,9 @@ export type TreeGridFilters = { [col: string]: string; }; -export type TreeGridProps = { +export type TreeGridProps = { columns: TreeGridColumn[]; - data: TreeGridData; + data: TreeGridData; defaultExpanded?: TreeGridId[]; highlightId?: TreeGridId; rowCustomization?: ( @@ -109,7 +110,9 @@ export type TreeGridProps = { onFilter?: (filters: TreeGridFilters) => void; }; -export function TreeGrid(props: TreeGridProps) { +export function TreeGrid( + props: TreeGridProps +) { const theme = useTheme(); const [state, updateState] = useImmer( @@ -391,6 +394,18 @@ export function useArrayAsTreeGridData< }, [array]); } +export function fromProtoColumns( + columns: columnProto.Column[] +): TreeGridColumn[] { + return columns.map((c) => ({ + key: c.key, + width: c.widthString ?? c.widthDouble ?? 100, + title: c.title, + sortable: c.sortable, + filterable: c.filterable, + })); +} + function renderChildren( theme: Theme, props: TreeGridProps, diff --git a/ui/src/criteria/classification.tsx b/ui/src/criteria/classification.tsx index db02822ed..7f05c8219 100644 --- a/ui/src/criteria/classification.tsx +++ b/ui/src/criteria/classification.tsx @@ -11,6 +11,7 @@ import Loading from "components/loading"; import { Search } from "components/search"; import { useSimpleDialog } from "components/simpleDialog"; import { + fromProtoColumns, TreeGrid, TreeGridColumn, TreeGridData, @@ -1022,18 +1023,6 @@ function fromProtoSortOrder(sortOrder: sortOrderProto.SortOrder): SortOrder { }; } -function fromProtoColumns( - columns: configProto.EntityGroup_Column[] -): TreeGridColumn[] { - return columns.map((c) => ({ - key: c.key, - width: c.widthString ?? c.widthDouble ?? 100, - title: c.title, - sortable: c.sortable, - filterable: c.filterable, - })); -} - function nameAttribute(config: configProto.EntityGroup) { return ( config.nameAttribute ?? config.columns[config.nameColumnIndex ?? 0].key diff --git a/ui/src/criteria/survey.tsx b/ui/src/criteria/survey.tsx new file mode 100644 index 000000000..a2ef3e635 --- /dev/null +++ b/ui/src/criteria/survey.tsx @@ -0,0 +1,860 @@ +import DeleteIcon from "@mui/icons-material/Delete"; +import Button from "@mui/material/Button"; +import IconButton from "@mui/material/IconButton"; +import Paper from "@mui/material/Paper"; +import Typography from "@mui/material/Typography"; +import { CriteriaPlugin, registerCriteriaPlugin } from "cohort"; +import Checkbox from "components/checkbox"; +import Empty from "components/empty"; +import Loading from "components/loading"; +import { Search } from "components/search"; +import { useSimpleDialog } from "components/simpleDialog"; +import { + fromProtoColumns, + TreeGrid, + TreeGridColumn, + TreeGridData, + TreeGridId, + TreeGridItem, + TreeGridRowData, +} from "components/treegrid"; +import { + ANY_VALUE_DATA, + decodeValueData, + encodeValueData, + ValueData, + ValueDataEdit, +} from "criteria/valueData"; +import { + ROLLUP_COUNT_ATTRIBUTE, + SortDirection, + SortOrder, +} from "data/configuration"; +import { + CommonSelectorConfig, + dataKeyFromProto, + EntityNode, + protoFromDataKey, + UnderlaySource, +} from "data/source"; +import { compareDataValues, DataEntry, DataKey } from "data/types"; +import { useUnderlaySource } from "data/underlaySourceContext"; +import { useUpdateCriteria } from "hooks"; +import emptyImage from "images/empty.svg"; +import produce from "immer"; +import { GridBox } from "layout/gridBox"; +import GridLayout from "layout/gridLayout"; +import * as configProto from "proto/criteriaselector/configschema/survey"; +import * as dataProto from "proto/criteriaselector/dataschema/survey"; +import * as sortOrderProto from "proto/sort_order"; +import { useCallback, useEffect, useMemo } from "react"; +import useSWRImmutable from "swr/immutable"; +import { useImmer } from "use-immer"; +import { base64ToBytes } from "util/base64"; +import { safeRegExp } from "util/safeRegExp"; +import { useLocalSearchState } from "util/searchState"; +import { isValid } from "util/valid"; + +type Selection = { + key: DataKey; + name: string; + entityGroup: string; + questionKey?: DataKey; + questionName: string; +}; + +enum EntityNodeItemType { + Question = "QUESTION", + Answer = "ANSWER", + Topic = "TOPIC", +} + +// A custom TreeGridItem allows us to store the EntityNode along with +// the rest of the data. +type EntityNodeItem = TreeGridItem & { + node: EntityNode; + entityGroup: string; + type: EntityNodeItemType; + parentKey?: DataKey; +}; + +type EntityTreeGridData = TreeGridData; + +// Exported for testing purposes. +export interface Data { + selected: Selection[]; + valueData: ValueData; +} + +// "survey" plugins are designed to handle medium sized (~<100k rows) amount of +// survey data in an optimized fashion. +@registerCriteriaPlugin( + "survey", + ( + underlaySource: UnderlaySource, + c: CommonSelectorConfig, + dataEntry?: DataEntry + ) => { + const config = decodeConfig(c); + + const data: Data = { + selected: [], + valueData: { ...ANY_VALUE_DATA }, + }; + + if (dataEntry) { + const name = String(dataEntry[nameAttribute(config)]); + const entityGroup = String(dataEntry.entityGroup); + if (!name || !entityGroup) { + throw new Error( + `Invalid parameters from search [${name}, ${entityGroup}].` + ); + } + + // TODO(tjennison): There's no way to get the question information for + // answers added via global search. We're currently not showing answers + // there so this isn't an issue, but if we were, that information would + // need to be made available at index time. + data.selected.push({ + key: dataEntry.key, + name, + entityGroup, + questionName: "", + }); + } + + return encodeData(data); + }, + search +) +// eslint-disable-next-line @typescript-eslint/no-unused-vars +class _ implements CriteriaPlugin { + public data: string; + private selector: CommonSelectorConfig; + private config: configProto.Survey; + + constructor(public id: string, selector: CommonSelectorConfig, data: string) { + this.selector = selector; + this.config = decodeConfig(selector); + this.data = data; + } + + renderEdit( + doneAction: () => void, + setBackAction: (action?: () => void) => void + ) { + return ( + + ); + } + + renderInline(groupId: string) { + const decodedData = decodeData(this.data); + + if (!this.config.valueConfigs.length || decodedData.selected.length) { + return null; + } + + return ( + + ); + } + + displayDetails() { + const decodedData = decodeData(this.data); + + const sel = groupSelection(decodedData.selected); + if (sel.length > 0) { + return { + title: + sel.length > 1 + ? `${sel[0].name} and ${sel.length - 1} more` + : sel[0].name, + standaloneTitle: true, + additionalText: sel.map( + (s) => + s.name + + (s.children.length + ? " (" + s.children.map((child) => child.name).join(", ") + ")" + : "") + ), + }; + } + + return { + title: "(any)", + }; + } +} + +function dataKey(key: DataKey, entityGroup: string): string { + return JSON.stringify({ + entityGroup, + key, + }); +} + +type SearchState = { + // The query entered in the search box. + query?: string; +}; + +type SurveyEditProps = { + data: string; + config: configProto.Survey; + doneAction: () => void; + setBackAction: (action?: () => void) => void; +}; + +function SurveyEdit(props: SurveyEditProps) { + const underlaySource = useUnderlaySource(); + const updateEncodedCriteria = useUpdateCriteria(); + const updateCriteria = useCallback( + (data: Data) => updateEncodedCriteria(encodeData(data)), + [updateEncodedCriteria] + ); + + const decodedData = useMemo(() => decodeData(props.data), [props.data]); + + const [localCriteria, updateLocalCriteria] = useImmer(decodedData); + + const selectedSets = useMemo(() => { + const sets = new Map>(); + localCriteria.selected.forEach((s) => { + if (!sets.has(s.entityGroup)) { + sets.set(s.entityGroup, new Set()); + } + sets.get(s.entityGroup)?.add(s.key); + }); + return sets; + }, [localCriteria]); + + const updateCriteriaFromLocal = useCallback(() => { + updateCriteria(produce(decodedData, () => localCriteria)); + }, [updateCriteria, localCriteria]); + + const [searchState, updateSearchState] = useLocalSearchState(); + + const [unconfirmedChangesDialog, showUnconfirmedChangesDialog] = + useSimpleDialog(); + + const unconfirmedChangesCallback = useCallback( + () => + showUnconfirmedChangesDialog({ + title: "Unsaved changes", + text: "Unsaved changes will be lost if you go back without saving.", + buttons: ["Cancel", "Go back", "Save"], + onButton: (button) => { + if (button === 1) { + props.doneAction(); + } else if (button === 2) { + updateCriteriaFromLocal(); + props.doneAction(); + } + }, + }), + [updateCriteriaFromLocal] + ); + + useEffect(() => { + // The extra function works around React defaulting to treating a function + // as an update function. + props.setBackAction(() => { + if (isDataEqual(decodedData, localCriteria)) { + return undefined; + } else { + return unconfirmedChangesCallback; + } + }); + }, [searchState, localCriteria]); + + const processEntities = useCallback( + (allEntityGroups: [string, EntityNode[]][]) => { + const data: EntityTreeGridData = {}; + + allEntityGroups.forEach(([entityGroup, nodes]) => { + nodes.forEach((node) => { + const rowData: TreeGridRowData = { ...node.data }; + const key = dataKey(node.data.key, entityGroup); + + let parentKey = "root"; + if (node.ancestors?.length) { + parentKey = dataKey(node.ancestors[0], entityGroup); + } + + const cItem: EntityNodeItem = { + data: rowData, + children: data[key]?.children ?? [], + node: node, + entityGroup, + type: + data[key]?.type ?? + (node.childCount === 0 + ? EntityNodeItemType.Answer + : EntityNodeItemType.Topic), + parentKey: parentKey, + }; + data[key] = cItem; + + if (data[parentKey]) { + data[parentKey].children?.push(key); + if (cItem.type === EntityNodeItemType.Answer) { + data[parentKey].type = EntityNodeItemType.Question; + } + } else { + const d = { + key: parentKey, + }; + data[parentKey] = { + data: d, + node: { + data: d, + entity: "loading", + }, + entityGroup, + type: + cItem.type === EntityNodeItemType.Answer + ? EntityNodeItemType.Question + : EntityNodeItemType.Topic, + children: [key], + }; + } + }); + }); + + return data; + }, + [] + ); + + const attributes = useMemo( + () => [ + ...new Set( + [ + props.config.columns.map(({ key }) => key), + nameAttribute(props.config), + ] + .flat() + .filter(isValid) + ), + ], + [props.config.columns] + ); + + const calcSortOrder = useCallback( + (primaryEntityGroupId?: string) => { + if (primaryEntityGroupId) { + const egSortOrder = props.config.entityGroups.find( + (c) => c.id === primaryEntityGroupId + )?.sortOrder; + if (egSortOrder) { + return egSortOrder; + } + } + + return props.config.defaultSort ?? DEFAULT_SORT_ORDER; + }, + [underlaySource] + ); + + const fetchInstances = useCallback(async () => { + const raw: [string, EntityNode[]][] = await Promise.all( + props.config.entityGroups.map(async (c) => [ + c.id, + ( + await underlaySource.searchEntityGroup( + attributes, + c.id, + fromProtoSortOrder(calcSortOrder(c.id)), + { + hierarchy: true, + fetchAll: true, + } + ) + ).nodes, + ]) + ); + + return processEntities(raw); + }, [underlaySource, attributes, processEntities]); + + const instancesState = useSWRImmutable( + { + type: "entityGroupInstances", + entityGroupIds: [...props.config.entityGroups].map((eg) => eg.id), + attributes, + }, + fetchInstances + ); + + const filteredData = useMemo(() => { + const data = instancesState.data; + if (!data || !searchState?.query) { + return data ?? {}; + } + + // TODO(tjennison): Handle RegExp errors. + const [re] = safeRegExp(searchState?.query); + const matched = new Set(); + + const matchNode = (key: TreeGridId) => { + const node = data[key]; + if (node.type != EntityNodeItemType.Topic) { + for (const k in node.data) { + if (re.test(String(node.data[k]))) { + matched.add(key); + node.node.ancestors?.forEach((a) => + matched.add(dataKey(a, node.entityGroup)) + ); + break; + } + } + } + + node.children?.forEach((c) => matchNode(c)); + }; + matchNode("root"); + + const filtered = produce(data, (data) => { + for (const key in data) { + const node = data[key]; + data[key].children = + node.type !== EntityNodeItemType.Topic + ? node.children + : node.children?.filter((c) => matched.has(c)); + } + }); + return filtered; + }, [instancesState.data, searchState?.query]); + + const columns: TreeGridColumn[] = useMemo( + () => fromProtoColumns(props.config.columns), + [props.config.columns] + ); + + const groupedSelection = useMemo( + () => groupSelection(localCriteria.selected ?? []), + [localCriteria.selected] + ); + + return ( + theme.palette.background.paper, + }} + > + + + + { + updateSearchState((data: SearchState) => { + data.query = query; + }); + }} + initialValue={searchState?.query} + /> + + + {!filteredData?.root?.children?.length ? ( + + ) : ( + + columns={columns} + data={filteredData} + expandable + reserveExpansionSpacing + rowCustomization={( + id: TreeGridId, + rowData: TreeGridRowData + ) => { + if (!instancesState.data) { + return undefined; + } + + // TODO(tjennison): Make TreeGridData's type generic so we can + // avoid this type assertion. Also consider passing the + // TreeGridItem to the callback instead of the TreeGridRowData. + const item = instancesState.data[id]; + if (!item) { + return undefined; + } + + const entityGroupSet = selectedSets.get(item.entityGroup); + const found = !!entityGroupSet?.has(item.node.data.key); + const foundAncestor = !!item.node.ancestors?.reduce( + (acc, cur) => acc || !!entityGroupSet?.has(cur), + false + ); + + return [ + { + column: 0, + prefixElements: ( + { + updateLocalCriteria((data) => { + if (found) { + data.selected = data.selected.filter( + (s) => + item.node.data.key !== s.key || + item.entityGroup !== s.entityGroup + ); + } else { + const question = + item.parentKey && + item.type === EntityNodeItemType.Answer + ? instancesState.data?.[item.parentKey] + : undefined; + const name = + rowData[nameAttribute(props.config)]; + const questionName = + question?.node?.data?.[ + nameAttribute(props.config) + ]; + data.selected.push({ + key: item.node.data.key, + name: !!name ? String(name) : "", + entityGroup: item.entityGroup, + questionKey: question?.node?.data?.key, + questionName: !!questionName + ? String(questionName) + : "", + }); + } + data.valueData = ANY_VALUE_DATA; + }); + }} + /> + ), + }, + ]; + }} + /> + )} + + + theme.palette.background.default, + }} + > + + + {groupedSelection.length ? ( + + + Selected items: + + + {groupedSelection.map((s, i) => ( + + + `0 -1px 0 ${theme.palette.divider}` + : undefined, + }} + > + {s.name} + {s.index >= 0 ? ( + + updateLocalCriteria((data) => { + data.selected.splice(s.index, 1); + }) + } + > + + + ) : null} + + + {s.children.map((child) => ( + + + {child.name} + + {child.index >= 0 ? ( + + updateLocalCriteria((data) => { + data.selected.splice(child.index, 1); + }) + } + > + + + ) : null} + + ))} + + + ))} + + + + + ) : ( + + )} + + + + + + + + {unconfirmedChangesDialog} + + ); +} + +type SurveyInlineProps = { + groupId: string; + criteriaId: string; + data: string; + config: configProto.Survey; +}; + +function SurveyInline(props: SurveyInlineProps) { + const underlaySource = useUnderlaySource(); + const updateEncodedCriteria = useUpdateCriteria(); + const updateCriteria = useCallback( + (data: Data) => updateEncodedCriteria(encodeData(data)), + [updateEncodedCriteria] + ); + + const decodedData = useMemo(() => decodeData(props.data), [props.data]); + + if (!props.config.valueConfigs.length || !decodedData.selected.length) { + return null; + } + + const entityGroup = underlaySource.lookupEntityGroup( + decodedData.selected[0].entityGroup + ); + + return ( + + updateCriteria( + produce(decodedData, (data) => { + data.valueData = valueData[0]; + }) + ) + } + /> + ); +} + +async function search( + underlaySource: UnderlaySource, + c: CommonSelectorConfig, + query: string +): Promise { + const config = decodeConfig(c); + const results = await Promise.all( + (config.entityGroups ?? []).map((eg) => + underlaySource + .searchEntityGroup( + config.columns.map(({ key }) => key), + eg.id, + fromProtoSortOrder(config.defaultSort ?? DEFAULT_SORT_ORDER), + { + query, + isLeaf: false, + } + ) + .then((res) => + res.nodes.map((node) => ({ + ...node.data, + entityGroup: eg.id, + })) + ) + ) + ); + + return results.flat(); +} + +function isDataEqual(data1: Data, data2: Data) { + // TODO(tjennison): In future the ValueData may need to be compared as well. + if (data1.selected.length != data2.selected.length) { + return false; + } + return data1.selected.reduce( + (acc, cur, i) => + acc && + cur.key === data2.selected[i].key && + cur.entityGroup === data2.selected[i].entityGroup, + true + ); +} + +function decodeData(data: string): Data { + const message = + data[0] === "{" + ? dataProto.Survey.fromJSON(JSON.parse(data)) + : dataProto.Survey.decode(base64ToBytes(data)); + + return { + selected: + message.selected?.map((s) => ({ + key: dataKeyFromProto(s.key), + name: s.name, + entityGroup: s.entityGroup, + questionKey: s.questionKey + ? dataKeyFromProto(s.questionKey) + : undefined, + questionName: s.questionName, + })) ?? [], + valueData: decodeValueData(message.valueData), + }; +} + +function encodeData(data: Data): string { + const message: dataProto.Survey = { + selected: data.selected.map((s) => ({ + key: protoFromDataKey(s.key), + name: s.name, + entityGroup: s.entityGroup, + questionKey: s.questionKey ? protoFromDataKey(s.questionKey) : undefined, + questionName: s.questionName, + })), + valueData: encodeValueData(data.valueData), + }; + return JSON.stringify(dataProto.Survey.toJSON(message)); +} + +const DEFAULT_SORT_ORDER = { + attribute: ROLLUP_COUNT_ATTRIBUTE, + direction: sortOrderProto.SortOrder_Direction.SORT_ORDER_DIRECTION_DESCENDING, +}; + +function decodeConfig(selector: CommonSelectorConfig): configProto.Survey { + return configProto.Survey.fromJSON(JSON.parse(selector.pluginConfig)); +} + +function fromProtoSortOrder(sortOrder: sortOrderProto.SortOrder): SortOrder { + return { + attribute: sortOrder.attribute, + direction: + sortOrder.direction === + sortOrderProto.SortOrder_Direction.SORT_ORDER_DIRECTION_DESCENDING + ? SortDirection.Desc + : SortDirection.Asc, + }; +} + +function nameAttribute(config: configProto.Survey) { + return config.nameAttribute ?? config.columns[0].key; +} + +type GroupedSelectionItem = { + index: number; + key: DataKey; + name: string; + children: GroupedSelectionItem[]; +}; + +function groupSelection(selected: Selection[]): GroupedSelectionItem[] { + const map = new Map(); + + selected.forEach((s, index) => { + const item: GroupedSelectionItem = { + index, + key: s.key, + name: s.name, + children: [], + }; + + if (!s.questionKey) { + const existing = map.get(s.key); + if (existing) { + // An answer and its question are both selected. + item.children = existing.children; + } + map.set(s.key, item); + } else { + const question = map.get(s.questionKey); + if (question) { + question.children.push(item); + } else { + map.set(s.questionKey, { + index: -1, + key: s.questionKey, + name: s.questionName ?? "Unknown", + children: [item], + }); + } + } + }); + + return Array.from(map, ([, item]) => { + item.children?.sort((a, b) => compareDataValues(a.key, b.key)); + return item; + }).sort((a, b) => compareDataValues(a.key, b.key)); +} diff --git a/ui/src/data/source.tsx b/ui/src/data/source.tsx index 22e6dd481..aa0f23147 100644 --- a/ui/src/data/source.tsx +++ b/ui/src/data/source.tsx @@ -27,6 +27,8 @@ export type SearchEntityGroupOptions = { parent?: DataKey; limit?: number; hierarchy?: boolean; + fetchAll?: boolean; + isLeaf?: boolean; }; export type SearchEntityGroupResult = { @@ -638,9 +640,7 @@ export class BackendUnderlaySource implements UnderlaySource { entityGroup, entity, sortOrder, - options?.query, - options?.parent, - options?.limit + options ) ) .then((res) => ({ @@ -895,14 +895,12 @@ export class BackendUnderlaySource implements UnderlaySource { entityGroup: EntityGroupData, entity: tanagraUnderlay.SZEntity, sortOrder: SortOrder, - query?: string, - parent?: DataValue, - limit?: number + options?: SearchEntityGroupOptions ): tanagra.ListInstancesRequest { const hierarchy = entity.hierarchies?.[0]?.name; const operands: tanagra.Filter[] = []; - if (entityGroup.relatedEntityId && parent) { + if (entityGroup.relatedEntityId && options?.parent) { const groupingEntity = this.lookupEntity(entityGroup.entityId); operands.push({ filterType: tanagra.FilterFilterTypeEnum.Relationship, @@ -915,7 +913,7 @@ export class BackendUnderlaySource implements UnderlaySource { attributeFilter: { attribute: groupingEntity.idAttribute, operator: tanagra.AttributeFilterOperatorEnum.Equals, - values: [literalFromDataValue(parent)], + values: [literalFromDataValue(options?.parent)], }, }, }, @@ -923,30 +921,30 @@ export class BackendUnderlaySource implements UnderlaySource { }, }); } else { - if (hierarchy && parent) { + if (hierarchy && options?.parent) { operands.push({ filterType: tanagra.FilterFilterTypeEnum.Hierarchy, filterUnion: { hierarchyFilter: { hierarchy, operator: tanagra.HierarchyFilterOperatorEnum.ChildOf, - values: [literalFromDataValue(parent)], + values: [literalFromDataValue(options?.parent)], }, }, }); - } else if (isValid(query)) { - if (query !== "") { + } else if (isValid(options?.query)) { + if (options?.query !== "") { operands.push({ filterType: tanagra.FilterFilterTypeEnum.Text, filterUnion: { textFilter: { matchType: tanagra.TextFilterMatchTypeEnum.ExactMatch, - text: query, + text: options?.query, }, }, }); } - } else if (hierarchy) { + } else if (hierarchy && !options?.fetchAll) { operands.push({ filterType: tanagra.FilterFilterTypeEnum.Hierarchy, filterUnion: { @@ -970,9 +968,33 @@ export class BackendUnderlaySource implements UnderlaySource { }, }, }); + + if (options && isValid(options.isLeaf)) { + let isLeafFilter: tanagra.Filter | null = { + filterType: tanagra.FilterFilterTypeEnum.Hierarchy, + filterUnion: { + hierarchyFilter: { + hierarchy, + operator: tanagra.HierarchyFilterOperatorEnum.IsLeaf, + }, + }, + }; + + if (!options.isLeaf) { + isLeafFilter = makeBooleanLogicFilter( + tanagra.BooleanLogicFilterOperatorEnum.Not, + [isLeafFilter] + ); + } + + if (isLeafFilter) { + operands.push(isLeafFilter); + } + } } } + const limit = options?.fetchAll ? 100000 : options?.limit; const req = { entityName: entity.name, underlayName: this.underlay.name, @@ -1705,7 +1727,10 @@ function makeBooleanLogicFilter( if (!subfilters || subfilters.length === 0) { return null; } - if (subfilters.length === 1) { + if ( + subfilters.length === 1 && + operator !== tanagra.BooleanLogicFilterOperatorEnum.Not + ) { return subfilters[0]; } return { diff --git a/ui/src/plugins.ts b/ui/src/plugins.ts index 9b3cc512d..d13e8a462 100644 --- a/ui/src/plugins.ts +++ b/ui/src/plugins.ts @@ -7,6 +7,7 @@ import "criteria/classification"; import "criteria/textSearch"; import "criteria/unhintedValue"; import "criteria/outputUnfiltered"; +import "criteria/survey"; // Cohort review plugins import "cohortReview/plugins/occurrenceTable"; diff --git a/ui/src/tanagra-underlay/underlayConfig.ts b/ui/src/tanagra-underlay/underlayConfig.ts index a7e110d0f..58f688123 100644 --- a/ui/src/tanagra-underlay/underlayConfig.ts +++ b/ui/src/tanagra-underlay/underlayConfig.ts @@ -26,6 +26,7 @@ export enum SZCorePlugin { ENTITY_GROUP = "ENTITY_GROUP", MULTI_ATTRIBUTE = "MULTI_ATTRIBUTE", OUTPUT_UNFILTERED = "OUTPUT_UNFILTERED", + SURVEY = "SURVEY", TEXT_SEARCH = "TEXT_SEARCH", UNHINTED_VALUE = "UNHINTED_VALUE", }; diff --git a/underlay/src/main/java/bio/terra/tanagra/api/shared/Literal.java b/underlay/src/main/java/bio/terra/tanagra/api/shared/Literal.java index 3e2540cc3..fde0014e4 100644 --- a/underlay/src/main/java/bio/terra/tanagra/api/shared/Literal.java +++ b/underlay/src/main/java/bio/terra/tanagra/api/shared/Literal.java @@ -14,7 +14,7 @@ import java.util.Objects; @SuppressWarnings("PMD.ImmutableField") -public final class Literal { +public final class Literal implements Comparable { private final boolean isNull; private final DataType dataType; private String stringVal; diff --git a/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/EntityGroupFilterBuilder.java b/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/EntityGroupFilterBuilder.java index 37d0f5bfe..8af225c53 100644 --- a/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/EntityGroupFilterBuilder.java +++ b/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/EntityGroupFilterBuilder.java @@ -1,330 +1,22 @@ package bio.terra.tanagra.filterbuilder.impl.core; -import static bio.terra.tanagra.filterbuilder.impl.core.utils.AttributeSchemaUtils.IGNORED_ATTRIBUTE_NAME_UI_USE_ONLY; import static bio.terra.tanagra.utils.ProtobufUtils.deserializeFromJsonOrProtoBytes; -import bio.terra.tanagra.api.filter.BooleanAndOrFilter; -import bio.terra.tanagra.api.filter.EntityFilter; -import bio.terra.tanagra.api.filter.PrimaryWithCriteriaFilter; import bio.terra.tanagra.api.shared.Literal; -import bio.terra.tanagra.exception.InvalidQueryException; -import bio.terra.tanagra.exception.SystemException; -import bio.terra.tanagra.filterbuilder.EntityOutput; -import bio.terra.tanagra.filterbuilder.FilterBuilder; -import bio.terra.tanagra.filterbuilder.impl.core.utils.AttributeSchemaUtils; -import bio.terra.tanagra.filterbuilder.impl.core.utils.EntityGroupFilterUtils; -import bio.terra.tanagra.filterbuilder.impl.core.utils.GroupByCountSchemaUtils; +import bio.terra.tanagra.proto.criteriaselector.ValueDataOuterClass; import bio.terra.tanagra.proto.criteriaselector.configschema.CFEntityGroup; -import bio.terra.tanagra.proto.criteriaselector.configschema.CFUnhintedValue; import bio.terra.tanagra.proto.criteriaselector.dataschema.DTEntityGroup; -import bio.terra.tanagra.proto.criteriaselector.dataschema.DTUnhintedValue; -import bio.terra.tanagra.underlay.Underlay; -import bio.terra.tanagra.underlay.entitymodel.Attribute; -import bio.terra.tanagra.underlay.entitymodel.Entity; -import bio.terra.tanagra.underlay.entitymodel.entitygroup.CriteriaOccurrence; -import bio.terra.tanagra.underlay.entitymodel.entitygroup.EntityGroup; -import bio.terra.tanagra.underlay.entitymodel.entitygroup.GroupItems; import bio.terra.tanagra.underlay.uiplugin.CriteriaSelector; -import bio.terra.tanagra.underlay.uiplugin.SelectionData; -import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; -import java.util.ArrayList; -import java.util.Comparator; import java.util.HashMap; -import java.util.HashSet; import java.util.List; import java.util.Map; -import java.util.Optional; -import java.util.Set; import java.util.stream.Collectors; -import org.apache.commons.lang3.tuple.Pair; -@SuppressFBWarnings( - value = "NP_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD", - justification = "The config and data objects are deserialized by Jackson.") -public class EntityGroupFilterBuilder extends FilterBuilder { +public class EntityGroupFilterBuilder extends EntityGroupFilterBuilderBase { public EntityGroupFilterBuilder(CriteriaSelector criteriaSelector) { super(criteriaSelector); } - @Override - public EntityFilter buildForCohort(Underlay underlay, List selectionData) { - DTEntityGroup.EntityGroup entityGroupSelectionData = - deserializeData(selectionData.get(0).getPluginData()); - List modifiersSelectionData = selectionData.subList(1, selectionData.size()); - if (entityGroupSelectionData == null || entityGroupSelectionData.getSelectedList().isEmpty()) { - // Empty selection data = null filter for a cohort. - return null; - } - - // We want to build one filter per entity group, not one filter per selected id. - Map> selectedIdsPerEntityGroup = - selectedIdsPerEntityGroup(underlay, entityGroupSelectionData); - - List entityFilters = new ArrayList<>(); - selectedIdsPerEntityGroup.entrySet().stream() - .sorted(Comparator.comparing(entry -> entry.getKey().getName())) - .forEach( - entry -> { - EntityGroup entityGroup = entry.getKey(); - List selectedIds = entry.getValue(); - switch (entityGroup.getType()) { - case CRITERIA_OCCURRENCE: - entityFilters.add( - buildPrimaryWithCriteriaFilter( - underlay, - (CriteriaOccurrence) entityGroup, - selectedIds, - entityGroupSelectionData, - modifiersSelectionData)); - break; - case GROUP_ITEMS: - entityFilters.add( - buildGroupItemsFilter( - underlay, (GroupItems) entityGroup, selectedIds, modifiersSelectionData)); - break; - default: - throw new SystemException( - "Unsupported entity group type: " + entityGroup.getType()); - } - }); - - return entityFilters.size() == 1 - ? entityFilters.get(0) - : new BooleanAndOrFilter(BooleanAndOrFilter.LogicalOperator.OR, entityFilters); - } - - @Override - public List buildForDataFeature( - Underlay underlay, List selectionData) { - DTEntityGroup.EntityGroup entityGroupSelectionData = - deserializeData(selectionData.get(0).getPluginData()); - List modifiersSelectionData = selectionData.subList(1, selectionData.size()); - - if (entityGroupSelectionData == null || entityGroupSelectionData.getSelectedList().isEmpty()) { - // Empty selection data = output all occurrence entities with null filters. - // Use the list of all possible entity groups in the config. - CFEntityGroup.EntityGroup entityGroupConfig = deserializeConfig(); - Set outputEntities = new HashSet<>(); - entityGroupConfig - .getClassificationEntityGroupsList() - .forEach( - classificationEntityGroup -> { - EntityGroup entityGroup = - underlay.getEntityGroup(classificationEntityGroup.getId()); - switch (entityGroup.getType()) { - case CRITERIA_OCCURRENCE: - CriteriaOccurrence criteriaOccurrence = (CriteriaOccurrence) entityGroup; - outputEntities.addAll(criteriaOccurrence.getOccurrenceEntities()); - break; - case GROUP_ITEMS: - GroupItems groupItems = (GroupItems) entityGroup; - outputEntities.add( - groupItems.getItemsEntity().isPrimary() - ? groupItems.getGroupEntity() - : groupItems.getItemsEntity()); - break; - default: - throw new SystemException( - "Unsupported entity group type: " + entityGroup.getType()); - } - }); - return outputEntities.stream().map(EntityOutput::unfiltered).collect(Collectors.toList()); - } else { - // Check that there are no group by modifiers. - Optional> - groupByModifierConfigAndData = - GroupByCountSchemaUtils.getModifier(criteriaSelector, modifiersSelectionData); - if (groupByModifierConfigAndData.isPresent()) { - throw new InvalidQueryException("Group by modifiers are not supported for data features"); - } - - // We want to build filters per entity group, not per selected id. - Map> selectedIdsPerEntityGroup = - selectedIdsPerEntityGroup(underlay, entityGroupSelectionData); - - Map> filtersPerEntity = new HashMap<>(); - selectedIdsPerEntityGroup.forEach( - (entityGroup, selectedIds) -> { - Map> filtersForSingleEntityGroup; - switch (entityGroup.getType()) { - case CRITERIA_OCCURRENCE: - CriteriaOccurrence criteriaOccurrence = (CriteriaOccurrence) entityGroup; - filtersForSingleEntityGroup = - EntityGroupFilterUtils.addOccurrenceFiltersForDataFeature( - underlay, criteriaOccurrence, selectedIds); - EntityGroupFilterUtils.buildAllModifierFilters( - underlay, - criteriaOccurrence.getOccurrenceEntities(), - criteriaSelector, - entityGroupSelectionData, - modifiersSelectionData, - filtersForSingleEntityGroup); - break; - case GROUP_ITEMS: - GroupItems groupItems = (GroupItems) entityGroup; - Entity notPrimaryEntity = - groupItems.getGroupEntity().isPrimary() - ? groupItems.getItemsEntity() - : groupItems.getGroupEntity(); - - filtersForSingleEntityGroup = new HashMap<>(); - EntityFilter idSubFilter = - EntityGroupFilterUtils.buildIdSubFilter( - underlay, notPrimaryEntity, selectedIds); - if (idSubFilter == null) { - filtersForSingleEntityGroup.put(notPrimaryEntity, new ArrayList<>()); - } else { - filtersForSingleEntityGroup.put( - notPrimaryEntity, new ArrayList<>(List.of(idSubFilter))); - } - EntityGroupFilterUtils.buildAllModifierFilters( - underlay, - List.of(notPrimaryEntity), - criteriaSelector, - entityGroupSelectionData, - modifiersSelectionData, - filtersForSingleEntityGroup); - break; - default: - throw new SystemException( - "Unsupported entity group type: " + entityGroup.getType()); - } - - List entityOutputsForSingleEntityGroup = - EntityGroupFilterUtils.mergeFiltersForDataFeature( - filtersForSingleEntityGroup, BooleanAndOrFilter.LogicalOperator.AND); - entityOutputsForSingleEntityGroup.forEach( - entityOutput -> { - List filters = - filtersPerEntity.getOrDefault(entityOutput.getEntity(), new ArrayList<>()); - if (entityOutput.hasDataFeatureFilter()) { - filters.add(entityOutput.getDataFeatureFilter()); - } else { - filters = new ArrayList<>(); - } - filtersPerEntity.put(entityOutput.getEntity(), filters); - }); - }); - - // If there are multiple filters for a single entity, OR them together. - return EntityGroupFilterUtils.mergeFiltersForDataFeature( - filtersPerEntity, BooleanAndOrFilter.LogicalOperator.OR); - } - } - - private Map> selectedIdsPerEntityGroup( - Underlay underlay, DTEntityGroup.EntityGroup entityGroupSelectionData) { - Map> selectedIdsPerEntityGroup = new HashMap<>(); - for (DTEntityGroup.EntityGroup.Selection selectedId : - entityGroupSelectionData.getSelectedList()) { - EntityGroup entityGroup = underlay.getEntityGroup(selectedId.getEntityGroup()); - List selectedIds = - selectedIdsPerEntityGroup.containsKey(entityGroup) - ? selectedIdsPerEntityGroup.get(entityGroup) - : new ArrayList<>(); - if (selectedId.hasKey()) { - selectedIds.add(Literal.forInt64(selectedId.getKey().getInt64Key())); - } - selectedIdsPerEntityGroup.put(entityGroup, selectedIds); - } - return selectedIdsPerEntityGroup; - } - - private EntityFilter buildPrimaryWithCriteriaFilter( - Underlay underlay, - CriteriaOccurrence criteriaOccurrence, - List selectedIds, - DTEntityGroup.EntityGroup entityGroupSelectionData, - List modifiersSelectionData) { - // Build the criteria sub-filter. - EntityFilter criteriaSubFilter = - EntityGroupFilterUtils.buildIdSubFilter( - underlay, criteriaOccurrence.getCriteriaEntity(), selectedIds); - - // Build the attribute modifier filters. - Map> subFiltersPerOccurrenceEntity = - EntityGroupFilterUtils.buildAttributeModifierFilters( - underlay, - criteriaSelector, - modifiersSelectionData, - criteriaOccurrence.getOccurrenceEntities()); - - // Build the instance-level modifier filters. - if (entityGroupSelectionData.hasValueData() - && !IGNORED_ATTRIBUTE_NAME_UI_USE_ONLY.equalsIgnoreCase( - entityGroupSelectionData.getValueData().getAttribute())) { - if (criteriaOccurrence.getOccurrenceEntities().size() > 1) { - throw new InvalidQueryException( - "Instance-level modifiers are not supported for entity groups with multiple occurrence entities: " - + criteriaOccurrence.getName()); - } - Entity occurrenceEntity = criteriaOccurrence.getOccurrenceEntities().get(0); - - EntityFilter attrFilter = - AttributeSchemaUtils.buildForEntity( - underlay, - occurrenceEntity, - occurrenceEntity.getAttribute(entityGroupSelectionData.getValueData().getAttribute()), - entityGroupSelectionData.getValueData()); - List subFilters = - subFiltersPerOccurrenceEntity.containsKey(occurrenceEntity) - ? subFiltersPerOccurrenceEntity.get(occurrenceEntity) - : new ArrayList<>(); - subFilters.add(attrFilter); - subFiltersPerOccurrenceEntity.put(occurrenceEntity, subFilters); - } - - Optional> - groupByModifierConfigAndData = - GroupByCountSchemaUtils.getModifier(criteriaSelector, modifiersSelectionData); - if (groupByModifierConfigAndData.isEmpty() - || groupByModifierConfigAndData.get().getRight() == null) { - return new PrimaryWithCriteriaFilter( - underlay, - criteriaOccurrence, - criteriaSubFilter, - subFiltersPerOccurrenceEntity, - null, - null, - null); - } - - // Build the group by filter information. - Map> groupByAttributesPerOccurrenceEntity = - GroupByCountSchemaUtils.getGroupByAttributesPerOccurrenceEntity( - underlay, groupByModifierConfigAndData, criteriaOccurrence.getOccurrenceEntities()); - DTUnhintedValue.UnhintedValue groupByModifierData = - groupByModifierConfigAndData.get().getRight(); - return new PrimaryWithCriteriaFilter( - underlay, - criteriaOccurrence, - criteriaSubFilter, - subFiltersPerOccurrenceEntity, - groupByAttributesPerOccurrenceEntity, - GroupByCountSchemaUtils.toBinaryOperator(groupByModifierData.getOperator()), - (int) groupByModifierData.getMin()); - } - - private EntityFilter buildGroupItemsFilter( - Underlay underlay, - GroupItems groupItems, - List selectedIds, - List modifiersSelectionData) { - Entity notPrimaryEntity = - groupItems.getGroupEntity().isPrimary() - ? groupItems.getItemsEntity() - : groupItems.getGroupEntity(); - - // Build the sub-filters on the non-primary entity. - List idFilterNonPrimaryEntity = new ArrayList<>(); - if (!selectedIds.isEmpty()) { - idFilterNonPrimaryEntity.add( - EntityGroupFilterUtils.buildIdSubFilter(underlay, notPrimaryEntity, selectedIds)); - } - return EntityGroupFilterUtils.buildGroupItemsFilter( - underlay, criteriaSelector, groupItems, idFilterNonPrimaryEntity, modifiersSelectionData); - } - @Override public CFEntityGroup.EntityGroup deserializeConfig() { return deserializeFromJsonOrProtoBytes( @@ -339,4 +31,35 @@ public DTEntityGroup.EntityGroup deserializeData(String serialized) { : deserializeFromJsonOrProtoBytes(serialized, DTEntityGroup.EntityGroup.newBuilder()) .build(); } + + @Override + protected List entityGroupIds() { + return deserializeConfig().getClassificationEntityGroupsList().stream() + .map( + classificationEntityGroup -> { + return classificationEntityGroup.getId(); + }) + .collect(Collectors.toList()); + } + + @Override + protected Map selectedIdsAndEntityGroups(String serializedSelectionData) { + Map idsAndEntityGroups = new HashMap<>(); + DTEntityGroup.EntityGroup selectionData = deserializeData(serializedSelectionData); + if (selectionData != null) { + for (DTEntityGroup.EntityGroup.Selection selectedId : selectionData.getSelectedList()) { + if (selectedId.hasKey()) { + idsAndEntityGroups.put( + Literal.forInt64(selectedId.getKey().getInt64Key()), selectedId.getEntityGroup()); + } + } + } + return idsAndEntityGroups; + } + + @Override + protected ValueDataOuterClass.ValueData valueData(String serializedSelectionData) { + DTEntityGroup.EntityGroup selectionData = deserializeData(serializedSelectionData); + return selectionData.hasValueData() ? selectionData.getValueData() : null; + } } diff --git a/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/EntityGroupFilterBuilderBase.java b/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/EntityGroupFilterBuilderBase.java new file mode 100644 index 000000000..f6af31f19 --- /dev/null +++ b/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/EntityGroupFilterBuilderBase.java @@ -0,0 +1,368 @@ +package bio.terra.tanagra.filterbuilder.impl.core; + +import static bio.terra.tanagra.filterbuilder.impl.core.utils.AttributeSchemaUtils.IGNORED_ATTRIBUTE_NAME_UI_USE_ONLY; + +import bio.terra.tanagra.api.filter.BooleanAndOrFilter; +import bio.terra.tanagra.api.filter.EntityFilter; +import bio.terra.tanagra.api.filter.PrimaryWithCriteriaFilter; +import bio.terra.tanagra.api.shared.Literal; +import bio.terra.tanagra.exception.InvalidQueryException; +import bio.terra.tanagra.exception.SystemException; +import bio.terra.tanagra.filterbuilder.EntityOutput; +import bio.terra.tanagra.filterbuilder.FilterBuilder; +import bio.terra.tanagra.filterbuilder.impl.core.utils.AttributeSchemaUtils; +import bio.terra.tanagra.filterbuilder.impl.core.utils.EntityGroupFilterUtils; +import bio.terra.tanagra.filterbuilder.impl.core.utils.GroupByCountSchemaUtils; +import bio.terra.tanagra.proto.criteriaselector.ValueDataOuterClass; +import bio.terra.tanagra.proto.criteriaselector.configschema.CFUnhintedValue; +import bio.terra.tanagra.proto.criteriaselector.dataschema.DTUnhintedValue; +import bio.terra.tanagra.underlay.Underlay; +import bio.terra.tanagra.underlay.entitymodel.Attribute; +import bio.terra.tanagra.underlay.entitymodel.Entity; +import bio.terra.tanagra.underlay.entitymodel.entitygroup.CriteriaOccurrence; +import bio.terra.tanagra.underlay.entitymodel.entitygroup.EntityGroup; +import bio.terra.tanagra.underlay.entitymodel.entitygroup.GroupItems; +import bio.terra.tanagra.underlay.uiplugin.CriteriaSelector; +import bio.terra.tanagra.underlay.uiplugin.SelectionData; +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Comparator; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.stream.Collectors; +import org.apache.commons.lang3.tuple.Pair; + +@SuppressFBWarnings( + value = "NP_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD", + justification = "The config and data objects are deserialized by Jackson.") +public abstract class EntityGroupFilterBuilderBase extends FilterBuilder { + public EntityGroupFilterBuilderBase(CriteriaSelector criteriaSelector) { + super(criteriaSelector); + } + + @Override + public EntityFilter buildForCohort(Underlay underlay, List selectionData) { + String criteriaSelectionData = selectionData.get(0).getPluginData(); + List modifiersSelectionData = selectionData.subList(1, selectionData.size()); + + // We want to build one filter per entity group, not one filter per selected id. + Map> selectedIdsPerEntityGroup = + selectedIdsPerEntityGroup(underlay, criteriaSelectionData); + if (selectedIdsPerEntityGroup.isEmpty() && modifiersSelectionData.isEmpty()) { + // Empty selection data = null filter for a cohort. + return null; + } + + List selectedEntityGroups = + selectedEntityGroups(underlay, selectedIdsPerEntityGroup); + + List entityFilters = new ArrayList<>(); + selectedEntityGroups.forEach( + entityGroup -> { + List selectedIds = selectedIdsPerEntityGroup.get(entityGroup); + if (selectedIds == null) { + selectedIds = new ArrayList<>(); + } + + switch (entityGroup.getType()) { + case CRITERIA_OCCURRENCE: + entityFilters.add( + buildPrimaryWithCriteriaFilter( + underlay, + (CriteriaOccurrence) entityGroup, + selectedIds, + criteriaSelectionData, + modifiersSelectionData)); + break; + case GROUP_ITEMS: + entityFilters.add( + buildGroupItemsFilter( + underlay, (GroupItems) entityGroup, selectedIds, modifiersSelectionData)); + break; + default: + throw new SystemException("Unsupported entity group type: " + entityGroup.getType()); + } + }); + + return entityFilters.size() == 1 + ? entityFilters.get(0) + : new BooleanAndOrFilter(BooleanAndOrFilter.LogicalOperator.OR, entityFilters); + } + + @Override + public List buildForDataFeature( + Underlay underlay, List selectionData) { + String criteriaSelectionData = selectionData.get(0).getPluginData(); + + Map> selectedIdsPerEntityGroup = + selectedIdsPerEntityGroup(underlay, criteriaSelectionData); + List modifiersSelectionData = selectionData.subList(1, selectionData.size()); + + List selectedEntityGroups = + selectedEntityGroups(underlay, selectedIdsPerEntityGroup); + + if (selectedIdsPerEntityGroup.isEmpty() && modifiersSelectionData.isEmpty()) { + // Empty selection data = output all occurrence entities with null filters. + // Use the list of all possible entity groups in the config. + Set outputEntities = new HashSet<>(); + selectedEntityGroups.forEach( + entityGroup -> { + switch (entityGroup.getType()) { + case CRITERIA_OCCURRENCE: + CriteriaOccurrence criteriaOccurrence = (CriteriaOccurrence) entityGroup; + outputEntities.addAll(criteriaOccurrence.getOccurrenceEntities()); + break; + case GROUP_ITEMS: + GroupItems groupItems = (GroupItems) entityGroup; + outputEntities.add( + groupItems.getItemsEntity().isPrimary() + ? groupItems.getGroupEntity() + : groupItems.getItemsEntity()); + break; + default: + throw new SystemException( + "Unsupported entity group type: " + entityGroup.getType()); + } + }); + return outputEntities.stream().map(EntityOutput::unfiltered).collect(Collectors.toList()); + } else { + // Check that there are no group by modifiers. + Optional> + groupByModifierConfigAndData = + GroupByCountSchemaUtils.getModifier(criteriaSelector, modifiersSelectionData); + if (groupByModifierConfigAndData.isPresent()) { + throw new InvalidQueryException("Group by modifiers are not supported for data features"); + } + + // We want to build filters per entity group, not per selected id. + ValueDataOuterClass.ValueData valueData = valueData(criteriaSelectionData); + + Map> filtersPerEntity = new HashMap<>(); + selectedEntityGroups.forEach( + entityGroup -> { + Map> filtersForSingleEntityGroup; + List selectedIds = selectedIdsPerEntityGroup.get(entityGroup); + if (selectedIds == null) { + selectedIds = new ArrayList<>(); + } + + switch (entityGroup.getType()) { + case CRITERIA_OCCURRENCE: + CriteriaOccurrence criteriaOccurrence = (CriteriaOccurrence) entityGroup; + filtersForSingleEntityGroup = + EntityGroupFilterUtils.addOccurrenceFiltersForDataFeature( + underlay, criteriaOccurrence, selectedIds); + EntityGroupFilterUtils.buildAllModifierFilters( + underlay, + criteriaOccurrence.getOccurrenceEntities(), + criteriaSelector, + valueData, + modifiersSelectionData, + filtersForSingleEntityGroup); + break; + case GROUP_ITEMS: + GroupItems groupItems = (GroupItems) entityGroup; + Entity notPrimaryEntity = + groupItems.getGroupEntity().isPrimary() + ? groupItems.getItemsEntity() + : groupItems.getGroupEntity(); + + filtersForSingleEntityGroup = new HashMap<>(); + EntityFilter idSubFilter = + EntityGroupFilterUtils.buildIdSubFilter( + underlay, notPrimaryEntity, selectedIds); + if (idSubFilter == null) { + filtersForSingleEntityGroup.put(notPrimaryEntity, new ArrayList<>()); + } else { + filtersForSingleEntityGroup.put( + notPrimaryEntity, new ArrayList<>(List.of(idSubFilter))); + } + EntityGroupFilterUtils.buildAllModifierFilters( + underlay, + List.of(notPrimaryEntity), + criteriaSelector, + valueData, + modifiersSelectionData, + filtersForSingleEntityGroup); + break; + default: + throw new SystemException( + "Unsupported entity group type: " + entityGroup.getType()); + } + + List entityOutputsForSingleEntityGroup = + EntityGroupFilterUtils.mergeFiltersForDataFeature( + filtersForSingleEntityGroup, BooleanAndOrFilter.LogicalOperator.AND); + entityOutputsForSingleEntityGroup.forEach( + entityOutput -> { + List filters = + filtersPerEntity.getOrDefault(entityOutput.getEntity(), new ArrayList<>()); + if (entityOutput.hasDataFeatureFilter()) { + filters.add(entityOutput.getDataFeatureFilter()); + } else { + filters = new ArrayList<>(); + } + filtersPerEntity.put(entityOutput.getEntity(), filters); + }); + }); + + // If there are multiple filters for a single entity, OR them together. + return EntityGroupFilterUtils.mergeFiltersForDataFeature( + filtersPerEntity, BooleanAndOrFilter.LogicalOperator.OR); + } + } + + private Map> selectedIdsPerEntityGroup( + Underlay underlay, String serializedSelectionData) { + Map> selectedIdsPerEntityGroup = new HashMap<>(); + Map selectedIdsAndEntityGroups = + selectedIdsAndEntityGroups(serializedSelectionData); + selectedIdsAndEntityGroups.forEach( + (key, entityGroupId) -> { + EntityGroup entityGroup = underlay.getEntityGroup(entityGroupId); + List selectedIds = + selectedIdsPerEntityGroup.containsKey(entityGroup) + ? selectedIdsPerEntityGroup.get(entityGroup) + : new ArrayList<>(); + selectedIds.add(key); + selectedIdsPerEntityGroup.put(entityGroup, selectedIds); + }); + + // Sort selected IDs so they're consistent for tests rather than returning them in the original + // selection order. + selectedIdsPerEntityGroup.forEach( + (entityGroup, selectedIds) -> { + Collections.sort(selectedIds); + }); + return selectedIdsPerEntityGroup; + } + + // Returns a list of the union of entity groups covered by the selected items or all configured + // entity groups if no items are selected. + private List selectedEntityGroups( + Underlay underlay, Map> selectedIdsPerEntityGroup) { + List selectedEntityGroups; + if (!selectedIdsPerEntityGroup.isEmpty()) { + selectedEntityGroups = new ArrayList<>(selectedIdsPerEntityGroup.keySet()); + } else { + selectedEntityGroups = + entityGroupIds().stream() + .map( + entityGroupId -> { + return underlay.getEntityGroup(entityGroupId); + }) + .collect(Collectors.toList()); + } + + return selectedEntityGroups.stream() + .sorted(Comparator.comparing(EntityGroup::getName)) + .collect(Collectors.toList()); + } + + private EntityFilter buildPrimaryWithCriteriaFilter( + Underlay underlay, + CriteriaOccurrence criteriaOccurrence, + List selectedIds, + String serializedSelectionData, + List modifiersSelectionData) { + // Build the criteria sub-filter. + EntityFilter criteriaSubFilter = + EntityGroupFilterUtils.buildIdSubFilter( + underlay, criteriaOccurrence.getCriteriaEntity(), selectedIds); + + // Build the attribute modifier filters. + Map> subFiltersPerOccurrenceEntity = + EntityGroupFilterUtils.buildAttributeModifierFilters( + underlay, + criteriaSelector, + modifiersSelectionData, + criteriaOccurrence.getOccurrenceEntities()); + + // Build the instance-level modifier filters. + ValueDataOuterClass.ValueData valueData = valueData(serializedSelectionData); + if (valueData != null + && !IGNORED_ATTRIBUTE_NAME_UI_USE_ONLY.equalsIgnoreCase(valueData.getAttribute())) { + if (criteriaOccurrence.getOccurrenceEntities().size() > 1) { + throw new InvalidQueryException( + "Instance-level modifiers are not supported for entity groups with multiple occurrence entities: " + + criteriaOccurrence.getName()); + } + Entity occurrenceEntity = criteriaOccurrence.getOccurrenceEntities().get(0); + + EntityFilter attrFilter = + AttributeSchemaUtils.buildForEntity( + underlay, + occurrenceEntity, + occurrenceEntity.getAttribute(valueData.getAttribute()), + valueData); + List subFilters = + subFiltersPerOccurrenceEntity.containsKey(occurrenceEntity) + ? subFiltersPerOccurrenceEntity.get(occurrenceEntity) + : new ArrayList<>(); + subFilters.add(attrFilter); + subFiltersPerOccurrenceEntity.put(occurrenceEntity, subFilters); + } + + Optional> + groupByModifierConfigAndData = + GroupByCountSchemaUtils.getModifier(criteriaSelector, modifiersSelectionData); + if (groupByModifierConfigAndData.isEmpty() + || groupByModifierConfigAndData.get().getRight() == null) { + return new PrimaryWithCriteriaFilter( + underlay, + criteriaOccurrence, + criteriaSubFilter, + subFiltersPerOccurrenceEntity, + null, + null, + null); + } + + // Build the group by filter information. + Map> groupByAttributesPerOccurrenceEntity = + GroupByCountSchemaUtils.getGroupByAttributesPerOccurrenceEntity( + underlay, groupByModifierConfigAndData, criteriaOccurrence.getOccurrenceEntities()); + DTUnhintedValue.UnhintedValue groupByModifierData = + groupByModifierConfigAndData.get().getRight(); + return new PrimaryWithCriteriaFilter( + underlay, + criteriaOccurrence, + criteriaSubFilter, + subFiltersPerOccurrenceEntity, + groupByAttributesPerOccurrenceEntity, + GroupByCountSchemaUtils.toBinaryOperator(groupByModifierData.getOperator()), + (int) groupByModifierData.getMin()); + } + + private EntityFilter buildGroupItemsFilter( + Underlay underlay, + GroupItems groupItems, + List selectedIds, + List modifiersSelectionData) { + Entity notPrimaryEntity = + groupItems.getGroupEntity().isPrimary() + ? groupItems.getItemsEntity() + : groupItems.getGroupEntity(); + + // Build the sub-filters on the non-primary entity. + List idFilterNonPrimaryEntity = new ArrayList<>(); + if (!selectedIds.isEmpty()) { + idFilterNonPrimaryEntity.add( + EntityGroupFilterUtils.buildIdSubFilter(underlay, notPrimaryEntity, selectedIds)); + } + return EntityGroupFilterUtils.buildGroupItemsFilter( + underlay, criteriaSelector, groupItems, idFilterNonPrimaryEntity, modifiersSelectionData); + } + + protected abstract List entityGroupIds(); + + protected abstract Map selectedIdsAndEntityGroups( + String serializedSelectionData); + + protected abstract ValueDataOuterClass.ValueData valueData(String serializedSelectionData); +} diff --git a/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/SurveyFilterBuilder.java b/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/SurveyFilterBuilder.java new file mode 100644 index 000000000..b7788cd5f --- /dev/null +++ b/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/SurveyFilterBuilder.java @@ -0,0 +1,62 @@ +package bio.terra.tanagra.filterbuilder.impl.core; + +import static bio.terra.tanagra.utils.ProtobufUtils.deserializeFromJsonOrProtoBytes; + +import bio.terra.tanagra.api.shared.Literal; +import bio.terra.tanagra.proto.criteriaselector.ValueDataOuterClass; +import bio.terra.tanagra.proto.criteriaselector.configschema.CFSurvey; +import bio.terra.tanagra.proto.criteriaselector.dataschema.DTSurvey; +import bio.terra.tanagra.underlay.uiplugin.CriteriaSelector; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +public class SurveyFilterBuilder extends EntityGroupFilterBuilderBase { + public SurveyFilterBuilder(CriteriaSelector criteriaSelector) { + super(criteriaSelector); + } + + @Override + public CFSurvey.Survey deserializeConfig() { + return deserializeFromJsonOrProtoBytes( + criteriaSelector.getPluginConfig(), CFSurvey.Survey.newBuilder()) + .build(); + } + + @Override + public DTSurvey.Survey deserializeData(String serialized) { + return (serialized == null || serialized.isEmpty()) + ? null + : deserializeFromJsonOrProtoBytes(serialized, DTSurvey.Survey.newBuilder()).build(); + } + + @Override + protected List entityGroupIds() { + return deserializeConfig().getEntityGroupsList().stream() + .map( + entityGroup -> { + return entityGroup.getId(); + }) + .collect(Collectors.toList()); + } + + @Override + protected Map selectedIdsAndEntityGroups(String serializedSelectionData) { + Map idsAndEntityGroups = new HashMap<>(); + DTSurvey.Survey selectionData = deserializeData(serializedSelectionData); + for (DTSurvey.Survey.Selection selectedId : selectionData.getSelectedList()) { + if (selectedId.hasKey()) { + idsAndEntityGroups.put( + Literal.forInt64(selectedId.getKey().getInt64Key()), selectedId.getEntityGroup()); + } + } + return idsAndEntityGroups; + } + + @Override + protected ValueDataOuterClass.ValueData valueData(String serializedSelectionData) { + DTSurvey.Survey selectionData = deserializeData(serializedSelectionData); + return selectionData.hasValueData() ? selectionData.getValueData() : null; + } +} diff --git a/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/utils/EntityGroupFilterUtils.java b/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/utils/EntityGroupFilterUtils.java index 1403e78f2..d2de7b56a 100644 --- a/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/utils/EntityGroupFilterUtils.java +++ b/underlay/src/main/java/bio/terra/tanagra/filterbuilder/impl/core/utils/EntityGroupFilterUtils.java @@ -14,10 +14,10 @@ import bio.terra.tanagra.api.shared.NaryOperator; import bio.terra.tanagra.exception.InvalidQueryException; import bio.terra.tanagra.filterbuilder.EntityOutput; +import bio.terra.tanagra.proto.criteriaselector.ValueDataOuterClass; import bio.terra.tanagra.proto.criteriaselector.configschema.CFAttribute; import bio.terra.tanagra.proto.criteriaselector.configschema.CFUnhintedValue; import bio.terra.tanagra.proto.criteriaselector.dataschema.DTAttribute; -import bio.terra.tanagra.proto.criteriaselector.dataschema.DTEntityGroup; import bio.terra.tanagra.proto.criteriaselector.dataschema.DTUnhintedValue; import bio.terra.tanagra.underlay.Underlay; import bio.terra.tanagra.underlay.entitymodel.Attribute; @@ -189,7 +189,7 @@ public static void buildAllModifierFilters( Underlay underlay, List occurrenceEntities, CriteriaSelector criteriaSelector, - DTEntityGroup.EntityGroup entityGroupSelectionData, + ValueDataOuterClass.ValueData valueData, List modifiersSelectionData, Map> filtersPerEntity) { // Build the attribute modifier filters. @@ -206,9 +206,8 @@ public static void buildAllModifierFilters( }); // Build the instance-level modifier filters. - if (entityGroupSelectionData.hasValueData() - && !IGNORED_ATTRIBUTE_NAME_UI_USE_ONLY.equalsIgnoreCase( - entityGroupSelectionData.getValueData().getAttribute())) { + if (valueData != null + && !IGNORED_ATTRIBUTE_NAME_UI_USE_ONLY.equalsIgnoreCase(valueData.getAttribute())) { if (occurrenceEntities.size() > 1) { throw new InvalidQueryException( "Instance-level modifiers are not supported for entity groups with multiple occurrence entities"); @@ -219,8 +218,8 @@ public static void buildAllModifierFilters( AttributeSchemaUtils.buildForEntity( underlay, occurrenceEntity, - occurrenceEntity.getAttribute(entityGroupSelectionData.getValueData().getAttribute()), - entityGroupSelectionData.getValueData()); + occurrenceEntity.getAttribute(valueData.getAttribute()), + valueData); List subFilters = filtersPerEntity.containsKey(occurrenceEntity) ? filtersPerEntity.get(occurrenceEntity) diff --git a/underlay/src/main/java/bio/terra/tanagra/underlay/serialization/SZCorePlugin.java b/underlay/src/main/java/bio/terra/tanagra/underlay/serialization/SZCorePlugin.java index 110c973ae..d762c2b05 100644 --- a/underlay/src/main/java/bio/terra/tanagra/underlay/serialization/SZCorePlugin.java +++ b/underlay/src/main/java/bio/terra/tanagra/underlay/serialization/SZCorePlugin.java @@ -25,7 +25,9 @@ public enum SZCorePlugin { @AnnotatedField( name = "SZCorePlugin.OUTPUT_UNFILTERED", markdown = "Use `plugin: \"outputUnfiltered\"`.") - OUTPUT_UNFILTERED("outputUnfiltered"); + OUTPUT_UNFILTERED("outputUnfiltered"), + @AnnotatedField(name = "SZCorePlugin.SURVEY", markdown = "Use `plugin: \"survey\"`.") + SURVEY("survey"); private final String idInConfig; SZCorePlugin(String idInConfig) { diff --git a/underlay/src/main/proto/column.proto b/underlay/src/main/proto/column.proto new file mode 100644 index 000000000..76bc37c39 --- /dev/null +++ b/underlay/src/main/proto/column.proto @@ -0,0 +1,29 @@ +syntax = "proto3"; + +package tanagra; + +option go_package = "github.com/DataBiosphere/tanagra/tanagrapb"; + +// Defines a column in the UI. +message Column { + // A unique key for the column. By default, used to look up attributes in + // the displayed data. + string key = 1; + + oneof width { + // Passed directly to the style of the column. "100%" can be used to take + // up space remaining after laying out fixed columns. + string width_string = 2; + // Units used by the UI library to standardize dimensions. + double width_double = 3; + } + + // The visible title of the column. + string title = 4; + + // Whether the column supports sorting. + bool sortable = 5; + + // Whether the column supports filtering. + bool filterable = 6; +} diff --git a/underlay/src/main/proto/criteriaselector/configschema/entity_group.proto b/underlay/src/main/proto/criteriaselector/configschema/entity_group.proto index bfc76745a..c12bbd43b 100644 --- a/underlay/src/main/proto/criteriaselector/configschema/entity_group.proto +++ b/underlay/src/main/proto/criteriaselector/configschema/entity_group.proto @@ -6,38 +6,15 @@ option go_package = "github.com/DataBiosphere/tanagra/criteriaselector/configsch option java_package = "bio.terra.tanagra.proto.criteriaselector.configschema"; option java_outer_classname = "CFEntityGroup"; -import "sort_order.proto"; +import "column.proto"; import "criteriaselector/value_config.proto"; +import "sort_order.proto"; // A criteria based on one or more entity groups. This allows the selection of // primary entities which are related to one or more of another entity which // match certain characteristics (e.g. people related to condition_occurrences // which have condition_name of "Diabetes"). message EntityGroup { - // Defines a column in the UI. - message Column { - // A unique key for the column. By default, used to look up attributes in - // the displayed data. - string key = 1; - - oneof width { - // Passed directly to the style of the column. "100%" can be used to take - // up space remaining after laying out fixed columns. - string width_string = 2; - // Units used by the UI library to standardize dimensions. - double width_double = 3; - } - - // The visible title of the column. - string title = 4; - - // Whether the column supports sorting. - bool sortable = 5; - - // Whether the column supports filtering. - bool filterable = 6; - } - // Columns displayed in the list view. repeated Column columns = 1; // Columns displayed in the hierarchy view. diff --git a/underlay/src/main/proto/criteriaselector/configschema/survey.proto b/underlay/src/main/proto/criteriaselector/configschema/survey.proto new file mode 100644 index 000000000..9b5b31f93 --- /dev/null +++ b/underlay/src/main/proto/criteriaselector/configschema/survey.proto @@ -0,0 +1,44 @@ +syntax = "proto3"; + +package tanagra.configschema; + +option go_package = "github.com/DataBiosphere/tanagra/criteriaselector/configschemapb"; +option java_package = "bio.terra.tanagra.proto.criteriaselector.configschema"; +option java_outer_classname = "CFSurvey"; + +import "column.proto"; +import "criteriaselector/value_config.proto"; +import "sort_order.proto"; + +message Survey { + // Columns displayed in the list view. + repeated Column columns = 1; + + message EntityGroupConfig { + // The id of the entity group. + string id = 1; + + // The sort order applied to this entity group when displayed in the + // hierarchy view. + SortOrder sort_order = 2; + } + + // Entity groups where the related entity is what is selected (e.g. + // surveyBasics when filtering surveyOccurrence). + repeated EntityGroupConfig entity_groups = 2; + + // Optional configuration of a categorical or numeric value associated with + // the selection (e.g. a numeric answer). Applied to the entire selection + // so generally not compatible with multi_select. Currently only one is + // supported. + repeated ValueConfig value_configs = 3; + + // The sort order to use in the list view, or in hierarchies where no sort + // order has been specified. + SortOrder default_sort = 4; + + // The attribute used to name selections if not the first column. This can be + // used to include extra context with the selected values that's not visible + // in the table view. + optional string nameAttribute = 5; +} diff --git a/underlay/src/main/proto/criteriaselector/dataschema/survey.proto b/underlay/src/main/proto/criteriaselector/dataschema/survey.proto new file mode 100644 index 000000000..ec789141d --- /dev/null +++ b/underlay/src/main/proto/criteriaselector/dataschema/survey.proto @@ -0,0 +1,39 @@ +syntax = "proto3"; + +package tanagra.dataschema; + +option go_package = "github.com/DataBiosphere/tanagra/criteriaselector/dataschemapb"; +option java_package = "bio.terra.tanagra.proto.criteriaselector.dataschema"; +option java_outer_classname = "DTSurvey"; + +import "criteriaselector/key.proto"; +import "criteriaselector/value_data.proto"; + +// Data for an entity group criteria is a list of selected values. +message Survey { + message Selection { + // The key of the selected value, which references a related entity (e.g. + // surveyBasics when filtering surveyOccurrence). + Key key = 1; + + // The visible name for the selection. This is stored to avoid extra lookups + // when rendering. + string name = 2; + + // The entity group is stored to differentiate between them when multiple + // are configured within a single criteria. + string entityGroup = 3; + + // If the selected item is an answer, the key of the question it belongs to. + Key question_key = 4; + + // If the selected item is an answer, the visible name of the question it + // belongs to. + string question_name = 5; + } + repeated Selection selected = 1; + + // Data for an additional categorical or numeric value associated with the + // selection (e.g. a numeric answer). + ValueData value_data = 2; +} diff --git a/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForCriteriaOccurrenceTest.java b/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForCriteriaOccurrenceTest.java index 26c70358b..13a9d0ae5 100644 --- a/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForCriteriaOccurrenceTest.java +++ b/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForCriteriaOccurrenceTest.java @@ -123,7 +123,7 @@ void criteriaOnlyCohortFilter() { underlay, underlay.getEntity("condition"), underlay.getEntity("condition").getHierarchy(Hierarchy.DEFAULT_NAME), - List.of(Literal.forInt64(201_826L), Literal.forInt64(201_254L))); + List.of(Literal.forInt64(201_254L), Literal.forInt64(201_826L))); expectedCohortFilter = new PrimaryWithCriteriaFilter( underlay, @@ -871,7 +871,7 @@ void criteriaOnlySingleOccurrenceDataFeatureFilter() { underlay, underlay.getEntity("condition"), underlay.getEntity("condition").getHierarchy(Hierarchy.DEFAULT_NAME), - List.of(Literal.forInt64(201_826L), Literal.forInt64(201_254L))); + List.of(Literal.forInt64(201_254L), Literal.forInt64(201_826L))); expectedDataFeatureFilter = new OccurrenceForPrimaryFilter( underlay, diff --git a/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForGroupTest.java b/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForGroupTest.java index b4e491b73..5ee7114c9 100644 --- a/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForGroupTest.java +++ b/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForGroupTest.java @@ -116,7 +116,7 @@ void criteriaOnlyCohortFilter() { underlay, underlay.getEntity("genotyping"), underlay.getEntity("genotyping").getHierarchy(Hierarchy.DEFAULT_NAME), - List.of(Literal.forInt64(30L), Literal.forInt64(3L))); + List.of(Literal.forInt64(3L), Literal.forInt64(30L))); expectedCohortFilter = new ItemInGroupFilter( underlay, @@ -638,7 +638,7 @@ void criteriaOnlyDataFeatureFilter() { underlay, underlay.getEntity("genotyping"), underlay.getEntity("genotyping").getHierarchy(Hierarchy.DEFAULT_NAME), - List.of(Literal.forInt64(30L), Literal.forInt64(3L))); + List.of(Literal.forInt64(3L), Literal.forInt64(30L))); expectedDataFeatureOutput = EntityOutput.filtered(underlay.getEntity("genotyping"), expectedDataFeatureFilter); assertEquals(expectedDataFeatureOutput, dataFeatureOutputs.get(0)); diff --git a/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForItemsTest.java b/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForItemsTest.java index 66cc45bf4..430c3e88c 100644 --- a/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForItemsTest.java +++ b/underlay/src/test/java/bio/terra/tanagra/filterbuilder/EntityGroupFilterBuilderForItemsTest.java @@ -52,7 +52,13 @@ void criteriaWithAttrModifiersCohortFilter() { true, SZCorePlugin.ATTRIBUTE.getIdInConfig(), serializeToJson(systolicConfig)); - CFEntityGroup.EntityGroup bloodPressureConfig = CFEntityGroup.EntityGroup.newBuilder().build(); + CFEntityGroup.EntityGroup bloodPressureConfig = + CFEntityGroup.EntityGroup.newBuilder() + .addClassificationEntityGroups( + CFEntityGroup.EntityGroup.EntityGroupConfig.newBuilder() + .setId("bloodPressurePerson") + .build()) + .build(); CriteriaSelector criteriaSelector = new CriteriaSelector( "bloodPressure", @@ -72,13 +78,7 @@ void criteriaWithAttrModifiersCohortFilter() { .build(); SelectionData systolicSelectionData = new SelectionData("systolic", serializeToJson(systolicData)); - DTEntityGroup.EntityGroup entityGroupData = - DTEntityGroup.EntityGroup.newBuilder() - .addSelected( - DTEntityGroup.EntityGroup.Selection.newBuilder() - .setEntityGroup("bloodPressurePerson") - .build()) - .build(); + DTEntityGroup.EntityGroup entityGroupData = DTEntityGroup.EntityGroup.newBuilder().build(); SelectionData entityGroupSelectionData = new SelectionData("bloodPressure", serializeToJson(entityGroupData)); EntityFilter cohortFilter = @@ -117,7 +117,13 @@ void criteriaWithGroupByModifierCohortFilter() { false, SZCorePlugin.UNHINTED_VALUE.getIdInConfig(), serializeToJson(groupByConfig)); - CFEntityGroup.EntityGroup bloodPressureConfig = CFEntityGroup.EntityGroup.newBuilder().build(); + CFEntityGroup.EntityGroup bloodPressureConfig = + CFEntityGroup.EntityGroup.newBuilder() + .addClassificationEntityGroups( + CFEntityGroup.EntityGroup.EntityGroupConfig.newBuilder() + .setId("bloodPressurePerson") + .build()) + .build(); CriteriaSelector criteriaSelector = new CriteriaSelector( "bloodPressure", @@ -139,13 +145,7 @@ void criteriaWithGroupByModifierCohortFilter() { .build(); SelectionData groupBySelectionData = new SelectionData("group_by_count", serializeToJson(groupByData)); - DTEntityGroup.EntityGroup entityGroupData = - DTEntityGroup.EntityGroup.newBuilder() - .addSelected( - DTEntityGroup.EntityGroup.Selection.newBuilder() - .setEntityGroup("bloodPressurePerson") - .build()) - .build(); + DTEntityGroup.EntityGroup entityGroupData = DTEntityGroup.EntityGroup.newBuilder().build(); SelectionData entityGroupSelectionData = new SelectionData("bloodPressure", serializeToJson(entityGroupData)); EntityFilter cohortFilter = @@ -185,7 +185,13 @@ void criteriaWithAttrAndGroupByModifiersCohortFilter() { false, SZCorePlugin.UNHINTED_VALUE.getIdInConfig(), serializeToJson(groupByConfig)); - CFEntityGroup.EntityGroup bloodPressureConfig = CFEntityGroup.EntityGroup.newBuilder().build(); + CFEntityGroup.EntityGroup bloodPressureConfig = + CFEntityGroup.EntityGroup.newBuilder() + .addClassificationEntityGroups( + CFEntityGroup.EntityGroup.EntityGroupConfig.newBuilder() + .setId("bloodPressurePerson") + .build()) + .build(); CriteriaSelector criteriaSelector = new CriteriaSelector( "bloodPressure", @@ -213,13 +219,7 @@ void criteriaWithAttrAndGroupByModifiersCohortFilter() { .build(); SelectionData groupBySelectionData = new SelectionData("group_by_count", serializeToJson(groupByData)); - DTEntityGroup.EntityGroup entityGroupData = - DTEntityGroup.EntityGroup.newBuilder() - .addSelected( - DTEntityGroup.EntityGroup.Selection.newBuilder() - .setEntityGroup("bloodPressurePerson") - .build()) - .build(); + DTEntityGroup.EntityGroup entityGroupData = DTEntityGroup.EntityGroup.newBuilder().build(); SelectionData entityGroupSelectionData = new SelectionData("bloodPressure", serializeToJson(entityGroupData)); EntityFilter cohortFilter = @@ -247,7 +247,13 @@ void criteriaWithAttrAndGroupByModifiersCohortFilter() { @Test void emptyCriteriaCohortFilter() { - CFEntityGroup.EntityGroup bloodPressureConfig = CFEntityGroup.EntityGroup.newBuilder().build(); + CFEntityGroup.EntityGroup bloodPressureConfig = + CFEntityGroup.EntityGroup.newBuilder() + .addClassificationEntityGroups( + CFEntityGroup.EntityGroup.EntityGroupConfig.newBuilder() + .setId("bloodPressurePerson") + .build()) + .build(); CriteriaSelector criteriaSelector = new CriteriaSelector( "bloodPressure", @@ -287,7 +293,13 @@ void emptyAttrModifierCohortFilter() { true, SZCorePlugin.ATTRIBUTE.getIdInConfig(), serializeToJson(systolicConfig)); - CFEntityGroup.EntityGroup bloodPressureConfig = CFEntityGroup.EntityGroup.newBuilder().build(); + CFEntityGroup.EntityGroup bloodPressureConfig = + CFEntityGroup.EntityGroup.newBuilder() + .addClassificationEntityGroups( + CFEntityGroup.EntityGroup.EntityGroupConfig.newBuilder() + .setId("bloodPressurePerson") + .build()) + .build(); CriteriaSelector criteriaSelector = new CriteriaSelector( "bloodPressure", @@ -300,13 +312,7 @@ void emptyAttrModifierCohortFilter() { List.of(systolicModifier)); EntityGroupFilterBuilder filterBuilder = new EntityGroupFilterBuilder(criteriaSelector); - DTEntityGroup.EntityGroup entityGroupData = - DTEntityGroup.EntityGroup.newBuilder() - .addSelected( - DTEntityGroup.EntityGroup.Selection.newBuilder() - .setEntityGroup("bloodPressurePerson") - .build()) - .build(); + DTEntityGroup.EntityGroup entityGroupData = DTEntityGroup.EntityGroup.newBuilder().build(); SelectionData entityGroupSelectionData = new SelectionData("bloodPressure", serializeToJson(entityGroupData)); EntityFilter expectedCohortFilter = @@ -349,7 +355,13 @@ void emptyGroupByModifierCohortFilter() { false, SZCorePlugin.UNHINTED_VALUE.getIdInConfig(), serializeToJson(groupByConfig)); - CFEntityGroup.EntityGroup bloodPressureConfig = CFEntityGroup.EntityGroup.newBuilder().build(); + CFEntityGroup.EntityGroup bloodPressureConfig = + CFEntityGroup.EntityGroup.newBuilder() + .addClassificationEntityGroups( + CFEntityGroup.EntityGroup.EntityGroupConfig.newBuilder() + .setId("bloodPressurePerson") + .build()) + .build(); CriteriaSelector criteriaSelector = new CriteriaSelector( "bloodPressure", @@ -362,13 +374,7 @@ void emptyGroupByModifierCohortFilter() { List.of(groupByModifier)); EntityGroupFilterBuilder filterBuilder = new EntityGroupFilterBuilder(criteriaSelector); - DTEntityGroup.EntityGroup entityGroupData = - DTEntityGroup.EntityGroup.newBuilder() - .addSelected( - DTEntityGroup.EntityGroup.Selection.newBuilder() - .setEntityGroup("bloodPressurePerson") - .build()) - .build(); + DTEntityGroup.EntityGroup entityGroupData = DTEntityGroup.EntityGroup.newBuilder().build(); SelectionData entityGroupSelectionData = new SelectionData("bloodPressure", serializeToJson(entityGroupData)); EntityFilter expectedCohortFilter = @@ -399,7 +405,13 @@ void emptyGroupByModifierCohortFilter() { @Test void criteriaOnlyDataFeatureFilter() { - CFEntityGroup.EntityGroup config = CFEntityGroup.EntityGroup.newBuilder().build(); + CFEntityGroup.EntityGroup bloodPressureConfig = + CFEntityGroup.EntityGroup.newBuilder() + .addClassificationEntityGroups( + CFEntityGroup.EntityGroup.EntityGroupConfig.newBuilder() + .setId("bloodPressurePerson") + .build()) + .build(); CriteriaSelector criteriaSelector = new CriteriaSelector( "bloodPressure", @@ -408,18 +420,12 @@ void criteriaOnlyDataFeatureFilter() { true, "core.EntityGroupFilterBuilder", SZCorePlugin.ENTITY_GROUP.getIdInConfig(), - serializeToJson(config), + serializeToJson(bloodPressureConfig), List.of()); EntityGroupFilterBuilder filterBuilder = new EntityGroupFilterBuilder(criteriaSelector); // No ids. - DTEntityGroup.EntityGroup data = - DTEntityGroup.EntityGroup.newBuilder() - .addSelected( - DTEntityGroup.EntityGroup.Selection.newBuilder() - .setEntityGroup("bloodPressurePerson") - .build()) - .build(); + DTEntityGroup.EntityGroup data = DTEntityGroup.EntityGroup.newBuilder().build(); SelectionData selectionData = new SelectionData("bloodPressure", serializeToJson(data)); List dataFeatureOutputs = filterBuilder.buildForDataFeature(underlay, List.of(selectionData)); @@ -439,7 +445,13 @@ void criteriaWithAttrModifierDataFeatureFilter() { true, SZCorePlugin.ATTRIBUTE.getIdInConfig(), serializeToJson(systolicConfig)); - CFEntityGroup.EntityGroup bloodPressureConfig = CFEntityGroup.EntityGroup.newBuilder().build(); + CFEntityGroup.EntityGroup bloodPressureConfig = + CFEntityGroup.EntityGroup.newBuilder() + .addClassificationEntityGroups( + CFEntityGroup.EntityGroup.EntityGroupConfig.newBuilder() + .setId("bloodPressurePerson") + .build()) + .build(); CriteriaSelector criteriaSelector = new CriteriaSelector( "bloodPressure", @@ -459,13 +471,7 @@ void criteriaWithAttrModifierDataFeatureFilter() { .build(); SelectionData systolicSelectionData = new SelectionData("systolic", serializeToJson(systolicData)); - DTEntityGroup.EntityGroup entityGroupData = - DTEntityGroup.EntityGroup.newBuilder() - .addSelected( - DTEntityGroup.EntityGroup.Selection.newBuilder() - .setEntityGroup("bloodPressurePerson") - .build()) - .build(); + DTEntityGroup.EntityGroup entityGroupData = DTEntityGroup.EntityGroup.newBuilder().build(); SelectionData entityGroupSelectionData = new SelectionData("bloodPressure", serializeToJson(entityGroupData)); List dataFeatureOutputs =