Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add clean commands to indexing readme. #468

Merged
merged 2 commits into from
Jul 7, 2023
Merged

Add clean commands to indexing readme. #468

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cbd535b
Add clean commands to indexing readme.
marikomedlock Jul 7, 2023
7a25a11
Allow UI integration tests to run for all PRs.
marikomedlock Jul 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .github/workflows/ui-integration-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@ on:
- 'docs/**'
pull_request:
branches: ["**"]
paths-ignore:
- 'README.md'
- 'docs/**'
workflow_dispatch:

jobs:
Expand Down
32 changes: 28 additions & 4 deletions docs/INDEXING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,10 @@
+ [Expand Underlay Config](#expand-underlay-config)
+ [Create Index Dataset](#create-index-dataset)
+ [Kickoff Jobs](#kickoff-jobs)
+ [Tips](#tips)
* [Troubleshooting](#troubleshooting)
+ [Concurrency](#concurrency)
+ [Re-Run Jobs](#re-run-jobs)
+ [Run dataflow locally](#run-dataflow-locally)
* [OMOP Example](#omop-example)


Expand Down Expand Up @@ -154,17 +157,38 @@ or entity group:
All the entities in a group should be indexed before the group. The `INDEX_ALL` command ensures this ordering, but keep
this in mind if you're running the jobs for each entity or entity group separately.

#### Concurrency
## Troubleshooting

### Concurrency
By default, the indexing jobs are run concurrently as much as possible. You can force it to run jobs serially by
appending `SERIAL` to the command:
```
./gradlew indexer:index -Dexec.args="INDEX_ALL $HOME/tanagra/service/src/main/resources/config/output/omop.json DRY_RUN SERIAL"
./gradlew indexer:index -Dexec.args="INDEX_ALL $HOME/tanagra/service/src/main/resources/config/output/omop.json NOT_DRY_RUN SERIAL"
```

### Tips
### Re-Run Jobs
Indexing jobs will not overwrite existing index tables. If you want to re-run indexing, either for a single entity/group
or for everything, you need to delete any existing index tables. You can either do that manually or using the clean
commands below. Similar to the indexing commands, the clean commands also respect the dry run flag.

To clean the generated index tables for everything:
```
./gradlew indexer:index -Dexec.args="CLEAN_ALL $HOME/tanagra/service/src/main/resources/config/output/omop.json DRY_RUN"
./gradlew indexer:index -Dexec.args="CLEAN_ALL $HOME/tanagra/service/src/main/resources/config/output/omop.json"
```
or a particular entity:
```
./gradlew indexer:index -Dexec.args="CLEAN_ENTITY $HOME/tanagra/service/src/main/resources/config/output/omop.json person DRY_RUN"
./gradlew indexer:index -Dexec.args="CLEAN_ENTITY $HOME/tanagra/service/src/main/resources/config/output/omop.json person"
```
or a particular entity group:
```
./gradlew indexer:index -Dexec.args="CLEAN_ENTITY_GROUP $HOME/tanagra/service/src/main/resources/config/output/omop.json person DRY_RUN"
./gradlew indexer:index -Dexec.args="CLEAN_ENTITY_GROUP $HOME/tanagra/service/src/main/resources/config/output/omop.json person"
```

#### Run dataflow locally
### Run dataflow locally

While developing a job, running locally is faster. Also, you can use Intellij debugger.
- Add to `BigQueryIndexingJob.buildDataflowPipelineOptions()`:
Expand Down