Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Changes to state manager and redis datastore schema #1489

Merged
merged 34 commits into from
Jun 13, 2024
Merged

Changes to state manager and redis datastore schema #1489

merged 34 commits into from
Jun 13, 2024

Conversation

jleaniz
Copy link
Collaborator

@jleaniz jleaniz commented May 31, 2024 •
edited
Loading

Description of the change

This pull request makes changes to the schema used to store Turbinia metadata in Redis. We are introducing a new key for Turbinia requests and changing TurbiniaTask keys to redis hashes instead of strings for optimization purposes.

Minor changes to the web UI.

Note: This is a breaking change. Redis databases from previous versions will not be compatible.

Applicable issues

Additional information

Checklist

  • All tests were successful.
  • Unit tests added.
  • [] Documentation updated.

Igor8mr and others added 15 commits May 30, 2024 14:34
@Igor8mr @jleaniz
Added the TurbiniaRequest hashed object in Redis
e4b703e
@Igor8mr @jleaniz
Converted TurbiniaTasks to hash objects in Redis
7f04f09
@Igor8mr @jleaniz
Made get_request_data more efficient
57a6964
@Igor8mr @jleaniz
Made get_requests_summary more efficient
e1358a8
@Igor8mr @jleaniz
Fixed some issues in request_status
e40d212
@jleaniz
Remove GCP dependencies (google#1440)
f220654 
* Remove gcp dependencies

* Update dockerfiles

* Update dockerfiles

* Update gcp error reporting

* Updates to formatting

* Add unit test

* Update unit test

* Clean up

* Update unit test

* Update error reporting

* Update file

* Update config template

* Catch exception

* Updates

* fix lint
@jleaniz
Lint fixes
47de4af
@jleaniz
Updates
a345ea4
@jleaniz
Updates
50f9084
@jleaniz
Various updates and fixes
ccb4cdf
@jleaniz
Updates
6961a19
@dependabot
--- (google#1483)
eb070a8 
updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@jleaniz
Updates
fb2edeb
@jleaniz
Update typo
843ff51
@jleaniz
Updates to unit tests
9f9fd0e
@jleaniz jleaniz changed the title Redis hashed requests and tasks Changes to state manager and redis datastore schema May 31, 2024
@jleaniz jleaniz marked this pull request as ready for review May 31, 2024 18:35
@jleaniz jleaniz requested a review from aarontp May 31, 2024 18:35
@jleaniz
Copy link
Collaborator Author

jleaniz commented May 31, 2024

Please ignore changes to files under turbinia/api/client/*

@jleaniz jleaniz self-assigned this May 31, 2024
@dependabot
Bump the pip group across 2 directories with 1 update (google#1487)
2601e17 
Bumps the pip group with 1 update in the / directory: [requests](https://github.com/psf/requests).
Bumps the pip group with 1 update in the /turbinia/api/cli directory: [requests](https://github.com/psf/requests).


Updates `requests` from 2.32.0 to 2.32.3
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.32.0...v2.32.3)

Updates `requests` from 2.32.0 to 2.32.3
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.32.0...v2.32.3)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@jleaniz jleaniz mentioned this pull request May 31, 2024
Closed
3 tasks
wajihyassine
wajihyassine reviewed May 31, 2024
View reviewed changes
Copy link
Member

@wajihyassine wajihyassine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from web ui side per our side chats around it. As discussed, will leave the truncate of statuses as is to fix issues with table being too long and look at a later time for a way to size the virtual scroll to the table width.

@aarontp
Copy link
Member

aarontp commented Jun 4, 2024

Quick update: I started reading through this today and will work on finishing the review tomorrow!

aarontp
aarontp reviewed Jun 5, 2024
View reviewed changes
Copy link
Member

@aarontp aarontp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, just some small nits/comments.

turbinia/message.py Show resolved Hide resolved
turbinia/task_manager.py Outdated Show resolved Hide resolved
turbinia/turbiniactl.py Outdated Show resolved Hide resolved
turbinia/workers/__init__.py Outdated Show resolved Hide resolved
turbinia/workers/__init__.py Outdated Show resolved Hide resolved
turbinia/state_manager.py Outdated Show resolved Hide resolved
turbinia/api/models/request_status.py Outdated Show resolved Hide resolved
turbinia/state_manager.py Outdated Show resolved Hide resolved
turbinia/state_manager.py Outdated Show resolved Hide resolved
turbinia/state_manager.py Show resolved Hide resolved
aarontp and others added 8 commits June 7, 2024 10:00
@aarontp
Set celery task limits and fix worker timeout (google#1486)
7678a63 
* Set celery task limits and fix worker timeout

* Remove solo pool and change concurrency=1

* Add soft/hard limit buffers

* Fix server timeout

* Late import for psutil

* small fixes, tests

* fix run tests

* test format string

* Fix process_result test

* fix execute test

* revert unnecessary config changes

* Handle soft timeout exception in execute()

* update timeout message

* Yaaaaaaaapf
@jleaniz
Review fixes
cde6842
@jleaniz
Updates and yapf fix
5d5f895
@jleaniz
Minor updates
baae935
@jleaniz
Update docstrings
09e6a01
@jleaniz
Updates to evidence_size
589eb4e
@jleaniz
Updates
b5ddefb
@jleaniz
Change log level for message
753afe2
@jleaniz jleaniz requested a review from aarontp June 8, 2024 01:32
aarontp
aarontp approved these changes Jun 12, 2024
View reviewed changes
Copy link
Member

@aarontp aarontp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a couple nits in comments and a question.

turbinia/message.py Outdated Show resolved Hide resolved
turbinia/message.py Outdated Show resolved Hide resolved
turbinia/state_manager.py Outdated Show resolved Hide resolved
@jleaniz jleaniz merged commit 0588367 into google:redis-hashed-requests-and-tasks Jun 13, 2024
5 checks passed
jleaniz added a commit that referenced this pull request Jun 13, 2024
@jleaniz @Igor8mr
@dependabot @aarontp
Changes to state manager and redis datastore schema (#1489) (#1494)
c351f9a 
* Added the TurbiniaRequest hashed object in Redis

* Converted TurbiniaTasks to hash objects in Redis

* Made get_request_data more efficient

* Made get_requests_summary more efficient

* Fixed some issues in request_status

* Remove GCP dependencies (#1440)

* Remove gcp dependencies

* Update dockerfiles

* Update dockerfiles

* Update gcp error reporting

* Updates to formatting

* Add unit test

* Update unit test

* Clean up

* Update unit test

* Update error reporting

* Update file

* Update config template

* Catch exception

* Updates

* fix lint

* Lint fixes

* Updates

* Updates

* Various updates and fixes

* Updates

* --- (#1483)

updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
...




* Updates

* Update typo

* Updates to unit tests

* Updates to unit tests and linter fixes

* Update table width UI

* Bump the pip group across 2 directories with 1 update (#1487)

Bumps the pip group with 1 update in the / directory: [requests](https://github.com/psf/requests).
Bumps the pip group with 1 update in the /turbinia/api/cli directory: [requests](https://github.com/psf/requests).


Updates `requests` from 2.32.0 to 2.32.3
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.32.0...v2.32.3)

Updates `requests` from 2.32.0 to 2.32.3
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.32.0...v2.32.3)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
...




* Fix small UI bug

* Minor typos/errors

* Set celery task limits and fix worker timeout (#1486)

* Set celery task limits and fix worker timeout

* Remove solo pool and change concurrency=1

* Add soft/hard limit buffers

* Fix server timeout

* Late import for psutil

* small fixes, tests

* fix run tests

* test format string

* Fix process_result test

* fix execute test

* revert unnecessary config changes

* Handle soft timeout exception in execute()

* update timeout message

* Yaaaaaaaapf

* Review fixes

* Updates and yapf fix

* Minor updates

* Update docstrings

* Updates to evidence_size

* Updates

* Change log level for message

* Lint

* bug fixes

* Address review comments

* Fix docstrings

* Minor UI update

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Igor Rodrigues <igormr@nyu.edu>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Aaron Peterson <aaronp@gmail.com>
This was referenced Jun 13, 2024
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wajihyassine wajihyassine wajihyassine left review comments

@aarontp aarontp aarontp approved these changes

Assignees

@jleaniz jleaniz

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jleaniz @aarontp @wajihyassine @Igor8mr