Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Self-hosted server sync #219

Merged
merged 855 commits into from
Jan 20, 2024
Merged

Self-hosted server sync #219

merged 855 commits into from
Jan 20, 2024

Conversation

tmpfs
Copy link
Collaborator

@tmpfs tmpfs commented Nov 11, 2023

  • Mutex in user storage, prevent writes whilst sync in progress
  • Support multiple remotes (self-hosted / platform backend)
  • Remove StorageProvider trait
  • Remove PatchFile type, just use in-memory patches
  • Make calls to sync_before_apply_change infallible (otherwise it would prevent local writes)
  • Return error of sync_send_events along with value so callers can see if their local changes were applied to a remote, this can be an additional Option<Error> return value
  • Sync across multiple remotes should return a list of errors and continue processing other remotes if an earlier one fails
  • Improve sos check log verbose output as a debugging aid
  • Merge AppPaths into UserPaths
  • Move storage modules into account
  • Use Arc for UserPaths after initialization (then we don't need to keep await in UserStorage - remove paths() and just clone the Arc)
  • Ensure running tests do not pollute the application debug path (some tests are still writing to it!)
  • Add sos check events for a verbose listing of all the events in a folder
  • Move rpc module from sos-sdk to sos-net
  • Handle sync of delegated passphrases as folders are created and removed
  • Exponential backoff for websocket re-connection
  • Shutdown channel in WebSocketHandle to close the connection
  • Identity Gatekeeper should be behind Arc+RwLock
  • Server locks should lock each account and not the entire server state
  • Move LocalProvider to sos_sdk
  • Make EventLogFile generic over the type of event
  • Support AccountEvent and AccountEventLog for account level events (eg: create/delete folders)
  • Remove lifetime from WriteEvent - we have to convert to owned anyhow, just clone it when needed
  • Remove libp2p and peer module
  • Create events subcommand (move from check events) and add support for account and file event logs
  • Initialize account and file logs from state on disc
  • Refactor AccountsList into Account
  • Support FileEvent and FileEventLog to track changes to external files
  • Save and load server definitions to disc
  • Assert on file event log
  • Put account/search behind a feature flag
  • Assert on account event log
  • Fold sos-migrate into sos-sdk behind the migrate feature flag
  • Add export_unsafe_archive and import_file to LocalAccount
  • Update audit trail test to use LocalAccount
  • Remove UpdateVault variant from WriteEvent
  • Log file events when deleting a folder!
  • Assert on file event log
    • Delete folder files
  • Assert on folder event log
    • Set vault meta
  • Move file management to folder storage so we can use FolderStorage on the server-side as well
  • Server manages account/file event logs
  • Move audit logs inside FolderStorage
  • Remove audit logs from server (now handled inside the FolderStorage)
  • Write device vault to different location as standard folders (vaults) (it doesn't have an event log so appears a bit weird when looking in the folders - write to device directory instead?)
  • Use remote instead of local for server-side directories?
  • Store trusted devices in the account devices vault
  • Use trusted device public keys to encrypt folder passwords for syncing #242
  • Remove SecureAccessKey and listen bridge channels
  • Use Folder type in IdentityVault so we track changes
  • Refactor to store identity vault server-side
  • Include commit proof for identity vault in account status
  • Support allow/deny config
  • Move audit trail behind a feature flag
  • Include and verify device signature for restricted service endpoints
  • Send device event log to server so trusted devices can be synced
  • Include device event log in diff/merge logic
  • Test spec for enrolling a device
  • Test spec for revoking a device
  • Remove noise protocol support
  • The Origin type can be a struct not an enum now (as the API will be the same for self-hosted and cloud)
  • Include external files in sync
  • Refactor to REST API
  • Add init command to server - write out config file
  • Add OpenAPI definitions
  • Remove RPC implementation, encoding (once REST API is ready)
  • Explicitly close listeners in listen test specs (either sign out or delete account)

Tests

  • Create secret
  • Update secret
  • Delete secret
  • Change folder name
  • Create folder
  • Delete folder
  • Import folder
  • Server offline
  • Changes listener
  • Write secrets after syncing folder via create/import folder events
  • Multiple listeners
  • Multiple remotes
  • Multiple remotes with failure beforehand
  • Shutdown all websocket listeners on sign out
  • Assert on number of server websocket connections (requires endpoint)
  • External files
  • Update file

  • Full sync handles new folders
  • Full sync handles deleted folders
  • Full sync handles file diffs

Closes #191 and closes #222 and closes #224 and closes #228.

@tmpfs tmpfs self-assigned this Nov 11, 2023
@tmpfs tmpfs force-pushed the server-sync branch 2 times, most recently from 263f8db to 746a4d5 Compare December 4, 2023 04:13
So that the client-side merge will be able to replay events on the local
account.
Use a Summary pointer not a Gatekeeper reference.
To be more consistent with function names in other types.
We need to merge events at the account level so may as well implement it
there.
With a view to diff/merge test spec that does not perform any
networking.
Add test spec for set_folder_description().
Always sign the URL path.
We need to handle the generic when the `sync` feature is not enabled so
use std::marker::PhantomData.
Start tidying the HTTP client implementation.
So that the API is consistent.
Always prefix with api/v1/sync path.
To ensure we shutdown the file transfers background task before teardown
tries to remove the integration test directory.
Requires work to show status comparison.
@tmpfs tmpfs mentioned this pull request Jan 19, 2024
5 tasks
Fix bug with loading files event log and make file transfer queue
loading from disc more fault tolerant.
@tmpfs tmpfs merged commit 5082c92 into main Jan 20, 2024
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant