rename modules to allow for further extension (#18)

CHANGELOG.md

@@ -0,0 +1,21 @@
+### 2022-01-30
+
+[Relevant PR](https://github.com/seanbreckenridge/HPI/pull/18)
+
+Renamed some modules to allow for future extension, and less possibilities for conflicts with other related HPI modules under a particular company/service/source
+
+For reference, [here is the current directory structure](https://github.com/seanbreckenridge/HPI/tree/eb425e653918d68eb9d41da29e791fe1ba554dc7/my) as of this commit
+
+In particular, anything with a `gdpr`/`data_export`/`privacy_export` is named to be that, instead of globally squashing the namespace module to the single `modulename.py` file
+
+Converting a single-file module to a namespace module [is always a breaking change](https://github.com/karlicoss/promnesia/pull/225#issuecomment-819773697), and though [one can do hacky traceback introspection](https://github.com/karlicoss/HPI/blob/master/my/reddit/__init__.py) (to possible delay the change from a non-namespace package to a namespace package. However, if anyone else is using this code, its likely in the background through promnesia, so most likely situation is that they don't see that till I deprecate it anyways), but its only a temporary solution until the `__init__.py`/`module.py` file is eventually removed -- so better to do them all now instead of waiting till it becomes 'too late'
+
+A user (or me) may want to write their own module with the same name, meaning they can't use both at the same time if mine is just `my.module_name.py`, since my module existing means any other namespace packages can't have the same base name (see [reorder_editable](https://github.com/seanbreckenridge/reorder_editable) for an explanation)
+
+The one motivating this change is `apple.py`, since the old `apple.py` was parsing the privacy export, but I wanted to add something to parse [`imessage`](https://github.com/seanbreckenridge/HPI/commit/e361ce8182d8be8b331875078ad17605d3f80a50) files. Someone else may want to add other `apple/file.py` files to parse other parts of apple/mac behaviour, but me having the single `apple.py` either means they have to have their repo before mine on their path (but doing so means they overwrite the current `apple.py` file, so they cant use that to parse their privacy export, even if they were trying to do something else entirely), or they have to rename their code to something like `my_apple/file.py` to create a new namespace module
+
+Possible 'Exceptions' to this:
+
+- For some files, if the possibility for conflict is low (I can't imagine anyone exporting data from the source in any other way, e.g., `ipython`, `project_euler`) or the name is so specific to the source that its not needed (e.g. `ttt`, `window_watcher`)
+- For files where I can't imagine you'd want both mine and your/custom implementation the same time, e.g. if you override `bash`, `zsh`, you're probably creating your own solution to parse that source, and don't need mine (If that's not the case, feel free to open an issue)
+- For some of my modules, I've renamed them from what they do to their service/project names instead (`albums` to `nextalbums`; `money` to `mint`), so I'm not holding the generic name of some function when I don't really need to

README.md

@@ -17,40 +17,40 @@ This is built on top of [`karlicoss/HPI`](https://github.com/karlicoss/HPI). It
 
 ### My Modules
 
-- `my.browsing`, using [`browserexport`](https://github.com/seanbreckenridge/browserexport) to backup/parse firefox/chrome/safari history
+- `my.browser.export`, using [`browserexport`](https://github.com/seanbreckenridge/browserexport) to backup/parse firefox/chrome/safari history
 - `my.zsh` and `my.bash`, access to my shell history w/ timestamps
 - `my.imap` to parse local IMAP sync's of my email
 - `my.google_takeout`, parses lots of (~500,000) events (youtube, searches, phone usage, comments, location history) from [google takeouts](https://takeout.google.com/), using [`google_takeout_parser`](https://github.com/seanbreckenridge/google_takeout_parser)
-- `my.mpv`, accesses movies/music w/ activity/metdata that have played on my machine, facilitated by a [mpv history daemon](https://github.com/seanbreckenridge/mpv-history-daemon)
-- `my.discord`, parses ~1,000,000 messages/events from the discord data export, parser [here](https://github.com/seanbreckenridge/discord_data)
-- `my.money`, bank account transactions/balance history from [my personal budget tool](https://github.com/seanbreckenridge/mint)
+- `my.mpv.history_daemon`, accesses movies/music w/ activity/metdata that have played on my machine, facilitated by a [mpv history daemon](https://github.com/seanbreckenridge/mpv-history-daemon)
+- `my.discord.data_export`, parses ~1,000,000 messages/events from the discord data export, parser [here](https://github.com/seanbreckenridge/discord_data)
+- `my.mint`, bank account transactions/balance history from [my personal budget tool](https://github.com/seanbreckenridge/mint)
 - `my.todotxt`, to track my to-do list history (using backups of my [`todotxt`](http://todotxt.org/) files)
-- `my.newsboat`, keeps track of when I added/removed RSS feeds (for [`newsboat`](https://newsboat.org/))
+- `my.rss.newsboat`, keeps track of when I added/removed RSS feeds (for [`newsboat`](https://newsboat.org/))
 - `my.ipython`, for timestamped python REPL history
 - `my.ttt`, to parse shell/system history tracked by [`ttt`](https://github.com/seanbreckenridge/ttt)
 - `my.window_watcher`, to parse active window events (what application I'm using/what the window title is) using [`window_watcher`](https://github.com/seanbreckenridge/aw-watcher-window)
 - `my.location`, merges data from [`gpslogger`](https://github.com/mendhak/gpslogger), `apple`, `google`, `discord`, `blizzard`, and `facebook` to provide location data (goes back ~10 years)
-- `my.chess`, to track my [chess.com](https://www.chess.com)/[lichess.org](https://lichess.org/) games, using [`chess_export`](https://github.com/seanbreckenridge/chess_export)
-- `my.trakt`, providing me a history/my ratings for Movies/TV Show (episodes) using [`traktexport`](https://github.com/seanbreckenridge/traktexport)
-- `my.listenbrainz`, exporting my music scrobbling history from [ListenBrainz](https://listenbrainz.org/) (open-source Last.fm) using [`listenbrainz_export`](https://github.com/seanbreckenridge/listenbrainz_export)
-- `my.mal`, for anime/manga history using [`malexport`](https://github.com/seanbreckenridge/malexport)
-- `my.grouvee`, for my video game history/backlog using [`grouvee_export`](https://github.com/seanbreckenridge/grouvee_export)
-- `my.runelite`, parses data from the [automatic runelite screenshots](https://github.com/runelite/runelite/wiki/Screenshot)
+- `my.chess.export`, to track my [chess.com](https://www.chess.com)/[lichess.org](https://lichess.org/) games, using [`chess_export`](https://github.com/seanbreckenridge/chess_export)
+- `my.trakt.export`, providing me a history/my ratings for Movies/TV Show (episodes) using [`traktexport`](https://github.com/seanbreckenridge/traktexport)
+- `my.listenbrainz.export`, exporting my music scrobbling history from [ListenBrainz](https://listenbrainz.org/) (open-source Last.fm) using [`listenbrainz_export`](https://github.com/seanbreckenridge/listenbrainz_export)
+- `my.mal.export`, for anime/manga history using [`malexport`](https://github.com/seanbreckenridge/malexport)
+- `my.grouvee.export`, for my video game history/backlog using [`grouvee_export`](https://github.com/seanbreckenridge/grouvee_export)
+- `my.runelite.screenshots`, parses data from the [automatic runelite screenshots](https://github.com/runelite/runelite/wiki/Screenshot)
 - `my.project_euler`, when I solved [Project Euler](https://projecteuler.net/) problems
-- `my.albums`, grabbing when I listened to music albums/my ratings using my [giant spreadsheet](https://sean.fish/s/albums). Handled by [`nextalbums export`](https://github.com/seanbreckenridge/albums)
+- `my.nextalbums`, grabbing when I listened to music albums/my ratings using my [giant spreadsheet](https://sean.fish/s/albums). Handled by [`nextalbums export`](https://github.com/seanbreckenridge/albums)
 
 #### 'Historical' Modules
 
 These are modules to parse GDPR exports/data from services I used to use, but don't anymore. They're here to provide more context into the past.
 
-- `my.apple`, parses Game Center and location data from the apple GDPR export
-- `my.facebook`, to parse the GDPR export from Facebook
-- `my.league`, gives League of Legends game history using [`lolexport`](https://github.com/seanbreckenridge/lolexport)
-- `my.steam`, for steam achievement data and game playtime using [`steamscraper`](https://github.com/seanbreckenridge/steamscraper)
-- `my.blizzard`, for general battle.net event data [parsed from a GDPR export](https://github.com/seanbreckenridge/blizzard_gdpr_parser)
+- `my.apple.privacy_export`, parses Game Center and location data from the apple GDPR export
+- `my.facebook.gdpr`, to parse the GDPR export from Facebook
+- `my.league.export`, gives League of Legends game history using [`lolexport`](https://github.com/seanbreckenridge/lolexport)
+- `my.steam.scraper`, for steam achievement data and game playtime using [`steamscraper`](https://github.com/seanbreckenridge/steamscraper)
+- `my.blizzard.gdpr`, for general battle.net event data [parsed from a GDPR export](https://github.com/seanbreckenridge/blizzard_gdpr_parser)
 - `my.old_forums`, parses random forum posts and achievements from sites I used to use in the past, see [`old_forums`](https://github.com/seanbreckenridge/old_forums)
-- `my.skype` to parse a couple datetimes from the Skype GDPR export
-- `my.spotify`, to parse the GDPR export from Spotify, mostly to access songs from my playlists from years ago
+- `my.skype.gdpr` to parse a couple datetimes from the Skype GDPR export (seems all my data from years ago is long gone)
+- `my.spotify.gdpr`, to parse the GDPR export from Spotify, mostly to access songs from my playlists from years ago
 - `my.twitch`, merging the [data export](https://www.twitch.tv/p/en/legal/privacy-choices/#user-privacy-requests) and my messages parsed from the [overrustle logs dump](https://github.com/seanbreckenridge/overrustle_parser)
 
 See [here](https://github.com/seanbreckenridge/dotfiles/blob/master/.config/my/my/config/__init__.py) for config.
@@ -111,8 +111,8 @@ Most common shell commands?
 What websites do I visit most?
 
 ```python
->>> import collections, pprint, my.browsing, urllib
->>> pprint.pprint(collections.Counter([urllib.parse.urlparse(h.url).netloc for h in my.browsing.history()]).most_common(5))
+>>> import collections, pprint, my.browser.export, urllib
+>>> pprint.pprint(collections.Counter([urllib.parse.urlparse(h.url).netloc for h in my.browser.export.history()]).most_common(5))
 [('github.com', 20953),
  ('duckduckgo.com', 10146),
  ('www.youtube.com', 10126),
@@ -123,9 +123,9 @@ What websites do I visit most?
 Song I've listened to most?
 
 ```python
->>> import collections, my.mpv
->>> collections.Counter([m.path for m in my.mpv.history()]).most_common(1)[0][0]
-'/home/sean/Music/Toby Fox/Toby Fox - UNDERTALE Soundtrack (2015) [V0]/085 - Fallen Down (Reprise).mp3'
+>>> import collections, my.mpv.history_daemon
+>>> collections.Counter([m.path for m in my.mpv.history_daemon.history()]).most_common(1)[0][0]
+'/home/sean/Music/JPEFMAFIA/JPEGMAFIA - LP! - 2021 - V0/JPEGMAFIA - LP! - 05 HAZARD DUTY PAY!.mp3'
 ```
 
 Movie I've watched most?
@@ -150,7 +150,7 @@ The [`install` script here](https://github.com/seanbreckenridge/HPI/blob/a6495ad
 
 For more information on that, and some of the complications one can run into, see [reorder_editable](https://github.com/seanbreckenridge/reorder_editable#editable-namespace-packages), and the [module design](https://github.com/karlicoss/HPI/blob/master/doc/MODULE_DESIGN.org#adding-new-modules) docs for HPI.
 
-Disregarding setting up all the dependencies for individual modules (which is why the [`install`](install) script exists), this is setup by doing:
+Disregarding setting up all the dependencies for individual (e.g. `my.ipython`) modules (which is why the [`install`](install) script exists), this is setup by doing:
 
 ```bash
 # clone and install upstream as an editable package
@@ -168,6 +168,8 @@ python3 -m reorder_editable reorder ./HPI ./HPI-fork
 
 Those directories are editable installs, meaning any changes I make to them get applied immediately, which is very convenient for debugging and developing new modules.
 
+If you have issues installing, check the [CHANGELOG](CHANGELOG.md) for any possible breaking changes
+
 [`jobs`](./jobs) contains anacron-like jobs that are run periodically, using [`bgproc`](https://github.com/seanbreckenridge/bgproc) and [`evry`](https://github.com/seanbreckenridge/evry). So, this repo contains both the [DAL](https://beepb00p.xyz/exports.html#dal) and scripts to backup my data. I run the jobs in the background using supervisor, see [here](https://github.com/seanbreckenridge/dotfiles/tree/master/.local/scripts/supervisor) for the config, and/or [`run_jobs`](https://github.com/seanbreckenridge/dotfiles/blob/master/.local/scripts/supervisor/run_jobs) for the `bgproc` wrapper. (They likely won't work out of the box for you, as they depend on tokens/environment variables that are set on my system - In particular the `HPIDATA` environment variable, which for me is `~/data`)
 
 ### TODO:

functions.sh

@@ -18,8 +18,8 @@ filter_unique() {
 # my.albums
 #############
 
-alias albums-history='hpi query my.albums.history'
-alias albums-to-listen='hpi query my.albums.to_listen'
+alias albums-history='hpi query my.nextalbums.history'
+alias albums-to-listen='hpi query my.nextalbums.to_listen'
 # how many albums I have on my list that I havent listened to yet
 alias albums-left='albums-to-listen | jq length'
 # pipe a list of album blobs to this to describe them
@@ -30,15 +30,15 @@ albums-describe-score() {
 	jq -r '"[\(.score) | \(.listened_on)] \(.cover_artists) - \(.album_name) (\(.year))"'
 }
 # any albums which I can't find/have to order physical copies for to listen to
-alias albums-cant-find="hpi query -s my.albums._albums_cached | jq -r 'select(.note==\"cant find\")' | albums-describe"
+alias albums-cant-find="hpi query -s my.nextalbums._albums_cached | jq -r 'select(.note==\"cant find\")' | albums-describe"
 # list any albums I have yet to listen to, sorted by how many awards they've won
 albums-awards() {
 	local COUNT="${1:-10}"
 	albums-to-listen | jq -r "sort_by(.reasons | length) | reverse | .[0:${COUNT}] | .[] | \"[\(.reasons | length)] \(.album_name) - \(.cover_artists) (\(.year))\""
 }
 # just the next albums I should listen to chronologically
 albums-next() {
-	hpi query my.albums.to_listen -s --limit "${1:-10}" | albums-describe
+	hpi query my.nextalbums.to_listen -s --limit "${1:-10}" | albums-describe
 }
 alias albums-next-all='albums-next 99999'
 alias albums-history-desc='albums-history -s | albums-describe-score'
@@ -62,7 +62,7 @@ albums-filter-genre() {
 ###################
 
 scrobbles() {
-	hpi query my.listenbrainz.history -s "$@"
+	hpi query my.listenbrainz.export.history -s "$@"
 }
 scrobble-describe() {
 	jq -r '"\(.listened_at) \(.artist_name) - \(.track_name)"'
@@ -74,7 +74,7 @@ scrobble-describe() {
 
 # functions to replay music I've listened to recently
 mpv-recent() {
-	hpi query my.mpv.history --order-type datetime --reverse -s --limit "${1:-1}"
+	hpi query my.mpv.history_daemon.history --order-type datetime --reverse -s --limit "${1:-1}"
 }
 mpv-recent-path() {
 	mpv-recent "$1" | jq -r .path
@@ -98,12 +98,12 @@ alias zsh-unique-fzf='zsh-unique | fzf'
 
 # e.g. trakt-movies --recent 4w | trakt-describe-movie
 trakt-movies() {
-	hpi query 'my.trakt.history' -s "$@" | trakt-filter-movies
+	hpi query 'my.trakt.export.history' -s "$@" | trakt-filter-movies
 }
 
 # e.g. trakt-episodes --recent 4w | trakt-describe-episode
 trakt-episodes() {
-	hpi query 'my.trakt.history' -s "$@" | trakt-filter-episodes
+	hpi query 'my.trakt.export.history' -s "$@" | trakt-filter-episodes
 }
 
 trakt-filter-movies() {

install

@@ -107,24 +107,24 @@ module_dependencies() {
 	hpi_module my.pdfs || return $?
 	hpi_module my.window_watcher || return $?
 	hpi_module my.smscalls || return $?
-	hpi_module my.browsing || return $?
-	hpi_module my.discord || return $?
+	hpi_module my.browser.export || return $?
+	hpi_module my.discord.data_export || return $?
 	hpi_module my.google_takeout || return $?
 	hpi_module my.time.tz.via_location || return $?
 	hpi_module my.coding.commits || return $?
-	hpi_module my.todotxt || return $?
+	hpi_module my.todotxt.file_backups || return $?
 	hpi_module my.location.ip || return $?
 	hpi_module my.location.gpslogger || return $?
-	hpi_module my.chess || return $?
-	hpi_module my.mpv || return $?
-	hpi_module my.league || return $?
-	hpi_module my.trakt || return $?
-	hpi_module my.imap || return $?
-	hpi_module my.grouvee || return $?
-	hpi_module my.mal || return $?
-	hpi_module my.listenbrainz || return $?
+	hpi_module my.chess.export || return $?
+	hpi_module my.mpv.history_daemon || return $?
+	hpi_module my.league.export || return $?
+	hpi_module my.trakt.export || return $?
+	hpi_module my.mail.imap || return $?
+	hpi_module my.grouvee.export || return $?
+	hpi_module my.mal.export || return $?
+	hpi_module my.listenbrainz.export || return $?
 	hpi_module my.old_forums || return $?
-	hpi_module my.skype || return $?
+	hpi_module my.skype.gdpr || return $?
 }
 
 verify_personal_python_packages() {

my/bash.py

@@ -36,7 +36,7 @@
 from my.core import get_files, Stats, LazyLogger, Paths, dataclass
 from my.core.common import mcachew
 from my.utils.time import parse_datetime_sec
-from my.utils.common import InputSource
+from my.utils.input_source import InputSource
 
 
 @dataclass

my/blizzard.py → my/blizzard/gdpr.py

@@ -22,8 +22,8 @@ class config(user_config):
 from itertools import chain
 
 from my.core import get_files, Stats
-from .utils.time import parse_datetime_sec
-from .utils.common import InputSource
+from my.utils.time import parse_datetime_sec
+from my.utils.input_source import InputSource
 
 
 logger = LazyLogger(__name__, level="warning")

my/chess.py → my/chess/export.py

@@ -19,7 +19,7 @@
 
 from my.core import get_files, Stats, LazyLogger, Paths, dataclass
 from my.core.common import mcachew
-from my.utils.common import InputSource
+from my.utils.input_source import InputSource
 
 
 @dataclass

my/facebook.py → my/facebook/gdpr.py

@@ -21,7 +21,7 @@ class config(user_config):
 
 
 from my.core import get_files, Stats, Res, Json, LazyLogger
-from .utils.time import parse_datetime_sec
+from my.utils.time import parse_datetime_sec
 
 
 logger = LazyLogger(__name__, level="warning")

my/ipython.py

@@ -39,7 +39,7 @@ class config(user_config):
 from IPython.core.history import HistoryAccessor  # type: ignore[import]
 
 from my.core import get_files, warn_if_empty, Stats, Res
-from .utils.common import InputSource
+from my.utils.input_source import InputSource
 
 
 class Command(NamedTuple):

my/league.py → my/league/export.py

@@ -28,8 +28,8 @@ class config(user_config):
 from itertools import chain
 
 from my.core import get_files, Stats, Res, Json, warn_if_empty
-from .utils.time import parse_datetime_millis
-from .utils.common import InputSource
+from my.utils.time import parse_datetime_millis
+from my.utils.input_source import InputSource
 
 
 def inputs() -> Sequence[Path]:

my/listenbrainz.py → my/listenbrainz/export.py

@@ -17,7 +17,7 @@
 from more_itertools import unique_everseen
 
 from my.core import get_files, Stats, LazyLogger, Paths, dataclass
-from my.utils.common import InputSource
+from my.utils.input_source import InputSource
 
 
 @dataclass

my/location/all.py

@@ -9,12 +9,11 @@
 from .models import Location
 
 # sources
-from .ip import ips
-from .gpslogger import history as gpslogger_history
-from ..apple import events as apple_events
-from ..apple import Location as AppleLocation
+from my.location.ip import ips
+from my.location.gpslogger import history as gpslogger_history
+from my.apple.privacy_export import events as apple_events, Location as AppleLocation
 
-from ..google_takeout import (
+from my.google_takeout import (
     events as google_events,
     _cachew_depends_on as _google_cachew_depends_on,
 )

my/location/gpslogger.py

@@ -25,7 +25,7 @@ class config(user_config):
 from my.core import Stats, LazyLogger, Res
 from my.core.common import get_files, warn_if_empty, mcachew
 from my.core.warnings import high
-from ..utils.common import InputSource
+from my.utils.input_source import InputSource
 
 
 logger = LazyLogger(__name__, level="warning")