-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rekt pps? problem? => piker.accounting
#489
Conversation
3fa212f
to
fa3cdbe
Compare
Lul, gotta drop that |
2e2f49e
to
c913adc
Compare
To be compat with the `Symbol` (for now) and generally allow for reading the (derivative) contract specific part of the fqme. Adjust `contract_info: list[str]` and make `src: str = ''` by default.
Initial attempt at getting the sampling and shm layer to use the new mkt info meta-data type. Draft out a potential `BackendInitMsg: msgspec.Struct` for validating the init msg returned from the `stream_quotes()` start value; obvs don't actually use it yet.
To make nested `msgspec.Struct`s work we need to tell the codec that the `.symbol` is some struct def, since we don't really need to enforce that (yet) we're just going to enc/dec as `str` until we further formalize and/or need something more complex.
# TODO: instead see if we can hack tomli and tomli-w to do the same: | ||
# - https://github.com/hukkin/tomli | ||
# - https://github.com/hukkin/tomli-w | ||
class PpsEncoder(toml.TomlEncoder): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
entirely replaced by tomlkit
conf_path: Path | ||
conf, conf_path = config.load_account(brokername, acctid) | ||
|
||
if brokername in conf: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this was originally for converting legacy account files: pps.toml
but we can probably remove it in next patch set for this subsys.
@@ -439,6 +519,35 @@ async def get_ohlc( | |||
yield get_ohlc, {'erlangs': 3, 'rate': 3} | |||
|
|||
|
|||
@async_lifo_cache() | |||
async def get_mkt_info( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is the new required endpoint for all backends which delivers the MktPair
(ideally with embedded .src
/.dst
attrs which are Asset
s) translation for any searched market/symbol/ticker pattern.
@@ -713,6 +771,98 @@ def normalize( | |||
return data | |||
|
|||
|
|||
@async_lifo_cache() | |||
async def get_mkt_info( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm the least happy with this implementation (in terms of existing backends) just because as a legacy market broker ib tends to assume a source asset for every exchange..
Ideally we can work toward getting this more refined going forward particularly if we're going to be re-doing our tsdb subsys backend choice(s) and don't have to rely on the table keys in the existing replicated data sets..
longer run i would like to see mnq/usd.cme.ib
over no usd in the fqme.
# but likely we want all backends to eventually handle | ||
# ``dst/src.venue.`` style !? | ||
src: str | Asset = '' | ||
if atype == 'forex': |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bleh, yeah we need some backend specific tests i think to be able to get this to the new proper fqme nomenclature i'm thinking...
c72b998
to
7011187
Compare
Ok so as per discussion in dev chat I'm going to cowboy land this massive change set without extensive review bc,
Hopefully this won't happen again 😹 |
Even moar fixes after #470, #475, #484 which mega broke the old
pps.toml
management for live backend 💥 😶🌫️ 🤕Turns out this might be the reason to start assaulting the revamp ofprolly positioning: the accounting of "what/when (is) owned"
our accounting tooling as per our now long list of desirables:
accounting
Testing enhancements (some tasks deferred)
rework test set to use
pytest.mark.parametrize
so we can moreeasily use common harness logic for darks and alert testing.
reorg the ahab API docker supervisor endpoints to be decoupled
from
open_pikerd()
and able to accept arbitrary config overrides forboth testing and distributed setups.
dark order and alert clearinglines
add a basic accounting of position only test set?tasks and bugs getting deferred..
pre-existing live order loading is borked on first chart edit from
order mode: Editting initial live order load fails? #502
final removal of
Symbol
and.symbol
semantics -> summarized in Final removal of.accounting._mktinfo.Symbol
#517order_mode
.symbol
final usage..?make a follow up issue to add a
Flume.Cascade
type for fspswhich describes stream IO between flumes:
deribit still need full
get_mkt_info()
andFeedInit
sync up the new
data.validate
stuff with old brokercheck idea inBackend test suite #387, Brokercheck an all backends #393 -> add an actual test?
dependency changes needed for this to land
move to new
TOML
libs (tomlkit
for styling andtomli-w
forfast writing) B)
toml
writing hacks we have into a new submod..toml
dep and go withtomli
,tomli-w
and
tomlkit
B)tomlkit
which does support writing inlinetables (as well as lotsa other great stuff) as requested in Better
TOML
libs ->tomlkit
for style,tomli(-w)
for speed #496 B)(https://github.com/pikers/tomlkit/tree/piker_pin)
Decimal
andMutableMapping
codecs, add a multiline-Array
indent control python-poetry/tomlkit#294Context
semantics for cross-actor-task cancellation and overruns goodboy/tractor#357Unrelated to
.accounting
changes.data._web_bs
implementation to avoid the useof backend code using async generators within async context managers.
too many devs are too busy to review it rn so there's not a lot of
point 😂
NEW SUBSYSTEM: the
piker.accounting
(subpkg) 🏄move all
.data._source
andpiker.pp
mod contents into a muchbetter organized subsystem API ->
piker.accounting
🏄._mktinfo
containsMktPair
andAsset
primitive as well as new.fqme
acronym (fully qualified market endpoint) processing.._pos
containsPosition
andPpTable
loading and processing._ledger
addsTransactionLedger
andTransaction
loading andsync.
.cli
adds aledger <broker>.<accountname>
syncing prototype cliwhich can be used to dump position summaries to console.
try to see how many bullets we can nail from Positioning system refinements: paper engine, file writing, toml style and perf #345
offer
.write_config()
to (the new)TransactionLedger
andPpTable
for immediate writes.further follow ups: Change
Position.clears
->.events
#510, Offline *executable* storage format #511, Move totyper
for all clis #513, Position accounting refinements, bug fixes, src-asset-pnls,ledgerd
daemon #515drop the singleton
pps.toml
instead breaking into multiple account filesper backend,
$HOME/.config/piker/accounting/account.<brokername>.<(sub)accountname>.toml
account.binance.paper.toml
,account.kraken.spot.toml
which are named from the accounts listed in a user's
brokers.toml
B)table schema: i.e. no longer any
kraken.paper
in any section keys.Prototype
ledger
cli usingtyper
with follow up to come with Ledger (syncing) CLI and offline toolset #512typer
andrich
as deps!Move our data feed layer to new
MktPair
market info struct aswell as support a new "backend validation system" using
the new
.data.validate.FeedInit
we now expect
stream_quotes()
to deliver a pedanticlist[FeedInit]
onstartup instead of a random
dict
of some schema 🏄all backends should provide a
get_mkt_info()
endpoint now whichdoes the work of constructing a
MktPair
from an input searchpattern (
str
) and is also expected to construct theMktPair.src
/
.dst
fields which in the ideal case are.accounting._mktinfo.Asset
refs.move all internal usage of the old
Symbol
type over to this aswell as rename as much as we can of
fqsn
to the newfqme
(fullyqualified market endpoint)
Flume.symbol
->.mkt: MktPair
Position.symbol
->.mkt
_chart
,_display
_fsp
.. etc.ui._chart.LinkedSplits.symbol
outright and finally!port all backends as much as is possible to this new interface
.data.validate
since it should bedefault now B)
implementation, but i don't have a live feed..
questrade
: use ws api for live rt quotes #509bring in backend checks from Brokercheck an all backends #393 and add to a test set for
superficial audit of all broker/data providers.
data.validate
stuff and not sure we really need explicit test set?
Outstanding/Related issues/bugs that should be resolved here:
kraken
: balance <-> position disparities with crypto/crypto pairs #373 closed for Position accounting refinements, bug fixes, src-asset-pnls,ledgerd
daemon #515Pps toml header leaking after zeroing positions #472 fixed here and prior
Better
TOML
libs ->tomlkit
for style,tomli(-w)
for speed #496ib forex is currently borked since the move to
MktPair
(b44b091)behave as before!
burning a shm allocation...
type matching.
kucoin
borked on search result feed selection-load. (ff74d47)discovered BUGs tofix, issues likely to re-issued and defer, new feature ideas:
kraken
position tracking gotchas #263 -> moved to Position accounting refinements, bug fixes, src-asset-pnls,ledgerd
daemon #515extra follow ToDos (birds with stones):
follow up
.accounting
subsys summary refinements and bug fixesin Position accounting refinements, bug fixes, src-asset-pnls,
ledgerd
daemon #515EMS layer: unify the messaging layer to a single msg set: Unify EMS messaging set #514
CLI
piker ledger (sync)
development and nice-to-havestyper
for all CLIs: Move totyper
for all clis #513