Override or fallback exchanges

[topherbuckley]

I'm currently in the US and geolocked from using the binance API. I have some BNB trades on binance when this was still ok (before the binance.us split). Currently, it appears the ccxt has an alt market pair of BNBUSDT defined here which is causing it to fail to retrieve the market data.

If I were to just overwrite that line and use another market, I would be removing the market with the highest volume by far, so not likely a great choice for all users. Two possible ideas I'm thinking about so far (please tell me if they are flawed)

1. Change the _ALT_MARKET_EXCHANGES_DICT and _ALT_MARKET_BY_BASE_DICT to something list-like rather than a dict, such that I can have multiple entries for a single asset type. This would entail some sort of priority, such that binance would remain top priority, but only fallback to the alternative if this 451 http error (geolock) occurs.
2. Have something in the user's config file that can override these two Dicts such that each user can decide their own alt exchanges. Is this already possible and I'm just not understanding it?

[macanudo527]

Do you need to file taxes soon?

The reason I ask is that there is a lot in the pipeline that will just make these errors go away. We have already roughly hashed out how this is going to work. It's a bit complicated, but maybe I can give you a basic run down.

1. Right now I'm working on completing automatic graph optimization Add Optimization to Djikstra #146 . This is a massive PR that will auto-prioritize markets with the highest volume by week. Basically week by week it will process pricing through the markets with the highest volume that week.

2. After that, I'll be implementing aliases for renamed coins and 'micro'-assets (e.g. mBTC, etc..) Implement aliases for assets on specific exchanges #140

3. Then, I'll be adding in 'teleporting' which will merge graphs (the routing map for pricing) from all exchanges into one graph. At that point, I'll be implementing automatic market discovery, and will be deleting most if not all the hard-coded alternative markets as well as _ALT_MARKET_EXCHANGES_DICT and _ALT_MARKET_BY_BASE_DICT. My hope is that all the hard coding except for supported exchanges will no longer be necessary.

If you can wait for that to be completed, which should be this year, it should fix basically all the issues you are currently having.

If you do need to file taxes this year. You have two options:

1. monkey patch it with your alt markets and get the prices you need, not submitting a PR. This is by far the fastest option, but you'll be getting the price from a market that you didn't actually use to make the trade.
   or
2. Building a csv reader/parser for Binance.com since the CSV files are still accessible from the States, and probably will remain so since it seems to be a publically posted document. It would need to implement 1 week candle emulation and we will need to extract the chunking code into a new utility class. It would be a tricky first project, but would be useful to the project as well as solve your issues since the posted candles/klines are only missing the last 48 hours or so.

I did this for Kraken. Very soon, I should have a newer version that implements 1 week candle emulation up for you to look at. I just need to write a few more tests for #146 .

EDIT: It looks like you would just need to absorb this script, add chunking, which is already implemented and just needs to be extracted and adapted, and add 1 week candle emulation, which you should be able to adapt from my code. It shouldn't be that tough if you have the time/willingness to do it. I, for one, would find it incredibly useful and would be grateful if you could do that.

[topherbuckley]

Thanks for the summary of what is going on! I was reading through all those PRs/Issues trying to catch up, but this helps a lot!

Do you need to file taxes soon?

Yeah, I already filed my Japanese ones, but still need to finish up the crypto end of my US ones. I was using Cryptact before discovering this repo. That made it easy to complete the Japanese taxes as I'm only dealing with a small realized loss (as calculated by Cryptact), and as I'm sure @macanudo527 knows, losses on crypto don't count for much in Japanese tax law atm (unless you have a bunch of Msc. Income to offset) so I just claimed a zero on my Japanese taxes. In all reality, I can do the same on my US ones and not lose any significant deduction. I was just being stubborn and wanted to get all my records straight before moving on, but I can always delay that to next year ¯_(ツ)_/¯

Building a csv reader/parser for Binance.com since the CSV files are still accessible from the States, and probably will remain so since it seems to be a publically posted document.

Can you help me understand the conceptual difference between this and the geolocked REST API? I don't have a whole lot of experience with network programming so is this csv download script something just outside of the REST API that can be accessed by everyone? Happy to work on this though to learn.

monkey patch it with your alt markets and get the prices you need, not submitting a PR. This is by far the fastest option, but you'll be getting the price from a market that you didn't actually use to make the trade.

I was also curious if anyone here knew if the data on binance.com is significantly different than binance.us? I know I can access the api from binance.us from the US, but I assume I should treat this an entirely different set of data (unless someone knows better).

As for what I plan to do, I'd likely just look up the prices and put them in my manual csv imports as I only have a handful of them, and was just overly relying on the __unknown keyword, despite having a lot of this price data locally in my records. I need to complete PR 134 before doing that though, so I can ensure my fiat_tickers are correct in the various manual csv files I'm using. I have some ideas for reusing the Configuration class that I'd like some comments on in the PR after I push a few more commits.

Edit: Feel free to close this issue if appropriate, and your other PRs will address this issue, just wanted to keep it open long enough for you to respond to my questions above, but we can put those in another issue specific to each if you'd like.

[macanudo527]

Can you help me understand the conceptual difference between this and the geo-locked REST API? I don't have a whole lot of experience with network programming so is this csv download script something just outside of the REST API that can be accessed by everyone? Happy to work on this though to learn.

For whatever reason, Binance.com has made the csv files public and free for download, but not the API. I would guess that this is due to it just not being all that server intensive to generate them every day and post them for such a significant exchange. It's some kind of goodwill gesture I suppose. A REST API would use up more resources since it needs to respond to incoming JSON, and if they aren't currently serving the States, they might as well shut it off.

You can manually download the csv files yourself. They just made a script available for you to programmatically do that.

I was also curious if anyone here knew if the data on binance.com is significantly different than binance.us? I know I can access the api from binance.us from the US, but I assume I should treat this an entirely different set of data (unless someone knows better).

I would guess that the Binance.us data is for trades and pricing on the Binance.us servers only. I would guess that the pricing would be similar because of liquidity providers and such, but it could differ during severe market volatility.

[topherbuckley]

Thanks again @macanudo527 that is a very helpful reply!