Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

indexer: use multiple processes, e.g. process per indexer #172

Open
karlicoss opened this issue Nov 20, 2020 · 0 comments
Open

indexer: use multiple processes, e.g. process per indexer #172

karlicoss opened this issue Nov 20, 2020 · 0 comments
Labels
backend Related to indexing/serving performance
Projects

Comments

@karlicoss
Copy link
Owner

At the moment only auto provider has some ad-hoc indexing

cores = use_cores()
if cores is None: # do not use cores
pool = nullcontext()
mapper = map # dummy pool
else:
workers = None if cores == 0 else cores
pool = Pool(workers) # type: ignore
mapper = pool.map # type: ignore

Need a flag/optional config option to allow specifying number of cores to run on + make sure that single core isn't spawning a separate process (otherwise breaks debugging). By default should be single threaded to avoid potential issues and simplify initial troubleshooting

Would require a change somewhere around here

def iter_all_visits() -> Iterator[Res[DbVisit]]:
for idx in indexers:

@karlicoss karlicoss added backend Related to indexing/serving performance labels Nov 20, 2020
@karlicoss karlicoss added this to enhancements in board Nov 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Related to indexing/serving performance
Projects
board
optional enhancements
Development

No branches or pull requests

1 participant