-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CopyCSV taskscheduler improvement #1404
Comments
Here's my suggestion for how to do this. Unless someone has a comment before I start, this is my current plan: OverviewAll tasks that are meant to be executed by multiple threads are based on reading a chunk of a disk file (e.g., arrow or parquet). Currently this works as follow. The main thread Tm (user's thread) that prepares and assigns the tasks by scanning 1 block (~8MB) of the disk file and puts it into a task and schedules it. Tm does this for multiple tasks Instead, I'll make these parts of the code morsel-driven in a simple way: I'll add a lock that registered threads needs to grab to scan a batch of 8MB, and then after that release the lock and run the rest of the copying function. For example,
It would change as follows:
Something like this. Note that to solve this current issue, I'll have to add some code to check if an error was thrown before grabbing a new batch and throw an exception if there was an error to stop working. I would do similar changes in other places that we can "morselize" both in node_copier.cpp and rel_copier.cpp. As part of this, I expect to remove several functions from TaskScheduler, such as In addition to these changes I want to also do the following (maybe in a separate PR):
I need to think of a way to test these changes though, i.e., that the mechanisms to stop early when there are timeouts and exceptions are working as expected. |
There are two issues related to the current taskScheduler:
1. CopyCSV taskScheduler exception handling.
When executing copycsv, the main thread may put multiple tasks in the task queue.
If an exception occurs on a task, we will remove that task from the task queue and throw the exception. However, the other tasks which are planned by the copyCSV are not removed from the queue. The worker thread will continue to grab task from the taskQueue and execute, which causes seg fault.
2. CopyCSV taskScheduler waitAllTasksToCompleteOrError problem:
The copyCSV calls the
waitAllTasksToCompleteOrError
to wait for all copyCSV tasks to finish. However, the processor may put some processor tasks into the task queue at the same time. The copyCSV will wait for the processor task to finish as well.The text was updated successfully, but these errors were encountered: