Concurrency implementation using batch processor

[berland]

Very much work-in-progress.

[berland]

Code runs through test. Using concurrency on the test workloads leads to slow-down, the penalty of sending dataframes back and forth between Python processes outweighs the benefit from concurrency.

[berland]

Self-assessment, concurrent code is missing in add_from_runpathfile(), apply() and get_smry() in the ScratchEnsemble object.

There is no concurrent code in EnsembleSet. Paralellizing this has very little potential, assuming the there are always more realizations in an ensemble, than ensembles in an ensembleset.

[asnyv]

There is no concurrent code in EnsembleSet. Paralellizing this has very little potential, assuming the there are always more realizations in an ensemble, than ensembles in an ensembleset.

Flownet might be an exception to that assumption @anders-kiaer ?

[anders-kiaer]

Flownet might be an exception to that assumption @anders-kiaer ?

I actually think @wouterjdb did some test on concurrency earlier, using flownet + fmu-ensemble?

[wouterjdb]

Self-assessment, concurrent code is missing in add_from_runpathfile(), apply() and get_smry() in the ScratchEnsemble object.

There is no concurrent code in EnsembleSet. Paralellizing this has very little potential, assuming the there are always more realizations in an ensemble, than ensembles in an ensembleset.

This holds for FlowNet just as well.

[berland]

This is now potentially done, except for a Py27 blocker as it turned out I have used a Py3-only feature in a part of the code that is only to be run on Py3.

[asnyv]

Looks good 👍 Left some comments

[berland]

get_volumetric_rates() needs parallelization.

[asnyv]

Not really concurrent, as the .result() waits for the subprocess to finish for each iteration. Should rather do something like below (which is how it is done in most of the rest of this PR, so this is probably just something you forgot to fix).

Suggested change

	loaded_reals = [
	executor.submit(
	ScratchRealization,
	realdir,
	realidxregexp=realidxregexp,
	autodiscovery=autodiscovery,
	batch=batch,
	).result()
	for realdir in globbedpaths
	]
	reals_futures = [
	executor.submit(
	ScratchRealization,
	realdir,
	realidxregexp=realidxregexp,
	autodiscovery=autodiscovery,
	batch=batch,
	)
	for realdir in globbedpaths
	]
	loaded_reals = [x.result() for x in reals_futures]

[asnyv]

Same as above

Suggested change

	with ProcessPoolExecutor() as executor:
	loaded_reals = [
	executor.submit(
	ScratchRealization,
	row.runpath,
	index=int(row.index),
	autodiscovery=False,
	find_files=[row.eclbase + ".DATA", row.eclbase + ".UNSMRY",],
	batch=batch,
	).result()
	for row in runpath_df.itertuples()
	]
	with ProcessPoolExecutor() as executor:
	reals_futures = [
	executor.submit(
	ScratchRealization,
	row.runpath,
	index=int(row.index),
	autodiscovery=False,
	find_files=[row.eclbase + ".DATA", row.eclbase + ".UNSMRY",],
	batch=batch,
	).result()
	for row in runpath_df.itertuples()
	]
	loaded_reals = [x.result() for x in reals_futures]

[asnyv]

This change is significant. Each call to ScratchRealization.get_eclsum() is quite heavy, and many methods are written with several calls to get_eclsum(), such that the loss of caching might actually increase runtime even with concurrency.

[asnyv]

Also tested setting lazy_loading=True in the EclSum in get_eclsum to reduce cost of each get_eclsum() call, but that was not a game changer (in some cases slower).

As I see it, all methods that currently run get_eclsum() somewhere in their dependencies are not safe to merge as it is now, without the risk of performance decrease for some users.

[ertomatic]

Can one of the admins verify this patch?

[berland]

Rebased on top of #182

[oysteoh]

maybe this looks better and reduces the usage of str(..) and lower() ?

env_var = str(os.environ[ENV_NAME]).lower()
if( env_var == "0" or env_var == "false" or env_var == "no"):
    ..
    ..

[berland]

Superseded by #206