[DESIGN] "cross-feeding" vs "performance" -- ie dispatch vs specialized

[dehann]

DFG is about having access to various graphing technologies in a way that allows info/data/etc to be easily transferred between technologies in a standard way. For example:

# copy from one dfg to another, regardless of different SolverParams types
copyGraph!(localdfg, cgdfg, args...)

merge!(CloudDFG, FileDFG)
merge!(FileDFG, CloudDFG)
merge!(CloudDFG, LightDFG)
merge!(GraphsDFG, LightDFG)
merge!(GraffDFG, CloudDFG) # seems excessive, but its about the philosophy 
# ...

updateVariables!(LigthDFG, CloudDFG)
updateVariables!(FileDFG, GraphDFG) # to illustrate point
updateVariables!(FuseDFG, LightDFG) # to illustrate point
# ...

buildSubgraph(::LightDFG, GraphDFG, ...)
buildSubgraph(::CloudDFG, LightDFG, ...)
buildSubgraph(::FileDFG, CloudDFG, ...)
# ...

Originates from hard discussions on:

• [BUG] subgraph functions not working for LightDFG #261
• Mini cleanup on solvable for GetSubgraphAroundNode on LightGraphs #214
• Copying CGDFG -> local fails with cryptic error #507

cc @GearsAD , @Affie

EDIT: GraphsDFG has since been archived in favor of LightDFG.

[dehann]

Decision is whether we want to support the kinds of "cross-feeding" functions I'm suggesting here -- or which subset? Or, if any decisions are going to actively exclude any of the features suggested here?

[dehann]

In reponse to a discussion in #261:

This is more speaking to the room: I want to stress that DFG should not just be a clambering together of silo'ed drivers for graph technologies -- the most value comes from true "Distributed" nature in a variety of workloads that one user probably wont' see in their individual use-cases. I fully acknowledge its hard to get right though.

[dehann]

would you guys say the current design in DFG v0.8.0 achieves this -- so we can close?

[Affie]

It’s definitely a step in that direction. I think it’s going to be a continuous effort.

[dehann]

Cannot close until at least #507 is resolved

[dehann]

#315 (comment)