Long distance restrictions

[CBroz1]

Description

This PR refactors dj_chains and dj_graph to use centralized logic for navigating graphs. See new docs for new notation using << and >> operators.

Other minor changes

• Adds camel_name as a Mixin property
• Adds fuzzy_get to dj_helper to permit indexing of custom objects
• Moves PERIPHERAL_TABLES to dj_helper to avoid circular import
• Removes redundant check on on loading the graph - graph handles this check
• Adds import_merge_table backup based on user feedback to search_descendants

Checklist:

• This PR should be accompanied by a release: No
• N/A. If release, I have updated the CITATION.cff
• No. This PR makes edits to table definitions: (yes/no)
• N/A. If table edits, I have included an alter snippet for release notes.
• I have updated the CHANGELOG.md with PR number and description.
• I have added/edited docs/notebooks to reflect the changes

[samuelbray32]

One required bug fix, and a few usability ideas.

One other more general idea:

• From testing, it seems like a multi-part restriction only works through this if all the keys are found in the same up(down)stream table
• I assume the correct way for a user to do this would be call restrict_by separately for each key and then combine the results.
• Could be nice to happen inside the restrict_by function (e.g. split the restriction into parts, call restrict_by for each, then return the intersection fo the results

[samuelbray32]

A usability thought/suggestion. As you mentioned, the shortest path option can get tricky with merge tables. In testing this I would end up manually repeating the call when it went the "wrong" way with no results and adding to the banned list each time.

When an empty restriction set is found would it make sense to add whatever the top of the cahin was to the banned list and then recursively call restrict_by?

[samuelbray32]

The stop condition would either be finding a non-empty set or hitting the error where the key is no longer in the graph

[CBroz1]

My plan for the immediate future is to get a sense of how useful it is from folks. If useful, if the table banning process is a chore, I would continue the TableChain search until all tables in the path had a valid restriction. Rather than deciding the path, I would depth-first search until there was a valid restriction the whole way.

It seemed like a lot of work to put in without that initial feedback, but I can revisit if you think it's needed

[samuelbray32]

That's fair. I agree it's not required for this PR.

As another option that doesn't require action right now; maybe a more efficient solution would be something like a required_tables_list where a chain is only valid if it contains the tables in that list. Would potentially solve the case of picking from a certain source of a merge table.

[CBroz1]

• Could be nice to happen inside the restrict_by function (e.g. split the restriction into parts, call restrict_by for each, then return the intersection fo the results

That's true - I had assumed that the passed restriction was valid on exactly one table elsewhere. If the user wanted to restrict by multiple, I think they could (Table << restr1) << restr2. I think the splitting into parts would be tricky. DJ already uses lists to indicate OR and parsing the possibilities can be tricky. What end-user syntax did you have in mind?

[samuelbray32]

What end-user syntax did you have in mind?

I was trying something like table << {"restr1": restr1_val, "restr_2": restr_2_val} being parsed by the function into (table << {"restr1": restr1_val}) << {"restr_2": restr_2_val} as you described.

This is another that was just me trying to test it as a naive user so it's fair for it not all implemented at release if the core function is working.

[CBroz1]

1. (table << {"restr1": restr1_val}) << {"restr_2": restr_2_val}
2. table << {"restr1": restr1_val, "restr_2": restr_2_val}

Across these cases, (1) uses additional info from the user that these likely come from different tables, and implementing (2) would assume they do not, which incurs the time cost of checking each table to see where they belong. I'd consider that time cost an annoyance, but maybe the average user wouldn't? Worth asking when I collect feedback