Modularizing loopback-datasource-juggler

[superkhau]

We are in the process of refactoring loopback-datasource-juggler to reduce it's concerns and modularize the project. I would like to gather some ideas from the community as to what you would like to see. The proposal (WIP) is at https://github.com/strongloop/loopback-datasource-juggler/wiki/%5BWIP%5D-Proposal-for-modularizing-loopback-datasource-juggler.

• Move dao.js to loopback-connector
• Make the in-memory connector it's own module (loopback-connector-memory)
• Type registry (type.js) should stay with Juggler, but be exposed so that LoopBack can expose it to the app
  • Allows users to register their own custom types during runtime
  • The registry should also read in datatypes from a config/setting that allows users to register types at boot time (ie. datatypes.json, with some configs that points to a custom datatype.js)
  • Probably need to define a standard interface for custom datatypes

[doublemarked]

One improvement I see is to somehow extract the parameter processing code out of the core data access functions (find, findOne, etc) so that it's easier to wrap those methods on individual models without copying/pasting a bunch of code from dao.js into your wrapper method. For example, these lines: https://github.com/strongloop/loopback-datasource-juggler/blob/master/lib/dao.js#L1349-L1374

[superkhau]

@doublemarked +1, these things should be mixed in on an as needed basis (as not all connectors are CRUD based anyways).

[bajtos]

I read the proposal and it looks to me like it's trying to completely re-engineer the current design, possibly also over-engineering in some places. I guess I was expecting more of an incremental, evolutional approach to this issue, instead of what I perceive as a revolution. Having said that, it's possible that I just misunderstood what's being proposed and was not able to connect the dots (the proposal and the current implementation) to realise that the amount of changes is small. I fully admit that I am not good at working in the "big design up front" style, which is probably not helpful here.

I think we should treat the proposal as a high-level description of the final design/architecture where we would like to eventually end up, in which case I don't have any particular objections against what's being proposed.

I'd like to see a list of incremental user stories to be able to better understand what changes are being proposed and be able to actually provide useful feedback.

@superkhau when compiling the list of user stories, please create the stories in such way that each story delivers some well-defined value/feature, either to loopback users or loopback maintainers. That way we can keep the discussion focused on what each change brings us, instead of getting lost in low-level technical details.

[superkhau]

I read the proposal and it looks to me like it's trying to completely re-engineer the current design, possibly also over-engineering in some places. I guess I was expecting more of an incremental, evolutional approach to this issue, instead of what I perceive as a revolution. Having said that, it's possible that I just misunderstood what's being proposed and was not able to connect the dots (the proposal and the current implementation) to realise that the amount of changes is small. I fully admit that I am not good at working in the "big design up front" style, which is probably not helpful here.

Sorry for the confusion, this doc was only created to bring up the finer points of the actual move (and to understand the entire landscape). I don't prefer to work in the BDUF style, but it helps with having something concrete for discussions. I plan to have a meeting with @raymondfeng today to discuss more specifically for how to create tasks from these ideas.

I think we should treat the proposal as a high-level description of the final design/architecture where we would like to eventually end up, in which case I don't have any particular objections against what's being proposed.

This is exactly the idea. The final output from this proposal (after speaking with @raymondfeng) is to create a list of actual stories in waffle. Before that though, I will create a small list of tasks/spikes to figure out.

when compiling the list of user stories, please create the stories in such way that each story delivers some well-defined value/feature, either to loopback users or loopback maintainers. That way we can keep the discussion focused on what each change brings us, instead of getting lost in low-level technical details.

Roger. Each waffle item will be high-level, with possible spikes to ensure deliverability.

[superkhau]

Breakdown

@bajtos This list has been verified by @raymondfeng over a hangout discussion we had today. There are too many details to list here, but the general idea is to break out Juggler into three modules:

• loopback-datasource-juggler - 8 points
• loopback-model - 8 points
• loopback-model-persistence - 8 points
• docs to wiki - 1 point
• examples to it's own repo/merge with other example - 2 points

Details will be listed in the individual waffle items

I will probably need a 2 point spike for each task (except move docs/examples) to make sure my estimates are correct.

@bajtos @raymondfeng ^

[ffflabs]

You can already register your own custom types with registerType. I did so based on geoPoint.

However, the custom behavior for geoPoint is hardcoded everywhere and it's pretty hard to override. You can't acrually extend geoPoint because its properties are directly accesed.

I also found out that, in postgres conector, in order to pass in custom functions to use PostGIS without resorting to the native pg conector, you have to use ParametrizedSQL, which is defined in loopback-connector.

Now, it turns out that if you don't have loopback-connector as a dependency, you won't be able to require it from boot scripts. If you do, loopback-postgres-connector includes it already, so two versions of loopback-connector may coexist, so instanceof doesn't work.

[superkhau]

@amenadiel Can you provide more details specifically as to how this is related to the Juggler modularization?

[superkhau]

@bajtos Please see my planning column in waffle for the individual tasks + spikes.