Add examples of ingest mapping for old OriginListing contract

[Zerim]

Submitting WIP PR for early feedback. I initially started with the spec, but I think the better approach for me is to work through several examples to get a broad feel for what the surface area of the API will be.

Comments in the example YAML file describe how those parts of the config might work.

Questions for feedback:

• Do we want one data source, entity, mapping strategy, per file or support many?
• Should ABI be included in-band (as shown) or out of band by some other means?
• How best to distinguish between event source data sources and data source that map directly? (is using "kind" clear to you)
• Does the overall structure/ hierarchy of the mapping make sense?

[Zerim]

Another question... Do we want the mapping to define everything required to index an entity (i.e. also point to the schema, and the raw data), or will there be some third configuration/ registry that links the mapping to raw data and a schema? I'm currently leaning towards maybe having the mapping define the entire data source...

[yanivtal]

Do we need to include the event params to disambiguate between overloaded events with different parameters?

[Zerim]

Sure, "event" field was just an example, but my thinking is that we would be able to filter by any of the fields listed here:
https://github.com/ethereum/wiki/wiki/JavaScript-API#callback-return

And perhaps metadata which can be filtered on as listed here:
https://github.com/ethereum/wiki/wiki/JavaScript-API#web3ethfilter

[yanivtal]

I would much rather see this be an IPLD URL with the ABI stored on IPFS.

[Zerim]

I guess we'll need to decide in general where these mappings are going to be stored. If the mapping is already stored in IPFS, and then registered on Ethereum, then it makes no difference if it's one IPFS file or many that the mapping comprises. For anything that we take a dependency on IPFS for, we'll need to worry about data availability, and how the network handles one or more files required to index a data source being missing.

[Zerim]

Maybe for now, an IPFS hash would be best as a reference since the IPLD spec is still a draft/ incomplete.

[yanivtal]

I don't think this would work with the schema registry idea

[yanivtal]

Are the quotes necessary?

[Zerim]

They are, because of the @ symbol... might be another way to escape it though.

[yanivtal]

Is this supposed to correspond to the getByIndex on line 32? How?

[Zerim]

The entity: @OriginProtocol/Listing parameter indicates to use the accessor for that entity that's listed above.

[Jannis]

This shouldn't be necessary if mappings are implicitly linked to entities by being part of a data source that describes what entities it provides.

[Zerim]

Yeah this example sort of evolved from a mappings spec, to a data source spec.

[Jannis]

It may be beneficial to look at mapping design in isolation first, before combining it with the concept of data sets and data sources.

[Zerim]

I had started with pure mappings, but understanding the information it would be combined with felt like important context to take into account.

[Jannis]

The way this is done here may not work. I feel "mapping" and "structure" are inherently linked. The challenge of keeping them separate is to find a way to define the mapping in way that is agnostic of the structure. So e.g. you'd have to describe the mapping for an Ethereum smart contract in a way that is completely unaware of smart contracts. If we can make that happen, it may be useful, but it may not be worth it or necessary.

Looking at the mapping in this file, you can already see that structure and mapping are not truly independent: both know it's a smart contract.

I also think data should not even exist here—unless this were a data source specification. But data sources are be managed in the data source registry and are unlikely to be defined in YAML.

[Zerim]

That's a good point on the structure vs. mappings, I went back and forth on that a few times myself. Part of my motivation for separating was reuse, for example if you have several mappings that get entities from the same data, do we want to redefine the structure each time in the spec?

As I mentioned above, after my earlier commits this did sort of evolve into a data source spec, so I apologize for the confusion there.

I do think there is value in having a data source spec that is in YAML, and that doesn't require a running Ethereum blockchain with an instance of a data source registry deployed. If the users of The Graph are querying data from IPFS, in a local network or trusted node setup (not requiring exchange of tokens), it would be a shame to require an entire running Ethereum blockchain just to store the data source.

[Jannis]

Ok, I can buy the idea of a data source registry that points to a file on IPFS for the actual spec of the data source. As long as the data source can be curated and voted on at the right degree of granularity.

[Jannis]

There is a problem with this approach: it does not describe how a listing entity is assembled from the result of this getter. Listings are a combination of a Listing struct and a JSON file on IPFS. An indexer needs to know that and it needs to know how these two pieces are combined into a ForSaleListing, HousingListing etc. entity.

[Jannis]

So we actually have at least two problems to solve here:

1. How does the result of getListing map to an entity (including all its fields)?
2. What is the type of this entity (e.g. ForSaleListing, HousingListing)?

[Zerim]

This example didn't account for IPFS, and was only trying to account for what an Ethereum or IPFS mappings might look like separately, but these are good points.

[Jannis]

Comment removed, I was wrong there.

[Jannis]

Also, @OriginProtocol/Listing is not a type. It's an interface. What's being updated in response to an event is a specific type of listing that needs to be determined based on the fields of the listing.

[Zerim]

Right, didn't deal with traversing the IPFS relationship w/ this example, but you are correct

[Jannis]

I see a bunch of problems with this proposal. I've taken some time to try to come up with an alternative that solely focuses on the mapping aspect, based on my experience with indexing the Origin listings.

I'll post it here for inspiration. It may not cover everything yet (e.g. fields may require optional pipelines to transform values) and it may be a little too specific in some places (e.g. sometimes the kind fields are namespaced, sometimes they aren't; that's a bit messy).

It addresses an important issue though: indexers need to know what to get out from data sources and how. That's what we aim to solve with mappings.

Note: I think versioning at the mapping spec level is good but we may need to version parts of the mappings too in order to be able to uprade them independently. E.g. the spec for each mapping kind should have its own version and the top-level mapping spec should only specify the minimum we need to be able to use arbitrary mapping kinds. In short: modularity is what we are after.

Another note: I think it's absolutely crucial that we evaluate any mapping design we come up with against CryptoKitties as well. The case studies demo document I wrote a while ago covers how the CK contracts are structured and they are significantly more complex than Origin listings.

specVersion: mapping/v1
mappings:
  - kind: ethereum/array-collection
    entity: ForSaleListing

    # Collection name and size define the state variables or
    # functions that provide the array collection and its length
    collectionName: listings
    collectionSize: listingsLength

    # Fields defines fields that can be retrieved from the
    # structs returned for each item in the array collection;
    #
    # Fields starting with an underscore (e.g. _ipfsHash) are
    # considered internal and are stripped of before passing
    # entities on to the indexer's data source event handler
    fields:
      lister:
        kind: ethereum/struct-field-by-index
        index: 0
      _ipfsHash:
        kind: ethereum/struct-field-by-index
        index: 1
      price:
        kind: ethereum/struct-field-by-index
        index: 2
      unitsAvailable:
        kind: ethereum/struct-field-by-index
        index: 3

    # Extra fields are additional fields that can come from additional
    # smart contract function calls, external files e.g. on IPFS and more;
    # they can refer to values of regular fields
    extraFields: 
      id:
        kind: ethereum/array-collection-item-index
      _schema:
        kind: ipfs/field-in-json-file
        hash:
          source: field
          name: _ipfsHash
        path: ["data", "schema"]
      category:
        kind: ipfs/field-in-json-file
        hash:
          source: field
          name: _ipfsHash
        path: ["data", "category"]
      description:
        kind: ipfs/field-in-json-file
        hash:
          source: field
          name: _ipfsHash
        path: ["data", "description"]

    # Transformations are applied to entities after all of their
    # fields have been collected; in this example, these items
    # are filtered based on a filter called is-for-sale-listing, which
    # is defined further below
    transformations:
      - kind: filter
        name: is-for-sale-listing

    # Events describe events that can lead to entities being reindexed
    events:
      - name: NewListing
        pipeline:
          - kind: ethereum/event-param
            name: _index
          - kind: ethereum/array-collection-item
          - kind: filter
            name: is-for-sale-listing
          - kind: update-entity
      - name: ListingPurchased
        pipeline:
          - kind: ethereum/event-param
            name: _index
          - kind: ethereum/array-collection-item
          - kind: filter
            name: is-for-sale-listing
          - kind: update-entity

    # Filters can be used in transformations and events and are defined
    # in a single place
    filters: 
      is-for-sale-listing:
        kind: filter-by-field-value
        field: _schema
        value: https://localhost:3000/schemas/for-sale.json

  - kind: ethereum/array-collection
    entity: AnnouncementListing
    collectionName: listings
    collectionSize: listingsLength
    fields:
      lister:
        kind: ethereum/struct-field-by-index
        index: 0
      _ipfsHash:
        kind: ethereum/struct-field-by-index
        index: 1
      price:
        kind: ethereum/struct-field-by-index
        index: 2
      unitsAvailable:
        kind: ethereum/struct-field-by-index
        index: 3
    extraFields: 
      id:
        kind: ethereum/array-collection-item-index
      _schema:
        kind: ipfs/field-in-json-file
        hash:
          source: field
          name: _ipfsHash
        path: ["data", "schema"]
      category:
        kind: ipfs/field-in-json-file
        hash:
          source: field
          name: _ipfsHash
        path: ["data", "category"]
      description:
        kind: ipfs/field-in-json-file
        hash:
          source: field
          name: _ipfsHash
        path: ["data", "description"]
    transformations:
      - kind: filter
        name: filter-announcement
    events:
      - name: NewListing
        pipeline:
          - kind: ethereum/event-param
            name: _index
          - kind: ethereum/array-collection-item
          - kind: filter
            name: is-announcement-listing
          - kind: update-entity
      - name: ListingPurchased
        pipeline:
          - kind: ethereum/event-param
            name: _index
          - kind: ethereum/array-collection-item
          - kind: filter
            name: is-announcement-listing
          - kind: update-entity
    filters: 
      is-announcement-listing:
        kind: filter-by-field-value
        field: _schema
        value: https://localhost:3000/schemas/announcement.json

[Zerim]

Okay @yanivtal @Jannis , PR has been updated and is ready for another look.

What's changed:

• I renamed my initial mapping to EventSourced-GenericDSL.yaml
• I added Jannis' suggestion as StateSnapshot-SpecializedDSL.yaml
• I added a Javascript mapping named ETL.js

The idea here is to explore the three possible approaches:

• A highly specialized DSL that handles every use case
• A generic DSL that handles simple use cases
• Using a generic programming language to define mappings.

These approaches can be used in combination with one another, though we'd likely want to settle on either a generic DSL or specialized DSL.

The Javascript mapping that I wrote, made some simplifying assumptions, primarily that we can switch on the category field in the IPFS data structure, since the example I'm looking at requires fetching data from localhost, which I'm guessing won't be their long-term implementation.

Overall I think the Javascript approach turned out OK. It's simple enough to reason about for me at least, and could probably be made even cleaner using async/await syntax. I'm generally still worried about the approach of reindexing everything every time we update, but it may be that we need to support that use case for certain types of contracts. I think it's still worth exploring an event-sourced approach in Javascript in addition.

[yanivtal]

Great start. I'd like to see this made a little more ergonomic. I agree async/await will help (should we do that now or on a subsequent PR?)

One thing that could be nice is using 0X's smart contract -> TypeScript type generator. It would be awesome if the values you got out of web3 were typed.

[yanivtal]

Where's this function defined?

[Zerim]

Should be loadEntity. Missed this in refactor.

[yanivtal]

Where's this used?

[Zerim]

All of these exported functions would be consumed as part of the module, presumably.

[Zerim]

Another thing I forgot to mention on the PR: This approach would suffer from memory problems if the data in the contract was sufficiently large (unlikely to happen on Ethereum), or if the input data was a large data set on IPFS.

For large datasets we could use a NodeJS Stream and take advantage of the backpresssure mechanism to address the problem.

[Jannis]

I don't think this should be stuck in a PR. Looks like a promising start.

[Jannis]

I have to say, I like the JS-based mappings. Definitely use async/await though.

[Zerim]

Another drawback of the current ETL.js approach shown is that it doesn't adequately decouple the mappings from the data. Maybe that's OK, but if this mapping was for an ERC20 contract, then it would need to be modified for each new ERC20 contract that is deployed. That might make it less convenient to distribute mappings as packages for really popular contracts, and decrease the amount of reuse we get from individual IPFS files/hashes.

[Zerim]

@Jannis A process question for you: I'm happy to merge this PR, but I'm concerned about losing the context of the conversation we're having here and merging something that still hasn't solved all the problems we've identified for the mappings.

Would you prefer to keep this PR alive as we evolve and explore different designs, until we reach a satisfactory v0.1, or would you prefer to get this merged and I can track the problems with the current approaches as Github Issues in this repo?

[Zerim]

Added #8 #9 #10 & #11 to track outstanding feedback/ action items on this PR. Merging after discussing with @Jannis and @yanivtal.