QueryDSL #207

pomadchin · 2020-02-17T22:35:18Z

Overview

This PR adds a concept of RasterSource catalog that is possible to query via they special query DSL

Checklist

Description of PR is in an appropriate section of the CHANGELOG and grouped with similar changes if possible

Closes #162
Closes #163
Closes #204

pomadchin · 2020-02-17T22:36:15Z

core/src/main/scala/geotrellis/store/GeoTrellisRasterSourceLegacy.scala

 import geotrellis.vector._

+import java.time.ZonedDateTime
+
+case class LayerLegacy(id: LayerId, metadata: TileLayerMetadata[_], bandCount: Int) {


Since we have GeoTrellisLayers here, why not to keep all the changes here and after approval port them into GeoTrellis core

pomadchin · 2020-02-17T22:36:36Z

core/src/main/scala/geotrellis/store/GeoTrellisRasterSourceLegacy.scala

+    "layerName"  -> layerId.name,
+    "zoomLevel"  -> layerId.zoom.toString,
+    "bandCount"  -> bandCount.toString
+  ) ++ time.map(t => ("time", t.toString)).toMap


The idea is that we can add time into attributes in case it is a temporal layer

pomadchin · 2020-02-17T22:37:07Z

core/src/main/scala/geotrellis/store/GeoTrellisRasterSourceLegacy.scala

-        case (SpatialKeyClass, TileClass) => spatialTileRead
-        case (SpatialKeyClass, MultibandTileClass) => spatialMultibandTileRead
+        case (SpatialKeyClass, TileClass)            => spatialTileRead
+        case (SpatialKeyClass, MultibandTileClass)   => spatialMultibandTileRead


I definitely feel bad about GeoTrellisRasterSource

Because of this match or other reasons too ?

core/src/main/scala/geotrellis/store/query/QueryCollection.scala

ogc/src/main/scala/geotrellis/server/ogc/wcs/WcsParams.scala

pomadchin · 2020-02-19T20:13:58Z

ogc/src/main/scala/geotrellis/server/ogc/Render.scala

-      }
+      val points: List[Double] =
+        if (step == 0) List.empty[Double]
+        else Range.BigDecimal.inclusive(currentEdge, nextEdge, step).map(_.toDouble).toList


@moradology ^ I changed the syntax due to deprecation warnings

pomadchin · 2020-02-19T20:16:05Z

build.sbt

+    libraryDependencies ++= {
+      CrossVersion.partialVersion(scalaVersion.value) match {
+        case Some((2, scalaMajor)) if scalaMajor == 11 =>
+          Seq(circeJava8.value)


In Scala 2.12 java8.time codecs are built in, that is why we have a compat object

echeipesh · 2020-02-25T13:14:22Z

core/src/main/scala/geotrellis/store/query/package.scala

+  implicit class ProjectedExtentOps(self: ProjectedExtent) {
+    def intersects(projectedExtent: ProjectedExtent): Boolean = self.extent.intersects(projectedExtent.reproject(self.crs))
+    def covers(projectedExtent: ProjectedExtent): Boolean     = self.extent.covers(projectedExtent.reproject(self.crs))
+    def contains(projectedExtent: ProjectedExtent): Boolean   = self.extent.contains(projectedExtent.reproject(self.crs))
+  }


@pomadchin think these are worth just moving to GT main after this? Seems a shame that we still don't have a way to abstract over both extent and ProjectedExtent but also a shame to have to write utility code in apps.

@echeipesh 💯 agree, along with all GeoTrellisRasterSources enhancements that are implemented here and probably with QueryDSL as well

echeipesh · 2020-02-25T13:36:01Z

ogc/src/main/scala/geotrellis/server/ogc/wcs/GetCoverage.scala

            LayerExtent(IO.pure(algebra), IO.pure(simpleLayers), ConcurrentInterpreter.DEFAULT)
-        }
+        }.getOrElse(throw new Exception("bad interpreter"))


This needs to be a better exception message. I believe this will show up in response to request that can't be satisfied (ie: the layer name doesn't produce any sources). Does this error filter back up to WCS response? (I'll come back to double check).

I think you're correct; Originally it was just unsafe .get from a map & probably it deserves a separate issue to walk though all the exceptions we throw in the code of gt-server?

echeipesh · 2020-02-25T14:29:55Z

ogc-example/src/main/scala/geotrellis/server/ogc/conf/OgcServiceConf.scala

    val simpleLayers =
-      layerDefinitions.collect { case ssc@SimpleSourceConf(_, _, _, _) => ssc.model }
+      layerDefinitions.collect { case ssc @ SimpleSourceConf(_, _, _, _) => ssc.models }.flatten


This looks like it may result in each layer being listed once per time-stamp in GetCapabilities ?

echeipesh · 2020-02-25T14:41:44Z

core/src/main/scala/geotrellis/store/GeoTrellisRasterSourceLegacy.scala

+        case KeyBounds(minKey, maxKey) =>
+          (minKey.time.toInstant.toEpochMilli to maxKey.time.toInstant.toEpochMilli by temporalResolution).toList.map { time =>
+            new GeoTrellisRasterSourceLegacy(attributeStore, dataPath, sourceLayers, Some(ZonedDateTime.ofInstant(time, ZoneOffset.UTC)), targetCellType)
+          }


This unfortunately is not correct for temporal layers. There could be time buckets that have no records stored in them and there could be time buckets which have multiple records stored in them. The only way to know which is which is to have use store the used values in metadata. Thinking of a concrete suggestion for this.

@echeipesh provide time as an input into this build function? (this can be a part of configuration)

But what to do if you don't know anything about the temporal layer?

Can't really report anything you don't know. The alternative is that you report a date for which there is no data and query comes back empty.

echeipesh · 2020-02-25T14:43:49Z

core/src/main/scala/geotrellis/store/GeoTrellisRasterSourceLegacy.scala

-        case (SpatialKeyClass, TileClass) => spatialTileRead
-        case (SpatialKeyClass, MultibandTileClass) => spatialMultibandTileRead
+        case (SpatialKeyClass, TileClass)            => spatialTileRead
+        case (SpatialKeyClass, MultibandTileClass)   => spatialMultibandTileRead


Because of this match or other reasons too ?

moradology · 2020-02-25T15:13:17Z

core/src/main/scala/geotrellis/store/query/QueryF.scala

+  @JsonCodec case class Between[A](from: ZonedDateTime, to: ZonedDateTime, fieldName: String = "time") extends QueryF[A]
+  @JsonCodec case class WithName[A](name: String) extends QueryF[A]
+  @JsonCodec case class WithNames[A](names: Set[String]) extends QueryF[A]
+  @JsonCodec case class Nothing[A]() extends QueryF[A]


Perhaps None is a better complement to All? Nothing seems to me like literally 'no thing' whereas this is just none of the instances of w.e particular thing we're talking about.

@moradology None is a reserved word; and the none function potentially can confuse users. I'm happy to try Any / Nothing though but probably any can also confuse users seeing that as a part of a Query DSL

moradology · 2020-02-25T15:42:56Z

ogc-example/src/main/scala/geotrellis/server/ogc/conf/RasterSourceConf.scala

@@ -25,7 +25,7 @@ import geotrellis.store.{GeoTrellisPath, GeoTrellisRasterSourceLegacy}
 *  a geotrellis-contrib [[RasterSource]].
 */
 sealed trait RasterSourceConf {
-  def toRasterSource: RasterSource
+  def toRasterSources: List[RasterSource]


This seems like a significant alteration, but I couldn't figure out what it was for (the only implementation I saw here just attached the lone RS to Nil)

It was to make use of the Repository interface. The whole thing was to make queriable temporal layers and a temporal layer is just a set of RasterSources. So having a List of RasterSources read from the configuration can allow us to create an instance of a Repository that we can query by whatever is supported by the dsl and by time as well.

echeipesh

Good enough for now. We will fix the behavior for querying space-time layers in next step.

pomadchin commented Feb 17, 2020

View reviewed changes

core/src/main/scala/geotrellis/store/query/QueryCollection.scala Outdated Show resolved Hide resolved

pomadchin commented Feb 17, 2020

View reviewed changes

ogc/src/main/scala/geotrellis/server/ogc/wcs/WcsParams.scala Show resolved Hide resolved

pomadchin force-pushed the feature/query-dsl branch 8 times, most recently from 9ea8811 to de503be Compare February 19, 2020 20:12

pomadchin commented Feb 19, 2020

View reviewed changes

pomadchin changed the title ~~[WIP] QueryDSL~~ QueryDSL Feb 19, 2020

pomadchin requested review from moradology and echeipesh February 19, 2020 20:43

pomadchin force-pushed the feature/query-dsl branch 3 times, most recently from 4eff2d5 to c754bf7 Compare February 20, 2020 14:17

echeipesh self-assigned this Feb 20, 2020

pomadchin mentioned this pull request Feb 20, 2020

Recursion Schemes geotrellis/maml#52

Open

echeipesh reviewed Feb 25, 2020

View reviewed changes

moradology reviewed Feb 25, 2020

View reviewed changes

echeipesh mentioned this pull request Feb 25, 2020

Review common WCS/WMS errors #218

Open

moradology reviewed Feb 25, 2020

View reviewed changes

echeipesh approved these changes Feb 25, 2020

View reviewed changes

DSL examples

b869982

pomadchin added 22 commits February 25, 2020 12:28

Add query DSL examples

be96a01

First pass in implementing OGCSourceCollection

c4e8cf5

Code cleanup

f6bef0f

Fix Algebra types

e08a115

Implement QueryF API for all OGC services

66f33e4

Futher API simpliication

30f972d

Code cleanup

133580b

Minor code cleanup and headers generation

d9bb58e

Force cats dependencies since droste overrides them for Scala 2.11

7a3f9cd

Update CHANGELOG

15d367f

Rename SourceCollections to Repository

8774ee4

Add a GeoTrellis layer builder to derive temporal layers

72266d7

Cover QueryF with tests

b72b463

Add Circe Query codecs

ff43e9b

Fix deprecation warning

d1f985c

Add circe-java8 compact for Scala 2.11 build

1bdebe9

Add some eval for JSON sugar

7990f63

Fix tests for Scala 2.11

ecf46ac

Add test case

3050ad4

Remove unused type parameter

4bbdf78

Fix typo in circe compat object

8dba8ac

Get rid of an exception thrown

b5daf5e

pomadchin force-pushed the feature/query-dsl branch from ac1c4b5 to b5daf5e Compare February 25, 2020 17:29

pomadchin merged commit 89f297c into geotrellis:develop Feb 25, 2020

pomadchin deleted the feature/query-dsl branch June 10, 2021 20:02

pomadchin restored the feature/query-dsl branch June 10, 2021 20:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

QueryDSL #207

QueryDSL #207

pomadchin commented Feb 17, 2020 •

edited

Loading

pomadchin Feb 17, 2020

pomadchin Feb 17, 2020

pomadchin Feb 17, 2020

echeipesh Feb 25, 2020

pomadchin Feb 19, 2020 •

edited

Loading

pomadchin Feb 19, 2020 •

edited

Loading

echeipesh Feb 25, 2020

pomadchin Feb 25, 2020

echeipesh Feb 25, 2020

pomadchin Feb 25, 2020

echeipesh Feb 25, 2020

echeipesh Feb 25, 2020

pomadchin Feb 25, 2020

echeipesh Feb 25, 2020

echeipesh Feb 25, 2020

moradology Feb 25, 2020 •

edited

Loading

pomadchin Feb 25, 2020 •

edited

Loading

moradology Feb 25, 2020

moradology Feb 25, 2020

pomadchin Feb 25, 2020 •

edited

Loading

echeipesh left a comment

QueryDSL #207

QueryDSL #207

Conversation

pomadchin commented Feb 17, 2020 • edited Loading

Overview

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pomadchin Feb 19, 2020 • edited Loading

Choose a reason for hiding this comment

pomadchin Feb 19, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moradology Feb 25, 2020 • edited Loading

Choose a reason for hiding this comment

pomadchin Feb 25, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pomadchin Feb 25, 2020 • edited Loading

Choose a reason for hiding this comment

echeipesh left a comment

Choose a reason for hiding this comment

pomadchin commented Feb 17, 2020 •

edited

Loading

pomadchin Feb 19, 2020 •

edited

Loading

pomadchin Feb 19, 2020 •

edited

Loading

moradology Feb 25, 2020 •

edited

Loading

pomadchin Feb 25, 2020 •

edited

Loading

pomadchin Feb 25, 2020 •

edited

Loading