Add import support to Mentix (#1332)

changelog/unreleased/mentix-import.md

@@ -0,0 +1,5 @@
+Enhancement: Add import support to Mentix
+
+This update adds import support to Mentix, transforming it into a **Mesh Entity Exchanger**. To properly support vendor site management, a new connector that works on a local file has been added as well.
+
+https://github.com/cs3org/reva/pull/1332

docs/content/en/docs/config/http/services/mentix/_index.md

@@ -7,7 +7,7 @@ description: >
 ---
 
 {{% pageinfo %}}
-Mentix (_**Me**sh E**nti**ty E**x**porter_) is a service to read mesh topology data from a source (e.g., a GOCDB instance) and export it to various targets like an HTTP endpoint or Prometheus.
+Mentix (_**Me**sh E**nti**ty E**x**changer_) is a service to read and write mesh topology data to and from one or more sources (e.g., a GOCDB instance) and export it to various targets like an HTTP endpoint or Prometheus.
 {{% /pageinfo %}}
 
 {{% dir name="prefix" type="string" default="mentix" %}}
@@ -18,24 +18,37 @@ prefix = "/mentix"
 {{< /highlight >}}
 {{% /dir %}}
 
-{{% dir name="connector" type="string" default="gocdb" %}}
-Mentix is decoupled from the actual source of the mesh data by using a so-called _connector_. A connector is used to gather the data from a certain source, which are then converted into Mentix' own internal format.
-
-Supported values are:
-
-- **gocdb** 
-The [GOCDB](https://wiki.egi.eu/wiki/GOCDB/Documentation_Index) is a database specifically designed to organize the topology of a mesh of distributed sites and services. In order to use GOCDB with Mentix, its instance address has to be configured (see [here](gocdb)).    
-
+{{% dir name="update_interval" type="string" default="1h" %}}
+How frequently Mentix should pull and update the mesh data. Supports common time duration strings, like "1h30m", "1d" etc.
 {{< highlight toml >}}
 [http.services.mentix]
-connector = "gocdb"
+update_interval = "15m"
 {{< /highlight >}}
 {{% /dir %}}
 
-{{% dir name="exporters" type="[]string" default="[webapi,cs3api,siteloc,promsd]" %}}
+## Connectors
+Mentix is decoupled from the actual sources of the mesh data by using so-called _connectors_. A connector is used to gather the data from a certain source, which are then converted into Mentix' own internal format.
+
+_Supported connectors:_
+
+- **gocdb** 
+The [GOCDB](https://wiki.egi.eu/wiki/GOCDB/Documentation_Index) is a database specifically designed to organize the topology of a mesh of distributed sites and services. In order to use GOCDB with Mentix, its instance address has to be configured (see [here](gocdb)).
+
+- **localfile**
+The [localfile](localfile) connector reads sites from a local JSON file. The file must contain an array of sites adhering to the `meshdata.Site` structure.
+
+## Importers
+Mentix can import mesh data from various sources and write it to one or more targets through the corresponding connectors.
+
+__Supported importers:__
+
+- **webapi**
+Mentix can import mesh data via an HTTP endpoint using the `webapi` importer. Data can be sent to the configured relative endpoint (see [here](webapi)).
+
+## Exporters
 Mentix exposes its gathered data by using one or more _exporters_. Such exporters can, for example, write the data to a file in a specific format, or offer the data via an HTTP endpoint.
 
-Supported values are:
+__Supported exporters:__
 
 - **webapi**
 Mentix exposes its data via an HTTP endpoint using the `webapi` exporter. Data can be retrieved at the configured relative endpoint (see [here](webapi)). The web API currently doesn't support any parameters but will most likely be extended in the future.
@@ -49,17 +62,3 @@ Mentix exposes its data via an HTTP endpoint using the `webapi` exporter. Data c
           - files:
              - '/usr/share/prom/sciencemesh_services.json'
   ```
-
-{{< highlight toml >}}
-[http.services.mentix]
-exporters = ["webapi", "promsd"]
-{{< /highlight >}}
-{{% /dir %}}
-
-{{% dir name="update_interval" type="string" default="1h" %}}
-How frequently Mentix should pull and update the mesh data. Supports common time duration strings, like "1h30m", "1d" etc.
-{{< highlight toml >}}
-[http.services.mentix]
-update_interval = "15m"
-{{< /highlight >}}
-{{% /dir %}}

docs/content/en/docs/config/http/services/mentix/cs3api/_index.md

@@ -13,7 +13,15 @@ The CS3API exporter exposes Mentix data in a format that is compliant with the C
 {{% dir name="endpoint" type="string" default="/" %}}
 The endpoint where the mesh data can be queried.
 {{< highlight toml >}}
-[http.services.mentix.cs3api]
+[http.services.mentix.exporters.cs3api]
 endpoint = "/data"
 {{< /highlight >}}
 {{% /dir %}}
+
+{{% dir name="enabled_connectors" type="[]string" default="*" %}}
+A list of all enabled connectors for the exporter.
+{{< highlight toml >}}
+[http.services.mentix.exporters.cs3api]
+enabled_connectors = ["gocdb"]
+{{< /highlight >}}
+{{% /dir %}}

docs/content/en/docs/config/http/services/mentix/gocdb/_index.md

@@ -13,15 +13,15 @@ When using the [GOCDB](https://wiki.egi.eu/wiki/GOCDB/Documentation_Index) conne
 {{% dir name="address" type="string" default="" %}}
 The address of the GOCDB instance; must be a valid URL (e.g., http://gocdb.uni-muenster.de). **Note:** The public API must be reachable under `<address>/gocdbpi/public`.
 {{< highlight toml >}}
-[http.services.mentix.gocdb]
+[http.services.mentix.connectors.gocdb]
 address = "http://gocdb.example.com"
 {{< /highlight >}}
 {{% /dir %}}
 
 {{% dir name="scope" type="string" default="SM" %}}
 The scope to use for filtering sites and services.
 {{< highlight toml >}}
-[http.services.mentix.gocdb]
+[http.services.mentix.connectors.gocdb]
 scope = "SM"
 {{< /highlight >}}
 {{% /dir %}}

docs/content/en/docs/config/http/services/mentix/localfile/_index.md

@@ -0,0 +1,19 @@
+---
+title: "localfile"
+linkTitle: "localfile"
+weight: 10
+description: >
+    Configuration for the local file connector of the Mentix service
+---
+
+{{% pageinfo %}}
+The local file connector reads sites from a local JSON file adhering to the `meshdata.Site` structure.
+{{% /pageinfo %}}
+
+{{% dir name="file" type="string" default="" %}}
+The file path.
+{{< highlight toml >}}
+[http.services.mentix.connectors.localfile]
+file = "/etc/reva/sites.json"
+{{< /highlight >}}
+{{% /dir %}}

docs/content/en/docs/config/http/services/mentix/promsd/_index.md

@@ -13,15 +13,23 @@ When using the Prometheus SD exporter, the output filenames have to be configure
 {{% dir name="metrics_output_file" type="string" default="" %}}
 The target filename of the generated Prometheus File SD scrape config for metrics.
 {{< highlight toml >}}
-[http.services.mentix.promsd]
+[http.services.mentix.exporters.promsd]
 metrics_output_file = "/var/shared/prometheus/sciencemesh.json"
 {{< /highlight >}}
 {{% /dir %}}
 
 {{% dir name="blackbox_output_file" type="string" default="" %}}
 The target filename of the generated Prometheus File SD scrape config for the blackbox exporter.
 {{< highlight toml >}}
-[http.services.mentix.promsd]
+[http.services.mentix.exporters.promsd]
 blackbox_output_file = "/var/shared/prometheus/blackbox.json"
 {{< /highlight >}}
 {{% /dir %}}
+
+{{% dir name="enabled_connectors" type="[]string" default="*" %}}
+A list of all enabled connectors for the exporter.
+{{< highlight toml >}}
+[http.services.mentix.exporters.promsd]
+enabled_connectors = ["gocdb"]
+{{< /highlight >}}
+{{% /dir %}}

docs/content/en/docs/config/http/services/mentix/siteloc/_index.md

@@ -13,7 +13,15 @@ The Site Locations exporter exposes location information of all sites to be cons
 {{% dir name="endpoint" type="string" default="/" %}}
 The endpoint where the locations data can be queried.
 {{< highlight toml >}}
-[http.services.mentix.siteloc]
+[http.services.mentix.exporters.siteloc]
 endpoint = "/loc"
 {{< /highlight >}}
 {{% /dir %}}
+
+{{% dir name="enabled_connectors" type="[]string" default="*" %}}
+A list of all enabled connectors for the exporter.
+{{< highlight toml >}}
+[http.services.mentix.exporters.siteloc]
+enabled_connectors = ["gocdb"]
+{{< /highlight >}}
+{{% /dir %}}

docs/content/en/docs/config/http/services/mentix/webapi/_index.md

@@ -7,13 +7,55 @@ description: >
 ---
 
 {{% pageinfo %}}
-The WebAPI exporter exposes the _plain_ Mentix data via an HTTP endpoint.
+The WebAPI of Mentix supports both importing and exporting of mesh data via an HTTP endpoint. Both the im- and exporter are configured separately.
 {{% /pageinfo %}}
 
+## Importer
+
+The WebAPI importer receives a single _plain_ Mentix site through an HTTP `POST` request; service types are currently not supported.
+
+The importer supports two actions that must be passed in the URL:
+```
+https://sciencemesh.example.com/mentix/webapi/?action=<value>
+```
+Currently, the following actions are supported:
+- `register`: Registers a new site
+- `unregister`: Unregisters an existing site
+
+For all actions, the site data must be sent as JSON data. If the call succeeded, status 200 is returned.
+
+{{% dir name="endpoint" type="string" default="/" %}}
+The endpoint where the mesh data can be sent to.
+{{< highlight toml >}}
+[http.services.mentix.importers.webapi]
+endpoint = "/data"
+{{< /highlight >}}
+{{% /dir %}}
+
+{{% dir name="enabled_connectors" type="[]string" default="" %}}
+A list of all enabled connectors for the importer. Must always be provided.
+{{< highlight toml >}}
+[http.services.mentix.importers.webapi]
+enabled_connectors = ["localfile"]
+{{< /highlight >}}
+{{% /dir %}}
+
+## Exporter
+
+The WebAPI exporter exposes the _plain_ Mentix data via an HTTP endpoint.
+
 {{% dir name="endpoint" type="string" default="/" %}}
 The endpoint where the mesh data can be queried.
 {{< highlight toml >}}
-[http.services.mentix.webapi]
+[http.services.mentix.exporters.webapi]
 endpoint = "/data"
 {{< /highlight >}}
 {{% /dir %}}
+
+{{% dir name="enabled_connectors" type="[]string" default="*" %}}
+A list of all enabled connectors for the exporter.
+{{< highlight toml >}}
+[http.services.mentix.exporters.webapi]
+enabled_connectors = ["gocdb"]
+{{< /highlight >}}
+{{% /dir %}}

examples/mentix/mentix.toml

@@ -6,27 +6,33 @@ address = "0.0.0.0:9600"
 enabled_services = ["mentix"]
 
 [http.services.mentix]
-connector = "gocdb"
-exporters = ["webapi", "cs3api", "siteloc"]
-# Enable the Prometheus Service Discovery:
-# exporters = ["webapi", "cs3api", "siteloc", "promsd"]
 update_interval = "15m"
 
-[http.services.mentix.gocdb]
+[http.services.mentix.connectors.gocdb]
 address = "http://sciencemesh-test.uni-muenster.de"
 
-[http.services.mentix.webapi]
+# Sites can also be stored in a local file
+[http.services.mentix.connectors.localfile]
+file = "/usr/share/revad/sites.json"
+
+# Enable the WebAPI exporter
+[http.services.mentix.exporters.webapi]
 endpoint = "/"
 
-[http.services.mentix.cs3api]
-endpoint = "/cs3"
+# Enable the CS3API and Site Locations exporters
+[http.services.mentix.exporters.cs3api]
+[http.services.mentix.exporters.siteloc]
+# If this setting is omitted, all connectors will be used as data sources
+enabled_connectors = ["gocdb"]
 
-[http.services.mentix.siteloc]
-endpoint = "/loc"
+# Enable the WebAPI importer
+[http.services.mentix.importers.webapi]
+# For importers, this is obligatory; the connectors will be used as the target for data updates
+enabled_connectors = ["localfile"]
 
 # Configure the Prometheus Service Discovery:
-# [http.services.mentix.promsd]
+[http.services.mentix.exporters.promsd]
 # The following files must be made available to Prometheus.
 # They can then be used as the file_sd source of a job.
-# metrics_output_file = "/usr/share/prom/sciencemesh_metrics.json"
-# blackbox_output_file = "/usr/share/prom/sciencemesh_blackbox.json"
+metrics_output_file = "/usr/share/prom/sciencemesh_metrics.json"
+blackbox_output_file = "/usr/share/prom/sciencemesh_blackbox.json"

internal/http/services/mentix/mentix.go

@@ -27,7 +27,6 @@ import (
 
 	"github.com/cs3org/reva/pkg/mentix"
 	"github.com/cs3org/reva/pkg/mentix/config"
-	"github.com/cs3org/reva/pkg/mentix/exporters"
 	"github.com/cs3org/reva/pkg/rhttp/global"
 )
 
@@ -60,9 +59,14 @@ func (s *svc) Prefix() string {
 }
 
 func (s *svc) Unprotected() []string {
-	// Get all endpoints exposed by the RequestExporters
+	// Get all endpoints exposed by the RequestExchangers
+	importers := s.mntx.GetRequestImporters()
 	exporters := s.mntx.GetRequestExporters()
-	endpoints := make([]string, len(exporters))
+
+	endpoints := make([]string, len(importers)+len(exporters))
+	for idx, importer := range importers {
+		endpoints[idx] = importer.Endpoint()
+	}
 	for idx, exporter := range exporters {
 		endpoints[idx] = exporter.Endpoint()
 	}
@@ -88,42 +92,70 @@ func parseConfig(m map[string]interface{}) (*config.Configuration, error) {
 	if err := mapstructure.Decode(m, &cfg); err != nil {
 		return nil, errors.Wrap(err, "mentix: error decoding configuration")
 	}
+	applyInternalConfig(m, cfg)
 	applyDefaultConfig(cfg)
 	return cfg, nil
 }
 
+func applyInternalConfig(m map[string]interface{}, conf *config.Configuration) {
+	getSubsections := func(section string) []string {
+		subsections := make([]string, 0, 5)
+		if list, ok := m[section].(map[string]interface{}); ok {
+			for name := range list {
+				subsections = append(subsections, name)
+			}
+		}
+		return subsections
+	}
+
+	conf.EnabledConnectors = getSubsections("connectors")
+	conf.EnabledImporters = getSubsections("importers")
+	conf.EnabledExporters = getSubsections("exporters")
+}
+
 func applyDefaultConfig(conf *config.Configuration) {
+	// General
 	if conf.Prefix == "" {
 		conf.Prefix = serviceName
 	}
 
-	if conf.Connector == "" {
-		conf.Connector = config.ConnectorIDGOCDB // Use GOCDB
+	if conf.UpdateInterval == "" {
+		conf.UpdateInterval = "1h" // Update once per hour
 	}
 
-	if len(conf.Exporters) == 0 {
-		conf.Exporters = exporters.RegisteredExporterIDs() // Enable all exporters
+	// Connectors
+	if conf.Connectors.GOCDB.Scope == "" {
+		conf.Connectors.GOCDB.Scope = "SM" // TODO(Daniel-WWU-IT): This might change in the future
 	}
 
-	if conf.UpdateInterval == "" {
-		conf.UpdateInterval = "1h" // Update once per hour
+	// Importers
+	if conf.Importers.WebAPI.Endpoint == "" {
+		conf.Importers.WebAPI.Endpoint = "/"
 	}
 
-	if conf.GOCDB.Scope == "" {
-		conf.GOCDB.Scope = "SM" // TODO(Daniel-WWU-IT): This might change in the future
+	// Exporters
+	addDefaultConnector := func(enabledList *[]string) {
+		if len(*enabledList) == 0 {
+			*enabledList = append(*enabledList, "*")
+		}
 	}
 
-	if conf.WebAPI.Endpoint == "" {
-		conf.WebAPI.Endpoint = "/"
+	if conf.Exporters.WebAPI.Endpoint == "" {
+		conf.Exporters.WebAPI.Endpoint = "/"
 	}
+	addDefaultConnector(&conf.Exporters.WebAPI.EnabledConnectors)
 
-	if conf.CS3API.Endpoint == "" {
-		conf.CS3API.Endpoint = "/cs3"
+	if conf.Exporters.CS3API.Endpoint == "" {
+		conf.Exporters.CS3API.Endpoint = "/cs3"
 	}
+	addDefaultConnector(&conf.Exporters.CS3API.EnabledConnectors)
 
-	if conf.SiteLocations.Endpoint == "" {
-		conf.SiteLocations.Endpoint = "/loc"
+	if conf.Exporters.SiteLocations.Endpoint == "" {
+		conf.Exporters.SiteLocations.Endpoint = "/loc"
 	}
+	addDefaultConnector(&conf.Exporters.SiteLocations.EnabledConnectors)
+
+	addDefaultConnector(&conf.Exporters.PrometheusSD.EnabledConnectors)
 }
 
 // New returns a new Mentix service.