Merge pull request #342 from DivanteLtd/feature/es7

Elastic 7 Search support

.gitignore

@@ -11,7 +11,10 @@ src/config.json
 config/local.json
 config/certs/*.pem
 var/magento2-sample-data/
+var/*
 .migrate
 *.iml
 /docker/elasticsearch/data/
 !/docker/elasticsearch/data/.gitkeep
+/docker/elasticsearch7/data/
+!/docker/elasticsearch7/data/.gitkeep

CHANGELOG.md

@@ -7,6 +7,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [1.11.0-rc.1] - UNRELEASED
 
 ### Added
+- Experimental Elastic 7 support - @pkarw (#342)
 - Output cache support with tagging and cache invalidate requests forwarding - @pkarw @resubaka (https://github.com/DivanteLtd/vue-storefront/issues/3367, #333)
 - Constant for Mailchimp subscription status - @KonstantinSoelch (#294)
 - mage2vs import now has optional `--generate-unique-url-keys` parameter which defaults to `false` to enable/disable the url key generation with name and id for categories - @rain2o (#232)
@@ -15,8 +16,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added jest support - @resubaka (#321)
 - Added caching factory and action factory for the image endpoint. This gives the possibility to use other services for caching or image processing - @resubaka (#317, #315)
 - Added support for tax calculation where the values from customer_tax_class_ids is used - @resubaka (#307)
+- The `db` context object - passed to every api endpoint now has two usefull methods: `getElasticClient` and `getRedisClient` for accesing the data stores - @pkarw (#328)
+- The `lib/utils` got two new methods `getStoreCode(req: Express.Request)` and `getStoreView(code: string)` for getting the current multistore context from `vue-storefront` frontend requests - @pkarw
+
+### Removed
+- The `scripts/seo.js` tool has been removed, the legacy `migrations` scripts have been removed, the unused legacy extensions (`gls-parcelshop-dk`, `postnord-parcelshop-dk`) - @pkarw (#342)
 
 ### Fixed
+- The way Elastic and Redis clients have been fixed and code duplication removed across the app - @pkarw (#327)
 - The `product.price_*` fields have been normalized with the backward compatibility support (see `config.tax.deprecatedPriceFieldsSupport` which is by default true) - @pkarw (#289)
 - The `product.final_price` field is now being taken into product price calcualtion. Moreover, we've added the `config.tax.finalPriceIncludesTax` - which is set to `true` by default. All the `price`, `original_price` and `special_price` fields are calculated accordingly. It was required as Magento2 uses `final_price` to set the catalog pricing rules after-prices - @pkarw (#289)
 - Force ES connections to use protocol config option - @cewald (#303, #304)

README.md

@@ -65,6 +65,68 @@ A [Kibana](https://www.elastic.co/products/kibana) service is available to explo
 
 At first access it will ask to specify an index pattern, insert `vue_storefront*`
 
+**Note:** Kibana is not provided with the Elastic 7 docker file. Please install it on your own when needed or check the [`es-head`](https://chrome.google.com/webstore/detail/elasticsearch-head/ffmkiejjmecolpfloofpjologoblkegm) Chrome plugin.
+
+## Elastic 7 support
+
+By default, Vue Storefront API docker files and config are based on Elastic 5.6. We plan to change the default Elastic version to 7 with the 1.11 stable release. As for now, the [Elastic 7 support](https://github.com/DivanteLtd/vue-storefront-api/pull/342) is marked as **experimental**. 
+
+**How to use it?**
+
+In order to use the new Elastic 7 please make sure your `vue-storefront-api` has been checked out from the [PR#342](https://github.com/DivanteLtd/vue-storefront-api/pull/342) branch.
+To start the Elastic 7 docker service please use the `docker-compose.elastic7.yml` file provided:
+
+```bash
+docker-compose -f docker-compose.elastic7.yml up
+```
+
+Then, please do change the `config/local.json` to start using the new Elastic API. Key properties in the `elasticsearch` section are: `indexTypes` and `apiVersion` (to be set to 7.1). If you're using the multistore configuration please make sure you adjusted the `storeViews.*.elasticsearch` section as well - per each separate store.
+
+```json
+  "elasticsearch": {
+    "host": "localhost",
+    "port": 9200,
+    "protocol": "http",
+    "min_score": 0.01,
+    "indices": [
+      "vue_storefront_catalog",
+      "vue_storefront_catalog_de",
+      "vue_storefront_catalog_it"
+    ],
+    "indexTypes": [
+      "product",
+      "category",
+      "cms_block",
+      "cms_page",
+      "attribute",
+      "taxrule",
+      "review"
+    ],
+    "apiVersion": "7.1"
+  }
+```
+
+Starting from [Elasitc 6 and 7](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html) we can have **just single** document type per single index. Vue Storefront used to have `product`, `category` ... types defined in the `vue_storefront_catalog`.
+
+From now on, we're using the separate indexes per each entity type. The convention is: `${indexName}_${entityType}`. If your' **logical index name** is `vue_storefront_catalog` then it will be mapped to the **physical indexes** of: `vue_storefront_catalog_product`, `vue_storefront_catalog_category` ...
+
+To take the advantage of this new logical->physical index distinction we've provided new Elastic tools: `db7`, `migrate7`, `dump7`, `restore7` tools. They can be used exactly the same way [like the old tools](https://docs.vuestorefront.io/guide/data/database-tool.html) were. The only distinction is that they work on separate indexes.
+
+**Create new index**
+
+Before restoreing or importing data you might want to create a new Elastic index with the proper data types/schema applied. You can run just the `yarn db7 new` command in order to do that.
+
+**Restore the data**
+
+After you ran the docker file and have Elastic7 up and running you might want to:
+
+a) restore the demo data by running `yarn restore7` and `yarn restore7_it; yarn restore7_de` for default multistores. The data is restored from `var/catalog_product.json`, `var/catalog_category.json` and so on...
+
+b) import the data from Magento to proper physical indexes. To do so, currently you can do this only with [mage2vuestorefront](https://github.com/DivanteLtd/mage2vuestorefront/pull/96).
+
+**Note:** After 1.11 stable release (around November, 2019) we'll **replace** the standard tools: `db`, `migrate`, `dump`, `restore` with the Elastic 7 equivalents and it will become the default.
+
+
 ## API access
 Catalog API calls are compliant with ElasticSearch (it works like a filtering proxy to ES). More on ES queries: [ElasticSearch queries tutorial](http://okfnlabs.org/blog/2013/07/01/elasticsearch-query-tutorial.html)
 

config/default.json

@@ -21,8 +21,6 @@
     "host": "localhost",
     "port": 9200,
     "protocol": "http",
-    "user": "elastic",
-    "password": "changeme",
     "min_score": 0.01,
     "indices": [
       "vue_storefront_catalog",
@@ -32,7 +30,8 @@
     "indexTypes": [
       "product",
       "category",
-      "cms",
+      "cms_block",
+      "cms_page",
       "attribute",
       "taxrule",
       "review"

config/elastic.schema.attribute.json

@@ -9,8 +9,7 @@
         "options": {
             "properties": {
                 "value": {
-                    "type": "text",
-                    "index": "not_analyzed"
+                    "type": "text"
                 }
             }
         }

config/elastic.schema.category.json

@@ -6,6 +6,7 @@
        "is_active": {"type": "boolean"},
        "product_count": {"type": "integer"},
        "parent_id": {"type": "integer"},
+       "position": {"type": "long"},
        "created_at": { 
            "type": "date",           
            "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"

config/elastic.schema.product.json

@@ -7,7 +7,7 @@
        "size": {"type": "integer"},
        "size_options": {"type": "integer"},
        "price": {"type": "float"},
-       "has_options": {"type": "boolean"},            
+       "has_options": {"type": "keyword"},            
        "special_price": {"type": "float"},
        "color": {"type": "integer"},
        "color_options": {"type": "integer"},
@@ -46,7 +46,7 @@
            "properties": {
                "url_key": {"type": "keyword"},
                "sku": {"type": "keyword"},
-               "has_options": {"type": "boolean"},
+               "has_options": {"type": "keyword"},
                "price": {"type": "float"},
                "special_price": {"type": "float"}
             }

docker-compose.elastic7.yml

@@ -0,0 +1,24 @@
+version: '3.0'
+services:
+  es7:
+    container_name: es7
+    build: docker/elasticsearch7/
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1    
+    volumes:
+      - ./docker/elasticsearch7/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
+    ports:
+      - '9200:9200'
+      - '9300:9300'
+    environment:
+      - discovery.type=single-node
+      - cluster.name=docker-cluster
+      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xmx512m -Xms512m"
+
+  redis:
+    image: 'redis:4-alpine'
+    ports:
+      - '6379:6379'

docker-compose.yml

@@ -3,13 +3,20 @@ services:
   es1:
     container_name: elasticsearch
     build: docker/elasticsearch/
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1    
     volumes:
       - ./docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
     ports:
       - '9200:9200'
       - '9300:9300'
     environment:
-      ES_JAVA_OPTS: "-Xmx512m -Xms512m"
+      - discovery.type=single-node
+      - cluster.name=docker-cluster
+      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xmx512m -Xms512m"
 
   kibana:
     build: docker/kibana/

docker/elasticsearch7/Dockerfile

@@ -0,0 +1,2 @@
+FROM docker.elastic.co/elasticsearch/elasticsearch:7.3.2
+

docker/elasticsearch7/config/elasticsearch.yml

@@ -0,0 +1,16 @@
+---
+## Default Elasticsearch configuration from elasticsearch-docker.
+## from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml
+#
+cluster.name: "docker-cluster"
+network.host: 0.0.0.0
+
+# minimum_master_nodes need to be explicitly set when bound on a public IP
+# set to 1 to allow single node clusters
+# Details: https://github.com/elastic/elasticsearch/pull/17288
+discovery.zen.minimum_master_nodes: 1
+
+## Use single node discovery in order to disable production mode and avoid bootstrap checks
+## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
+#
+discovery.type: single-node

migrations/.common.js

@@ -1,24 +1,9 @@
 
-let config = require('config')
-let kue = require('kue')
-let queue = kue.createQueue(Object.assign(config.kue, { redis: config.redis }))
-
-let es = require('elasticsearch')
-const esConfig = {
-  host: {
-    host: config.elasticsearch.host,
-    port: config.elasticsearch.port,
-    protocol: config.elasticsearch.protocol
-  },
-  log: 'debug',
-  apiVersion: config.elasticsearch.apiVersion,
-  requestTimeout: 1000 * 60 * 60,
-  keepAlive: false
-}
-if (config.elasticsearch.user) {
-  esConfig.httpAuth = config.elasticsearch.user + ':' + config.elasticsearch.password
-}
-let client = new es.Client(esConfig)
+const config = require('config')
+const kue = require('kue')
+const queue = kue.createQueue(Object.assign(config.kue, { redis: config.redis }))
+const es = require('../src/lib/elastic')
+const client = es.getClient(config)
 
 exports.db = client
 exports.queue = queue

package.json

@@ -13,16 +13,22 @@
     "build:copy:schema": "cpx 'src/**/*.schema*.json' dist",
     "start": "pm2 start ecosystem.json $PM2_ARGS",
     "db": "node scripts/db.js",
-    "seo": "node scripts/seo.js",
+    "db7": "node scripts/db7.js",
     "cache": "node ./scripts/cache",
     "mage2vs": "node scripts/mage2vs.js",
-    "restore": "node scripts/elastic.js restore",
+    "restore": "node scripts/elastic.js restore && npm run db rebuild",
+    "restore7": "node scripts/elastic7.js restore --output-index=vue_storefront_catalog && npm run db7 rebuild --indexName=vue_storefront_catalog",
+    "restore7_it": "npm run restore7 -- --input-file=var/catalog_it.json --output-index=vue_storefront_catalog_it && npm run db7 rebuild -- --indexName=vue_storefront_catalog_it",
+    "restore7_de": "npm run restore7 -- --input-file=var/catalog_de.json --output-index=vue_storefront_catalog_de && npm run db7 rebuild -- --indexName=vue_storefront_catalog_de",
     "restore_it": "npm run restore -- --input-file=var/catalog_it.json --output-index=vue_storefront_catalog_it && npm run db rebuild -- --indexName=vue_storefront_catalog_it",
     "restore_de": "npm run restore -- --input-file=var/catalog_de.json --output-index=vue_storefront_catalog_de && npm run db rebuild -- --indexName=vue_storefront_catalog_de",
     "restore2main": "npm run restore -- --output-index=vue_storefront_catalog",
     "dump": "node scripts/elastic.js dump",
     "dump_it": "npm run dump -- --input-index=vue_storefront_catalog_it --output-file=var/catalog_it.json",
     "dump_de": "npm run dump -- --input-index=vue_storefront_catalog_de --output-file=var/catalog_de.json",
+    "dump7": "node scripts/elastic7.js dump",
+    "dump7_it": "npm run dump7 -- --input-index=vue_storefront_catalog_it --output-file=var/catalog_it.json",
+    "dump7_de": "npm run dump7 -- --input-index=vue_storefront_catalog_de --output-file=var/catalog_de.json",
     "kue": "node scripts/kue.js",
     "kue-dashboard": "npm run kue dashboard -- --port=3050",
     "o2m": "node src/worker/order_to_magento2.js start",
@@ -46,6 +52,7 @@
   "author": "Piotr Karwatka <pkarwatka@divante.pl>",
   "license": "MIT",
   "dependencies": {
+    "@elastic/elasticsearch": "^7.3.0",
     "@google-cloud/storage": "^3.0.3",
     "ajv": "^6.4.0",
     "ajv-keywords": "^3.4.0",
@@ -55,8 +62,7 @@
     "compression": "^1.7.2",
     "config": "^1.30.0",
     "cors": "^2.8.4",
-    "elasticdump": "^3.3.12",
-    "elasticsearch": "^15.2.0",
+    "elasticdump": "^6.9.1",
     "email-check": "^1.1.0",
     "express": "^4.16.3",
     "fs-extra": "^8.1.0",
@@ -80,6 +86,7 @@
     "nodemailer": "^4.6.8",
     "oauth-1.0a": "^1.0.1",
     "pm2": "^2.10.4",
+    "query-string": "^6.8.3",
     "redis-tag-cache": "^1.2.1",
     "request": "^2.85.0",
     "request-promise-native": "^1.0.5",

scripts/db.js

@@ -17,7 +17,7 @@ program
     const tempIndex = originalIndex + '_' + Math.round(+new Date() / 1000)
 
     console.log(`** Creating temporary index ${tempIndex}`)
-    es.createIndex(common.db, tempIndex, (err) => {
+    es.createIndex(common.db, tempIndex, '', (err) => {
       if (err) {
         console.log(err)
       }
@@ -58,7 +58,7 @@ program
 
     console.log('** Hello! I am going to create NEW ES index')
     const indexName = cmd.indexName
-    es.createIndex(common.db, indexName, (err) => {
+    es.createIndex(common.db, indexName, '', (err) => {
       if (err) {
         console.log(err)
       }