Ignore non index fields in default_field for Elasticsearch (elastic#9549

)

We recently started to use fields with `index: false`. By default all fields are defined as keyword internally which meant these fields were also added to the list of default_fields for Elasticsearch. With this PR these fields are now excluded.

(cherry picked from commit 77c34fc)

CHANGELOG.asciidoc

@@ -49,6 +49,7 @@ https://github.com/elastic/beats/compare/v6.5.0...6.x[Check the HEAD diff]
 - Log events at the debug level when dropped by encoding problems. {pull}9251[9251]
 - Refresh host metadata in add_host_metadata. {pull}9359[9359]
 - When collecting swap metrics for beats telemetry or system metricbeat module handle cases of free swap being bigger than total swap by assuming no swap is being used. {issue}6271[6271] {pull}9383[9383]
+- Ignore non index fields in default_field for Elasticsearch. {pull}9549[9549]
 
 *Auditbeat*
 

libbeat/template/processor.go

@@ -167,7 +167,9 @@ func (p *Processor) keyword(f *common.Field) common.MapStr {
 		fullName = f.Path + "." + f.Name
 	}
 
-	defaultFields = append(defaultFields, fullName)
+	if f.Index == nil || (f.Index != nil && *f.Index) {
+		defaultFields = append(defaultFields, fullName)
+	}
 
 	property["type"] = "keyword"
 
@@ -201,7 +203,9 @@ func (p *Processor) text(f *common.Field) common.MapStr {
 		fullName = f.Path + "." + f.Name
 	}
 
-	defaultFields = append(defaultFields, fullName)
+	if f.Index == nil || (f.Index != nil && *f.Index) {
+		defaultFields = append(defaultFields, fullName)
+	}
 
 	properties["type"] = "text"