adding documentation and an additional config option.

docs/clients/promtail/README.md

@@ -32,9 +32,19 @@ Just like Prometheus, `promtail` is configured using a `scrape_configs` stanza.
 drop, and the final metadata to attach to the log line. Refer to the docs for
 [configuring Promtail](configuration.md) for more details.
 
+## Loki Push API
+
+With the [Push Target](./configuration.md#push_config) Promtail can also expose the [Loki Push API](../../api.md#post-lokiapiv1push) to allow sending logs to a Promtail instance as if it were a Loki server. Those logs will then be sent to another Promtail or Loki server.
+
+There are a few instances where this might be helpful:
+
+* complex network infrastructures where many machines having egress is not desirable.
+* using the Docker Logging Driver and wanting to provide a complex pipeline or to extract metrics from logs.
+* serverless setups where many ephemeral log sources want to send to Loki, sending to a Promtail instance with `use_incoming_timestamp` == false can avoid out of order errors and avoid having to use high cardinality labels.
+
 ## Receiving logs From Syslog
 
-When the [Syslog Target](./scraping.md#syslog-target) is being used, logs
+When the [Syslog Target](./configuration.md#syslog_config) is being used, logs
 can be written with the syslog protocol to the configured port.
 
 ## Labeling and parsing

docs/clients/promtail/configuration.md

@@ -26,6 +26,7 @@ and how to scrape logs from files.
         * [tenant](#tenant)
     * [journal_config](#journal_config)
     * [syslog_config](#syslog_config)
+    * [push_config](#push_config)
     * [relabel_config](#relabel_config)
     * [static_config](#static_config)
     * [file_sd_config](#file_sd_config)
@@ -270,6 +271,9 @@ job_name: <string>
 # Describes how to receive logs from syslog.
 [syslog: <syslog_config>]
 
+# Describes how to receive logs via the Loki push API, (e.g. from other Promtails or the Docker Logging Driver)
+[push: <push_config>]
+
 # Describes how to relabel targets to determine if they should
 # be processed.
 relabel_configs:
@@ -716,6 +720,31 @@ labels:
 * `__syslog_message_msg_id`: The [msgid field](https://tools.ietf.org/html/rfc5424#section-6.2.7) parsed from the message.
 * `__syslog_message_sd_<sd_id>[_<iana_enterprise_id>]_<sd_name>`: The [structured-data field](https://tools.ietf.org/html/rfc5424#section-6.3) parsed from the message. The data field `[custom@99770 example="1"]` becomes `__syslog_message_sd_custom_99770_example`.
 
+### push_config
+
+The `push_config` block configures Promtail to expose a [Loki push API](../../api.md#post-lokiapiv1push) server.
+
+Each job configured with a `push_config` will expose this API and will require a separate port.
+
+Note the `server` configuration is the same as [server_config](#server_config)
+
+
+
+```yaml
+# The push server configuration options
+[server: <server_config>]
+
+# Label map to add to every log line sent to the push API
+labels:
+  [ <labelname>: <labelvalue> ... ]
+
+# If promtail should pass on the timestamp from the incoming log or not.
+# When false promtail will assign the current timestamp to the log when it was processed
+[use_incoming_timestamp: <bool> | default = false]
+```
+
+See [Example Push Config](#example-push-config)
+
 ### relabel_config
 
 Relabeling is a powerful tool to dynamically rewrite the label set of a target
@@ -1159,3 +1188,34 @@ scrape_configs:
       - source_labels: ['__syslog_message_hostname']
         target_label: 'host'
 ```
+
+## Example Push Config
+
+The example starts Promtail as a Push receiver and will accept logs from other Promtail instances or the Docker Logging Dirver:
+
+```yaml
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: http://ip_or_hostname_where_Loki_run:3100/loki/api/v1/push
+
+scrape_configs:
+- job_name: push1
+  push:
+    server:
+      http_listen_port: 3500
+      grpc_listen_port: 3600
+    labels:
+      pushserver: push1
+```
+
+Please note the `job_name` must be provided and must be unique between multiple `push` scrape_configs, it will be used to register metrics.
+
+A new server instance is created so the `http_listen_port` and `grpc_listen_port` must be different from the promtail `server` config section (unless it's disabled)
+
+You can set `grpc_listen_port` to `0` to have a random port assigned if not using httpgrpc.

pkg/promtail/scrapeconfig/scrapeconfig.go

@@ -75,6 +75,9 @@ type PushTargetConfig struct {
 
 	// Labels optionally holds labels to associate with each record received on the push api.
 	Labels model.LabelSet `yaml:"labels"`
+
+	// If promtail should maintain the incoming log timestamp or replace it with the current time.
+	KeepTimestamp bool `yaml:"use_incoming_timestamp"`
 }
 
 // DefaultScrapeConfig is the default Config.

pkg/promtail/targets/push/pushtarget.go

@@ -144,7 +144,13 @@ func (t *PushTarget) handle(w http.ResponseWriter, r *http.Request) {
 		}
 
 		for _, entry := range stream.Entries {
-			err := t.handler.Handle(filtered, time.Now(), entry.Line)
+			var err error
+			if t.config.KeepTimestamp {
+				err = t.handler.Handle(filtered, entry.Timestamp, entry.Line)
+			} else {
+				err = t.handler.Handle(filtered, time.Now(), entry.Line)
+			}
+
 			if err != nil {
 				lastErr = err
 				continue

pkg/promtail/targets/push/pushtarget_test.go

@@ -53,6 +53,7 @@ func TestPushTarget(t *testing.T) {
 			"pushserver": "pushserver1",
 			"dropme":     "label",
 		},
+		KeepTimestamp: true,
 	}
 
 	rlbl := []*relabel.Config{
@@ -107,6 +108,9 @@ func TestPushTarget(t *testing.T) {
 	// Spot check the first value in the result to make sure relabel rules were applied properly
 	require.Equal(t, expectedLabels, eh.Messages[0].Labels)
 
+	// With keep timestamp enabled, verify timestamp
+	require.Equal(t, time.Unix(99, 0).Unix(), eh.Messages[99].Time.Unix())
+
 	_ = pt.Stop()
 
 }