Timestamp Handling

[remithomasn7]

Environment:
I am using GRAFANA 7.0.6, LOKI/PROMTAIL 1.5.0
They are deployed in docker container, in a docker network.
I have a local application, and my log file is bind in my Promtail container

Describe the bug
I have trouble to understand custom timestamp handling.

My app's log are based on a JSON format, as follows :
{"time":"Wed Jul 15 11:13:07.315 CEST 2020", "level":"INF", "message":"Something Info", "_ctx":"demo"}

The Promtail configuration file content is:
server:
http_listen_port: 9080
grpc_listen_port: 0

positions:
filename: /tmp/positions.yaml

clients:

• url: http://172.21.0.3:3100/api/prom/push

scrape_configs:

• job_name: a_job
  static_configs:

  • targets:
    • localhost
      labels:
      job: a_job
      env: dev
      host: local
      path: /var/log/*log

  pipeline_stages:

  • match:
    selector: '{job="a_job"}'
    stages:
    • json:
      expressions:
      timestamp: time
      level: level
      message: message
      ctx: _ctx
    • timestamp:
      source: time
      format: Wed Jul 15 10:11:07.999 CEST 2020

I am able to set Loki as a Datasource in Grafana. I am able to see my logs in Grafana (from Loki), but I have trouble to understand how timestamps are handled.

Here is a screenshot of my current view in Grafana Explore:

I can see that my JSON log is well parsed. However, I do not understand why Promtail adds fields 'ts', and 'tsNS'. Those fields are confusing, I would like Promtail to use my scrap_config (where I specified my custom timestamp format).

Is there a way to remove Promtail timestamp ? How can I use my custom timestamp ?

In addition, when I use the DashBoard view instead of the Explore view, I am not able to set a visualization similar to the one in Explore. I am only able to use 'Table' and 'Logs'.

In 'Table' visualisation, the column 'Time ns' content is 'Invalid date', while the content of the column 'Time' is ok :

Is there a reason why the 'Time ns' is invalid ? How can I configure the 'Time' column to use the timestamp I specified in the Promtail configuration, instead of the Promtail default timestamp ?

Expected behavior
I would have expect Grafana to allow me to use my custom timestamp instead of using Promtail timestamp.

[fredczj]

I am testing Loki atm and I have same behavior. Does anyone have an idea how to make promtail use timestamp from scraped logs?

[owen-d]

If I recall correctly, the ts and tsNS fields are actually created by grafana from the timestamp's associated with each log line. It looks like your timestamp stage is causing the values to be parsed incorrectly, which then results in promtail assigning timestamps based on when it receives logs.

timestamp:
source: time
format: Wed Jul 15 10:11:07.999 CEST 2020

I'll adjust the docs to make it clearer, but the reference time in the format field must be Mon Jan 2 15:04:05 -0700 MST 2006. Internally, we use golang's time.Parse function. You can use this to test: https://play.golang.org/p/GrmdYnR4ooX

[remithomasn7]

Thanks for your answer.
I tried to switch on the format @owen-d suggested (Mon Jan 2 15:04:05 -0700 MST 2006, which seems to match UnixDate). I was able to synchronise ts and tsNs with my parsed timestamp.

@owen-d, you said that the format field must be Mon Jan 2 15:04:05 -0700 MST 2006, as it is not what is said in the official doc https://github.com/grafana/loki/blob/master/docs/clients/promtail/stages/timestamp.md, I tried with format RFC3339Nano. This time again, I was able to synch ts and tsNs with my parsed timestamp.

I think that the documentation is fine. The only part that would need to be clarified should be how to properly set a custom format. I understand it is possible, but we must follow a specific format.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.