v2.2.0

Listener batch size handling

• Fail config validation if listener batch size is greater than
  1MB. NATS will only accept messages of up to 1MB in size.
• Ensure the listener will never send a batch to NATS that is bigger
  than the 1MB which NATS will accept. This was previously possible
  under some circumstances.

InfluxDB HTTP basic auth support

influx-spout writers can now authenticate to InfluxDB backends if
required. The username and password are read from two environment
variables (INFLUXDB_USER and INFLUXDB_PASS).

Write retries

influx-spout writers will now retry failed writes for a configurable
number of times. Retries are configured by the following new options:

• writer_retry_batches: The maximum number of batches that failed to
  write to InfluxDB to track for retry at any given time. The oldest retry batch will be
  discarded if necessary to keep the set of batches being retried within this
  limit. Default is 1. Set to 0 to disable write retries.
• writer_retry_interval: The amount of time between write retry
  attempts. Default is "10s".
• writer_retry_timeout: The maximum amount of time to keep retrying
  a given batch. Default is "1m"

Enforce maximum writer batch size

When writing a batch to InfluxDB it is possible for the batch size to
exceed the maximum batch size because incoming lines are unlikely to
exactly fit inside the desired batch size. A BatchSplitter is now used
to ensure that the configured batch size is never exceeded.

This is important because InfluxDB has a maximum body size that it will accept.

Other

• Fixed escape handling bugs in downsampler component.
• The Prometheus metrics published by each influx-spout component is
  now documented in README.md.
• InfluxDB Line Protocol unescaping is now about 4% faster.
• Batches recevied by the listener component will now have a newline
  added if it is missing.
• Fixed incorrect accounting of received batches for the UDP listener.
• Fix potential issues with batch memory growth. Batch buffers now
  grow by at least the maximum UDP packet size to ensure that there is
  always sufficient memory available.

v2.1.0

New downsampler component

The downsamper is useful for creating rolled up versions of metrics streams for long term archive.

It reads measurements from one or more NATS subjects, averaging numeric fields over a sampling period (as set by downsample_period), emitting the averaged values every sampling period. For non-numeric fields, the last value seen for each sampling period is emitted. Measurements are emitted on clean time boundaries regardless of when the downsampler was started.

Downsampled lines are emitted to a NATS subject with the same name as the input subject but with a suffix as set by the downsample_suffix configuration option (default is -archive)

filter: New "tags rule type

A new rule type has been introduced which allows matching against one or more exact tags on measurement lines. The approach taken is much faster than using regexes to achieve the same kinds of matches (80-1000% depending on number tags to match and line size). This approach is also much safer as matching is independent of tag order and matches tag keys and values precisely.

filter: Tag sorting

The filter now orders tag keys in all measurement lines passing through it. This ensures the best performance when lines are inserted into InfluxDB and is also required for the downsampler component.

filter: Avoid unnecessary escaping

Unescaping of lines is now avoided where possible avoiding unnecessary computation and memory allocation. This improves performance significantly in many cases.

filter: Measurement name hash caching

The hash of measurement names in lines are now cached. This speeds up matching when there are multiple "basic" filter rules.

Other

• Fixed invalid TOML in README
• Fix race in probes package tests
• Retry gnatsd startup in tests (a common reason for CI failure)
• Automatically manage probe port allocations in end-to-end test
• Generate releases out of Travis CI

v2.0.0

Configuration cleanup

Compatibility

Some significant changes to influx-spout's configuration file have
been made to improvement usability, clarity and consistency. These
changes break compatibility with the previous configuration file
format. Some configuration options have been renamed, some take a
different value type and one has been removed.

Here's a summary of the changes:

Old Name	New Name	Old Format	New Format
batch_max_mb	batch_max_size	integer (MB)	string, flexible units (e.g. "20MB")
nats_pending_max_mb	nats_max_pending_size	integer (MB)	string, flexible units (e.g. "1GB")
read_buffer_bytes	read_buffer_size	integer (bytes)	string, flexible units (e.g. "512K")
batch	batch_max_count	integer count	(unchanged)
listener_batch_size	Removed	-	use batch_max_size instead
batch_max_secs	batch_max_age	integer (seconds)	string, flexible units (e.g. "5m")
max_time_delta_secs	max_time_delta	integer (seconds)	string, flexible units (e.g. "30s")
write_timeout_secs	write_timeout	integer (seconds)	string, flexible units (e.g. "10s")

Listener batch aging

The listener component will now send its batch if the age of the batch
exceeds a configured limit, as set by the batch_max_age option. This
prevents batched lines from being held up if the rate of data received
by the listener drops. The default listener batch age is 5 minutes.

Hostname included in internal metrics

The metrics that each influx-spout component exposes (in Prometheus
format) now include a hostname label.

Configurable stats interval

The interval at which each influx-spout component publishes its own
metrics is now configurable using the new stats_interval
option. This was previously fixed at 3 seconds.

Configuration validation

Various errors in influx-spout's configuration are now checked
for. influx-spout will fail to start if a configuration error is
detected.

Performance improvements for internal stats

• Avoid unnecessary name lookups for index based metrics.
• Use atomic counter updates instead of mutex.
• stats package microbenchmark is now 78% faster.

Developer tooling for manual tests

• Added slow (fake) influxd tool (cmd/slowinfluxd). This is useful
  for testing how influx-spout behaves when dealing with slow
  consumers.
• Added watchmem tool for monitoring memory consumption.
• Documented how to run a manual slow consumer test.

Other

• Code structure: all binaries now live as subpackages under the cmd
  top level package.
• The writer and listener now share the same batch type instead of
  having independent but similar implementations.
• Cleaner handling for internal Prometheus labels.
• Fixed minor linter issues

---

Automated with GoReleaser
Built with go version go1.10.3 linux/amd64

v1.1.0

Changelog

9bec111 Merge pull request #82 from mjs/pprof-support
3fd57d9 Implement pprof support
3616d83 Merge pull request #81 from mjs/bump-perfcheck-reference
22b95b7 Bump the default perfcheck reference branch
258aed4 Fix HTTP listener test & benchmarks
84ccaa7 Merge pull request #80 from mjs/disable-benchmark-logs
633065d Merge pull request #79 from mjs/http-ts-precision
6c0f8f3 listener: Avoid unnecessary use of TestMain
870e084 perfcheck: No need to manually exclude the vendor directory
3dcf3d7 Suppress logs during benchmarks
5c2062f influx: Faster time conversions
d7a711e listener: Add benchmarks for HTTP listener
842bcce listener: Add support for HTTP "precision" argument
20a8916 influx: ExtractTimestamp and ExtractNanos
e6cd848 influx: Add functions for timestamp precision calculation
7414c61 filter: Parse timestamps using influx.ExtractTimestamp
ec77db3 influx: Timestamp extraction moved to new influx package

---

Automated with GoReleaser
Built with go version go1.10.3 linux/amd64

v1.0.0

Changelog

5c8fc8a Merge pull request #78 from oplehto/http_listener_204
bccbcc0 Changed HTTP listener to return 204 instead of 200
693993a Merge pull request #77 from mjs/http-listener-concurrency
cd7972a listener: Protect batch buffer from concurrent access
259feb2 Merge pull request #76 from mjs/listener-buf-handling
4a53ca9 listener: Prevent large HTTP POSTs from being broken up
04dd6f1 listener: Introduce batch abstraction
5aee4d3 Merge pull request #75 from mjs/k8s-probes
830386c Be systematic about ports used in tests
69b3777 Test readiness probes in end-to-end test
55bf27b monitor: Expose k8s probe endpoints
3fa8edd writer: Expose k8s probe endpoints
de9b218 Ensure component is stopped if readiness check fails
a536c49 filter: Expose k8s probe endpoints
b473079 listener: Expose k8s probe endpoints
8b32c5e probes: Return nullListener when port is 0
e2b485e probes: Make Probes an interface
0461ccf config: Add probe_port
ac7b76a probes: Added listener for k8s health probes
9e95499 Merge pull request #74 from mjs/multiple-filters
2ba1aef Merge pull request #73 from mjs/listener-failed-nats-publish
543bb9b filter: Allow for multiple filters
617a250 listener: Add failed_nats_publish metric
393a8c3 Merge pull request #71 from mjs/clean-up-writer-drop-metrics
d2cc5c0 writer: Clean up subscription dropped reporting
ca0c897 Merge pull request #70 from mjs/filter-nats-pending-limits
b2eb25e Merge pull request #69 from mjs/always-retry-nats-connections
75a5c51 filter: Use nats_pending_max_mb
f30ffb7 Always retry connections to NATS
f836098 Merge pull request #68 from mjs/filter-publish-errors
9be52aa filter: New "failed_nats_publish" metric
4ae2fed Merge pull request #67 from mjs/filter-nats-dropped-metric
3582e39 filter: Expose NATS dropped messages metric
840e8da Merge pull request #65 from mjs/writer-maxpending
14fed65 spouttest: Adjust end-to-end test to account for max_pending metric
29ee717 writer: Publish new max_pending metric
3992db0 stats: Add tracking of maximum values
203cc15 Merge pull request #64 from mjs/use-monitor
ee7defc Separate component type and name in Prometheus metrics
8f7c80e spouttest: Include the monitor in the end-to-end test
bfac4dd Document the monitor component
380fbfd config: Set a default port for the monitor.
f5e652b stats: Remove the now unused Stats.Clone method
1617812 lineformatter: Removed
56abc7f writer: Use Prometheus metric for subscription drop stats
e037211 filter: Clean up generation of per rule metrics
7782183 stats: Add CounterToPrometheus
b7e443f spouttest: Remove unused AssertRecvMulti helper
7cbff78 spouttest: Clearer failure messages from AssertMonitor
746055f writer: Emit metrics in Prometheus format
8f96283 filter: Emit monitor metrics in Prometheus format
7a80da7 listener: Convert to sending Prometheus metrics to monitor subject
849759e stats: Add SnapshotToPrometheus
4df0864 stats: Add Snapshot method
767b788 Merge pull request #62 from mjs/writer-test-reliability
8aa8500 Merge pull request #63 from mjs/remove-listener-state-metrics
0efbc53 listener: Remove unused state metrics
2d4ee5e Wait for fake influxd to have started
4c5d1a8 Merge pull request #61 from mjs/perfcheck-tweaks
340aebe benchcheck: Tweak output
bd8cee0 perfcheck: Have benchcheck always print results
64595a9 perfcheck: Try multiple (shorter) attempts
1c065ba Merge pull request #59 from jumptrading/monitor-test-failure
f568c8d monitor: Flush after subscribing
1b80854 Merge pull request #60 from jumptrading/fix-perfcheck-ref-branch
2e58392 Fetch reference branch in reference clone
9256be5 Merge pull request #57 from mjs/monitor-component
02f71fe monitor: Initial version of monitor implemented
5cf0926 prometheus: Add UpdateFromSet to MetricSet
021fe84 prometheus: Add ToBytes to MetricSet
b4234d0 prometheus: Implement ParseMetrics
a910bad Merge pull request #54 from mjs/prometheus-protocol
fe0045f prometheus: Implement MetricSet
0bfe7c8 Update testify to 1.2.1
c96cc1a Move prometheus helpers to own package
3f384e7 Implement turning metrics into Promethesus protocol lines
4cded0c monitor: Tag prometheus tests & add license headers
8c088e7 Move filter.fastParseInt to convert.ToInt
34be8c8 monitor: Implement Prometheus metric parsing
8cd2e23 Merge pull request #52 from oplehto/travis_slack
2a4b22d Added Travis Slack key
b1ffd23 Merge branch 'master' of https://github.com/jumptrading/influx-spout
23dd2ad Merge pull request #51 from mjs/pull-perfcheck-reference
1b6352a Merge pull request #50 from mjs/fix-travis-badge
55c3448 Pull perfcheck reference branch from origin
d3eeb31 Update TravisCI badge now that project is public
aa1869f Merge pull request #49 from jumptrading/go_report_card
eef3cac Merge pull request #48 from oplehto/filter_linelength_fix
f4c670d Merge pull request #47 from oplehto/nats_import_fix
5c82794 Update README.md
24f198f Merge pull request #45 from mjs/cleanup-writer-tests
b980faf filter: Incremented the minimum line length threshold
dada5c0 Changed the nats import to the correct path
9052b11 Changed the nats import to the correct path
2aafa8e writer: Clean up tests
cbc51ed Merge pull request #44 from mjs/remove-writer-state-metrics
a67dc47 Merge pull request #43 from mjs/timestamp-filtering
dd8a516 writer: Remove unused state metrics
ffc45b8 Update perfcheck revision to one with a realistic filter benchmark
7abf52f Reject metrics with timestamps that are either too old or too new
8162eb7 Remove previous benchmark output
15c080c Merge pull request #41 from jumptrading/travis_mods
b1f95eb Streamlined testing a bit
02605c0 Merge pull request #40 from jumptrading/improve_readme
1b35933 Update README.md
0fb95a6 Update README.md
703a17b Improved the beginning of the README a bit
5249beb Merge pull request #38 from mjs/enable-perfcheck
8257202 Add copyright header to benchcheck
aa09388 perfcheck: Multiple benchmark iterations & uses new benchcmp
15b178f Change benchcheck backend - now calls benchstat
a35b9dd Run performance regression checks in TravisCI
708e3d0 Merge pull request #39 from mjs/writer-tags
8197215 writer: Include InfluxDB details in metrics tags
4f1d79f Moved filter test assert helpers to spouttest to allow reuse
938fbe4 Merge pull request #28 from jumptrading/filter-config-name
fd9814a filter: Check stats output multiple times
fd57289 filter: Include name from config as tag in stats
7890cf8 Merge pull request #26 from jumptrading/go-1.10
42f18cf Specify exactly Go 1.10 as TravisCI not handling .x syntax
1f80fc1 Test with Go 1.10

---

Automated with GoReleaser
Built with go version go1.10.3 linux/amd64

v1.0.0-beta1

Merge pull request #25 from jumptrading/end-to-end-test

Implement an end-to-end test