ConsumerGroup does not support 'buffer' encoding, corrupts binary data

[mishan]

Environment

• Node version: 10.4.0
• Kafka-node version: 3.0.1
• Kafka version: 0.10.2

For specific cases also provide

• Number of Brokers: 1
• Number partitions for topic: 1

Include Sample Code to reproduce behavior

const { ConsumerGroup } = require('kafka-node');

var options = {
  kafkaHost: 'kafka:9092',
  groupId: 'TestGroup',
};

var consumerGroup = new ConsumerGroup(options, 'mytopic');

consumerGroup.on('message', (message) => {
  // XXX: observe that message.value is not a Buffer
  // there is no option to make it one via ConsumerGroup.
  // binary data is corrupted from the UTF8 conversion
  console.log(typeof message.value);
  console.log(message.value);
});

Output is that message.value is a string. If you use binary encoded data like with AVRO, then you will end up with corrupted data.

Consumer offers an encoding option, but ConsumerGroup does not. It looks like ConsumerGroup needs to include this option and allow it to be passed down.

[TysonAndre]

It looks like it exists, and a followup comment on another ticket says it's poorly documented, but I'm not familiar with this codebase. See #470 (comment)

I agree with including it in the README. Changing the code makes easier to read in my opinion, but doesn't seem absolutely necessary for calling code to work properly

Aside: It's also already mentioned in ConsumerGroupOptions of types/index.d.ts, so types/index.d.ts doesn't need to be updated.