Beautify `kcat` consumer output by piping to `jq`

- 2 minutes read - 278 words

I was working on some demos recently on the Apache Kafka source connectors (hi #KafkaSummit!), and trying to display the stream of changes in the resulting Apache Kafka topic.

My standard approach is to vertically divide the terminal and have on the left the database connection where I can issue commands and on the right the output of kcat. This setup works really well for simple messages, but when using it with a Debezium source connector, the amount of fields pushed in a single message makes it hardly parsable by a live audience watching a screen.

Unparsable JSON

In the above image you can notice that the kcat output is a JSON format without syntax highlight making it almost unreadable.

jq to the rescue

I knew and was using since long time jq a tool that allows you to beautify and parse JSON outputs. After installing you can just use it as

echo '{"name":"Francesco", "preferences":["pasta","pizza","espresso"]}' | jq '.'

and you get your amazing JSON displayed nicely

{
  "name": "Francesco",
  "preferences": [
    "pasta",
    "pizza",
    "espresso"
  ]
}

jq and kcat

Now, how can I apply jq to the kcat consumer output? I tried a direct pipe like:

kcat -F kcat.config -C \
    -t my_pgsource.public.players | jq

But unfortunately I only got the output comments without the actual data being displayed

% Reading configuration from file kcat.config
% Reached end of topic my_pgsource.public.players [0] at offset 3

Seen on screen as

No output of JSON

I struggled long time, but then reached out to a friend (kudos @rmoff) who pointed me to the right solution: the -u flag ( Unbuffered output) in kcat. Once added the -u flag I’m now able to see my beautified JSON!

Beautified JSON