parquet_decode

Available in: Cloud, Self-Managed

Decodes Parquet files into a batch of structured messages.

# Configuration fields, showing default values
label: ""
parquet_decode:
  handle_logical_types: v1

Fields

handle_logical_types

Set to v2 to enable enhanced decoding of logical types, or keep the default value (v1) to ignore logical type metadata when decoding values.

In Parquet format, logical types are represented using standard physical types along with metadata that provides additional context. For example, UUIDs are stored as a FIXED_LEN_BYTE_ARRAY physical type, but the schema metadata identifies them as UUIDs. By enabling v2, this processor uses the metadata descriptions of logical types to produce more meaningful values during decoding.

For backward compatibility, this field enables logical-type handling for the specified Parquet format version, and all earlier versions. When creating new pipelines, Redpanda recommends that you use the newest documented version.

Type: string

Default: v1

Option Summary

v1

No special handling of logical types

v2

  • TIMESTAMP - decodes as an RFC3339 string describing the time. If the isAdjustedToUTC flag is set to true in the parquet file, the time zone will be set to UTC. If it is set to false the time zone will be set to local time.

  • UUID - decodes as a string, i.e. 00112233-4455-6677-8899-aabbccddeeff.

 
# Examples:
handle_logical_types: v2

Examples

Reading Parquet Files from AWS S3

In this example we consume files from AWS S3 as they’re written by listening onto an SQS queue for upload events. We make sure to use the to_the_end scanner which means files are read into memory in full, which then allows us to use a parquet_decode processor to expand each file into a batch of messages. Finally, we write the data out to local files as newline delimited JSON.

input:
  aws_s3:
    bucket: TODO
    prefix: foos/
    scanner:
      to_the_end: {}
    sqs:
      url: TODO
  processors:
    - parquet_decode: {}

output:
  file:
    codec: lines
    path: './foos/${! meta("s3_key") }.jsonl'