Storing offset

​

Introduction

Unlike databases like PostgreSQL, MySQL where we read the data from the WAL or BinLog which we can use to understand the latest checkpoing of the data, in cases with third-party services like AirTable or Stripe we must manully store the last checkpoint of the data we have read. This helps us to restart Blink instance or ptch it’s version without re-syncing all the data. This is where the offset storage comes in.

​

Offset storage in Blink

Offset storage in Blink implemented using Redis under the hood. Therefore you must provide Blink with the redis URI to store the offset. Here is an example of how to define the offset storage in the configuration file (this is partial config file. It doesn’t contain all the properties):

service:
  offset_storage_uri: redis://localhost:6379

​

Debugging offset storage

In case you want to check what is the last offset stored in the Redis you should look for the key with the following pattern: pipeline_${PIPELINE_ID}_stream_${NAME_OF_THE_STREAM}