Secrets

​

Introduction

Managing secrets is a crucial part of any application. In this article, we will walk you through how to manage secrets in Blink.

​

Secrets in Blink

There are two ways of storing secrets in Blink:

• Using environment variables
• Using remote secret manager

​

Using environment variables

In order to use environment variables, you need to define them in the Blink configuration file. Here is an example of how to define an environment variable in the configuration file (this is partial config file. It doesn’t contain all the properties):

service:
  pipeline_id: ${PIPELINE_ID}

Here we see that pipeline_id is defined as an environment variable. When you start the Blink instance, you need to pass the environment variable to the instance. Here is an example of how to start the Blink instance with the environment variable:

PIPELINE_ID=1 blink start -c config.yaml

​

Using remote secret manager

Blink supports remote secret manager. Currently, Blink supports AWS Secrets Manager. To use remote secret manager, you need to configure AWS Secrets Manager to retrieve the secrets. Here is an example of how to define a secrets section in the configuration file.

In the example blow, we are combining both environment variables and remote secret manager. We are defining the storage_type as aws_secret_manager and providing the AWS credentials as environment variables.

secrets:
  storage_type: aws_secret_manager
  config:
    aws_secret_key: ${AWS_SECRET_KEY}
    aws_secret_key_id: ${AWS_SECRET_KEY_ID}
    aws_region: ${AWS_REGION}

As soon as we have this block in the configuration file, Blink will automatically fetch the secrets from AWS Secrets Manager.

​

Reffering secrets in the configuration file

In order to refference remote secret from the secret manager you must include it to the config file using the following syntax: #{secret.NAME_OF_THE_SECRET_IN_AWS_SECRET_MANAGER}

service:
  etcd:
    host: #{secret.etcd/host}

The secret etcd/host will be fetched from the AWS Secrets Manager and will be used in the configuration file.

​

Full example

service:
  pipeline_id: 1
  etcd:
    host: #{secret.etcd/host}
secrets:
  storage_type: aws_secret_manager
  config:
    aws_secret_key: ${AWS_SECRET_KEY}
    aws_secret_key_id: ${AWS_SECRET_KEY_ID}
    aws_region: ${AWS_REGION}
source:
  driver: playground
  config:
    data_type: market
    publish_interval: 1
    historical_batch: false
  stream_schema:
    - stream: market
      columns:
        - name: company
          nativeConnectorType: String
          databrewType: String
          nullable: false
          pk: false
        - name: currency
          nativeConnectorType: String
          databrewType: String
          nullable: false
          pk: false
processors:
  - driver: sql
    config:
      query: "select * from streams.market where currency = 'USD'"
sink:
  driver: stdout
  config: {}