Secrets
Explaining how to manage secrets in Blink.
Introduction
Managing secrets is a crucial part of any application. In this article, we will walk you through how to manage secrets in Blink.
Secrets in Blink
There are two ways of storing secrets in Blink:
- Using environment variables
- Using remote secret manager
Using environment variables
In order to use environment variables, you need to define them in the Blink configuration file. Here is an example of how to define an environment variable in the configuration file (this is partial config file. It doesn’t contain all the properties):
service:
pipeline_id: ${PIPELINE_ID}
Here we see that pipeline_id is defined as an environment variable. When you start the Blink instance, you need to pass the environment variable to the instance. Here is an example of how to start the Blink instance with the environment variable:
PIPELINE_ID=1 blink start -c config.yaml
Using remote secret manager
Blink supports remote secret manager. Currently, Blink supports AWS Secrets Manager. To use remote secret manager, you need to configure AWS Secrets Manager to retrieve the secrets. Here is an example of how to define a secrets
section in the configuration file.
In the example blow, we are combining both environment variables and remote secret manager. We are defining the storage_type
as aws_secret_manager
and providing the AWS credentials as environment variables.
secrets:
storage_type: aws_secret_manager
config:
aws_secret_key: ${AWS_SECRET_KEY}
aws_secret_key_id: ${AWS_SECRET_KEY_ID}
aws_region: ${AWS_REGION}
As soon as we have this block in the configuration file, Blink will automatically fetch the secrets from AWS Secrets Manager.
Reffering secrets in the configuration file
In order to refference remote secret from the secret manager you must include it to the config file using the following syntax: #{secret.NAME_OF_THE_SECRET_IN_AWS_SECRET_MANAGER}
service:
etcd:
host: #{secret.etcd/host}
The secret etcd/host
will be fetched from the AWS Secrets Manager and will be used in the configuration file.
Full example
service:
pipeline_id: 1
etcd:
host: #{secret.etcd/host}
secrets:
storage_type: aws_secret_manager
config:
aws_secret_key: ${AWS_SECRET_KEY}
aws_secret_key_id: ${AWS_SECRET_KEY_ID}
aws_region: ${AWS_REGION}
source:
driver: playground
config:
data_type: market
publish_interval: 1
historical_batch: false
stream_schema:
- stream: market
columns:
- name: company
nativeConnectorType: String
databrewType: String
nullable: false
pk: false
- name: currency
nativeConnectorType: String
databrewType: String
nullable: false
pk: false
processors:
- driver: sql
config:
query: "select * from streams.market where currency = 'USD'"
sink:
driver: stdout
config: {}
Was this page helpful?