Anonymization Gateway for Datagrid VCTR

Quick Start

The examples below set up the gateway in its default configuration, which can be used immediately to access the Datagrid API and to send telemetry data safely without disclosing any host identifying information. Simply configure the Datagrid clients to use your gateway host instead of the default feed.datagridsys.com and api.datargirdys.com.

Docker Image

The gateway is available as a Docker application container, which can be run on any system that has Docker (see References for info on installing Docker)

Start with new key and auto-generated self-signed certificate:

sudo docker pull datagridsys/agate
sudo mkdir /etc/dgri
sudo touch /etc/dgri/agate.conf
sudo docker run -d --restart=always -e https_proxy=$https_proxy -v /etc/dgri:/etc/dgri \
       -p 8080-8081:8080-8081/tcp --name agate --hostname agate datagridsys/agate \
       server --autokey mygateway.example.com

Note: in the above example, replace mygateway.example.com with hostname or IP address for the gateway

RPM from Datagrid Yum Repo (CentOS-7, RHEL-7)

Configure access to the Datagrid yum repository and install with yum:

sudo rpm -U https://s3.amazonaws.com/dgri-bits/download/dgri-rpm-repo-latest.noarch.rpm
sudo yum install -y epel-release
sudo yum install -y dgri-gateway

First run - start with new key and auto-generated self-signed certificate.

sudo dgri-gateway server --autokey mygateway.example.com

NOTE: this runs the server in the foreground. You can stop it with Ctrl-C and run it as a service with 'sudo systemctl start dgri-gateway'. It will also start automatically on system boot.

Standalone Executable

(NOTE: tar file not published yet, ignore this section unless you received the dgri-gateway tar package from Datagrid directly)

Use this option to run on OS-es that are not supported by a pre-made package or if you want to download and run the gateway without having root/administrator privilege. Note that while you do not need to be root to run the gateway, installing some of the prerequisites may require this.

curl https://s3.amazonaws.com/files.datargidsys.com/dgri-gateway-1.0.0a4.tar.gz | tar -zxv
cd dgri-gateway-1.0.0a4
mkdir -p ~/.dgri
touch ~/.dgri/agate.conf

Install prerequisites:

  • openssl (sudo apt-get install openssl , sudo yum install openssl, or similar )
  • python-crypto (sudo apt-get install python-crypto , sudo yum install python2-crypto, ... )
  • web.py (sudo apt-get install python-webpy, sudo pip install web.py; note web.py is not available as RPM)
  • pyOpenSSL (sudo apt-get install python-openssl, sudo pip install pyOpenSSL; or yum install python-openssl after setting up the Datagrid repository - see RPM install above.)

You can now run the dgri-gateway executable from the current directory as ./dgri-gateway. Startin with a new key and auto-generated self-signed certificate:

./dgri-gateway server --autokey mygateway.example.com

Invocation

Command-line format:

dgri-gateway [--nossl] (version | genkey | getkeys | getconfig | setconfig | gencert)
dgri-gateway server [--nossl][ --autokey fqdn.hostname ]

All commands except server are transient and exit immediately upon completion.

Sub-command descriptions

version

Displays program version and exits

genkey

Generates a new encryption key and adds it to the configured keys. Up to three keys are kept. If there are already 3 keys stored, the oldest one is deleted.

If there is a running server, it needs to be reloaded or restarted for the new key to take effect. See Signals.

getconfig

Prints the current configuration to stdout. This command is intended for configuration backup and to replicate the same configuration to multiple instances of the gateway. See setconfig for a special note about the server certificate, if using multiple concurrently running copies of the gateway.

The getconfig command output can be given to setconfig to restore the same exact configuration.

getkeys

Prints the encryption keys to stdout. Similarly to getconfig, this command is intended for configuration backup and to replicate the same configuration to multiple instances of the gateway.

The output of getkeys can be fed to setconfig, it will cause only the keys to be replaced, leaving the server certificate alone. Use getkeys + setconfig when you need to synchronize the keys between two instances that have different server certificates.

setconfig

reads configuration from stdin and saves it in the gateway's configuration directory (/etc/dgri, or ~/.dgri).

The text given on stdin can be either one of:

  • the output of a previous run of dgri-gateway getconfig, or
  • a server certificate in PEM format, with the private key included.

if using setconfig to set or update the server certificate, keep the following in mind:

  • the certificate's common name (CN=...) must match the hostname that will be used by clients to connect to it (i.e., the name used in the dgri client configuration files for the api_url and feed_url settings.). If using more than one instance of the gateway server, each one will likely have a different hostname, but it may still be convenient to use the same certificate for all of them. To do this, the "Certificate Subject Alternative Name" extension can be used, to define multiple DNS names that are matched by the same certificate. Alternatively, if the hostnames allow for this, a wildcard certificate can be used, e.g., *.gateways.example.com.
  • the certificate must include the private key in unencrypted form. The private key must be first in the file, followed by the certificate. If you use openssl req -x509 to produce a self-signed certificate and give it the same file name for both the -keyout and the -out options, it will write the file in the correct order (key first, then certificate).

server

Starts the server. The command will run in foreground, with error messages printed to 'stderr'. This is intended primarily use in a Linux container, where the containerized process is expected to stay in the foreground and not fork(). System task/session managers such as Upstart or Systemd also provide convenient means for running a foreground process as a daemon (and will also provide logging of 'stderr' into file).

Before starting the server, the configuration directory must be populated with all required files, unless the --autokey option is used. It is also possible to start the server without SSL, using --nossl option. The latter is not recommended, except in cases when the gateway is running in a protected network in which all hosts are trusted.

The --autokey option can be used for a quick setup of a new gateway server. If the configuration directory does not contain any keys, it will generate a new key and a self-signed server certificate and then start the server immediately. If there are keys in the configuration directory, --autokey has no effect and the server will start with the existing configuration. This behavior is useful when dgri-gateway is ran in a container, because the command line for the container is set on first run and cannot be changed.

If --autokey is specified on the command line, it must be followed by a hostname. The hostname is used as the subject of the generated certificate. The hostname should be the exact name under which the server host is known to the clients that will access the gateway. This hostname will typically be a fully-qualified DNS domain name, but this is not a requirement. If your environment uses host resolution via /etc/hosts, use whatever name refers to the server in the /etc/hosts files of the clients that will access it.

gencert

Creates a new self-signed certificate and saves it for use on the next start/re-start. If a certificate was already installed with a previous gencert command, start --auto-key or setconfig, it will be renamed (with a warning message showing the name of the file) and the new certificate will be written in its place.

Configuring Local Authentication

By default, the gateway does not do any request authentication and forwards all incoming requests, expecting them to be authenticated by the back-end server. This requires that all hosts that connect to it (both for telemetry data reporting and for API access) to be configured with your Datagrid account credentials. Such a setup is suitable for the cases where the gateway is used as a single point of outgoing connections to the Datagrid servers, for an otherwise closed network.

If the network is open or it is otherwise undesirable to configure all reporting hosts with the Datagrid login, the gateway can be set up to hold the Datagrid credentials and be the only host that does so. Hosts that connect to the gateway are then configured to use a different user ID and pasword, which are verified at the gateway.

Enabling Local Authentication

Create a temporary text file containing the following settings:

[auth]
handler = basic_local
api_username = username for API input
feed_username = username for Feed input
api_password = password for API input
feed_password = password for Feed input
api_remote_username = user ID for API output
feed_remote_username = user ID for Feed output
api_remote_password = password for API output
feed_remote_password = pass for Feed output

Note the file should be safeguarded, given that it contains login information (both for accessing the gateway and for your Datagrid account). Edit it on a trusted host and if needed set the file permissions to your own user only.

Use the setconfig command with this file as input, e.g.:

cat /tmp/authfile | dgri-gateway setconfig

The temporary file can now be deleted.

If the gateway daemon was running, it needs to be restarted for the configuration to take effect.

It is also possible to use setconfig to edit part of the configuration (e.g., to change only the local passwords):

printf '[auth]\n api_password = newpassword\n' | dgri-gateway setconfig

If the daemon is used as a standalone executable and you have direct access to its configuration directory (see Configuration below), the auth. configuration file can be created or edited manually, as well. It is located in /etc/dgri/agate-auth.conf or ~/.dgri/agate-auth.conf. NOTE the file does not have the cleartext local passwords (api_password and feed_password). If you edit it directly, you will find api_hash and feed_hash, instead. The hashes are in the form used by the GNU crypt() function (in SHA-512 mode) and can be replaced manually, if you know how to use the crypt utility or the crypt() library function to hash your passwords.

Configuration (standalone usage)

The following information is only for using the gateway as a standalone executable (ignore this section if using the gateway as a Docker container; in that case all configuration is done only with the keygen, getkeys, getconfig and setconfig commands.)

The configuration is stored either in /etc/dgri or in ~/.dgri (checked in this order. The first directory containing agate.conf is assumed to be the configuration directory and all other files are read from it). Only one of these directories should be created on the host where you run the gateway, to avoid mistakes. Note that some of the configuration is done by running the gateway executable itself and it needs to have the directory writable. In addition, when running the server, it will require that the sensitive files are readable only to itself, therefore /etc/dgri can be used only if running as 'root'.

Before running dgri-gateway for the first time, create the configuration directory and an empty config file, e.g.:

mkdir ~/.dgri
touch ~/.dgri/agate.conf

Create a random encryption key using this command:

./dgri-gateway genkey

Create a self-signed server certificate (or have a certificate signing request sent to a CA and use the certificate signed by your CA) and store it in the configuration directory as agate-cert.pem. The certificate must be in PEM format and contain the private key (not locked with a password).

agate.conf Configuration Options

The configuration settings are defined in agate.conf in the form key = value. Note that no quotes should be placed around value. Spaces before or after the = are allowed but not required.

Example (the built-in defaults):

api_listen   = *:8080
feed_listen  = *:8081

The following settings are defined:

Name Description
api_listen hostname/address and port to listen to for API requests. This can be specified either as host or as host:port. If port is not set, 8080 is used. if host is * (the default), the gateway listens on all network interfaces.
feed_listen hostname/address and port to listen to for telemetry sending requests. The format is the same as for api_listen. Default is *:8081.

Handling the Encryption Keys

The encryption keys are stored in the configuration directory under variable names, in the form agate.??.key. These files should not be edited manually, nor renamed or copied by any means other than one of:

  • using the getkeys, getconfig and setconfig commands
  • saving and restoring the key files (or the entire config directory) with tar, rsync (with the archive option -a) or another command that preserves all file attributes.

New keys can be added with the genkey command.

The gateway will use the latest key added with genkey to encrypt outgoing data. When retrieving encrypted data and decrypting it, it will use the key that was used to encrypt it (if it is still available), allowing for up to 2 'old' keys to be used for older data that was encrypted before the genkey was ran.

If using multiple concurrently running copies of the gateway that serve the same account, the key sets should be kept in synch between them with minimal periods of time where one copy might be running with keys that are different from the others, to avoid the possibility that decryption of retrieved data fails.

The following sequence can be used to do a near-simultaneous update of keys at several locations:

Make a new key at one of the locations, but do not reload the daemon. Then get the resulting new set of keys with getkeys:

dgri-gateway genkey
dgri-gateway getkeys >/tmp/newkeys

Copy the file to all other locations, update the keysets, but do not reload the daemon:

dgri-gateway setconfig </tmp/newkeys

Reload all instances of the daemon (best done with a tool that can run the same command on multiple remote locations via ssh):

killall -HUP dgri-gateway

Usage Examples - Package Install or Standalone Executable

In the examples below, the current user is assumed to be 'root' and the dgri-gateway is assumed to be accessible in the shell path (this will be the case if it was installed from a package).

If you have just downloaded the tar file and unpacked it as a regular user (not 'root'), you can adapt the examples below by running the executable as ./dgri-gateway and replacing /etc/dgri with ~/.dgri (note you should not have an /etc/dgri directory if running as non-root).

Generate key if needed and run in the background

# default config (empty file is OK), if running as root:
mkdir -p /etc/dgri ; touch /etc/dgri/agate.conf
# start in background
dgri-gateway server --autokey </dev/null >logfile 2>&1 &

Save configuration

( umask 077 ; touch saved-config ; ) # make it not readable to others
dgri-gateway getconfig >saved-config

Set up when you have a saved configuration

mkdir -p /etc/dgri ; touch /etc/dgri/agate.conf
dgri-gateway setconfig <saved-config
dgri-gateway server </dev/null >logfile 2>&1 &

Set up with new key, when you have your own server certificate

mkdir -p /etc/dgri ; touch /etc/dgri/agate.conf
dgri-gateway server genkey
dgri-gateway setconfig </tmp/my-certificate.pem
dgri-gateway server </dev/null >logfile 2>&1 &

Re-configuration - add a new random key

dgri-gateway genkey
killall -HUP dgri-gateway

Re-configuration while running - update configuration from file

dgri-gateway setconfig <saved-config
killall -HUP dgri-gateway

Re-configuration while running - replace expired certificate

dgri-gateway setconfig </tmp/my-new-certificate.pem
killall -HUP dgri-gateway

Usage Examples - Docker Container

In the examples below, the current directory is assumed to contain the gateway files.

The examples use agate for the container that runs the server. The name is arbitrary, you can choose your own.

Quick start with new key and auto-generated self-signed certificate

sudo docker run -d --restart=always -e https_proxy=$https_proxy -v /etc/dgri:/etc/dgri \
       -p [bind-addr:]4433-4434:8080-8081/tcp --name agate --hostname agate datagridsys/agate \
       server --autokey mygateway.example.com

(Note: replace 4433-4434 with the ports on which you want the gateway's input entry points to appear on your host. Replace the hostname with your host's name)

Save configuration

( umask 077 ; touch /tmp/saved-config ; ) # make it not readable to others
sudo docker run --rm --volumes-from agate datagridsys/agate getconfig >/tmp/saved-config

First run when you have a saved configuration

# create a config directory
# sudo mkdir /etc/dgri
# sudo touch /etc/dgri/agate.conf
# create the container, but don't run it
sudo docker create --name agate --hostname agate --restart=always -e https_proxy=$https_proxy -v /etc/dgri:/etc/dgri -p 8080-8081:8080-8081/tcp datagridsys/agate
# apply configuration
sudo docker run --rm -i -v /etc/dgri:/etc/dgri datagridsys/agate setconfig </tmp/saved-config
# run server
sudo docker start agate

Upgrade, keeping the configuration

sudo docker pull datagridsys/agate
sudo docker create --restart=always -e https_proxy=$https_proxy -v /etc/dgri:/etc/dgri \
       --name agate-new --hostname agate -p 8080-8081:8080-8081/tcp datagridsys/agate:latest
sudo docker stop agate
sudo docker start agate-new
# optional, rename or delete old:
sudo docker rename agate agate-old
sudo docker rename agate-new agate
sudo docker rm agate

Reconfigure - add new random key

sudo docker run --rm -v /etc/dgri:/etc/dgri datagridsys/agate genkey
sudo docker kill -s HUP agate

Reconfigure - set configuration from file

cat /tmp/saved-config | docker run --rm -v /etc/dgri:/etc/dgri -i datagridsys/agate setconfig
sudo docker kill -s HUP agate

Usage Examples - Alternative

(Docker Use Without Host Directory Mapping)

Quick Start

sudo docker run -d --restart=always -e https_proxy=$https_proxy -p 8080-8081:8080-8081/tcp \
   --name agate --hostname agate datagridsys/agate server --autokey mygateway.example.com

First run when you have a saved configuration

# create the container, but don't run it
sudo docker create --name agate --hostname agate -p 8080-8081:8080-8081/tcp datagridsys/agate
# apply configuration
sudo docker run --rm -i --volumes-from agate datagridsys/agate setconfig </tmp/saved-config
# run server
sudo docker start agate

Upgrade, keeping the configuration

sudo docker pull datagridsys/agate
sudo docker create --restart=always -e https_proxy=$https_proxy --volumes-from agate \
       --name agate-new --hostname agate -p 8080-8081:8080-8081/tcp datagridsys/agate:latest
sudo docker stop agate
sudo docker start agate-new
# optional, rename or delete old:
sudo docker rename agate agate-old
sudo docker rename agate-new agate
sudo docker rm agate

Reconfigure - add new random key

sudo docker run --rm --volumes-from agate datagridsys/agate genkey
sudo docker kill -s HUP agate

Reconfigure - set configuration from file

cat /tmp/saved-config | docker run --rm --volumes-from agate -i datagridsys/agate setconfig
sudo docker kill -s HUP agate

References

  • Docker
    • Main https://www.docker.com/
    • Install https://docs.docker.com/engine/installation/
    • Quick setup on Linux OS-es: curl -fsSL https://get.docker.com/ | sh