Improve The CrowdSec Multi-Server Installation With HTTPS Between Agents

Posted on by admin

Prerequisites

This article is a follow-up from the Crowdsec multi-server setup. It applies to a configuration with at least two servers (referred to as server-1 and one of server-2 or server-3).

Goals

To address security issues posed by clear http communication in our previous crowdsec multi-server installation, we propose solutions to achieve communication between Crowdsec agents over encrypted channels. On top of that, the third solution allows server-2 or server-3 to trust server-1 identity, and avoid man-in -the -middle attacks.

Using self-signed certificates

Create the certificate

First we have to create a certificate. This can be achieved with the following one-liner.

openssl req -x509 -newkey rsa:4096 -keyout encrypted-key.pem -out cert.pem -days 365 -addext "subjectAltName = IP:172.31.100.242"

For now crowdsec is not able to ask for the passphrase of the private key when starting.  Thus we have the choice to decipher by hand the private key each time we start or reload crowdsec or store the key unencrypted. In any way to strip the passphrase one can do:

openssl rsa -in encrypted-key.pem -out key.pem

Then, the unencrypted key file can be safely deleted after Crowdsec is started.

Configure crowdsec for using a self-signed certificate

On server-1 we have to tell crowdsec to use the generated certificate. Hence, the  tls.cert_file and tls.key_file option in the api.server section of the following /etc/crowdec/config.yaml excerpt set to the generated certificate file.

api: server: log_level: info listen_uri: 10.0.0.1:8080 profiles_path: /etc/crowdsec/profiles.yaml online_client: # Crowdsec API credentials (to push signals and receive bad tls: cert_file: /etc/crowdsec/ssl/cert.pem key_file: /etc/crowdsec/ssl/key.pem

On the client side configuration changes happen in two files. First we have to modify /etc/crowdec/config.yaml to accept self-signed certificates by setting the insecure_skip_verify to true.

We have to change http for https in the  /etc/crowdsec/local_api_credentials.yaml file too in order to reflect the changes. This small change has to be done on all three servers (server-1, server-2 and server-3).

url: https://10.0.0.1:8080/ login: <login> password: <password>

Side note: Obviously using self-signed certificates doesn’t provide any confidence over ownership on the lapi server: servers using the service (server-2 or server-3 in our setup) are still vulnerable to man in the middle attack, but at least this setup provides encrypted communications. That’s the reason why the InsecureSkipVerify option is needed.

Using a certificate authority issued certificate

One could say that letsencrypt, or services like Amazon ACM could be leveraged to workaround the InsecureSkipVerify, by issuing a certificate for a fully qualified domain name that could be added to /etc/hosts or to a local dns server.  /etc/crowdsec/local_api_credentials.yaml could then be filled with this specified fully qualified domain name.

This indeed works and avoids the InsecureSkipVerify option to be set. This ensures that communication between client and server can’t be tampered with as long as dns configuration can be trusted, but should still be considered as a workaround.

Using a PKI

It’s not this blog post’s purpose to show how to configure and manage an SSL pki. Please have a look at openssl official openssl documentation. The simple pki scenario is enough to make the Crowdsec stuff work.

Following this documentation, there’s a few thing worth mentioning:

  • To be usable in our crowdsec tls scenario the certificate requests have to be issued with a subject alternative name corresponding to the IP of the Crowdsec LAPI server. This can be done by positioning the SAN environment variable when invoking openssl for the certificate request (cf step 3.3 in the openssl simple pki scenario):

SAN=IP:10.0.0.1 openssl req -new -config etc/server.conf -out certs/crowdsec.csr -keyout certs/crowdsec.key

  • The public part of the root and the signing certificates (bundle file created at step 4.5 in the openssl simple pki scenario) have to be added to the local certificate store before starting crowdsec agent. In this setup, this is required to connect to the LAPI server. There’re many ways to do so, golang sources specify where certificates are expected, or one can use the SSL_CERT_FILE environment variable in the systemd service file to specify where to find the certificate when launching crowdsec agent.

Conclusion

This article gives some highlights on how to ensure secure communications between our different crowdsec installations. The considered use-case is crowdsec installations in a private network, but this can be deployed as well on a public network with communication over the internet. In such a case a third party certificate would easily do the trick.

Depending on the needs, I proposed three different ways to achieve tls secure communications between crowdsec instances.

The first scenario with self-signed certificates if one only wants to ensure encrypted communication with no need for authentication. The second scenario proposed may only be considered as a workaround when one has the possibility to modify local dns resolutions. The third proposed scenario is the most complicated, but would fit in most use-cases and may be the way to go when security concerns are high.