Try on VM

This guide will walk you through a basic Camblet scenario. It will deploy Camblet into the kernel to assign strong identities to processes and transparently establish mTLS connections. Simple NGINX server with cURL will be used for demostration.

In this scenario there will be a NGINX server which will work as an echo server. On client side cURL will be used. All commands must be run inside the virtual machine. To return to Lima:

limactl shell quickstart

Install NGINX server:

sudo apt install nginx -y

Configure agent

Camblet consist of two building blocks:

Kernel module: Handles transparent TLS and enforces policies.
Agent: Signs certificates and collects metadata for processes.

The agent configuration resides in /etc/camblet/config.yaml.

By default, it looks like this:

agent:
  trustDomain: acme.corp
  defaultCertTTL: 2h
  metadataCollectors:
    procfs:
      enabled: true
      extractEnvs: false
    linuxos:
      enabled: true
    sysfsdmi:
      enabled: true
    azure:
      enabled: false
    ec2:
      enabled: false
    gcp:
      enabled: false
    kubernetes:
      enabled: false
    docker:
      enabled: false

In this file, the agent can be configured to use a trust domain of your choice, along with the certificate time-to-live (certTTL). Camblet support various metadata sources, those can be configured under the metadataCollectors block. Camblet can utilize metadata from these sources to identify processes and enforce policies. The procfs, linuxos, and sysfsdmi are enabled by default.

Let’s gather metadata for the NGINX web server that was just installed.

camblet agent augment $(pidof -s nginx)

linuxos:kernel:release:linux-6.5.0-14-generic
linuxos:kernel:version:#14-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:59:49 UTC 2023
linuxos:name:ubuntu
linuxos:version:Ubuntu 23.10
process:cmdline:nginx: worker process
process:gid:33
process:gid:additional:33
process:gid:effective:33
process:gid:real:33
process:name:nginx
process:pid:3731
process:uid:33
process:uid:effective:33
process:uid:real:33
sysfsdmi:bios:date:03/01/2023
sysfsdmi:bios:release:0.0
sysfsdmi:bios:vendor:EDK II
sysfsdmi:bios:version:edk2-stable202302-for-qemu
sysfsdmi:chassis:type:1
sysfsdmi:chassis:vendor:QEMU
sysfsdmi:chassis:version:pc-q35-8.2
sysfsdmi:product:name:Standard PC (Q35 + ICH9, 2009)
sysfsdmi:product:version:pc-q35-8.2

The command prints out every available metadata for the specified process. These metadata can be used in policies to describe identities and related parameters.

Create policy for NGINX

The policy can be written by hand, but the Camblet CLI can do the work for you with the generate-policy command. There are two mandatory parameters, the PID of the process and the workload ID.

Let’s run the Camblet CLI to generate a policy and save it into the default policy directory as nginx.yaml.

sudo camblet agent generate-policy $(pidof -s nginx) nginx | sudo tee /etc/camblet/policies/nginx.yaml

- certificate:
    ttl: 86400s
    workloadID: nginx
  connection:
    mtls: STRICT
  selectors:
  - process:binary:path: /usr/sbin/nginx
    process:gid: "33"
    process:name: nginx
    process:uid: "33"

Camblet will use the selectors to identify the NGINX server. The connection part configures the TLS settings where STRICT mTLS value means only clients with trusted certificates can communicate with it.

To verify that, let’s try connecting to NGINX with cURL.

curl localhost

curl: (56) Recv failure: Connection reset by peer

The connection failed, as it was supposed to. Since Camblet now protects NGINX with mTLS, so nothing can communicate with it without a trusted client certificate.

The protection can be simply checked by openssl s_client.

openssl s_client -connect localhost:80 2>/dev/null

CONNECTED(00000003)
---
Certificate chain
 0 s:CN = camblet-protected-workload
   i:CN = Camblet root CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Feb 21 13:44:20 2024 GMT; NotAfter: Feb 22 13:44:20 2024 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDSDCCAjCgAwIBAgIUGBEwCjEpqr+6IuiCLtk6TLIh2oIwDQYJKoZIhvcNAQEL
BQAwGjEYMBYGA1UEAxMPQ2FtYmxldCByb290IENBMB4XDTI0MDIyMTEzNDQyMFoX
DTI0MDIyMjEzNDQyMFowJTEjMCEGA1UEAxMaY2FtYmxldC1wcm90ZWN0ZWQtd29y
a2xvYWQwggEgMA0GCSqGSIb3DQEBAQUAA4IBDQAwggEIAoIBAQDCgZnRjiiSlrAZ
QqOr0C4o+2uWKqPQdFENNqp3qZAMkq6tSYlkEyf6q9KPFI+V9gOxwPkAFPamizTH
3d57974oBmNK2BwtYiyhLus0AnAVahSYgqefCJ7EyYXw62Ip93Bqjdvu+P4KLLHe
2Vdy/tT3T05aQTn9yguvhdQjIncUoPleU+UFFatv+/qkBUKQnr284ado+YQvTD2U
jCL/s/yhyhbZ0M3NugZn+dZuqDRB7dt6/rZOQ/McPJ0FzxZDK53dWz6xQb+5yGHm
YbJKNcGtPyPycKjhnmcEzowBT4L+uTnVPxSr8kP8KqzspfIBRLPC4vURyYwl9to9
fJBkY7AFAgEDo30wezAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUH
AwIGCCsGAQUFBwMBMB8GA1UdIwQYMBaAFKGmr3y8dJsskFwuowJaVVzBru2kMCkG
A1UdEQQiMCCGHnNwaWZmZTovL2FjbWUuY29ycC9lY2hvLXNlcnZlcjANBgkqhkiG
9w0BAQsFAAOCAQEAFm5svzOUQAMR79d+MbCOHtrixGzipPCuIa2locGo7vLk/mdk
PYSU1dQsR0pCBpXuSfuTIhLG1WvSN6weIHisVhikWKKq0WCw02MwhyyZoh0E3PXu
3iE+nY2w6RUCAR6Ok2goH0bw6vVpmHklFJl4XHSzmzVbvNwqg3IL1+lR2cf67Feo
N3Wmi0wD0Pe0E5Z5ObQ1uHaxOvoDl4OqBfu7M5HOyq1eUcxWyj15Zbp/aUq/Uil0
5YYSndYJ49nDAki5jWVjXPlnehtvVV0pHtSzuOcXDlOaBEpuAiadO6/gqPQFm4++
0uQsSZHkbGhwqHCfsBVudroOusDXAKvDIuLmwA==
-----END CERTIFICATE-----
subject=CN = camblet-protected-workload
issuer=CN = Camblet root CA
---
Acceptable client certificate CA names
CN = Camblet root CA
Client Certificate Types: RSA sign
Requested Signature Algorithms: RSA+SHA256:RSA+SHA384:RSA+SHA512:RSA+SHA224:RSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:RSA+SHA384:RSA+SHA512:RSA+SHA224
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1294 bytes and written 398 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 64217D9B0EE7BE52F229F9D30BD7949C3494019E957A76C9385F60C146030F09
    Session-ID-ctx:
    Master-Key: 0C1F47031A5C1408C3B85D09F492FAC3086112F5A8AA800145AB1E49163A446A09CD56F7EC59EF195453E1E158EB0A25
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1708523445
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
---

Create policy for cURL

To create a new policy for cURL to communicate with the server, we need the PID of cURL. Since cURL is not running continuously like the server, a “dummy” command is needed to force it to run until the policy is generated.

curl 1.2.3.4 &

Let’s generate a policy for cURL

This must be done before cURL times out and the process terminates.

sudo camblet agent generate-policy $(pidof curl) curl | sudo tee /etc/camblet/policies/curl-simple.yaml

- certificate:
    ttl: 86400s
    workloadID: curl
  connection:
    mtls: STRICT
  selectors:
  - process:binary:path: /usr/bin/curl
    process:gid: "1000"
    process:name: curl
    process:uid: "501"

Try to connect to NGINX with a certificate

Using this policy, cURL gets an identity and will use mTLS. Let’s try to communicate with the NGINX once again.

curl localhost

curl: (56) Recv failure: Connection reset by peer

It still doesn’t work, one last piece of the puzzle is missing. Camblet must ascertain the target destination for the application of policies. Enforcing policies on every egress connection implies that cURL won’t be able to reach destinations beyond the Camblet-managed environment. Services has to be registered into the service registry through service definitions.

Sample Service discovery configuration file

- addresses:
  - address: localhost
    port: 80
  labels:
    app:label: app

Let’s ignore the labels part for now; it is meant for more advanced configuration. The addresses section specifies the registered destination addresses to which the policies should be applied.

Let’s create our own services.yaml where we place the nginx server’s service IP and port number.

sudo tee /etc/camblet/services/local.yaml >/dev/null <<EOF
- addresses:
  - address: 127.0.0.1
    port: 80
  labels:
    app:label: nginx
EOF

Try to connect to NGINX with a certificate and configured Camblet

With the service registry entry for NGINX in place, let’s check if its indeed fixed the error.

curl localhost

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

It finally works.

Cleanup

Exit from the Lima VM and delete it to cleanup.

limactl delete quickstart --force