Tutorial: Standalone Envoy

The tutorial shows how Envoy’s External authorization filter can be used with OPA as an authorization service to enforce security policies over API requests received by Envoy. The tutorial also covers examples of authoring custom policies over the HTTP request body.

Overview

In this tutorial we'll see how to use OPA as an External Authorization service for the Envoy proxy. We'll do this by:

• Running a local Kubernetes cluster
• Creating a simple authorization policy in Rego and serving it via the Bundle API
• Deploying a sample application with Envoy and OPA sidecars
• Run some sample requests to see the policy in action

Note that other than the HTTP client and bundle server, all components are co-located in the same pod.

Running a local Kubernetes cluster

To start a local Kubernetes cluster to run our demo, we'll be using kind.

info

If you haven't used kind before, you can find installation instructions in the project documentation.

Create a cluster with the following command:

$ kind create cluster --name opa-envoy --image kindest/node:v1.27.3
Creating cluster "opa-envoy" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
...

Once the cluster is created, make sure your kubectl context is set to connect to the new cluster:

$ kubectl cluster-info --context kind-opa-envoy
Kubernetes control plane is running at ...
CoreDNS is running at ...
...

Listing the cluster nodes, should show something like this:

$ kubectl get nodes
NAME                       STATUS   ROLES           AGE     VERSION
opa-envoy-control-plane   Ready    control-plane   2m35s   v1.27.3

Creating & Serving our Policy Bundle

This tutorial assumes you have some Rego knowledge, in summary the policy below does the following:

• Checks that the JWT token is valid
• Checks that the action is allowed based on the token payload role and the request path
• Guests have read-only access to the /people endpoint, admins can create users too as long as the name is not the same as the admin's name.

# policy.rego
package envoy.authz

import input.attributes.request.http as http_request

default allow := false

allow if {
	is_token_valid
	action_allowed
}

is_token_valid if {
	token.valid
	now := time.now_ns() / 1000000000
	token.payload.nbf <= now
	now < token.payload.exp
}

action_allowed if {
	http_request.method == "GET"
	token.payload.role == "guest"
	glob.match("/people", ["/"], http_request.path)
}

action_allowed if {
	http_request.method == "GET"
	token.payload.role == "admin"
	glob.match("/people", ["/"], http_request.path)
}

action_allowed if {
	http_request.method == "POST"
	token.payload.role == "admin"
	glob.match("/people", ["/"], http_request.path)
	lower(input.parsed_body.firstname) != base64url.decode(token.payload.sub)
}

token := {"valid": valid, "payload": payload} if {
	[_, encoded] := split(http_request.headers.authorization, " ")
	[valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"})
}

Create a file called policy.rego with the above content and store it in a ConfigMap:

kubectl create configmap authz-policy --from-file policy.rego

Now that the policy is stored in a ConfigMap, we can spin up an HTTP server to make it available as a Bundle to OPA when it's making decisions for our application:

# bundle-server.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bundle-server
  labels:
    app: bundle-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bundle-server
  template:
    metadata:
      labels:
        app: bundle-server
    spec:
      initContainers:
      - name: opa-builder
        image: openpolicyagent/opa:latest
        args:
        - "build"
        - "--bundle"
        - "/opt/policy/"
        - "--output"
        - "/opt/output/bundle.tar.gz"
        volumeMounts:
        - name: index
          mountPath: /opt/output/
        - name: policy
          mountPath: /opt/policy/
      containers:
      - name: bundle-server
        image: nginx:1.25
        ports:
        - containerPort: 80
          name: http
        volumeMounts:
        - name: index
          mountPath: /usr/share/nginx/html
      volumes:
      - name: index
        emptyDir: {}
      - name: policy
        configMap:
          name: authz-policy
---
apiVersion: v1
kind: Service
metadata:
  name: bundle-server
spec:
  selector:
    app: bundle-server
  ports:
  - protocol: TCP
    port: 80
    targetPort: http

Create a file called bundle-server.yaml with the above content and apply it to the cluster:

kubectl apply -f bundle-server.yaml

Once the deployment is running, we can check that the bundle is available by running:

kubectl port-forward service/bundle-server 8080:80

Before checking that the bundle has been generated correctly and is available to download:

$ curl -I localhost:8080/bundle.tar.gz
HTTP/1.1 200 OK
...

You may now exit the port-forwarding session, the bundle server will only be accessed from inside the cluster from now on.

Deploying an application with Envoy and OPA sidecars

In this tutorial, we are manually configuring the Envoy proxy sidecar to intermediate HTTP traffic from clients and our application. Envoy will consult OPA to make authorization decisions for each request by sending CheckRequest messages over a gRPC connection.

We will use the following Envoy configuration to achieve this. In summary, this configures Envoy to:

• Listen on port 8000 for HTTP traffic
• Consult OPA for authorization decisions at 127.0.0.1:9191 & deny failing requests
• Forward requests to the application at 127.0.0.1:8080 if ok.

# envoy.yaml
static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 8000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: service
          http_filters:
          - name: envoy.ext_authz
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
              transport_api_version: V3
              with_request_body:
                max_request_bytes: 8192
                allow_partial_message: true
              failure_mode_allow: false
              grpc_service:
                google_grpc:
                  target_uri: 127.0.0.1:9191
                  stat_prefix: ext_authz
                timeout: 0.5s
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: service
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 8080
admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001
layered_runtime:
  layers:
  - name: static_layer_0
    static_layer:
      envoy:
        resource_limits:
          listener:
            example_listener_name:
              connection_limit: 10000
      overload:
        global_downstream_max_connections: 50000

Create a ConfigMap containing the above configuration by running:

kubectl create configmap proxy-config --from-file envoy.yaml

Our application will be configured using a Deployment and Service. There are a few things to note:

• the pods have an initContainer that configures the iptables rules to redirect traffic to the Envoy proxy.
• the demo-test-server container is a simple user store using in-memory state.
• the envoy container is configured to use the proxy-config ConfigMap we created earlier.
• The OPA container is configured to download policy bundles from the in-cluster bundle server (bundle-server.default.svc.cluster.local).

# app.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      initContainers:
      - name: proxy-init
        image: openpolicyagent/proxy_init:v8
        # Configure the iptables bootstrap script to redirect traffic to the
        # Envoy proxy on port 8000. Envoy will be running as 1111, and port
        # 8282 will be excluded to support OPA health checks.
        args: ["-p", "8000", "-u", "1111", "-w", "8282"]
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
          runAsNonRoot: false
          runAsUser: 0
      containers:
      - name: app
        image: openpolicyagent/demo-test-server:v1
        ports:
        - containerPort: 8080
      - name: envoy
        image: envoyproxy/envoy:v1.26.3
        volumeMounts:
        - readOnly: true
          mountPath: /config
          name: proxy-config
        args:
        - "envoy"
        - "--config-path"
        - "/config/envoy.yaml"
        env:
        - name: ENVOY_UID
          value: "1111"
      - name: opa
        image: openpolicyagent/opa:latest-envoy
        args:
        - "run"
        - "--server"
        - "--addr=localhost:8181"
        - "--diagnostic-addr=0.0.0.0:8282"
        - "--set=services.default.url=http://bundle-server"
        - "--set=bundles.default.resource=bundle.tar.gz"
        - "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
        - "--set=plugins.envoy_ext_authz_grpc.path=envoy/authz/allow"
        - "--set=decision_logs.console=true"
        - "--set=status.console=true"
        - "--ignore=.*"
        livenessProbe:
          httpGet:
            path: /health?plugins
            scheme: HTTP
            port: 8282
          initialDelaySeconds: 5
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /health?plugins
            scheme: HTTP
            port: 8282
          initialDelaySeconds: 1
          periodSeconds: 3
      volumes:
      - name: proxy-config
        configMap:
          name: proxy-config
---
apiVersion: v1
kind: Service
metadata:
  name: example-app
spec:
  selector:
    app: example-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

Deploy the application and Kubernetes Service to the cluster with:

kubectl apply -f app.yaml

Check that everything is working by listing the pod (make sure that all three pods are running ok).

$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
bundle-server-5d7bfffdb6-bgn86   1/1     Running   0          1m
example-app-74b4bc88-5d4wh       3/3     Running   0          1m

See the Policy in Action

Run a shell inside the cluster to use for testing. We will use this in-cluster shell for the rest of the tutorial.

kubectl run curl --restart=Never -it --rm --image curlimages/curl:8.1.2 -- sh

Set two tokens for two users, Alice and Bob with different permissions. As defined by our policy:

export ALICE_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiZ3Vlc3QiLCAic3ViIjogIllXeHBZMlU9In0.Uk5hgUqMuUfDLvBLnlXMD0-X53aM_Hlziqg3vhOsCc8"
export BOB_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiYWRtaW4iLCAic3ViIjogIlltOWkifQ.5qsm7rRTvqFHAgiB6evX0a_hWnGbWquZC0HImVQPQo8"

Listing People

Send a request to list people. This should succeed for both Alice and Bob.

curl -i -H "Authorization: Bearer $ALICE_TOKEN" http://example-app/people

HTTP/1.1 200 OK
content-type: application/json
date: Tue, 18 Jul 2023 15:22:25 GMT
content-length: 96
x-envoy-upstream-service-time: 14
server: envoy

[{"id":"1","firstname":"John","lastname":"Doe"},{"id":"2","firstname":"Jane","lastname":"Doe"}]

And for Bob:

curl -i -H "Authorization: Bearer $BOB_TOKEN" http://example-app/people

HTTP/1.1 200 OK
...omitted...

Creating People

Send a request to create a new user. This should fail for Alice but not Bob:

curl -i -H "Authorization: Bearer $ALICE_TOKEN" \
  -d '{"firstname":"Foo", "lastname":"Bar"}' -H "Content-Type: application/json" \
  -X POST http://example-app/people

HTTP/1.1 403 Forbidden
date: Tue, 18 Jul 2023 15:25:28 GMT
server: envoy
content-length: 0

And for Bob, the request is permitted and the user is saved with an ID

curl -i -H "Authorization: Bearer $BOB_TOKEN" \
  -d '{"firstname":"Foo", "lastname":"Bar"}' -H "Content-Type: application/json" \
  -X POST http://example-app/people

HTTP/1.1 200 OK
content-type: application/json
date: Tue, 18 Jul 2023 15:28:20 GMT
content-length: 51
x-envoy-upstream-service-time: 11
server: envoy

{"id":"498081","firstname":"Foo","lastname":"Bar"}

Creating People: Conflict

Our policy also blocks users from creating users with the same name, test that functionality with this request:

curl -i -H "Authorization: Bearer $BOB_TOKEN" \
  -d '{"firstname":"Bob", "lastname":"Bar"}' -H "Content-Type: application/json" \
  -X POST http://example-app/people

HTTP/1.1 403 Forbidden
date: Tue, 18 Jul 2023 15:31:48 GMT
server: envoy
content-length: 0

Shutting Down

Exit the in-cluster shell by typing exit.

Delete the cluster by running:

$ kind delete cluster --name opa-envoy
Deleting cluster "opa-envoy" ...
Deleted nodes: ["opa-envoy-control-plane"]

Wrap Up

Congratulations on finishing the tutorial !

This tutorial showed how to use OPA as an External authorization service to enforce custom policies by leveraging Envoy’s External authorization filter.

This tutorial also showed a sample OPA policy that returns a boolean decision to indicate whether a request should be allowed or not.

Envoy's external authorization filter allows optional response headers and body to be sent to the downstream client or upstream. An example of a rule that returns an object that not only indicates if a request is allowed or not but also provides optional response headers, body and HTTP status that can be sent to the downstream client or upstream can be seen here.