Another Tutorial

log2rbac-operator

Kubernetes operator that helps you to set up the RBAC rules for your application. If requested, it scans the application’s log files for authorization errors and adds them as exceptions/rights to the associated Role. It is like having a sudo command for your service accounts. However, with great power comes great responsibility. The goal of the tool is to find the minimum set of rights that is needed for your workload to run instead of using the cluster admin for everything.

User has to allow this process by creating a RbacNegotiation custom resource where they need to specify the app* and Role. Role can be either existing one or operator can create a new one for you and bind it to the service account that’s configured with the deployment. Again if the service account is not there, it will be created by the operator.

* App can be one of the following:

• Deployment
• StatefulSet
• DaemonSet
• Service
• ReplicaSet
• or key-value pair specifying the pod selector

By creating RbacNegotiation we start the “recording” regime for our workload in which the operator will be building this allow list of verbs, resources (rbac entries) for the given role.

(pauseable demo)

This project is conceptually very similar to audit2rbac. The main distinction here is that log2rbac is based on the controller pattern and on the output from the workloads, while audit2rbac uses the k8s’ audit log and it’s a “one-pass” CLI tool.

Quick Start

# clone repo and
make deploy

alternatively install it using all-in-one yaml

kubectl apply -f http://bit.do/log2rbac

or helm:

helm repo add log2rbac https://jkremser.github.io/log2rbac-operator
helm repo update
helm -n log2rbac upgrade -i log2rbac log2rbac/log2rbac-operator --wait --create-namespace

Now when the operator was installed, let’s deploy something that needs the special RBAC. Prometheus Operator is a good example.

kubectl create ns monitoring
kubectl apply -n monitoring -f https://github.com/prometheus-operator/kube-prometheus/raw/v0.10.0/manifests/prometheusOperator-deployment.yaml

This deployment will fail to start because of the missing rights to do its stuff. Let’s request the RBAC negotiation process.

# create RbacNegotiation for Prometheus operator
cat <<CustomResource | kubectl apply -f -
apiVersion: kremser.dev/v1
kind: RbacNegotiation
metadata:
  name: for-prom
spec:
  for:
    namespace: monitoring
    kind: Deployment
    name: prometheus-operator
  role:
    name: foo
    isClusterRole: true
    createIfNotExist: true
CustomResource

# After some time, the Prometheus Operator should start and we should see.

k describe clusterrole foo
Name:         foo
Labels:       <none>
Annotations:  app.kubernetes.io/created-by=log2rbac
PolicyRule:
  Resources                                  Non-Resource URLs  Resource Names  Verbs
  ---------                                  -----------------  --------------  -----
  configmaps                                 []                 []              [list watch]
  namespaces                                 []                 []              [list watch]
  secrets                                    []                 []              [list watch]
  statefulsets.apps                          []                 []              [list watch]
  alertmanagerconfigs.monitoring.coreos.com  []                 []              [list]
  alertmanagers.monitoring.coreos.com        []                 []              [list]
  ...

Note: This set of rights was necessary only for the Prometheus operator to start. Once we start interacting with the subject of our RBAC negotiation process, the new code paths will be executed and possibly new rights will be requested. It might be a good idea to run e2e tests or at least have some script that calls the Prometheus operator’s functionality (CRUDing all its CRDs). Last but not least, once we are happy with the resulting role and the rights it got, we should delete the RBAC negotiation custom resource to reduce the attack surface on our cluster.

You may want to capture the role in yaml format and store it as part of your infrastructure code in git:

k get clusterrole foo -o yaml | k neat

Kubectl Plugin

Installation:

kubectl krew install log2rbac

It can help with creating those RbacNegotiation custom resources by interactive TUI api.

It’s located in this repo

Observability

Operator’s code has been instrumented by ~OpenTracing~ OpenTelemetry calls so that one can export the spans to Jaeger or Zipkin and connect the dots. There is an example deployment using open telemetry collector running as a side-car container that exports the traces to Jaeger that runs in its own deployment. To deploy this scenario, issue:

make deploy-otel

To check the Jaeger web UI for log2rbac traces, the easiest way is to

kubectl port-forward svc/jaeger-collector 16686
open http://localhost:16686

Configuration

Following options are available as env variables for the operator: