eBPF Summit 2022 CTF Challenge 3 Writeup
0x00 Introduction
This is the writeup for the third challenge of eBPF Summit 2022 CTF. See writeup for challenge 2 for the previous setup. The challenge details can be found here.
0x01 Information Gathering
The first step is to find out what’s running in the cluster.
Note: if you shell into the VM, you need to set KUBECONFIG before using kubectl.
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/challenge-3-7465fc8b59-vwgwf 1/1 Running 0 3h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 3h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/challenge-3 1/1 1 1 3h
NAME DESIRED CURRENT READY AGE
replicaset.apps/challenge-3-7465fc8b59 1 1 1 3h
There is only one pod running, let’s take a look at it.
$ kubectl describe pod/challenge-3-7465fc8b59-vwgwf
...
Init Containers:
ctfinit:
Container ID: containerd://20c218f7e87b78f947522ac7d5228282c45714562d5fa584ba2877f6d1b2651c
Image: ghcr.io/lizrice/ctfinit:drop
Image ID: ghcr.io/lizrice/ctfinit@sha256:70d741f7a58720c500348b7be70f5bed0d30bd4b63765404e85d651d061e5b49
...
Containers:
ctfapp:
Container ID: containerd://19f4d67df58afe4889a504d11684a5696f9ce008955962bc05c317153038ef99
Image: ghcr.io/lizrice/ctfapp:drop
Image ID: ghcr.io/lizrice/ctfapp@sha256:78934f9a1d4b4a70ec736f067c4b19e347b442232a8870da898d03b019e3f252
Port: <none>
Host Port: <none>
...
Similar to the previous challenge, there are two containers running in the pod ctfinit and ctfapp.
Let’s check the logs.
$ kubectl logs challenge-3-7465fc8b59-vwgwf -c ctfinit
Init complete
$ kubectl logs challenge-3-7465fc8b59-vwgwf -c ctfapp
I need internet access so I can learn about eBPF! But I'm getting errors:
get: Get "https://docs.cilium.io/en/stable/bpf/#xdp": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I'll give you another update in around 30 seconds
I need internet access so I can learn about eBPF! But I'm getting errors:
get: Get "https://docs.cilium.io/en/stable/bpf/#xdp": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I'll give you another update in around 30 seconds
Looks like the ctfapp container is trying to access the internet but failing.
Let’s try to validate the assumption by running a shell in the container.
$ kubectl exec -ti challenge-3-7465fc8b59-vwgwf -c ctfapp -- sh
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "d87692ae7dd5dd7be67e5d3f8afc4264a9fe6637436d18c50745203b8069af75": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown
Oops, no shell, let’s try kubectl debug
$ kubectl debug -ti challenge-3-7465fc8b59-vwgwf --target ctfapp --image=busybox
$ wget https://docs.cilium.io/en/stable/bpf/#xdp
Connecting to docs.cilium.io (104.17.33.82:443)
No access to the internet, try the same command in VM:
$ wget https://docs.cilium.io/en/stable/bpf/#xdp
Resolving docs.cilium.io (docs.cilium.io)... 104.17.32.82, 104.17.33.82
Connecting to docs.cilium.io (docs.cilium.io)|104.17.32.82|:443... connected.
HTTP request sent, awaiting response... 200 OK
So the host can access the internet but the pod cannot.
Can it be some bpf prog blocking the traffic? Let’s check some bpf logs.
$ sudo bpftool prog tracelog
...
ksoftirqd/3-33 [003] d.s.1 14236.774115: bpf_trace_printk: Dropping TCP packet
ksoftirqd/3-33 [003] d.s.1 14239.131882: bpf_trace_printk: Dropping TCP packet
Looks like there is something dropping packet. Is there any xdp prog?
$ sudo bpftool prog show | grep xdp
1301: xdp name a85d49 tag fc274aabdac1d306 gpl
What is it?
$ sudo bpftool prog dump x id 1301
int a85d49(struct xdp_md * ctx):
; int a85d49(struct xdp_md *ctx)
0: (b7) r0 = 0
; void *data = (void *)(long)ctx->data;
1: (79) r2 = *(u64 *)(r1 +0)
; void *data_end = (void *)(long)ctx->data_end;
2: (79) r1 = *(u64 *)(r1 +8)
; if (data + sizeof(struct ethhdr) > data_end)
3: (bf) r3 = r2
4: (07) r3 += 14
; if (data + sizeof(struct ethhdr) > data_end)
5: (2d) if r3 > r1 goto pc+11
6: (bf) r3 = r2
7: (07) r3 += 34
8: (2d) if r3 > r1 goto pc+8
9: (b7) r0 = 2
; if (iph->protocol == IPPROTO_ICMP) {
10: (71) r1 = *(u8 *)(r2 +23)
; if (iph->protocol == IPPROTO_ICMP) {
11: (55) if r1 != 0x6 goto pc+5
; bpf_printk("Dropping TCP packet\n");
12: (18) r1 = map[id:225][0]+0
14: (b7) r2 = 21
15: (85) call bpf_trace_printk#-73936
16: (b7) r0 = 1
; }
17: (95) exit
OK, this is the one blocking the traffic. Where is it attached?
$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 52:55:55:d5:a0:ff brd ff:ff:ff:ff:ff:ff
altname enp0s1
3: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether c2:6c:b4:f1:82:c0 brd ff:ff:ff:ff:ff:ff
4: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether c6:93:c4:1a:1c:d4 brd ff:ff:ff:ff:ff:ff
5: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 46:29:aa:ab:06:ff brd ff:ff:ff:ff:ff:ff
7: lxc_health@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether ba:3a:b1:93:73:b7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
9: lxc6deee4986637@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether d2:33:63:ce:63:10 brd ff:ff:ff:ff:ff:ff link-netns cni-2dce3bd9-38d2-cc9e-26d0-9115ed5611ea
11: lxc2fdeff222904@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 2e:5f:c2:91:4b:ba brd ff:ff:ff:ff:ff:ff link-netns cni-52ff6a2f-66c3-b385-5ed0-f5e922a87859
13: lxc618a1255c457@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether ea:ad:91:a5:08:3d brd ff:ff:ff:ff:ff:ff link-netns cni-e1de76c3-6d1f-e7e0-639a-f7f99a27fc52
19: lxc99d1afd98d54@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether fe:93:ea:ff:53:56 brd ff:ff:ff:ff:ff:ff link-netns cni-e9934a9c-656a-2a38-e26a-579c46a82af8
21: lxcec181dd23b0d@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether fa:f9:92:3b:83:9a brd ff:ff:ff:ff:ff:ff link-netns cni-f96d09b0-c1c4-dc19-2ff8-1ffdf0bfe0e1
23: lxc9fd8b423732d@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether d6:b6:78:a3:74:ce brd ff:ff:ff:ff:ff:ff link-netns cni-fc740045-2b6d-5855-22a3-5545722ddf8d
25: lxc52fa0150481e@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 12:83:88:f5:88:07 brd ff:ff:ff:ff:ff:ff link-netns cni-366802eb-f758-2697-1ffa-f4f5b2d9636b
It’s not attached to any interface from host, let’s see the other side of the veth pair.
$ sudo ip -all netns exec ip link show
...
netns: cni-52ff6a2f-66c3-b385-5ed0-f5e922a87859
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 52:eb:c4:6c:66:41 brd ff:ff:ff:ff:ff:ff link-netnsid 0
prog/xdp id 1301 tag fc274aabdac1d306 jited
...
Got you.
Note I did a shortcut by showing all namespaces, for precise tracing, we can also use nsenter.
First we need to know the pid of the container ctfapp in host.
$ sudo crictl inspect 19f4d67df58afe4889a504d11684a5696f9ce008955962bc05c317153038ef99
...
"info": {
"sandboxID": "10f228ff47aaa79b329e15797144ce347244333659d1a2e08c39d3e9688d3252",
"pid": 10481,
...
Then we can use nsenter to enter the namespace of the container.
sudo nsenter -t 10481 -n ip link show
0x02 Exploit
Detach the xdp program from the interface.
sudo ip netns exec cni-52ff6a2f-66c3-b385-5ed0-f5e922a87859 bpftool net detach xdp dev eth0
Check again
$ sudo ip netns exec cni-52ff6a2f-66c3-b385-5ed0-f5e922a87859 ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 52:eb:c4:6c:66:41 brd ff:ff:ff:ff:ff:ff link-netnsid 0
prog/xdp id 1301 tag fc274aabdac1d306 jited
It’s still there. Because it’s xdpgeneric, we need to do this.
$ sudo ip netns exec cni-52ff6a2f-66c3-b385-5ed0-f5e922a87859 bpftool net detach xdpgeneric dev eth0
$ sudo ip netns exec cni-52ff6a2f-66c3-b385-5ed0-f5e922a87859 ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 52:eb:c4:6c:66:41 brd ff:ff:ff:ff:ff:ff link-netnsid 0
Done! Let’s check the log.
$ kubectl logs challenge-3-7465fc8b59-vwgwf -c ctfapp
The first flag is ALDERAAN
Now I've got internet access, but I'm distracted by looking at https://www.starwars.com. Make me focus on reading about Cilium at docs.cilium.io.
I'll give you another update in around 30 seconds
Now we get the first flag, the next part is to limit internet access to docs.cilium.io only.
That’s the exact use case for FQDN cilium network policy.
Cilium Network Policy Editor is a great tool to help writing the policy.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: ctf-policy
spec:
endpointSelector: {}
egress:
- toEndpoints:
- {}
- toEntities:
- cluster
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
rules:
dns:
- matchPattern: "*"
- toFQDNs:
- matchName: docs.cilium.io
Apply the policy.
kubectl apply -f ctf-policy.yaml
Then check the log again.
$ kubectl logs challenge-3-7465fc8b59-vwgwf -c ctfapp -f
I need internet access so I can learn about eBPF! But I'm getting errors:
get: Get "https://docs.cilium.io/en/stable/bpf/#xdp": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I'll give you another update in around 30 seconds```
Looks like it’s blocked again. Due to how FQDN policy works, we need to wait for a while for the DNS cache to expire. Because data plane only understand ip address, it needs to resolve the domain name to ip address first. Once there is new DNS request, the ip cache will be refreshed.
Or we can just send the DNS request manually. Back to the debug shell again.
$ # kubectl debug -ti challenge-3-7465fc8b59-vwgwf --target=ctfapp --image=busybox
$ nslookup docs.cilium.io
Server: 10.43.0.10
Address: 10.43.0.10:53
Non-authoritative answer:
Name: docs.cilium.io
Address: 104.17.32.82
Name: docs.cilium.io
Address: 104.17.33.82
$ wget https://docs.cilium.io
Connecting to docs.cilium.io (104.17.33.82:443)
wget: note: TLS certificate validation not implemented
Connecting to docs.cilium.io (104.17.33.82:443)
saving to 'index.html'
$ wget https://www.starwars.com
Connecting to www.starwars.com (149.135.80.25:443)
Then check the log.
$ kubectl logs challenge-3-7465fc8b59-vwgwf -c ctfapp -f
...
The second flag is DANTOOINE
Case closed.
0x03 Conclusion
This challenge is a little harder than the previous one, it requires some knowledge about XDP and Cilium network policy. There are 2 good eCHO episodes covering both.