Creating custom SELinux policy for a small network service

17 Mar 2023

Introduction

Security-Enhanced Linux (SELinux) is a security architecture for Linux® systems that allows administrators to have more control over who can access the system.

In this article I will try to show how a small custom network service can be hardened with help of SELinux policies.

Note that this article is not an introduction to SELinux and assumes some basics.

Bit of context

Lets say we are developing a small single virtual machine contained application. In this case I’m targeting smallest Hetzner instance: 1VCPU, 2Gb RAM, 20Gb disk space. It will run multiple components of which one will be a network service pushing data into a database. It’s a small standalone binary (written in Rust) which serves as a mail delivery agent on port 25. All submitted messages are pushed to PostgreSQL database.

Usually we could simply drop the binary into /usr/local/bin, create a SystemD unit file and be done with it. But what happens if our service gets exploited over the network or one of third party library components is shipped with credentials stealing code? Our first line of defence has failed. Can we still contain it?

A common solution to isolate our service from the rest of the system is to containerize it. This will limit the “blast area”. It’s also a valid and familiar way to deal with it. However it has some small issues - container disk usage, extra infrastructure to ship container images. This may become important if your software is to be deployed more as an limited resource appliance (small office server, small VM, embedded Linux machine) than a “cloud application”.

There’s also not much indication of any out of the ordinary service actions in case it’s been compromised.

Lets see how we can limit that “blast area” using SELinux.

The service

As mentioned this service is a standalone binary. It:

Listens for TCP connections on port 25.
Connects to PostgreSQL via local UNIX socket.
Logs to STDOUT and leaves log handling to SystemD.
Has its own dedicated unprivileged user.
To bind to port 25 binary has cap_net_bind_service capability flag (i.e. setcap 'cap_net_bind_service=ep' /path/to/binary).

And here’s the systemd unit file:

[Unit]
Description=A dummy mail delivery server
After=network.target

[Service]
EnvironmentFile=-/etc/sysconfig/catch-smtpd
ExecStart=/usr/bin/catch-smtpd --listen=0.0.0.0:25 --hostname=dev.local
Type=simple
User=catch-smtpd
Group=catch-smtpd

[Install]
WantedBy=multi-user.target

Before any kind of confinement

Binary file label:

# ls -Z /usr/bin/catch-smtpd
system_u:object_r:bin_t:s0 /usr/bin/catch-smtpd

Process label:

# ps -Zaux | grep catch-smtpd
system_u:system_r:unconfined_service_t:s0 catch-s+ 8527 0.0  0.2 351476 5936 ?   Ssl  18:18   0:00 /usr/bin/catch-smtpd --listen=0.0.0.0:25 --hostname=example.org

TCP port label:

# netstat -Z -lntp | grep catch-smtpd
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      8527/catch-smtpd     system_u:system_r:unconfined_service_t:s0

SELinux policy scaffold

Let’s follow instructions in this RedHat guide and set up a policy scaffold:

# sepolicy generate --init /usr/bin/catch-smtpd
Created the following files:
/root/catch-smtpd-selinux/catch_smtpd.te # Type Enforcement file
/root/catch-smtpd-selinux/catch_smtpd.if # Interface file
/root/catch-smtpd-selinux/catch_smtpd.fc # File Contexts file
/root/catch-smtpd-selinux/catch_smtpd_selinux.spec # Spec file
/root/catch-smtpd-selinux/catch_smtpd.sh # Setup Script

Build and load it:

# ./catch_smtpd.sh
Building and Loading Policy
+ make -f /usr/share/selinux/devel/Makefile catch_smtpd.pp
make: 'catch_smtpd.pp' is up to date.
+ /usr/sbin/semodule -i catch_smtpd.pp
+ sepolicy manpage -p . -d catch_smtpd_t
./catch_smtpd_selinux.8
+ /sbin/restorecon -F -R -v /usr/bin/catch-smtpd
++ pwd
... <rest of output skipped: it's the rpm package build> ...

Notice the build script ran restorecon on our binary. It now has a new label:

# ls -Z /usr/bin/catch-smtpd
system_u:object_r:catch_smtpd_exec_t:s0 /usr/bin/catch-smtpd

After restarting the service, process label is changed:

# ps -Zaux | grep catch-smtpd
system_u:system_r:catch_smtpd_t:s0 catch-s+ 9174  0.0  0.3 351476  6224 ?        Ssl  18:39   0:00 /usr/bin/catch-smtpd --listen=0.0.0.0:25 --hostname=dev.local

It is no longer running as unconfined_service_t. However, because of directive permissive catch_smtpd_t; in the generated type enforcement file catch_smtpd.te it is still running in permissive mode. This means policy rules will be tested, but not denied.

If we look at the logs we find the following:

# journalctl -n200 | grep -E "catch(-|_)smtpd"
Mar 15 18:39:10 dev.local systemd[1]: Stopping catch-smtpd.service - Catch any mail MDA server...
Mar 15 18:39:10 dev.local systemd[1]: catch-smtpd.service: Deactivated successfully.
Mar 15 18:39:10 dev.local systemd[1]: Stopped catch-smtpd.service - Catch any mail MDA server.
Mar 15 18:39:10 dev.local audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=catch-smtpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 15 18:39:17 dev.local systemd[1]: Started catch-smtpd.service - Catch any mail MDA server.
Mar 15 18:39:17 dev.local audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=catch-smtpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 15 18:39:17 dev.local audit[9174]: AVC avc:  denied  { search } for  pid=9174 comm="catch-smtpd" name="/" dev="cgroup2" ino=1 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:cgroup_t:s0 tclass=dir permissive=1
Mar 15 18:39:17 dev.local audit[9174]: AVC avc:  denied  { name_bind } for  pid=9174 comm="catch-smtpd" src=25 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:smtp_port_t:s0 tclass=tcp_socket permissive=1
Mar 15 18:39:17 dev.local audit[9174]: AVC avc:  denied  { node_bind } for  pid=9174 comm="catch-smtpd" src=25 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=tcp_socket permissive=1
Mar 15 18:39:17 dev.local audit[9174]: AVC avc:  denied  { net_bind_service } for  pid=9174 comm="catch-smtpd" capability=10  scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:system_r:catch_smtpd_t:s0 tclass=capability permissive=1
Mar 15 18:39:17 dev.local audit[9174]: AVC avc:  denied  { listen } for  pid=9174 comm="catch-smtpd" lport=25 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:system_r:catch_smtpd_t:s0 tclass=tcp_socket permissive=1
Mar 15 18:39:17 dev.local audit[9174]: AVC avc:  denied  { read } for  pid=9174 comm="catch-smtpd" name="passwd" dev="dm-0" ino=135080190 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file permissive=1
Mar 15 18:39:17 dev.local audit[9174]: AVC avc:  denied  { open } for  pid=9174 comm="catch-smtpd" path="/etc/passwd" dev="dm-0" ino=135080190 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file permissive=1
Mar 15 18:39:17 dev.local audit[9174]: AVC avc:  denied  { getattr } for  pid=9174 comm="catch-smtpd" path="/etc/passwd" dev="dm-0" ino=135080190 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file permissive=1
Mar 15 18:39:17 dev.local audit[9174]: AVC avc:  denied  { search } for  pid=9174 comm="catch-smtpd" name="catch-smtpd" dev="dm-0" ino=135104552 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=user_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1
Mar 15 18:39:17 dev.local audit[9174]: AVC avc:  denied  { write } for  pid=9174 comm="catch-smtpd" name=".s.PGSQL.5432" dev="tmpfs" ino=1329 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:postgresql_var_run_t:s0 tclass=sock_file permissive=1
Mar 15 18:39:17 dev.local audit[9174]: AVC avc:  denied  { connectto } for  pid=9174 comm="catch-smtpd" path="/run/postgresql/.s.PGSQL.5432" scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:system_r:postgresql_t:s0 tclass=unix_stream_socket permissive=1
Mar 15 18:39:17 dev.local catch-smtpd[9174]: Listening on: 0.0.0.0:25
Mar 15 18:39:17 dev.local catch-smtpd[9174]: Started stats monitor thread

This shows all kinds of rules violated that would end up in access denials if SELinux was running in ‘enforcing’ mode.

We can also see similar entries in audit damon logs:

# ausearch --message AVC --comm catch-smtpd
----
time->Fri Mar 17 07:57:30 2023
type=AVC msg=audit(1679039850.452:559): avc:  denied  { search } for  pid=2780 comm="catch-smtpd" name="/" dev="cgroup2" ino=1 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:cgroup_t:s0 tclass=dir permissive=1
----
time->Fri Mar 17 07:57:30 2023
type=AVC msg=audit(1679039850.453:560): avc:  denied  { name_bind } for  pid=2780 comm="catch-smtpd" src=25 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:smtp_port_t:s0 tclass=tcp_socket permissive=1
----
time->Fri Mar 17 07:57:30 2023
type=AVC msg=audit(1679039850.453:561): avc:  denied  { node_bind } for  pid=2780 comm="catch-smtpd" src=25 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=tcp_socket permissive=1
----
time->Fri Mar 17 07:57:30 2023
type=AVC msg=audit(1679039850.453:562): avc:  denied  { net_bind_service } for  pid=2780 comm="catch-smtpd" capability=10  scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:system_r:catch_smtpd_t:s0 tclass=capability permissive=1
----
time->Fri Mar 17 07:57:30 2023
type=AVC msg=audit(1679039850.453:563): avc:  denied  { listen } for  pid=2780 comm="catch-smtpd" lport=25 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:system_r:catch_smtpd_t:s0 tclass=tcp_socket permissive=1
----
time->Fri Mar 17 07:57:30 2023
type=AVC msg=audit(1679039850.453:564): avc:  denied  { read } for  pid=2780 comm="catch-smtpd" name="passwd" dev="dm-0" ino=135080190 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file permissive=1
----
time->Fri Mar 17 07:57:30 2023
type=AVC msg=audit(1679039850.453:565): avc:  denied  { open } for  pid=2780 comm="catch-smtpd" path="/etc/passwd" dev="dm-0" ino=135080190 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file permissive=1
----
time->Fri Mar 17 07:57:30 2023
type=AVC msg=audit(1679039850.453:566): avc:  denied  { getattr } for  pid=2780 comm="catch-smtpd" path="/etc/passwd" dev="dm-0" ino=135080190 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file permissive=1
----
time->Fri Mar 17 07:57:30 2023
type=AVC msg=audit(1679039850.453:567): avc:  denied  { search } for  pid=2780 comm="catch-smtpd" name="catch-smtpd" dev="dm-0" ino=135104552 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=user_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1
----
time->Fri Mar 17 07:57:30 2023
type=AVC msg=audit(1679039850.453:568): avc:  denied  { write } for  pid=2780 comm="catch-smtpd" name=".s.PGSQL.5432" dev="tmpfs" ino=1017 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:postgresql_var_run_t:s0 tclass=sock_file permissive=1
----
time->Fri Mar 17 07:57:30 2023
type=AVC msg=audit(1679039850.453:569): avc:  denied  { connectto } for  pid=2780 comm="catch-smtpd" path="/run/postgresql/.s.PGSQL.5432" scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:system_r:postgresql_t:s0 tclass=unix_stream_socket permissive=1

Note that I’m doing this after flushing audit logs. There’s little bit of trickery to look up correct recent audit logs that are relevant for us.

Now we can use audit2allow tool to generate type enforcement rules:

# ausearch --message AVC --comm catch-smtpd | audit2allow --reference

require {
        type catch_smtpd_t;
        class capability net_bind_service;
        class tcp_socket listen;
}

#============= catch_smtpd_t ==============
allow catch_smtpd_t self:capability net_bind_service;
allow catch_smtpd_t self:tcp_socket listen;
auth_read_passwd_file(catch_smtpd_t)
corenet_tcp_bind_generic_node(catch_smtpd_t)
corenet_tcp_bind_smtp_port(catch_smtpd_t)
fs_search_cgroup_dirs(catch_smtpd_t)
postgresql_stream_connect(catch_smtpd_t)
userdom_list_user_home_dirs(catch_smtpd_t)

And merge it with existing pre-generated module catch_smtpd.te:

policy_module(catch_smtpd, 1.0.0)

########################################
#
# Declarations
#

type catch_smtpd_t;
type catch_smtpd_exec_t;
init_daemon_domain(catch_smtpd_t, catch_smtpd_exec_t)

permissive catch_smtpd_t;

########################################
#
# catch_smtpd local policy
#
allow catch_smtpd_t self:fifo_file rw_fifo_file_perms;
allow catch_smtpd_t self:unix_stream_socket create_stream_socket_perms;

domain_use_interactive_fds(catch_smtpd_t)

files_read_etc_files(catch_smtpd_t)

miscfiles_read_localization(catch_smtpd_t)

sysnet_dns_name_resolve(catch_smtpd_t)

Resulting something like this:

policy_module(catch_smtpd, 1.0.0)

require {
        class capability net_bind_service;
        class tcp_socket listen;
}

########################################
#
# Declarations
#

type catch_smtpd_t;
type catch_smtpd_exec_t;
init_daemon_domain(catch_smtpd_t, catch_smtpd_exec_t)

permissive catch_smtpd_t;

########################################
#
# catch_smtpd local policy
#
allow catch_smtpd_t self:fifo_file rw_fifo_file_perms;
allow catch_smtpd_t self:unix_stream_socket create_stream_socket_perms;

domain_use_interactive_fds(catch_smtpd_t)

files_read_etc_files(catch_smtpd_t)

miscfiles_read_localization(catch_smtpd_t)

sysnet_dns_name_resolve(catch_smtpd_t)

allow catch_smtpd_t self:capability net_bind_service;
allow catch_smtpd_t self:tcp_socket listen;
auth_read_passwd_file(catch_smtpd_t)
corenet_tcp_bind_generic_node(catch_smtpd_t)
corenet_tcp_bind_smtp_port(catch_smtpd_t)
fs_search_cgroup_dirs(catch_smtpd_t)
postgresql_stream_connect(catch_smtpd_t)
userdom_list_user_home_dirs(catch_smtpd_t)

We can now load the rules again through catch_smtpd.sh set up script and re-test the service again to see if service triggers any access denials. If there’s none - remove statement permissive catch_smtpd_t; and build again. Now we can ship those rules to our server as a rpm package.

It so happens I did not fully test everything at the first cycle and missed one rule. My final result after some tweaks are reformatting was:

policy_module(catch_smtpd, 1.0.0)

require {
        class capability net_bind_service;
        class tcp_socket { accept listen };
}

########################################
#
# Declarations
#

type catch_smtpd_t;
type catch_smtpd_exec_t;
init_daemon_domain(catch_smtpd_t, catch_smtpd_exec_t)

########################################
#
# catch_smtpd local policy
#
allow catch_smtpd_t self:fifo_file rw_fifo_file_perms;
allow catch_smtpd_t self:unix_stream_socket create_stream_socket_perms;
allow catch_smtpd_t self:tcp_socket { accept listen };

domain_use_interactive_fds(catch_smtpd_t)
files_read_etc_files(catch_smtpd_t)
miscfiles_read_localization(catch_smtpd_t)
sysnet_dns_name_resolve(catch_smtpd_t)
auth_read_passwd_file(catch_smtpd_t)
corenet_tcp_bind_generic_node(catch_smtpd_t)
corenet_tcp_bind_smtp_port(catch_smtpd_t)
fs_search_cgroup_dirs(catch_smtpd_t)
postgresql_stream_connect(catch_smtpd_t)
userdom_list_user_home_dirs(catch_smtpd_t)

Just for fun - demonstrating confinement

Let’s say someone develops a rust library stealing SSH keys. Our malicious dependency tries to open ~/.ssh/id_rsa and submit it to some remote server.

Deep in the code it has something like this:

let home = env::var("HOME").unwrap();
let path = format!("{home}/.ssh/id_rsa");
println!("Will try to open: {path}");
let key = read_to_string(path).await.unwrap_or("no luck".to_string());
println!("Got the key: {key}");

(note how our evil library developer added some helpful logging statements to help us with demonstration)

For purpose of this demo our service user catch-smtpd has an ssh key in its home.

After starting the service I we can find the following in the logs:

Mar 17 09:32:28 dev.local systemd[1]: Started catch-smtpd.service - Catch any mail MDA server.
Mar 17 09:32:28 dev.local catch-smtpd[16513]: Will try to open: /home/catch-smtpd/.ssh/id_rsa
Mar 17 09:32:28 dev.local audit[16513]: AVC avc:  denied  { search } for  pid=16513 comm="tokio-runtime-w" name=".ssh" dev="dm-0" ino=67292352 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=user_u:object_r:ssh_home_t:s0 tclass=dir permissive=0
Mar 17 09:32:28 dev.local catch-smtpd[16513]: Got the key: no luck
Mar 17 09:32:28 dev.local catch-smtpd[16513]: Listening on: 0.0.0.0:25
Mar 17 09:32:28 dev.local catch-smtpd[16513]: Started stats monitor thread

Neat. Now let’s say our malicious code tries to launch curl as a child process and download more malicious code. This is what we find in logs then:

Mar 17 09:51:23 dev.local systemd[1]: Started catch-smtpd.service - Catch any mail MDA server.
Mar 17 09:51:23 dev.local audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=catch-smtpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 17 09:51:23 dev.local audit[29191]: AVC avc:  denied  { execute } for  pid=29191 comm="catch-smtpd" name="curl" dev="dm-0" ino=135187103 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:bin_t:s0 tclass=file permissive=0
Mar 17 09:51:23 dev.local catch-smtpd[29186]: Listening on: 0.0.0.0:25
Mar 17 09:51:23 dev.local catch-smtpd[29186]: Started stats monitor thread

What about “calling home” over outgoing TCP connection? Stopped as well:

Mar 17 10:11:06 dev.local systemd[1]: Started catch-smtpd.service - Catch any mail MDA server.
Mar 17 10:11:06 dev.local audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=catch-smtpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 17 10:11:06 dev.local audit[33469]: AVC avc:  denied  { name_connect } for  pid=33469 comm="catch-smtpd" dest=80 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:http_port_t:s0 tclass=tcp_socket permissive=0
Mar 17 10:11:06 dev.local audit[33469]: AVC avc:  denied  { name_connect } for  pid=33469 comm="catch-smtpd" dest=80 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:http_port_t:s0 tclass=tcp_socket permissive=0
Mar 17 10:11:06 dev.local audit[33469]: AVC avc:  denied  { name_connect } for  pid=33469 comm="catch-smtpd" dest=80 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:http_port_t:s0 tclass=tcp_socket permissive=0
Mar 17 10:11:06 dev.local audit[33469]: AVC avc:  denied  { name_connect } for  pid=33469 comm="catch-smtpd" dest=80 scontext=system_u:system_r:catch_smtpd_t:s0 tcontext=system_u:object_r:http_port_t:s0 tclass=tcp_socket permissive=0
Mar 17 10:11:06 dev.local catch-smtpd[33469]: payload download failed
Mar 17 10:11:06 dev.local catch-smtpd[33469]: Listening on: 0.0.0.0:25
Mar 17 10:11:06 dev.local catch-smtpd[33469]: Started stats monitor thread

Conclusion

Using SELinux to harden your own in-house developed services is not as difficult as it used to be. Tooling around it is pretty good and does many things for you.

Admittedly it’s only slightly more complicated example than the one demonstrated in RedHat documentation. Larger monolithic applications are likely to need permission to launch various system tools, connect to external services and similar. Crafting a good rule set in those cases may get complicated. Or if we’re not careful - result in a rule set which does not contain malicious examples shown above.

Nevertheless when circumstances are right it’s a technique worth consideration.

Tadas Sasnauskas Tech/Engineering Blog