public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pve-devel] [PATCH container/docs/firewall/manager/proxmox-firewall/qemu-server v3 00/39] proxmox firewall nftables implementation
@ 2024-04-18 16:13 Stefan Hanreich
  2024-04-18 16:13 ` [pve-devel] [PATCH proxmox-firewall v3 01/39] config: add proxmox-ve-config crate Stefan Hanreich
                   ` (39 more replies)
  0 siblings, 40 replies; 42+ messages in thread
From: Stefan Hanreich @ 2024-04-18 16:13 UTC (permalink / raw)
  To: pve-devel

## Introduction
This RFC provides a drop-in replacement for the current pve-firewall package
that is based on Rust and nftables.

It consists of three crates:
* proxmox-ve-config
  for parsing firewall and guest configuration files, as well as some helpers
  to access host configuration (particularly networking)
* proxmox-nftables
  contains bindings for libnftables as well as types that implement the JSON
  schema defined by libnftables-json
* proxmox-firewall
  uses the other two crates to read the firewall configuration and create the
  respective nftables configuration


## Installation
* Build & install all deb packages on your PVE instance
* Enable the nftables firewall by going to
  Web UI > <Host> > Firewall > Options > nftables
* Enable the firewall datacenter-wide if you haven't already
* Restarting running VMs/CTs is required so the changes to the fwbr creation
  go into effect

For your convenience I have provided pre-built packages on our share under
`shanreich-proxmox-firewall`.

The source code is also available on my staff repo as `proxmox-firewall`.


## Configuration
The firewall should work as a drop-in replacement for the pve-firewall, so you
should be able to configure the firewall as usual via the Web UI or
configuration files.


## Known Issues
There is currently one major issue that we still need to solve: REJECTing
packets from the guest firewalls is currently not possible for incoming traffic
(it will instead be dropped).

This is due to the fact that we are using the postrouting hook of nftables in a
table with type bridge for incoming traffic. In the bridge table in the
postrouting hook we cannot tell whether the packet has also been sent to other
ports in the bridge (e.g. when a MAC has not yet been learned and the packet
then gets flooded to all bridge ports). If we would then REJECT a packet in the
postrouting hook this can lead to a bug where the firewall rules for one guest
REJECT a packet and send a response (RST for TCP, ICMP port/host-unreachable
otherwise).

This has also been explained in the respective commit introducing the
restriction [1].

We were able to circumvent this restriction in the old firewall due to using
firewall bridges and rejecting in the firewall bridge itself. Doing this leads
to the behavior described above, which has tripped up some of our users before
[2] [3] and which is, frankly, wrong.

I currently see two possible solutions for this, both of which carry downsides.
Your input on this matter would be much appreciated, particularly if you can
think of another solution which I cannot currently see:

1. Only REJECT packets in the prerouting chain of the firewall bridge with the
destination MAC address set to the MAC address of the network device, otherwise
DROP

The downside of this is that we, once again, will have to resort to using
firewall bridges, which we wanted to eliminate. This would also be the sole
reason for still having to resort to using firewall bridges.

2. Only allow DROP in the guest firewall for incoming traffic

This would be quite awkward since, well, rejecting traffic would be quite nice
for a firewall I'd say ;)

I'm happy for all input regarding this matter.


## Useful Commands

You can check if firewall rules got created by running

```
nft list ruleset
```

You can also check that `iptables` rules are not created via
```
iptables-save
```

Further info about the services:
```
systemctl status proxmox-firewall.{service,timer}
```

You can grab the debug output from the new firewall like so:

```
RUST_LOG=trace proxmox-firewall
```

## Upcoming
There are many ideas for further features, here is a (non-exhaustive) list:
* SNAT/DNAT handling
* SDN integration + bridge/vnet-level firewalling
* counters
* flow offloading
* synproxy support
* rate-limiting for rules
* connlimit support
* brouting support

The first thing I'll be working on though is proper rustdocs for both libraries
as well as the firewall.


Changes from v2 -> v3:
* move from libnftables to nft CLI
* daemonize proxmox-firewall
* prettify chains.json
* move location of proxmox-firewall to /usr/libexec/proxmox
* update documentation to reflect the changes

Changes from v1 -> v2:
* now builds cleanly in sbuild (thanks @Fabian)
* made base ruleset more efficient
* fixed issues with some rules in the base ruleset
* refactored config loading in order to make it mockable (thanks @Lukas)
* added an integration test that spans the whole pipeline from config ->
  nftables rules
* changed many maps from HashMap to BTreeMap to make the generated nftables
  output stable (particularly important for the integration test)
* improved logging output
* implemented ARP spoofing prevention
* failing to change sysctl settings now only emits a warning instead of aborting
  the rule generation process
* fixed some bugs wrt rule generation
* added a few macros (HTTP/3, PBS)
* added documentation to pve-docs
* EUI64 link-local addresses are now added to the automatically generated ip
  filters
* incorporated suggestions from @Max and @Lukas (tyvm!)


[1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/net/bridge/netfilter/nft_reject_bridge.c?h=v6.8.2&id=127917c29a432c3b798e014a1714e9c1af0f87fe
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=4964
[3] https://forum.proxmox.com/threads/proxmox-claiming-mac-address.52601/page-2#post-415493

proxmox-firewall:

Stefan Hanreich (34):
  config: add proxmox-ve-config crate
  config: firewall: add types for ip addresses
  config: firewall: add types for ports
  config: firewall: add types for log level and rate limit
  config: firewall: add types for aliases
  config: host: add helpers for host network configuration
  config: guest: add helpers for parsing guest network config
  config: firewall: add types for ipsets
  config: firewall: add types for rules
  config: firewall: add types for security groups
  config: firewall: add generic parser for firewall configs
  config: firewall: add cluster-specific config + option types
  config: firewall: add host specific config + option types
  config: firewall: add guest-specific config + option types
  config: firewall: add firewall macros
  config: firewall: add conntrack helper types
  nftables: add crate for libnftables bindings
  nftables: add helpers
  nftables: expression: add types
  nftables: expression: implement conversion traits for firewall config
  nftables: statement: add types
  nftables: statement: add conversion traits for config types
  nftables: commands: add types
  nftables: types: add conversion traits
  nftables: add nft client
  firewall: add firewall crate
  firewall: add base ruleset
  firewall: add config loader
  firewall: add rule generation logic
  firewall: add object generation logic
  firewall: add ruleset generation logic
  firewall: add proxmox-firewall binary and move existing code into lib
  firewall: add files for debian packaging
  firewall: add integration test


qemu-server:

Stefan Hanreich (1):
  firewall: add handling for new nft firewall

 vm-network-scripts/pve-bridge | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)


pve-container:

Stefan Hanreich (1):
  firewall: add handling for new nft firewall

 src/PVE/LXC.pm | 5 +++++
 1 file changed, 5 insertions(+)


pve-firewall:

Stefan Hanreich (1):
  add configuration option for new nftables firewall

 src/PVE/Firewall.pm | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)


pve-manager:

Stefan Hanreich (1):
  firewall: expose configuration option for new nftables firewall

 www/manager6/grid/FirewallOptions.js | 1 +
 1 file changed, 1 insertion(+)


pve-docs:

Stefan Hanreich (1):
  firewall: add documentation for proxmox-firewall

 pve-firewall.adoc | 185 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 185 insertions(+)


Summary over all repositories:
  5 files changed, 214 insertions(+), 6 deletions(-)

-- 
Generated by git-murpp 0.6.0

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel


^ permalink raw reply	[flat|nested] 42+ messages in thread

end of thread, other threads:[~2024-04-19  7:48 UTC | newest]

Thread overview: 42+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-18 16:13 [pve-devel] [PATCH container/docs/firewall/manager/proxmox-firewall/qemu-server v3 00/39] proxmox firewall nftables implementation Stefan Hanreich
2024-04-18 16:13 ` [pve-devel] [PATCH proxmox-firewall v3 01/39] config: add proxmox-ve-config crate Stefan Hanreich
2024-04-18 16:13 ` [pve-devel] [PATCH proxmox-firewall v3 02/39] config: firewall: add types for ip addresses Stefan Hanreich
2024-04-18 16:13 ` [pve-devel] [PATCH proxmox-firewall v3 03/39] config: firewall: add types for ports Stefan Hanreich
2024-04-18 16:13 ` [pve-devel] [PATCH proxmox-firewall v3 04/39] config: firewall: add types for log level and rate limit Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 05/39] config: firewall: add types for aliases Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 06/39] config: host: add helpers for host network configuration Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 07/39] config: guest: add helpers for parsing guest network config Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 08/39] config: firewall: add types for ipsets Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 09/39] config: firewall: add types for rules Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 10/39] config: firewall: add types for security groups Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 11/39] config: firewall: add generic parser for firewall configs Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 12/39] config: firewall: add cluster-specific config + option types Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 13/39] config: firewall: add host specific " Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 14/39] config: firewall: add guest-specific " Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 15/39] config: firewall: add firewall macros Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 16/39] config: firewall: add conntrack helper types Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 17/39] nftables: add crate for libnftables bindings Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 18/39] nftables: add helpers Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 19/39] nftables: expression: add types Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 20/39] nftables: expression: implement conversion traits for firewall config Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 21/39] nftables: statement: add types Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 22/39] nftables: statement: add conversion traits for config types Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 23/39] nftables: commands: add types Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 24/39] nftables: types: add conversion traits Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 25/39] nftables: add nft client Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 26/39] firewall: add firewall crate Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 27/39] firewall: add base ruleset Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 28/39] firewall: add config loader Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 29/39] firewall: add rule generation logic Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 30/39] firewall: add object " Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 31/39] firewall: add ruleset " Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 32/39] firewall: add proxmox-firewall binary and move existing code into lib Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 33/39] firewall: add files for debian packaging Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH proxmox-firewall v3 34/39] firewall: add integration test Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH qemu-server v3 35/39] firewall: add handling for new nft firewall Stefan Hanreich
2024-04-18 21:08   ` Thomas Lamprecht
2024-04-18 16:14 ` [pve-devel] [PATCH pve-container v3 36/39] " Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH pve-firewall v3 37/39] add configuration option for new nftables firewall Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH pve-manager v3 38/39] firewall: expose " Stefan Hanreich
2024-04-18 16:14 ` [pve-devel] [PATCH pve-docs v3 39/39] firewall: add documentation for proxmox-firewall Stefan Hanreich
2024-04-18 20:05 ` [pve-devel] partially-applied-series: [PATCH container/docs/firewall/manager/proxmox-firewall/qemu-server v3 00/39] proxmox firewall nftables implementation Thomas Lamprecht

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal