Re: [pve-devel] [PATCH docs/firewall/manager/proxmox{-ve-rs, -firewall, -perl-rs} v3 00/24] autogenerate ipsets for sdn objects

[Stefan Hanreich <s.hanreich@proxmox.com>]

v4 here:
https://lore.proxmox.com/pve-devel/20241115120937.169342-1-s.hanreich@proxmox.com/T/

On 11/12/24 13:25, Stefan Hanreich wrote:
> This patch series adds support for autogenerating ipsets for SDN objects. It
> autogenerates ipsets for every VNet as follows:
> 
> * ipset containing all IP ranges of the VNet
> * ipset containing all gateways of the VNet
> * ipset containing all IP ranges of the subnet - except gateways
> * ipset containing all dhcp ranges of the vnet
> 
> Additionally it generates an IPSet for every guest that has one or more IPAM
> entries in the pve IPAM.
> 
> Those can then be used in the cluster / host / guest firewalls. Firewall rules
> automatically update on changes of the SDN / IPAM configuration. This patch
> series works for the old firewall as well as the new firewall.
> 
> The ipsets in nftables currently get generated as named ipsets in every table,
> this means that the `nft list ruleset` output can get quite crowded for large
> SDN configurations or large IPAM databases. Another option would be to only
> include them as anonymous IPsets in the rules, which would make the nft output
> far less crowded but this way would use more memory when making extensive use of
> the sdn ipsets, since everytime it is used in a rule we create an entirely new
> ipset.
> 
> The base for proxmox-ve-rs (which is a filtered version of the proxmox-firewall
> repository can be found here:)
> 
> staff/s.hanreich/proxmox-ve-rs.git master
> 
> Dependencies:
> * proxmox-perl-rs and proxmox-firewall depend on proxmox-ve-rs
> * pve-firewall depends on proxmox-perl-rs
> * pve-manager depends on pve-firewall
> 
> Changes from v2:
> * rename end in IpRange to last to avoid confusion - thanks @Wolfgang
> * bump Rust to 1.82 - thanks @Wolfgang
> * improvements to the code generating IPSets - thanks @Wolfgang
> * implement AsRef<str> for SDN name types - thanks @Wolfgang
> * improve docstrings (proper capitalization and punctuation) - thanks @Wolfgang
> * included a patch that removes proxmox-ve-config from proxmox-firewall
> 
> Changes from RFC:
> * added documentation
> * added separate SDN scope for IPSets
> * rustfmt fixes
> 
> proxmox-ve-rs:
> 
> Stefan Hanreich (16):
>   debian: add files for packaging
>   firewall: add sdn scope for ipsets
>   firewall: add ip range types
>   firewall: address: use new iprange type for ip entries
>   ipset: add range variant to addresses
>   iprange: add methods for converting an ip range to cidrs
>   ipset: address: add helper methods
>   firewall: guest: derive traits according to rust api guidelines
>   common: add allowlist
>   sdn: add name types
>   sdn: add ipam module
>   sdn: ipam: add method for generating ipsets
>   sdn: add config module
>   sdn: config: add method for generating ipsets
>   tests: add sdn config tests
>   tests: add ipam tests
> 
>  .cargo/config.toml                            |    5 +
>  .gitignore                                    |    8 +
>  Cargo.toml                                    |   17 +
>  Makefile                                      |   69 +
>  build.sh                                      |   35 +
>  bump.sh                                       |   44 +
>  proxmox-ve-config/Cargo.toml                  |   19 +-
>  proxmox-ve-config/debian/changelog            |    5 +
>  proxmox-ve-config/debian/control              |   46 +
>  proxmox-ve-config/debian/copyright            |   19 +
>  proxmox-ve-config/debian/debcargo.toml        |    4 +
>  proxmox-ve-config/src/common/mod.rs           |   31 +
>  .../src/firewall/types/address.rs             | 1171 ++++++++++++++++-
>  proxmox-ve-config/src/firewall/types/alias.rs |    4 +-
>  proxmox-ve-config/src/firewall/types/ipset.rs |   32 +-
>  proxmox-ve-config/src/firewall/types/rule.rs  |    6 +-
>  proxmox-ve-config/src/guest/types.rs          |    7 +-
>  proxmox-ve-config/src/guest/vm.rs             |   11 +-
>  proxmox-ve-config/src/lib.rs                  |    2 +
>  proxmox-ve-config/src/sdn/config.rs           |  640 +++++++++
>  proxmox-ve-config/src/sdn/ipam.rs             |  368 ++++++
>  proxmox-ve-config/src/sdn/mod.rs              |  251 ++++
>  proxmox-ve-config/tests/sdn/main.rs           |  189 +++
>  proxmox-ve-config/tests/sdn/resources/ipam.db |   26 +
>  .../tests/sdn/resources/running-config.json   |   54 +
>  25 files changed, 2976 insertions(+), 87 deletions(-)
>  create mode 100644 .cargo/config.toml
>  create mode 100644 .gitignore
>  create mode 100644 Cargo.toml
>  create mode 100644 Makefile
>  create mode 100755 build.sh
>  create mode 100755 bump.sh
>  create mode 100644 proxmox-ve-config/debian/changelog
>  create mode 100644 proxmox-ve-config/debian/control
>  create mode 100644 proxmox-ve-config/debian/copyright
>  create mode 100644 proxmox-ve-config/debian/debcargo.toml
>  create mode 100644 proxmox-ve-config/src/common/mod.rs
>  create mode 100644 proxmox-ve-config/src/sdn/config.rs
>  create mode 100644 proxmox-ve-config/src/sdn/ipam.rs
>  create mode 100644 proxmox-ve-config/src/sdn/mod.rs
>  create mode 100644 proxmox-ve-config/tests/sdn/main.rs
>  create mode 100644 proxmox-ve-config/tests/sdn/resources/ipam.db
>  create mode 100644 proxmox-ve-config/tests/sdn/resources/running-config.json
> 
> 
> proxmox-firewall:
> 
> Stefan Hanreich (3):
>   add proxmox-ve-rs crate - move proxmox-ve-config there
>   config: tests: add support for loading sdn and ipam config
>   ipsets: autogenerate ipsets for vnets and ipam
> 
>  Cargo.toml                                    |    4 +-
>  Makefile                                      |    2 +-
>  proxmox-firewall/Cargo.toml                   |    2 +-
>  proxmox-firewall/src/config.rs                |   69 +
>  proxmox-firewall/src/firewall.rs              |   22 +-
>  proxmox-firewall/src/object.rs                |   41 +-
>  .../tests/input/.running-config.json          |   45 +
>  proxmox-firewall/tests/input/ipam.db          |   32 +
>  proxmox-firewall/tests/integration_tests.rs   |   10 +
>  .../integration_tests__firewall.snap          | 1288 +++++++++++++++++
>  proxmox-nftables/Cargo.toml                   |    2 +-
>  proxmox-nftables/src/expression.rs            |   17 +-
>  proxmox-nftables/src/types.rs                 |    2 +-
>  proxmox-ve-config/Cargo.toml                  |   25 -
>  proxmox-ve-config/resources/ct_helper.json    |   52 -
>  proxmox-ve-config/resources/macros.json       |  923 ------------
>  proxmox-ve-config/src/firewall/cluster.rs     |  374 -----
>  proxmox-ve-config/src/firewall/common.rs      |  184 ---
>  proxmox-ve-config/src/firewall/ct_helper.rs   |  115 --
>  proxmox-ve-config/src/firewall/fw_macros.rs   |   69 -
>  proxmox-ve-config/src/firewall/guest.rs       |  237 ---
>  proxmox-ve-config/src/firewall/host.rs        |  372 -----
>  proxmox-ve-config/src/firewall/mod.rs         |   10 -
>  proxmox-ve-config/src/firewall/parse.rs       |  494 -------
>  proxmox-ve-config/src/firewall/ports.rs       |   80 -
>  .../src/firewall/types/address.rs             |  615 --------
>  proxmox-ve-config/src/firewall/types/alias.rs |  174 ---
>  proxmox-ve-config/src/firewall/types/group.rs |   36 -
>  proxmox-ve-config/src/firewall/types/ipset.rs |  349 -----
>  proxmox-ve-config/src/firewall/types/log.rs   |  222 ---
>  proxmox-ve-config/src/firewall/types/mod.rs   |   14 -
>  proxmox-ve-config/src/firewall/types/port.rs  |  181 ---
>  proxmox-ve-config/src/firewall/types/rule.rs  |  412 ------
>  .../src/firewall/types/rule_match.rs          |  977 -------------
>  proxmox-ve-config/src/guest/mod.rs            |  115 --
>  proxmox-ve-config/src/guest/types.rs          |   38 -
>  proxmox-ve-config/src/guest/vm.rs             |  510 -------
>  proxmox-ve-config/src/host/mod.rs             |    1 -
>  proxmox-ve-config/src/host/utils.rs           |   70 -
>  proxmox-ve-config/src/lib.rs                  |    3 -
>  40 files changed, 1517 insertions(+), 6671 deletions(-)
>  create mode 100644 proxmox-firewall/tests/input/.running-config.json
>  create mode 100644 proxmox-firewall/tests/input/ipam.db
>  delete mode 100644 proxmox-ve-config/Cargo.toml
>  delete mode 100644 proxmox-ve-config/resources/ct_helper.json
>  delete mode 100644 proxmox-ve-config/resources/macros.json
>  delete mode 100644 proxmox-ve-config/src/firewall/cluster.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/common.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/ct_helper.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/fw_macros.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/guest.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/host.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/mod.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/parse.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/ports.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/types/address.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/types/alias.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/types/group.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/types/ipset.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/types/log.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/types/mod.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/types/port.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/types/rule.rs
>  delete mode 100644 proxmox-ve-config/src/firewall/types/rule_match.rs
>  delete mode 100644 proxmox-ve-config/src/guest/mod.rs
>  delete mode 100644 proxmox-ve-config/src/guest/types.rs
>  delete mode 100644 proxmox-ve-config/src/guest/vm.rs
>  delete mode 100644 proxmox-ve-config/src/host/mod.rs
>  delete mode 100644 proxmox-ve-config/src/host/utils.rs
>  delete mode 100644 proxmox-ve-config/src/lib.rs
> 
> 
> pve-firewall:
> 
> Stefan Hanreich (2):
>   add support for loading sdn firewall configuration
>   api: load sdn ipsets
> 
>  src/PVE/API2/Firewall/Cluster.pm |  8 +++--
>  src/PVE/API2/Firewall/Rules.pm   | 12 ++++---
>  src/PVE/API2/Firewall/VM.pm      |  3 +-
>  src/PVE/Firewall.pm              | 59 ++++++++++++++++++++++++++++----
>  4 files changed, 67 insertions(+), 15 deletions(-)
> 
> 
> proxmox-perl-rs:
> 
> Stefan Hanreich (1):
>   add PVE::RS::Firewall::SDN module
> 
>  pve-rs/Cargo.toml          |   1 +
>  pve-rs/Makefile            |   1 +
>  pve-rs/src/firewall/mod.rs |   1 +
>  pve-rs/src/firewall/sdn.rs | 130 +++++++++++++++++++++++++++++++++++++
>  pve-rs/src/lib.rs          |   1 +
>  5 files changed, 134 insertions(+)
>  create mode 100644 pve-rs/src/firewall/mod.rs
>  create mode 100644 pve-rs/src/firewall/sdn.rs
> 
> 
> pve-manager:
> 
> Stefan Hanreich (1):
>   firewall: add sdn scope to IPRefSelector
> 
>  www/manager6/form/IPRefSelector.js | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> 
> pve-docs:
> 
> Stefan Hanreich (1):
>   sdn: add documentation for firewall integration
> 
>  pvesdn.adoc | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 92 insertions(+)
> 
> 
> Summary over all repositories:
>   76 files changed, 4793 insertions(+), 6774 deletions(-)
> 



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel