From: Stefan Hanreich <s.hanreich@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: Re: [PATCH pve-network v3 15/32] sdn: add optional VRFs for simple zones
Date: Wed, 18 Nov 2026 17:18:33 +0100 [thread overview]
Message-ID: <4fd4b511-43f7-4cde-80c6-dd387280517c@proxmox.com> (raw)
In-Reply-To: <20261116161154.343743-16-g.goller@proxmox.com>
On 11/16/26 5:16 PM, Gabriel Goller wrote:
> Allow simple zones to isolate routing with dedicated-vrf. It is
> disabled by default to preserve existing behavior.
>
> Create the VRF even when the zone has no VNets, so fabrics can use it
> without requiring a VNet.
>
> Add unreachable IPv4 and IPv6 default routes with the highest metric
> to the VRF, as EVPN does on non-exit nodes. Otherwise lookups that find
> no route in the VRF table fall through to the main table, e.g. if no
> default route is configured or before FRR has installed it.
>
> Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
> ---
> src/PVE/API2/Network/SDN/Zones.pm | 6 ++++
> src/PVE/Network/SDN/Zones.pm | 19 +++++++++--
> src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 5 +--
> src/PVE/Network/SDN/Zones/Plugin.pm | 11 ++++++
> src/PVE/Network/SDN/Zones/SimplePlugin.pm | 32 +++++++++++++++--
> .../hetzner-vrf/expected_sdn_interfaces | 27 +++++++++++++++
> src/test/zones/simple/hetzner-vrf/interfaces | 6 ++++
> src/test/zones/simple/hetzner-vrf/sdn_config | 34 +++++++++++++++++++
> .../simple/vrf-empty/expected_sdn_interfaces | 7 ++++
> src/test/zones/simple/vrf-empty/interfaces | 2 ++
> src/test/zones/simple/vrf-empty/sdn_config | 12 +++++++
> .../zones/simple/vrf/expected_sdn_interfaces | 14 ++++++++
> src/test/zones/simple/vrf/interfaces | 2 ++
> src/test/zones/simple/vrf/sdn_config | 17 ++++++++++
> 14 files changed, 187 insertions(+), 7 deletions(-)
> create mode 100644 src/test/zones/simple/hetzner-vrf/expected_sdn_interfaces
> create mode 100644 src/test/zones/simple/hetzner-vrf/interfaces
> create mode 100644 src/test/zones/simple/hetzner-vrf/sdn_config
> create mode 100644 src/test/zones/simple/vrf-empty/expected_sdn_interfaces
> create mode 100644 src/test/zones/simple/vrf-empty/interfaces
> create mode 100644 src/test/zones/simple/vrf-empty/sdn_config
> create mode 100644 src/test/zones/simple/vrf/expected_sdn_interfaces
> create mode 100644 src/test/zones/simple/vrf/interfaces
> create mode 100644 src/test/zones/simple/vrf/sdn_config
>
> diff --git a/src/PVE/API2/Network/SDN/Zones.pm b/src/PVE/API2/Network/SDN/Zones.pm
> index 74b10f032bf4..38d950fda076 100644
> --- a/src/PVE/API2/Network/SDN/Zones.pm
> +++ b/src/PVE/API2/Network/SDN/Zones.pm
> @@ -61,6 +61,12 @@ my $ZONE_PROPERTIES = {
> optional => 1,
> description => 'Name of DHCP server backend for this zone.',
> },
> + 'dedicated-vrf' => {
> + type => 'boolean',
> + default => 0,
> + optional => 1,
> + description => 'Create a dedicated VRF for a simple zone.',
> + },
> 'rt-import' => {
> type => 'string',
> optional => 1,
> diff --git a/src/PVE/Network/SDN/Zones.pm b/src/PVE/Network/SDN/Zones.pm
> index c189652a79eb..cf81c002c4a5 100644
> --- a/src/PVE/Network/SDN/Zones.pm
> +++ b/src/PVE/Network/SDN/Zones.pm
> @@ -124,6 +124,9 @@ sub generate_etc_network_config {
> my $controller_cfg = $cfg->{controllers};
> return if !$vnet_cfg && !$zone_cfg;
>
> + my $vnet_ids = $vnet_cfg ? ($vnet_cfg->{ids} // {}) : {};
> + my $zone_ids = $zone_cfg ? ($zone_cfg->{ids} // {}) : {};
> +
> my $interfaces_config = PVE::INotify::read_file('interfaces');
> my $bridge_ports = PVE::IPRoute2::get_bridge_port_mapping();
>
> @@ -131,8 +134,18 @@ sub generate_etc_network_config {
> my $config = {};
> my $nodename = PVE::INotify::nodename();
>
> - for my $id (sort keys %{ $vnet_cfg->{ids} }) {
> - my $vnet = $vnet_cfg->{ids}->{$id};
> + for my $zone_id (sort keys %$zone_ids) {
> + my $plugin_config = $zone_ids->{$zone_id};
> + next if $plugin_config->{type} ne 'simple';
> + next if defined($plugin_config->{nodes}) && !$plugin_config->{nodes}->{$nodename};
> +
> + PVE::Network::SDN::Zones::SimplePlugin->generate_zone_config(
fine for now, but shouldn't we make this a method of the ZonePlugin?
> + $plugin_config, $zone_id, $config,
> + );
> + }
> +
> + for my $id (sort keys %$vnet_ids) {
> + my $vnet = $vnet_ids->{$id};
> my $zone = $vnet->{zone};
>
> if (!$zone) {
> @@ -140,7 +153,7 @@ sub generate_etc_network_config {
> next;
> }
>
> - my $plugin_config = $zone_cfg->{ids}->{$zone};
> + my $plugin_config = $zone_ids->{$zone};
>
> if (!defined($plugin_config)) {
> warn "can't generate vnet '$id': zone $zone don't exist\n";
> diff --git a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> index 45a628d6a69d..534ac9ffa5b1 100644
> --- a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> +++ b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> @@ -319,8 +319,9 @@ sub generate_sdn_config {
> push @iface_config, "vrf-table auto";
> my $route_action = $is_evpn_gateway ? 'del' : 'add';
> push @iface_config,
> - "post-up ip route $route_action vrf $vrf_iface unreachable default metric 4278198272",
> - "post-up ip -6 route $route_action vrf $vrf_iface unreachable default metric 4278198272";
> + PVE::Network::SDN::Zones::Plugin::vrf_unreachable_default_cmds(
> + $vrf_iface, $route_action,
> + );
>
> push(@{ $config->{$vrf_iface} }, @iface_config) if !$config->{$vrf_iface};
>
> diff --git a/src/PVE/Network/SDN/Zones/Plugin.pm b/src/PVE/Network/SDN/Zones/Plugin.pm
> index f3844ffda3b5..b3b4261c58af 100644
> --- a/src/PVE/Network/SDN/Zones/Plugin.pm
> +++ b/src/PVE/Network/SDN/Zones/Plugin.pm
> @@ -99,6 +99,17 @@ sub encode_value {
> return $value;
> }
>
> +# Unreachable IPv4 and IPv6 default routes with the highest metric, so that
> +# lookups without a matching route in the VRF table do not fall through to the
> +# main table. $action is 'add' or 'del'.
> +sub vrf_unreachable_default_cmds {
> + my ($vrf_iface, $action) = @_;
> +
> + return map {
> + "post-up ip${_} route $action vrf $vrf_iface unreachable default metric 4278198272"
> + } ('', ' -6');
> +}
> +
> sub generate_sdn_config {
> my (
> $class,
> diff --git a/src/PVE/Network/SDN/Zones/SimplePlugin.pm b/src/PVE/Network/SDN/Zones/SimplePlugin.pm
> index 347eee9a46ba..bc591839819d 100644
> --- a/src/PVE/Network/SDN/Zones/SimplePlugin.pm
> +++ b/src/PVE/Network/SDN/Zones/SimplePlugin.pm
> @@ -35,6 +35,11 @@ sub properties {
> type => 'string',
> enum => PVE::Network::SDN::Dhcp->plugin_types(),
> },
> + 'dedicated-vrf' => {
> + description => 'Create a dedicated VRF for the zone.',
> + type => 'boolean',
> + default => 0,
> + },
> };
> }
>
> @@ -47,15 +52,34 @@ sub options {
> dnszone => { optional => 1 },
> ipam => { optional => 1 },
> dhcp => { optional => 1 },
> + 'dedicated-vrf' => { optional => 1 },
> };
> }
>
> +sub generate_zone_config {
> + my ($class, $plugin_config, $zone_id, $config) = @_;
> +
> + if ($plugin_config->{'dedicated-vrf'}) {
> + my $vrf_iface = "vrf_$zone_id";
> + if (!defined($config->{$vrf_iface})) {
> + $config->{$vrf_iface} = [
> + 'vrf-table auto',
> + PVE::Network::SDN::Zones::Plugin::vrf_unreachable_default_cmds(
> + $vrf_iface, 'add',
> + ),
> + ];
> + }
> + }
> +
> + return $config;
> +}
> +
> # Plugin implementation
> sub generate_sdn_config {
> my (
> $class,
> $plugin_config,
> - $zoneid,
> + $zone_id,
> $vnetid,
> $vnet,
> $controller,
> @@ -110,7 +134,10 @@ sub generate_sdn_config {
> }
>
> #add route for /32 pointtopoint
> - push @iface_config, "up ip route add $cidr dev $vnetid" if $mask == 32 && $ipversion == 4;
> + if ($mask == 32 && $ipversion == 4) {
> + my $vrf = $plugin_config->{'dedicated-vrf'} ? " vrf vrf_$zone_id" : '';
> + push @iface_config, "up ip route add $cidr dev $vnetid$vrf";
> + }
> if ($subnet->{snat}) {
> #find outgoing interface
> my ($outip, $outiface);
> @@ -150,6 +177,7 @@ sub generate_sdn_config {
> push @iface_config, "alias $alias" if $alias;
> push @iface_config, "ip-forward on" if $enable_forward_v4;
> push @iface_config, "ip6-forward on" if $enable_forward_v6;
> + push @iface_config, "vrf vrf_$zone_id" if $plugin_config->{'dedicated-vrf'};
>
> push @{ $config->{$vnetid} }, @iface_config;
>
> diff --git a/src/test/zones/simple/hetzner-vrf/expected_sdn_interfaces b/src/test/zones/simple/hetzner-vrf/expected_sdn_interfaces
> new file mode 100644
> index 000000000000..a94d091dcf50
> --- /dev/null
> +++ b/src/test/zones/simple/hetzner-vrf/expected_sdn_interfaces
> @@ -0,0 +1,27 @@
> +#version:1
> +
> +auto myvnet
> +iface myvnet
> + address 144.76.100.65/29
> + bridge_ports none
> + bridge_stp off
> + bridge_fd 0
> + ip-forward on
> + vrf vrf_myzone
> +
> +auto myvnet2
> +iface myvnet2
> + address 144.76.0.1/32
> + up ip route add 144.76.200.65/32 dev myvnet2 vrf vrf_myzone
> + up ip route add 144.76.200.66/32 dev myvnet2 vrf vrf_myzone
> + bridge_ports none
> + bridge_stp off
> + bridge_fd 0
> + ip-forward on
> + vrf vrf_myzone
> +
> +auto vrf_myzone
> +iface vrf_myzone
> + vrf-table auto
> + post-up ip route add vrf vrf_myzone unreachable default metric 4278198272
> + post-up ip -6 route add vrf vrf_myzone unreachable default metric 4278198272
> diff --git a/src/test/zones/simple/hetzner-vrf/interfaces b/src/test/zones/simple/hetzner-vrf/interfaces
> new file mode 100644
> index 000000000000..53e894aaa4ef
> --- /dev/null
> +++ b/src/test/zones/simple/hetzner-vrf/interfaces
> @@ -0,0 +1,6 @@
> +auto eth0
> +iface eth0 inet static
> + address 144.76.0.1
> + netmask 255.255.255.255
> + pointopoint 172.31.1.1
> + gateway 172.31.1.1
> diff --git a/src/test/zones/simple/hetzner-vrf/sdn_config b/src/test/zones/simple/hetzner-vrf/sdn_config
> new file mode 100644
> index 000000000000..847c42da81a4
> --- /dev/null
> +++ b/src/test/zones/simple/hetzner-vrf/sdn_config
> @@ -0,0 +1,34 @@
> +{
> + version => 1,
> + vnets => {
> + ids => {
> + myvnet => { type => "vnet", zone => "myzone" },
> + myvnet2 => { type => "vnet", zone => "myzone" },
> + },
> + },
> + zones => {
> + ids => { myzone => { ipam => "pve", type => "simple", "dedicated-vrf" => 1 } },
> + },
> +
> + subnets => {
> + ids => {
> + 'myzone-144.76.100.64-29' => {
> + 'type' => 'subnet',
> + 'vnet' => 'myvnet',
> + 'gateway' => '144.76.100.65',
> + },
> + 'myzone-144.76.200.65-32' => {
> + 'type' => 'subnet',
> + 'vnet' => 'myvnet2',
> + 'gateway' => '144.76.0.1',
> + },
> + 'myzone-144.76.200.66-32' => {
> + 'type' => 'subnet',
> + 'vnet' => 'myvnet2',
> + 'gateway' => '144.76.0.1',
> + },
> + }
> + }
> +}
> +
> +
> diff --git a/src/test/zones/simple/vrf-empty/expected_sdn_interfaces b/src/test/zones/simple/vrf-empty/expected_sdn_interfaces
> new file mode 100644
> index 000000000000..d3b7de26d460
> --- /dev/null
> +++ b/src/test/zones/simple/vrf-empty/expected_sdn_interfaces
> @@ -0,0 +1,7 @@
> +#version:1
> +
> +auto vrf_myzone
> +iface vrf_myzone
> + vrf-table auto
> + post-up ip route add vrf vrf_myzone unreachable default metric 4278198272
> + post-up ip -6 route add vrf vrf_myzone unreachable default metric 4278198272
> diff --git a/src/test/zones/simple/vrf-empty/interfaces b/src/test/zones/simple/vrf-empty/interfaces
> new file mode 100644
> index 000000000000..f1bd92ed2b43
> --- /dev/null
> +++ b/src/test/zones/simple/vrf-empty/interfaces
> @@ -0,0 +1,2 @@
> +auto lo
> +iface lo inet loopback
> diff --git a/src/test/zones/simple/vrf-empty/sdn_config b/src/test/zones/simple/vrf-empty/sdn_config
> new file mode 100644
> index 000000000000..32fd5ea50432
> --- /dev/null
> +++ b/src/test/zones/simple/vrf-empty/sdn_config
> @@ -0,0 +1,12 @@
> +{
> + version => 1,
> + vnets => { ids => {} },
> + zones => {
> + ids => {
> + myzone => {
> + type => "simple",
> + "dedicated-vrf" => 1,
> + },
> + },
> + },
> +}
> diff --git a/src/test/zones/simple/vrf/expected_sdn_interfaces b/src/test/zones/simple/vrf/expected_sdn_interfaces
> new file mode 100644
> index 000000000000..777adafa9f48
> --- /dev/null
> +++ b/src/test/zones/simple/vrf/expected_sdn_interfaces
> @@ -0,0 +1,14 @@
> +#version:1
> +
> +auto myvnet
> +iface myvnet
> + bridge_ports none
> + bridge_stp off
> + bridge_fd 0
> + vrf vrf_myzone
> +
> +auto vrf_myzone
> +iface vrf_myzone
> + vrf-table auto
> + post-up ip route add vrf vrf_myzone unreachable default metric 4278198272
> + post-up ip -6 route add vrf vrf_myzone unreachable default metric 4278198272
> diff --git a/src/test/zones/simple/vrf/interfaces b/src/test/zones/simple/vrf/interfaces
> new file mode 100644
> index 000000000000..f1bd92ed2b43
> --- /dev/null
> +++ b/src/test/zones/simple/vrf/interfaces
> @@ -0,0 +1,2 @@
> +auto lo
> +iface lo inet loopback
> diff --git a/src/test/zones/simple/vrf/sdn_config b/src/test/zones/simple/vrf/sdn_config
> new file mode 100644
> index 000000000000..c57074e1bf4a
> --- /dev/null
> +++ b/src/test/zones/simple/vrf/sdn_config
> @@ -0,0 +1,17 @@
> +{
> + version => 1,
> + vnets => {
> + ids => {
> + myvnet => { type => "vnet", zone => "myzone" },
> + },
> + },
> + zones => {
> + ids => {
> + myzone => {
> + ipam => "pve",
> + type => "simple",
> + "dedicated-vrf" => 1,
> + },
> + },
> + },
> +}
next prev parent reply other threads:[~2026-11-18 16:18 UTC|newest]
Thread overview: 65+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 16:11 [PATCH common/manager/network/proxmox{-ve-rs,-perl-rs,-widget-toolkit} v3 00/32] SDN VRF support Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-common v3 01/32] interfaces: preserve VRF membership across address family changes Gabriel Goller
2026-11-18 14:13 ` Stefan Hanreich
2026-11-18 15:41 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-ve-rs v3 02/32] frr: add VRF-aware OSPF rendering Gabriel Goller
2026-11-18 14:41 ` Stefan Hanreich
2026-11-18 16:21 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-ve-rs v3 03/32] sdn: add zone references to OSPF and BGP fabrics Gabriel Goller
2026-11-18 14:10 ` Stefan Hanreich
2026-11-18 16:55 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-ve-rs v3 04/32] sdn: generate fabric routing configuration in zone VRFs Gabriel Goller
2026-11-18 14:53 ` Stefan Hanreich
2026-11-18 17:26 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-ve-rs v3 05/32] sdn: fix VRF route-map scoping and zone ID validation Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-ve-rs v3 06/32] tests: fabrics: add test for fabrics in VRFs Gabriel Goller
2026-11-18 15:01 ` Stefan Hanreich
2026-11-18 17:33 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-ve-rs v3 07/32] sdn: allow static routes in simple zone VRFs Gabriel Goller
2026-11-18 15:29 ` Stefan Hanreich
2026-11-18 18:10 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-ve-rs v3 08/32] sdn: support static routes in EVPN " Gabriel Goller
2026-11-18 15:32 ` Stefan Hanreich
2026-11-19 13:12 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-perl-rs v3 09/32] pve-rs: fabrics: assign network interfaces to configured VRFs Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-perl-rs v3 10/32] pve-rs: fabrics: make per-fabric status queries VRF-aware Gabriel Goller
2026-11-18 15:42 ` Stefan Hanreich
2026-11-19 13:30 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-perl-rs v3 11/32] pve-rs: fabrics: include VRF routes in aggregate status Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-perl-rs v3 12/32] pve-rs: fabrics: expose host interface membership Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-perl-rs v3 13/32] sdn: expose static routes for simple zone uplinks Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-network v3 14/32] sdn: evpn: add unreachable IPv6 default route to zone VRFs Gabriel Goller
2026-11-18 16:08 ` Stefan Hanreich
2026-11-18 16:39 ` Stefan Hanreich
2026-11-19 13:38 ` Gabriel Goller
2026-11-19 15:51 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-network v3 15/32] sdn: add optional VRFs for simple zones Gabriel Goller
2026-11-18 16:18 ` Stefan Hanreich [this message]
2026-11-19 13:44 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-network v3 16/32] sdn: allow fabrics to use simple zone VRFs Gabriel Goller
2026-11-18 16:27 ` Stefan Hanreich
2026-11-19 14:14 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-network v3 17/32] sdn: require underlay fabrics to use the default VRF Gabriel Goller
2026-11-18 16:30 ` Stefan Hanreich
2026-11-19 14:33 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-network v3 18/32] sdn: always generate EVPN zone VRFs Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-network v3 19/32] api: sdn: allow EVPN zones as fabric VRFs Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-network v3 20/32] api: sdn: disallow BGP fabrics in EVPN zone VRFs Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-network v3 21/32] sdn: simple: support static routes in zone routing tables Gabriel Goller
2026-11-18 16:49 ` Stefan Hanreich
2026-11-19 14:53 ` Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-network v3 22/32] sdn: evpn: allow static routes Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-network v3 23/32] sdn: zones: SNAT via dedicated uplinks in zone VRFs Gabriel Goller
2026-11-18 18:07 ` Stefan Hanreich
2026-11-16 16:11 ` [PATCH pve-network v3 24/32] sdn: dhcp: only configure DHCP for zones on the local node Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-network v3 25/32] sdn: dhcp: run dnsmasq in the zone VRF Gabriel Goller
2026-11-16 16:11 ` [PATCH proxmox-widget-toolkit v3 26/32] ui: network: allow selecting a VRF for host interfaces Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-manager v3 27/32] ui: sdn: add VRF zone selection for fabrics Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-manager v3 28/32] ui: sdn: expose EVPN zones as fabric VRFs Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-manager v3 29/32] network: allow setting the VRF of host interfaces Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-manager v3 30/32] ui: sdn: allow static routes for Simple zone VRFs Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-manager v3 31/32] ui: sdn: allow static routes in EVPN zones Gabriel Goller
2026-11-16 16:11 ` [PATCH pve-manager v3 32/32] network: generate SDN DHCP config after ifreload Gabriel Goller
2026-11-18 18:27 ` [PATCH common/manager/network/proxmox{-ve-rs,-perl-rs,-widget-toolkit} v3 00/32] SDN VRF support Stefan Hanreich
2026-11-19 15:33 ` Gabriel Goller
2026-11-19 20:10 ` Gabriel Goller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4fd4b511-43f7-4cde-80c6-dd387280517c@proxmox.com \
--to=s.hanreich@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox