[PVE-User] Problem with Centos 5.X virtio ethernet drivers and last PVE updates

[PVE-User] Problem with Centos 5.X virtio ethernet drivers and last PVE updates

[Fabrizio Cuseo]

Good morning.
I have just upgraded a 6.2 cluster from 6.2.2 to 6.2.10; suddently, all my Centos 5.X guests, with virtio ethernet drivers, stop working. Changing ethernet from virtio to e1000 fix the problem.

Someone else have this problem ? Something to fix and come back to virtio drivers ? (I can't upgrade those guests to newer centos).

Regards, Fabrizio 

-- 
---
Fabrizio Cuseo - mailto:f.cuseo@panservice.it
Direzione Generale - Panservice InterNetWorking
Servizi Professionali per Internet ed il Networking
Panservice e' associata AIIP - RIPE Local Registry
Phone: +39 0773 410020 - Fax: +39 0773 470219
http://www.panservice.it  mailto:info@panservice.it
Numero verde nazionale: 800 901492

Re: [PVE-User] Problem with Centos 5.X virtio ethernet drivers and last PVE updates

[richard lucassen]

On Tue, 21 Jul 2020 08:32:55 +0200 (CEST)
Fabrizio Cuseo <f.cuseo@panservice.it> wrote:

> Good morning.
> I have just upgraded a 6.2 cluster from 6.2.2 to 6.2.10; suddently,
> all my Centos 5.X guests, with virtio ethernet drivers, stop working.
> Changing ethernet from virtio to e1000 fix the problem.
> 
> Someone else have this problem ? Something to fix and come back to
> virtio drivers ? (I can't upgrade those guests to newer centos).

I noticed the same thing on pm-5.4 when upgrading from 4.15.18-27-pve
to 4.15.18-30-pve. Suddenly on (old) Debian Lenny guests the virtio
drivers got very unstable. I changed them to rtl8139 and it seems to be
much stabler, but not resolved. I created a cronjob that stops the
network, unloads the kernel modules, reloads them and restarts the
network. There is definitly something wrong somewhere.

R.


-- 
richard lucassen
http://contact.xaq.nl/

Re: [PVE-User] Problem with Centos 5.X virtio ethernet drivers and last PVE updates

[Fabrizio Cuseo]

I know that CentOS 5.X is an old release, but sometimes, if you have some customer's application, and he can't upgrade the OS due to some compatibility issue with applications, is not so simple to upgrade the OS release.
I only want to tell that with the previous minor release of PVE (and qemu/kvm of course), i had no problems at all.

Fabrizio


----- Il 21-lug-20, alle 11:06, Michael Rasmussen mir@miras.org ha scritto:

> On Tue, 21 Jul 2020 10:41:54 +0200
> richard lucassen <mailinglists@lucassen.org> wrote:
> 
>> network. There is definitly something wrong somewhere.
>> 
> The only thing wrong here is that you a running a version of
> RHEL/CentOS which is in maintenance support 2. This means there will be
> no more updates except critical bugs. As of March 31, 2017 it is
> considered in deep freeze!
> 
> --
> Hilsen/Regards
> Michael Rasmussen
> 
> Get my public GnuPG keys:
> michael <at> rasmussen <dot> cc
> https://pgp.key-server.io/pks/lookup?search=0xD3C9A00E
> mir <at> datanom <dot> net
> https://pgp.key-server.io/pks/lookup?search=0xE501F51C
> mir <at> miras <dot> org
> https://pgp.key-server.io/pks/lookup?search=0xE3E80917
> --------------------------------------------------------------
> /usr/games/fortune -es says:
> All intelligent species own cats.

-- 
---
Fabrizio Cuseo - mailto:f.cuseo@panservice.it
Direzione Generale - Panservice InterNetWorking
Servizi Professionali per Internet ed il Networking
Panservice e' associata AIIP - RIPE Local Registry
Phone: +39 0773 410020 - Fax: +39 0773 470219
http://www.panservice.it  mailto:info@panservice.it
Numero verde nazionale: 800 901492

Re: [PVE-User] Problem with Centos 5.X virtio ethernet drivers and last PVE updates

[richard lucassen]

On Tue, 21 Jul 2020 11:06:27 +0200
Michael Rasmussen via pve-user <pve-user@lists.proxmox.com> wrote:

> On Tue, 21 Jul 2020 10:41:54 +0200
> richard lucassen <mailinglists@lucassen.org> wrote:
> 
> > network. There is definitly something wrong somewhere.
> > 
> The only thing wrong here is that you a running a version of
> RHEL/CentOS which is in maintenance support 2. This means there will
> be no more updates except critical bugs. As of March 31, 2017 it is
> considered in deep freeze!

I know. But sometimes you need to keep an old version and a good way to
handle this is to run such an old version in a virtual environment.
A vhost supplies virtual hardware and apparently this virtual hardware
has changed. This is not A Good Thing IMHO.

It runs well under subversion 27 of the pve kernel but has stopped
under version 30. So I think this is a bug as I do not expect design
changes between subversion 27 and 30.

R. 

-- 
richard lucassen
https://contact.xaq.nl/

Re: [PVE-User] Problem with Centos 5.X virtio ethernet drivers and last PVE updates

[Roland]

i see that my centos 5.11 VM should have support for virtio (see below)
and according to https://access.redhat.com/articles/2488201 rhel/centos
supports this since 5.3.

so if virtio worked before a proxmox update,  then i would call this a
regression.

can you tell which proxmox version worked for you with virtio ?

i tested centos 5.11 installer on my older pve 6.0-4 (5.0.15-1-pve
kernel) , but it does not find virtual disks.

roland

[root@hr-neu ~]# uname -aLinux hr-neu 2.6.18-398.el5 #1 SMP Tue Sep 16
20:50:52 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux


[root@centos5 ~]# cat /etc/redhat-release
CentOS release 5.11 (Final)
[root@centos5 ~]# rpm -q kernel --changelog|grep virtio|grep 446214
- [xen] virtio: include headers in kernel-headers package (Eduardo
Pereira Habkost ) [446214]
- [xen] virtio: add PV network and block drivers for KVM (Mark
McLoughlin ) [446214]
- [xen] virtio: include headers in kernel-headers package (Eduardo
Pereira Habkost ) [446214]
- [xen] virtio: add PV network and block drivers for KVM (Mark
McLoughlin ) [446214]

Am 23.07.20 um 10:22 schrieb richard lucassen:
> On Tue, 21 Jul 2020 11:06:27 +0200
> Michael Rasmussen via pve-user <pve-user@lists.proxmox.com> wrote:
>
>> On Tue, 21 Jul 2020 10:41:54 +0200
>> richard lucassen <mailinglists@lucassen.org> wrote:
>>
>>> network. There is definitly something wrong somewhere.
>>>
>> The only thing wrong here is that you a running a version of
>> RHEL/CentOS which is in maintenance support 2. This means there will
>> be no more updates except critical bugs. As of March 31, 2017 it is
>> considered in deep freeze!
> I know. But sometimes you need to keep an old version and a good way to
> handle this is to run such an old version in a virtual environment.
> A vhost supplies virtual hardware and apparently this virtual hardware
> has changed. This is not A Good Thing IMHO.
>
> It runs well under subversion 27 of the pve kernel but has stopped
> under version 30. So I think this is a bug as I do not expect design
> changes between subversion 27 and 30.
>
> R.
>

Re: [PVE-User] Problem with Centos 5.X virtio ethernet drivers and last PVE updates

[richard lucassen]

On Thu, 23 Jul 2020 12:11:19 +0200
Roland <devzero@web.de> wrote:

I run a Debian Lenny VM:

# cat /etc/issue.net 
Debian GNU/Linux 5.0

# modinfo virtio_net
filename:       /lib/modules/2.6.26-2-686/kernel/drivers/net/virtio_net.ko
license:        GPL
description:    Virtio network driver
alias:          virtio:d00000001v*
depends:        
vermagic:       2.6.26-2-686 SMP mod_unload modversions 686 
parm:           napi_weight:int
parm:           csum:bool
parm:           gso:bool

Proxmox host version 5.4:

# cat /proc/version
Linux version 4.15.18-30-pve (root@nora) (gcc version 6.3.0 20170516
(Debian 6.3.0-18+deb9u1)) #1 SMP PVE 4.15.18-58 (Fri, 12 Jun 2020
13:53:01 +0200)

I upgraded from 4.15.18-27-pve to 4.15.18-30-pve and then the virtio
driver became very unstable. This machine has run for many years on
multiple versions of Proxmox (IIRC from version 1) without any problem.

R.

> i see that my centos 5.11 VM should have support for virtio (see
> below) and according to https://access.redhat.com/articles/2488201
> rhel/centos supports this since 5.3.
> 
> so if virtio worked before a proxmox update,  then i would call this a
> regression.
> 
> can you tell which proxmox version worked for you with virtio ?
> 
> i tested centos 5.11 installer on my older pve 6.0-4 (5.0.15-1-pve
> kernel) , but it does not find virtual disks.
> 
> roland
> 
> [root@hr-neu ~]# uname -aLinux hr-neu 2.6.18-398.el5 #1 SMP Tue Sep 16
> 20:50:52 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> [root@centos5 ~]# cat /etc/redhat-release
> CentOS release 5.11 (Final)
> [root@centos5 ~]# rpm -q kernel --changelog|grep virtio|grep 446214
> - [xen] virtio: include headers in kernel-headers package (Eduardo
> Pereira Habkost ) [446214]
> - [xen] virtio: add PV network and block drivers for KVM (Mark
> McLoughlin ) [446214]
> - [xen] virtio: include headers in kernel-headers package (Eduardo
> Pereira Habkost ) [446214]
> - [xen] virtio: add PV network and block drivers for KVM (Mark
> McLoughlin ) [446214]
> 
> Am 23.07.20 um 10:22 schrieb richard lucassen:
> > On Tue, 21 Jul 2020 11:06:27 +0200
> > Michael Rasmussen via pve-user <pve-user@lists.proxmox.com> wrote:
> >
> >> On Tue, 21 Jul 2020 10:41:54 +0200
> >> richard lucassen <mailinglists@lucassen.org> wrote:
> >>
> >>> network. There is definitly something wrong somewhere.
> >>>
> >> The only thing wrong here is that you a running a version of
> >> RHEL/CentOS which is in maintenance support 2. This means there
> >> will be no more updates except critical bugs. As of March 31, 2017
> >> it is considered in deep freeze!
> > I know. But sometimes you need to keep an old version and a good
> > way to handle this is to run such an old version in a virtual
> > environment. A vhost supplies virtual hardware and apparently this
> > virtual hardware has changed. This is not A Good Thing IMHO.
> >
> > It runs well under subversion 27 of the pve kernel but has stopped
> > under version 30. So I think this is a bug as I do not expect design
> > changes between subversion 27 and 30.
> >
> > R.
> >


-- 
richard lucassen
https://contact.xaq.nl/

Re: [PVE-User] Problem with Centos 5.X virtio ethernet drivers and last PVE updates

[richard lucassen]

On Fri, 24 Jul 2020 08:07:34 +0200
Eneko Lacunza via pve-user <pve-user@lists.proxmox.com> wrote:

> What physical network interface make/model in Proxmox node?

# lspci | grep -i ether
02:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709
Gigabit Ethernet (rev 20)

02:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709
Gigabit Ethernet (rev 20)

07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)

07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)

The MAC adresses point to "Hewlett-Packard" in the MAC database and I'm
at 900km's from these machines so I can't tell you if it is the Intel or
the Broadcom. The loaded kernel modules are e1000e and bnx2.

R.

> El 23/7/20 a las 21:21, richard lucassen escribió:
> > On Thu, 23 Jul 2020 12:11:19 +0200
> > Roland <devzero@web.de> wrote:
> >
> > I run a Debian Lenny VM:
> >
> > # cat /etc/issue.net
> > Debian GNU/Linux 5.0
> >
> > # modinfo virtio_net
> > filename:       /lib/modules/2.6.26-2-686/kernel/drivers/net/virtio_net.ko
> > license:        GPL
> > description:    Virtio network driver
> > alias:          virtio:d00000001v*
> > depends:
> > vermagic:       2.6.26-2-686 SMP mod_unload modversions 686
> > parm:           napi_weight:int
> > parm:           csum:bool
> > parm:           gso:bool
> >
> > Proxmox host version 5.4:
> >
> > # cat /proc/version
> > Linux version 4.15.18-30-pve (root@nora) (gcc version 6.3.0 20170516
> > (Debian 6.3.0-18+deb9u1)) #1 SMP PVE 4.15.18-58 (Fri, 12 Jun 2020
> > 13:53:01 +0200)
> >
> > I upgraded from 4.15.18-27-pve to 4.15.18-30-pve and then the virtio
> > driver became very unstable. This machine has run for many years on
> > multiple versions of Proxmox (IIRC from version 1) without any
> > problem.
> >
> > R.
> >
> >> i see that my centos 5.11 VM should have support for virtio (see
> >> below) and according to https://access.redhat.com/articles/2488201
> >> rhel/centos supports this since 5.3.
> >>
> >> so if virtio worked before a proxmox update,  then i would call
> >> this a regression.
> >>
> >> can you tell which proxmox version worked for you with virtio ?
> >>
> >> i tested centos 5.11 installer on my older pve 6.0-4 (5.0.15-1-pve
> >> kernel) , but it does not find virtual disks.
> >>
> >> roland
> >>
> >> [root@hr-neu ~]# uname -aLinux hr-neu 2.6.18-398.el5 #1 SMP Tue
> >> Sep 16 20:50:52 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
> >>
> >>
> >> [root@centos5 ~]# cat /etc/redhat-release
> >> CentOS release 5.11 (Final)
> >> [root@centos5 ~]# rpm -q kernel --changelog|grep virtio|grep 446214
> >> - [xen] virtio: include headers in kernel-headers package (Eduardo
> >> Pereira Habkost ) [446214]
> >> - [xen] virtio: add PV network and block drivers for KVM (Mark
> >> McLoughlin ) [446214]
> >> - [xen] virtio: include headers in kernel-headers package (Eduardo
> >> Pereira Habkost ) [446214]
> >> - [xen] virtio: add PV network and block drivers for KVM (Mark
> >> McLoughlin ) [446214]
> >>
> >> Am 23.07.20 um 10:22 schrieb richard lucassen:
> >>> On Tue, 21 Jul 2020 11:06:27 +0200
> >>> Michael Rasmussen via pve-user <pve-user@lists.proxmox.com> wrote:
> >>>
> >>>> On Tue, 21 Jul 2020 10:41:54 +0200
> >>>> richard lucassen <mailinglists@lucassen.org> wrote:
> >>>>
> >>>>> network. There is definitly something wrong somewhere.
> >>>>>
> >>>> The only thing wrong here is that you a running a version of
> >>>> RHEL/CentOS which is in maintenance support 2. This means there
> >>>> will be no more updates except critical bugs. As of March 31,
> >>>> 2017 it is considered in deep freeze!
> >>> I know. But sometimes you need to keep an old version and a good
> >>> way to handle this is to run such an old version in a virtual
> >>> environment. A vhost supplies virtual hardware and apparently this
> >>> virtual hardware has changed. This is not A Good Thing IMHO.
> >>>
> >>> It runs well under subversion 27 of the pve kernel but has stopped
> >>> under version 30. So I think this is a bug as I do not expect
> >>> design changes between subversion 27 and 30.
> >>>
> >>> R.
> >>>
> >
> 
> 


-- 
richard lucassen
https://contact.xaq.nl/

Re: [PVE-User] Problem with Centos 5.X virtio ethernet drivers and last PVE updates

[Adam Thompson]

On 2020-07-24 09:01, richard lucassen wrote:
> On Fri, 24 Jul 2020 08:07:34 +0200
> Eneko Lacunza via pve-user <pve-user@lists.proxmox.com> wrote:
> 
>> What physical network interface make/model in Proxmox node?
> 
> # lspci | grep -i ether
> 02:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709
> Gigabit Ethernet (rev 20)
> [...]
> The MAC adresses point to "Hewlett-Packard" in the MAC database and I'm
> at 900km's from these machines so I can't tell you if it is the Intel 
> or
> the Broadcom. The loaded kernel modules are e1000e and bnx2.

Two easy ways to tell:
1. install and run "lshw", probably "lshw -c network" is most useful
2. Use the bash script in the 2nd answer at 
https://unix.stackexchange.com/questions/41817/linux-how-to-find-the-device-driver-used-for-a-device

-Adam

Re: [PVE-User] Problem with Centos 5.X virtio ethernet drivers and last PVE updates

[richard lucassen]

On Fri, 24 Jul 2020 09:19:25 -0500
Adam Thompson <athompso@athompso.net> wrote:

> 1. install and run "lshw", probably "lshw -c network" is most useful

It is this one (the active slave of a bond device):

--- 8< ---
  *-network:1
       description: Ethernet interface
       product: 82571EB Gigabit Ethernet Controller
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci@0000:07:00.1
       logical name: ens2f1
       version: 06
       serial: 00:26:55:ed:6f:4c
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi pciexpress bus_master cap_list rom ethernet
physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=e1000e
driverversion=3.4.1.1-NAPI duplex=full firmware=5.11-2 latency=0
link=yes multicast=yes port=twisted pair slave=yes speed=1Gbit/s
resources: irq:46 memory:fbfa0000-fbfbffff memory:fbf80000-fbf9ffff
ioport:5020(size=32) memory:fbf20000-fbf3ffff
--- 8< ---

-- 
richard lucassen
https://contact.xaq.nl/

Re: [PVE-User] Problem with Centos 5.X virtio ethernet drivers and last PVE updates

[richard lucassen]

On Mon, 3 Aug 2020 13:54:54 +0200
Eneko Lacunza via pve-user <pve-user@lists.proxmox.com> wrote:

> As reported 10 days ago, we have found a e1000e driver hang recently, 
> after upgrading from PVE 5.4 to 6.2, in an otherwise stable server.
> 
> It could be a driver issue and not a virtio network issue, but we 
> haven't seen another hang since the one reported.

[note] I just moved the images to a new proxmox 6.2.11 environment and
the problem remains. An RTL8169 NIC works well

-- 
richard lucassen
https://contact.xaq.nl/