[PVE-User] IS able to login through web interface, but the status of the VM is marked as ??

[PVE-User] IS able to login through web interface, but the status of the VM is marked as ??

[Joseph John]

Dear All ,
We are using PVE 8.X EE , we have 8 node cluster
today one of the  cluster , we were able to login through the web
interface, but the status of the VM machines were marked as ?
We tried to login in using SSH, were able not able to  init 6 , it was
showing the message

root@pve-4:~# init 6

*Failed to set wall message, ignoring: Transport endpoint is not
connectedCall to Reboot failed: Transport endpoint is not connected*

Later on we did hard reset and the system was back
How could we know what was the problem?, which log files  should we use? to
understand the issue

Advice requested


Thanks
Joseph John

Re: [PVE-User] IS able to login through web interface, but the status of the VM is marked as ??

[proxmox]

Hello, as i think this should be a linux issue you should just to have a look in
Journalctl
Dmesg
Messages/syslog/daemon
And if you have service in question you should look in the related logfile

Hth
Mehmet

Am 11. August 2023 07:03:48 MESZ schrieb Joseph John <jjk.saji@gmail.com>:
>Dear All ,
>We are using PVE 8.X EE , we have 8 node cluster
>today one of the  cluster , we were able to login through the web
>interface, but the status of the VM machines were marked as ?
>We tried to login in using SSH, were able not able to  init 6 , it was
>showing the message
>
>root@pve-4:~# init 6
>
>*Failed to set wall message, ignoring: Transport endpoint is not
>connectedCall to Reboot failed: Transport endpoint is not connected*
>
>Later on we did hard reset and the system was back
>How could we know what was the problem?, which log files  should we use? to
>understand the issue
>
>Advice requested
>
>
>Thanks
>Joseph John
>_______________________________________________
>pve-user mailing list
>pve-user@lists.proxmox.com
>https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>

Re: [PVE-User] IS able to login through web interface, but the status of the VM is marked as ??

[Roland]

there is a known problem when storage hang as pvestatd may get stuck in
a syscall and never returns :

https://forum.proxmox.com/threads/unavailable-storage-will-make-the-webui-unusable.119107/

https://bugzilla.proxmox.com/show_bug.cgi?id=3714

if you have even ssh login problems, i guess it was some issue/hang with
the root filesystem/disks/raid

furthermore, it could have been some out of memory condition.  to get
into the details of such, it's useful
to have external system monitoring, as it will tell you when ram did
fill up slowly...

roland

Am 11.08.23 um 07:03 schrieb Joseph John:
> Dear All ,
> We are using PVE 8.X EE , we have 8 node cluster
> today one of the  cluster , we were able to login through the web
> interface, but the status of the VM machines were marked as ?
> We tried to login in using SSH, were able not able to  init 6 , it was
> showing the message
>
> root@pve-4:~# init 6
>
> *Failed to set wall message, ignoring: Transport endpoint is not
> connectedCall to Reboot failed: Transport endpoint is not connected*
>
> Later on we did hard reset and the system was back
> How could we know what was the problem?, which log files  should we use? to
> understand the issue
>
> Advice requested
>
>
> Thanks
> Joseph John
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>

Re: [PVE-User] IS able to login through web interface, but the status of the VM is marked as ??

[Joseph John]

Thanks Roland,
We will setup a network monitoring tool ( zabbix ) and keep it monitored
for studying such occurrences

Thanks
Joseph John



On Fri, Aug 11, 2023 at 6:15 PM Roland <devzero@web.de> wrote:

> there is a known problem when storage hang as pvestatd may get stuck in
> a syscall and never returns :
>
>
> https://forum.proxmox.com/threads/unavailable-storage-will-make-the-webui-unusable.119107/
>
> https://bugzilla.proxmox.com/show_bug.cgi?id=3714
>
> if you have even ssh login problems, i guess it was some issue/hang with
> the root filesystem/disks/raid
>
> furthermore, it could have been some out of memory condition.  to get
> into the details of such, it's useful
> to have external system monitoring, as it will tell you when ram did
> fill up slowly...
>
> roland
>
> Am 11.08.23 um 07:03 schrieb Joseph John:
> > Dear All ,
> > We are using PVE 8.X EE , we have 8 node cluster
> > today one of the  cluster , we were able to login through the web
> > interface, but the status of the VM machines were marked as ?
> > We tried to login in using SSH, were able not able to  init 6 , it was
> > showing the message
> >
> > root@pve-4:~# init 6
> >
> > *Failed to set wall message, ignoring: Transport endpoint is not
> > connectedCall to Reboot failed: Transport endpoint is not connected*
> >
> > Later on we did hard reset and the system was back
> > How could we know what was the problem?, which log files  should we use?
> to
> > understand the issue
> >
> > Advice requested
> >
> >
> > Thanks
> > Joseph John
> > _______________________________________________
> > pve-user mailing list
> > pve-user@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>

Re: [PVE-User] IS able to login through web interface, but the status of the VM is marked as ??

[Roland]

https://bugzilla.proxmox.com/show_bug.cgi?id=1410
https://bugzilla.proxmox.com/show_bug.cgi?id=4519

Am 17.08.23 um 06:57 schrieb Joseph John:
> Dear Mark
> THANKS
> Like to give the feedback,
> Most probably the issue might have been occured because of the loal disk
> getting over , reason is that I was moving some images from my old
> instance  to the new instance, this may be the reason why this behaviour
> happened
> thanks
>
> On Mon, Aug 14, 2023 at 12:31 PM Wolf Noble <wolf@wolfspyre.com> wrote:
>
>> that tracks… the ‘why’ still could use a bit more digging, but I concur
>> that tracks error-condition-chain wise…
>>
>> oom feels unlikely coz other lower priority stuff’d break first…
>> network loss?
>>
>> time issues?
>>
>> stupid high latency/context switching?
>>
>> could be a lot of things
>>
>>
>> [= The contents of this message have been written, read, processed,
>> erased, sorted, sniffed, compressed, rewritten, misspelled,
>> overcompensated, lost, found, and most importantly delivered entirely with
>> recycled electrons =]
>>
>>> On Aug 14, 2023, at 02:15, Mark Schouten <mark@tuxis.nl> wrote:
>>>
>>> Hi,
>>>
>>> The message ’Transport endport is not connected’ points to a
>> fuse-mountpoint. Given that pvestatd can also not write its statusses (you
>> see question-marks, that points to pvestatd not working), I would assume
>> that the pve-cluster was broken. You should probably see messages in logs
>> from pmxcfs and corosync (on all nodes, I suspect).
>>> —
>>> Mark Schouten
>>> CTO, Tuxis B.V.
>>> +31 318 200208 / mark@tuxis.nl
>>>
>>>
>>> ------ Original Message ------
>>>  From "Joseph John" <jjk.saji@gmail.com>
>>> To pve-user@pve.proxmox.com
>>> Date 11/08/2023, 07:03:48
>>> Subject [PVE-User] IS able to login through web interface, but the
>> status of the VM is marked as ??
>>>> Dear All ,
>>>> We are using PVE 8.X EE , we have 8 node cluster
>>>> today one of the  cluster , we were able to login through the web
>>>> interface, but the status of the VM machines were marked as ?
>>>> We tried to login in using SSH, were able not able to  init 6 , it was
>>>> showing the message
>>>>
>>>> root@pve-4:~# init 6
>>>>
>>>> *Failed to set wall message, ignoring: Transport endpoint is not
>>>> connectedCall to Reboot failed: Transport endpoint is not connected*
>>>>
>>>> Later on we did hard reset and the system was back
>>>> How could we know what was the problem?, which log files  should we
>> use? to
>>>> understand the issue
>>>>
>>>> Advice requested
>>>>
>>>>
>>>> Thanks
>>>> Joseph John
>>>> _______________________________________________
>>>> pve-user mailing list
>>>> pve-user@lists.proxmox.com
>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>> _______________________________________________
>> pve-user mailing list
>> pve-user@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] IS able to login through web interface, but the status of the VM is marked as ??

[Joseph John]

Dear Mark
THANKS
Like to give the feedback,
Most probably the issue might have been occured because of the loal disk
getting over , reason is that I was moving some images from my old
instance  to the new instance, this may be the reason why this behaviour
happened
thanks

On Mon, Aug 14, 2023 at 12:31 PM Wolf Noble <wolf@wolfspyre.com> wrote:

> that tracks… the ‘why’ still could use a bit more digging, but I concur
> that tracks error-condition-chain wise…
>
> oom feels unlikely coz other lower priority stuff’d break first…
> network loss?
>
> time issues?
>
> stupid high latency/context switching?
>
> could be a lot of things
>
>
> [= The contents of this message have been written, read, processed,
> erased, sorted, sniffed, compressed, rewritten, misspelled,
> overcompensated, lost, found, and most importantly delivered entirely with
> recycled electrons =]
>
> > On Aug 14, 2023, at 02:15, Mark Schouten <mark@tuxis.nl> wrote:
> >
> > Hi,
> >
> > The message ’Transport endport is not connected’ points to a
> fuse-mountpoint. Given that pvestatd can also not write its statusses (you
> see question-marks, that points to pvestatd not working), I would assume
> that the pve-cluster was broken. You should probably see messages in logs
> from pmxcfs and corosync (on all nodes, I suspect).
> >
> > —
> > Mark Schouten
> > CTO, Tuxis B.V.
> > +31 318 200208 / mark@tuxis.nl
> >
> >
> > ------ Original Message ------
> > From "Joseph John" <jjk.saji@gmail.com>
> > To pve-user@pve.proxmox.com
> > Date 11/08/2023, 07:03:48
> > Subject [PVE-User] IS able to login through web interface, but the
> status of the VM is marked as ??
> >
> >> Dear All ,
> >> We are using PVE 8.X EE , we have 8 node cluster
> >> today one of the  cluster , we were able to login through the web
> >> interface, but the status of the VM machines were marked as ?
> >> We tried to login in using SSH, were able not able to  init 6 , it was
> >> showing the message
> >>
> >> root@pve-4:~# init 6
> >>
> >> *Failed to set wall message, ignoring: Transport endpoint is not
> >> connectedCall to Reboot failed: Transport endpoint is not connected*
> >>
> >> Later on we did hard reset and the system was back
> >> How could we know what was the problem?, which log files  should we
> use? to
> >> understand the issue
> >>
> >> Advice requested
> >>
> >>
> >> Thanks
> >> Joseph John
> >> _______________________________________________
> >> pve-user mailing list
> >> pve-user@lists.proxmox.com
> >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>