[PVE-User] PBS : is dirty-bitmap really accurate ?

[PVE-User] PBS : is dirty-bitmap really accurate ?

[Daniel Berteaud]

Hi 

I've started playing with PBS on some VM. So far, it's looking really promizing. 
There's one strange thing though : the percent of the dirty data. For example, I backup one VM every 2 or 3 days. It's a moderately busy server, mainly serving a MariaDB database (zabbix server + mariadb). On each backup, I get similar dirty values : 

INFO: using fast incremental mode (dirty-bitmap), 492.7 GiB dirty of 590.0 GiB total 

While I'm sur not even 10% of this has really been written. 

Get more or less the same problem on other VM. One which I know just sleep all day (my personnal OnlyOffice document server), and which I backup daily, and get values like : 

INFO: using fast incremental mode (dirty-bitmap), 5.0 GiB dirty of 10.0 GiB total 

Or another small one (personnal samba DC controler) : 

INFO: using fast incremental mode (dirty-bitmap), 13.0 GiB dirty of 20.0 GiB total 

The only write activity for those 2 are just a few KB or maybe MB of log lines. Respectivly 5 and 13GB of dirty blocks seems unreal. 

Am I the only one seeing this ? Could the dirty-bitmap mark dirty blocks without write activity somehow ? 

Regards, 
Daniel 

-- 


[ https://www.firewall-services.com/ ] 	
Daniel Berteaud 
FIREWALL-SERVICES SAS, La sécurité des réseaux 
Société de Services en Logiciels Libres 
Tél : +33.5 56 64 15 32 
Matrix: @dani:fws.fr 
[ https://www.firewall-services.com/ | https://www.firewall-services.com ] 

Re: [PVE-User] PBS : is dirty-bitmap really accurate ?

[Fabian Grünbichler]

On July 22, 2020 6:40 pm, Daniel Berteaud wrote:
> Hi 
> 
> I've started playing with PBS on some VM. So far, it's looking really promizing. 
> There's one strange thing though : the percent of the dirty data. For example, I backup one VM every 2 or 3 days. It's a moderately busy server, mainly serving a MariaDB database (zabbix server + mariadb). On each backup, I get similar dirty values : 
> 
> INFO: using fast incremental mode (dirty-bitmap), 492.7 GiB dirty of 590.0 GiB total 
> 
> While I'm sur not even 10% of this has really been written. 
> 
> Get more or less the same problem on other VM. One which I know just sleep all day (my personnal OnlyOffice document server), and which I backup daily, and get values like : 
> 
> INFO: using fast incremental mode (dirty-bitmap), 5.0 GiB dirty of 10.0 GiB total 
> 
> Or another small one (personnal samba DC controler) : 
> 
> INFO: using fast incremental mode (dirty-bitmap), 13.0 GiB dirty of 20.0 GiB total 
> 
> The only write activity for those 2 are just a few KB or maybe MB of log lines. Respectivly 5 and 13GB of dirty blocks seems unreal. 
> 
> Am I the only one seeing this ? Could the dirty-bitmap mark dirty blocks without write activity somehow ? 

possibly you haven't upgraded to pve-qemu-kvm 5.0-11 (or your VM hasn't 
been restarted yet since the upgrade):

 https://git.proxmox.com/?p=pve-qemu.git;a=commit;h=f257cc05f4fbf772cad3231021b3ce7587127a1b

the bitmap has a granularity of 4MB, so depending on the activity inside 
you can see quite a bit of amplification. also writing and then 
zeroing/reverting again to the old content would leave a mark in the 
bitmap without permanently changing the contents.

Re: [PVE-User] PBS : is dirty-bitmap really accurate ?

[Daniel Berteaud]

----- Le 23 Juil 20, à 8:43, Fabian Grünbichler f.gruenbichler@proxmox.com a écrit :

> possibly you haven't upgraded to pve-qemu-kvm 5.0-11 (or your VM hasn't
> been restarted yet since the upgrade):
> 
> https://git.proxmox.com/?p=pve-qemu.git;a=commit;h=f257cc05f4fbf772cad3231021b3ce7587127a1b

I'm running pve-qemu-kvm 5.0.0-11, and all the implied VM have been either (cold) rebooted, or migrated.

> 
> the bitmap has a granularity of 4MB, so depending on the activity inside
> you can see quite a bit of amplification. also writing and then
> zeroing/reverting again to the old content would leave a mark in the
> bitmap without permanently changing the contents.
> 

Yes, I'd expect some amplification, but not that much. For my Zabbix server, it's nearly canceling all the benefit of using a dirty bitmap.
One thing I've noted, is that I get expected values at least for one guest, running PfSense (where I get ~150MB of dirty blocks each days). Most of my other VM are Linux, I'll check if it could be related to the atime update or something

Cheers,
Daniel

-- 
[ https://www.firewall-services.com/ ] 	
Daniel Berteaud 
FIREWALL-SERVICES SAS, La sécurité des réseaux 
Société de Services en Logiciels Libres 
Tél : +33.5 56 64 15 32 
Matrix: @dani:fws.fr 
[ https://www.firewall-services.com/ | https://www.firewall-services.com ]

Re: [PVE-User] PBS : is dirty-bitmap really accurate ?

[Jorge Boncompte]

El 23/7/20 a las 8:53, Daniel Berteaud escribió:
> ----- Le 23 Juil 20, à 8:43, Fabian Grünbichler f.gruenbichler@proxmox.com a écrit :
> 
>> possibly you haven't upgraded to pve-qemu-kvm 5.0-11 (or your VM hasn't
>> been restarted yet since the upgrade):
>>
>> https://git.proxmox.com/?p=pve-qemu.git;a=commit;h=f257cc05f4fbf772cad3231021b3ce7587127a1b
> 
> I'm running pve-qemu-kvm 5.0.0-11, and all the implied VM have been either (cold) rebooted, or migrated.
> 
>>
>> the bitmap has a granularity of 4MB, so depending on the activity inside
>> you can see quite a bit of amplification. also writing and then
>> zeroing/reverting again to the old content would leave a mark in the
>> bitmap without permanently changing the contents.
>>
> 
> Yes, I'd expect some amplification, but not that much. For my Zabbix server, it's nearly canceling all the benefit of using a dirty bitmap.
> One thing I've noted, is that I get expected values at least for one guest, running PfSense (where I get ~150MB of dirty blocks each days). Most of my other VM are Linux, I'll check if it could be related to the atime update or something

	Hi, does the dirty-bitmap take somehow into account block discarding
and zeroing? Because the other thing I would look for in this case is
for a fstrim firing every day.

> 
> Cheers,
> Daniel
> 

Re: [PVE-User] PBS : is dirty-bitmap really accurate ?

[Fabian Grünbichler]

On July 23, 2020 11:00 am, Jorge Boncompte wrote:
> El 23/7/20 a las 8:53, Daniel Berteaud escribió:
>> ----- Le 23 Juil 20, à 8:43, Fabian Grünbichler f.gruenbichler@proxmox.com a écrit :
>> 
>>> possibly you haven't upgraded to pve-qemu-kvm 5.0-11 (or your VM hasn't
>>> been restarted yet since the upgrade):
>>>
>>> https://git.proxmox.com/?p=pve-qemu.git;a=commit;h=f257cc05f4fbf772cad3231021b3ce7587127a1b
>> 
>> I'm running pve-qemu-kvm 5.0.0-11, and all the implied VM have been either (cold) rebooted, or migrated.
>> 
>>>
>>> the bitmap has a granularity of 4MB, so depending on the activity inside
>>> you can see quite a bit of amplification. also writing and then
>>> zeroing/reverting again to the old content would leave a mark in the
>>> bitmap without permanently changing the contents.
>>>
>> 
>> Yes, I'd expect some amplification, but not that much. For my Zabbix server, it's nearly canceling all the benefit of using a dirty bitmap.
>> One thing I've noted, is that I get expected values at least for one guest, running PfSense (where I get ~150MB of dirty blocks each days). Most of my other VM are Linux, I'll check if it could be related to the atime update or something
> 
> 	Hi, does the dirty-bitmap take somehow into account block discarding
> and zeroing? Because the other thing I would look for in this case is
> for a fstrim firing every day.

also a possible candidate. trim/discard of course changes the blocks and 
thus dirties the bitmap.

Re: [PVE-User] PBS : is dirty-bitmap really accurate ?

[Daniel Berteaud]

----- Le 23 Juil 20, à 11:34, Fabian Grünbichler f.gruenbichler@proxmox.com a écrit :

> also a possible candidate. trim/discard of course changes the blocks and
> thus dirties the bitmap.
> 

Indeed, I do have a daily fstrim job running on all my Linux guests (and not on the PfSense one), that could explain it. So it would mean we have to choose between thin prov + reclaiming unused space, or efficient dirty bitmap ...

I'll run some test with a weekly fstrim instead of daily.

Cheers,
Daniel

-- 
[ https://www.firewall-services.com/ ] 	
Daniel Berteaud 
FIREWALL-SERVICES SAS, La sécurité des réseaux 
Société de Services en Logiciels Libres 
Tél : +33.5 56 64 15 32 
Matrix: @dani:fws.fr 
[ https://www.firewall-services.com/ | https://www.firewall-services.com ]

Re: [PVE-User] PBS : is dirty-bitmap really accurate ?

[Roland]

i just had a look on that, too - i was backing up a VM i backed up this
afternoon,

and for my curiosity 1GB out of 15GB was marked dirty.

that looks a quite much for me for a mostly idle system, because there
was definitely only a little bit of change on the system within logfiles
in /var/log

so i wonder what marked all that blocks dirty....

i'm also suspecting atime changes...will keep an eye on that....

regards
roland


Am 23.07.20 um 08:53 schrieb Daniel Berteaud:
> ----- Le 23 Juil 20, à 8:43, Fabian Grünbichler f.gruenbichler@proxmox.com a écrit :
>
>> possibly you haven't upgraded to pve-qemu-kvm 5.0-11 (or your VM hasn't
>> been restarted yet since the upgrade):
>>
>> https://git.proxmox.com/?p=pve-qemu.git;a=commit;h=f257cc05f4fbf772cad3231021b3ce7587127a1b
> I'm running pve-qemu-kvm 5.0.0-11, and all the implied VM have been either (cold) rebooted, or migrated.
>
>> the bitmap has a granularity of 4MB, so depending on the activity inside
>> you can see quite a bit of amplification. also writing and then
>> zeroing/reverting again to the old content would leave a mark in the
>> bitmap without permanently changing the contents.
>>
> Yes, I'd expect some amplification, but not that much. For my Zabbix server, it's nearly canceling all the benefit of using a dirty bitmap.
> One thing I've noted, is that I get expected values at least for one guest, running PfSense (where I get ~150MB of dirty blocks each days). Most of my other VM are Linux, I'll check if it could be related to the atime update or something
>
> Cheers,
> Daniel
>

Re: [PVE-User] PBS : is dirty-bitmap really accurate ?

[Daniel Berteaud]

----- Le 23 Juil 20, à 23:59, Roland devzero@web.de a écrit :

> i just had a look on that, too - i was backing up a VM i backed up this
> afternoon,
> 
> and for my curiosity 1GB out of 15GB was marked dirty.
> 
> that looks a quite much for me for a mostly idle system, because there
> was definitely only a little bit of change on the system within logfiles
> in /var/log
> 
> so i wonder what marked all that blocks dirty....
> 
> i'm also suspecting atime changes...will keep an eye on that....


It was my daily fstrim in my case. Each time you trim, it'll dirty all the blocks corresponding to unused space.
I've switched this to a weekly job so bacups can run most of the time efficiently. Since then, dirty blocks went from ~15GB per VM on average to something betwwen 800MB and 4GB, which is much closer to what I expect, considering the 4MB granularity of the bitmap.

I'll check on my biggest VM (Zabbix server) but I expect similar results

Thanks Fabian and Jorge for pointing this out

Cheers
Daniel

-- 
[ https://www.firewall-services.com/ ] 	
Daniel Berteaud 
FIREWALL-SERVICES SAS, La sécurité des réseaux 
Société de Services en Logiciels Libres 
Tél : +33.5 56 64 15 32 
Matrix: @dani:fws.fr 
[ https://www.firewall-services.com/ | https://www.firewall-services.com ]

Re: [PVE-User] PBS : is dirty-bitmap really accurate ?

[Ronny Aasen]

On 24.07.2020 09:02, Daniel Berteaud wrote:
> ----- Le 23 Juil 20, à 23:59, Roland devzero@web.de a écrit :
> 
>> i just had a look on that, too - i was backing up a VM i backed up this
>> afternoon,
>>
>> and for my curiosity 1GB out of 15GB was marked dirty.
>>
>> that looks a quite much for me for a mostly idle system, because there
>> was definitely only a little bit of change on the system within logfiles
>> in /var/log
>>
>> so i wonder what marked all that blocks dirty....
>>
>> i'm also suspecting atime changes...will keep an eye on that....
> 
> 
> It was my daily fstrim in my case. Each time you trim, it'll dirty all the blocks corresponding to unused space.
> I've switched this to a weekly job so bacups can run most of the time efficiently. Since then, dirty blocks went from ~15GB per VM on average to something betwwen 800MB and 4GB, which is much closer to what I expect, considering the 4MB granularity of the bitmap.
> 
> I'll check on my biggest VM (Zabbix server) but I expect similar results
> 
> Thanks Fabian and Jorge for pointing this out
> 
> Cheers
> Daniel
> 


would mounting the disk with discard help on this ? where it only trims 
blocks that are actually discarded ? instead of touching the whole disk 
with fstrim ?


Ronny

Re: [PVE-User] PBS : is dirty-bitmap really accurate ?

[Mark Schouten]

On Fri, Jul 24, 2020 at 09:54:13AM +0200, Ronny Aasen wrote:
> would mounting the disk with discard help on this ? where it only trims
> blocks that are actually discarded ? instead of touching the whole disk with
> fstrim ?

I would expect so, yes.

-- 
Mark Schouten     | Tuxis B.V.
KvK: 74698818     | http://www.tuxis.nl/
T: +31 318 200208 | info@tuxis.nl