[pve-devel] Compiling PBS on arm64

[pve-devel] Compiling PBS on arm64

[Kamil Trzciński]

ARM64 is becoming increasingly popular, especially that PBS seems at least
for my usage-pattern to be ideal to run on my arm64 NAS. In the end
I want to try to be able to recompile everything for arm64 and see how
nicely
it works there.

But first I decided to try to compile all packages for `amd64`. And I'm
scratching my head
to try to do that. This proves to be super hard due to multitude of
dependencies that are backported,
unordered, and require a very special set of commands to be executed
depending on a git repo
since PBS is rust-based.

Do you maybe happen to have a comprehensive guide that allows you to
compile everything
from the scratch without the use of `/devel` repo, ideally with some kind
of CI scripts that would do this heavy lifting?

Kamil Trzciński

Re: [pve-devel] Compiling PBS on arm64

[Kamil Trzciński]

I'm looking for some help on how to get it running and how to make it
reproducible :)

On Fri, Dec 4, 2020 at 4:21 PM Kamil Trzciński <ayufan@ayufan.eu> wrote:

> ARM64 is becoming increasingly popular, especially that PBS seems at least
> for my usage-pattern to be ideal to run on my arm64 NAS. In the end
> I want to try to be able to recompile everything for arm64 and see how
> nicely
> it works there.
>
> But first I decided to try to compile all packages for `amd64`. And I'm
> scratching my head
> to try to do that. This proves to be super hard due to multitude of
> dependencies that are backported,
> unordered, and require a very special set of commands to be executed
> depending on a git repo
> since PBS is rust-based.
>
> Do you maybe happen to have a comprehensive guide that allows you to
> compile everything
> from the scratch without the use of `/devel` repo, ideally with some kind
> of CI scripts that would do this heavy lifting?
>
> Kamil Trzciński
>
>
>

Re: [pve-devel] Compiling PBS on arm64

[Kamil Trzciński]

I'm slightly progressing, but I stumbled across some `debcargo` problem. It
appears that
Proxmox uses their own fork of `debcargo`, which is needed in order to
build crates
without the usage of crates.io. Is this patch published somewhere?

rust-debcargo (2.4.2-pve1) proxmox-rust; urgency=medium

  * allow overriding maintainer via debcargo.toml
  * implement local crate support.patch

 -- Proxmox Support Team <support@proxmox.com>  Tue, 14 Jan 2020 16:13:48
+0100


On Fri, Dec 4, 2020 at 4:21 PM Kamil Trzciński <ayufan@ayufan.eu> wrote:

> ARM64 is becoming increasingly popular, especially that PBS seems at least
> for my usage-pattern to be ideal to run on my arm64 NAS. In the end
> I want to try to be able to recompile everything for arm64 and see how
> nicely
> it works there.
>
> But first I decided to try to compile all packages for `amd64`. And I'm
> scratching my head
> to try to do that. This proves to be super hard due to multitude of
> dependencies that are backported,
> unordered, and require a very special set of commands to be executed
> depending on a git repo
> since PBS is rust-based.
>
> Do you maybe happen to have a comprehensive guide that allows you to
> compile everything
> from the scratch without the use of `/devel` repo, ideally with some kind
> of CI scripts that would do this heavy lifting?
>
> Kamil Trzciński
>
>
>

Re: [pve-devel] Compiling PBS on arm64

[Dietmar Maurer]

> On 12/06/2020 8:41 PM Kamil Trzciński <ayufan@ayufan.eu> wrote:
> 
>  
> I'm slightly progressing, but I stumbled across some `debcargo` problem. It
> appears that
> Proxmox uses their own fork of `debcargo`, which is needed in order to
> build crates
> without the usage of crates.io. 

I guess you can simply adopt/remove the .cargo/config file we ship with our
packages. Cargo will use normal repos then.

Re: [pve-devel] Compiling PBS on arm64

[Fabian Grünbichler]

On December 6, 2020 8:41 pm, Kamil Trzciński wrote:
> I'm slightly progressing, but I stumbled across some `debcargo` problem. It
> appears that
> Proxmox uses their own fork of `debcargo`, which is needed in order to
> build crates
> without the usage of crates.io. Is this patch published somewhere?
> 
> rust-debcargo (2.4.2-pve1) proxmox-rust; urgency=medium
> 
>   * allow overriding maintainer via debcargo.toml
>   * implement local crate support.patch
> 
>  -- Proxmox Support Team <support@proxmox.com>  Tue, 14 Jan 2020 16:13:48
> +0100

https://git.proxmox.com/?p=debcargo-conf.git;a=tree;f=src/debcargo/debian/patches;h=7345cadea4cbffc3174c04e34960e91767892100;hb=refs/heads/proxmox/buster

all of the toolchain backports are published as well on 
git.proxmox.com.. if you want to re-compile all of our devel repo for 
arm64, I suggest familiarizing yourself with debcargo and the 
debcargo-conf repo (especially the upstream and Proxmox readmes) ;)

> On Fri, Dec 4, 2020 at 4:21 PM Kamil Trzciński <ayufan@ayufan.eu> wrote:
> 
>> ARM64 is becoming increasingly popular, especially that PBS seems at least
>> for my usage-pattern to be ideal to run on my arm64 NAS. In the end
>> I want to try to be able to recompile everything for arm64 and see how
>> nicely
>> it works there.
>>
>> But first I decided to try to compile all packages for `amd64`. And I'm
>> scratching my head
>> to try to do that. This proves to be super hard due to multitude of
>> dependencies that are backported,
>> unordered, and require a very special set of commands to be executed
>> depending on a git repo
>> since PBS is rust-based.
>>
>> Do you maybe happen to have a comprehensive guide that allows you to
>> compile everything
>> from the scratch without the use of `/devel` repo, ideally with some kind
>> of CI scripts that would do this heavy lifting?
>>
>> Kamil Trzciński
>>
>>
>>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 

Re: [pve-devel] Compiling PBS on arm64

[Kamil Trzciński]

Thank you Fabian :)

I was looking for just that.

I was able to remove `.cargo/config` and use `crates.io` to compile
packages. I will post more results once it is done, but I got it rebuild at
least on amd64,
and waiting for arm64 to finish.

Kamil

On Mon, Dec 7, 2020 at 8:54 AM Fabian Grünbichler <
f.gruenbichler@proxmox.com> wrote:

> On December 6, 2020 8:41 pm, Kamil Trzciński wrote:
> > I'm slightly progressing, but I stumbled across some `debcargo` problem.
> It
> > appears that
> > Proxmox uses their own fork of `debcargo`, which is needed in order to
> > build crates
> > without the usage of crates.io. Is this patch published somewhere?
> >
> > rust-debcargo (2.4.2-pve1) proxmox-rust; urgency=medium
> >
> >   * allow overriding maintainer via debcargo.toml
> >   * implement local crate support.patch
> >
> >  -- Proxmox Support Team <support@proxmox.com>  Tue, 14 Jan 2020
> 16:13:48
> > +0100
>
>
> https://git.proxmox.com/?p=debcargo-conf.git;a=tree;f=src/debcargo/debian/patches;h=7345cadea4cbffc3174c04e34960e91767892100;hb=refs/heads/proxmox/buster
>
> all of the toolchain backports are published as well on
> git.proxmox.com.. if you want to re-compile all of our devel repo for
> arm64, I suggest familiarizing yourself with debcargo and the
> debcargo-conf repo (especially the upstream and Proxmox readmes) ;)
>
> > On Fri, Dec 4, 2020 at 4:21 PM Kamil Trzciński <ayufan@ayufan.eu> wrote:
> >
> >> ARM64 is becoming increasingly popular, especially that PBS seems at
> least
> >> for my usage-pattern to be ideal to run on my arm64 NAS. In the end
> >> I want to try to be able to recompile everything for arm64 and see how
> >> nicely
> >> it works there.
> >>
> >> But first I decided to try to compile all packages for `amd64`. And I'm
> >> scratching my head
> >> to try to do that. This proves to be super hard due to multitude of
> >> dependencies that are backported,
> >> unordered, and require a very special set of commands to be executed
> >> depending on a git repo
> >> since PBS is rust-based.
> >>
> >> Do you maybe happen to have a comprehensive guide that allows you to
> >> compile everything
> >> from the scratch without the use of `/devel` repo, ideally with some
> kind
> >> of CI scripts that would do this heavy lifting?
> >>
> >> Kamil Trzciński
> >>
> >>
> >>
> > _______________________________________________
> > pve-devel mailing list
> > pve-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> >
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>

Re: [pve-devel] Compiling PBS on arm64

[Kamil Trzciński]

OK. If anyone is interested, a little hacky, but works:

1. I was able to compile (after minor patching) without having to rebuild
the whole dependency tree
    as required by Debian, instead using `crates.io`. It builds on a Debian
Bullseye as it was simpler
    and required basically no backports

2. I did use stock `debcargo`, but had to include patches to add `debian/*`
files in some cases.
    This is suboptimal, but it was the fastest way to have something
working. Ideally a mentioned `debcargo`
    should be used to create this glue between Rust and Debian

3. It compiled a minimal PBS for `amd64` and `arm64v8`. `amd64` compiled
natively, where for
    compiling `arm64v8` I used `qemu-user`. It took a few hours for this to
finish

4. It builds a very simple Docker image that can be easily run via
docker-compose. I was successful
    running a bunch of backups from my Home PC to my NAS running on
RockPro64 (ARM64)

5. All details and precompiled images are here:

    - https://github.com/ayufan/pve-backup-server-dockerfiles
    - https://hub.docker.com/repository/docker/ayufan/proxmox-backup-server

6. I plan to occasionally update it to the newer version and use it
personally for backuping my personal PVE



On Mon, Dec 7, 2020 at 1:42 PM Kamil Trzciński <ayufan@ayufan.eu> wrote:

> Thank you Fabian :)
>
> I was looking for just that.
>
> I was able to remove `.cargo/config` and use `crates.io` to compile
> packages. I will post more results once it is done, but I got it rebuild at
> least on amd64,
> and waiting for arm64 to finish.
>
> Kamil
>
> On Mon, Dec 7, 2020 at 8:54 AM Fabian Grünbichler <
> f.gruenbichler@proxmox.com> wrote:
>
>> On December 6, 2020 8:41 pm, Kamil Trzciński wrote:
>> > I'm slightly progressing, but I stumbled across some `debcargo`
>> problem. It
>> > appears that
>> > Proxmox uses their own fork of `debcargo`, which is needed in order to
>> > build crates
>> > without the usage of crates.io. Is this patch published somewhere?
>> >
>> > rust-debcargo (2.4.2-pve1) proxmox-rust; urgency=medium
>> >
>> >   * allow overriding maintainer via debcargo.toml
>> >   * implement local crate support.patch
>> >
>> >  -- Proxmox Support Team <support@proxmox.com>  Tue, 14 Jan 2020
>> 16:13:48
>> > +0100
>>
>>
>> https://git.proxmox.com/?p=debcargo-conf.git;a=tree;f=src/debcargo/debian/patches;h=7345cadea4cbffc3174c04e34960e91767892100;hb=refs/heads/proxmox/buster
>>
>> all of the toolchain backports are published as well on
>> git.proxmox.com.. if you want to re-compile all of our devel repo for
>> arm64, I suggest familiarizing yourself with debcargo and the
>> debcargo-conf repo (especially the upstream and Proxmox readmes) ;)
>>
>> > On Fri, Dec 4, 2020 at 4:21 PM Kamil Trzciński <ayufan@ayufan.eu>
>> wrote:
>> >
>> >> ARM64 is becoming increasingly popular, especially that PBS seems at
>> least
>> >> for my usage-pattern to be ideal to run on my arm64 NAS. In the end
>> >> I want to try to be able to recompile everything for arm64 and see how
>> >> nicely
>> >> it works there.
>> >>
>> >> But first I decided to try to compile all packages for `amd64`. And I'm
>> >> scratching my head
>> >> to try to do that. This proves to be super hard due to multitude of
>> >> dependencies that are backported,
>> >> unordered, and require a very special set of commands to be executed
>> >> depending on a git repo
>> >> since PBS is rust-based.
>> >>
>> >> Do you maybe happen to have a comprehensive guide that allows you to
>> >> compile everything
>> >> from the scratch without the use of `/devel` repo, ideally with some
>> kind
>> >> of CI scripts that would do this heavy lifting?
>> >>
>> >> Kamil Trzciński
>> >>
>> >>
>> >>
>> > _______________________________________________
>> > pve-devel mailing list
>> > pve-devel@lists.proxmox.com
>> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>> >
>>
>>
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>>
>