From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <aderumier@odiso.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id D54F170E6E for <pve-user@lists.proxmox.com>; Sat, 26 Jun 2021 13:16:35 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C26C21D448 for <pve-user@lists.proxmox.com>; Sat, 26 Jun 2021 13:16:35 +0200 (CEST) Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 865011D424 for <pve-user@lists.proxmox.com>; Sat, 26 Jun 2021 13:16:34 +0200 (CEST) Received: by mail-wm1-x334.google.com with SMTP id w13so7866147wmc.3 for <pve-user@lists.proxmox.com>; Sat, 26 Jun 2021 04:16:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=odiso-com.20150623.gappssmtp.com; s=20150623; h=message-id:subject:from:to:cc:date:in-reply-to:references :user-agent:mime-version; bh=dmjkxVRskQEsLQoRnTF8V/JT1CMGrtmJ1iX0qIzc/tM=; b=OKtSvx25tw4LplKxPjg667liLm1x6Psh5r0PKybY9DJZL1WNEC7gqN7slTl41iLgph tmqNIzBZGZT/GSeRJTEb99fHh7h82nr428A3a0SCze4ED4F61UKrgrmJ5YlDh1yOuKo+ Dvt1q9jMDepnfEA/Wm09l5+fhJ0NRiZSTq1T4Pies1nkdTMQMY9NQVi+XF9TwGFliWk9 04nIrxLfF3AIMA6x/QdvoPOZItSl4DFluDp6RbVVKUdu/P1Id9f20jXNoD83GmC1oNc3 aynH9NcAT5LWrowAebl4hEA9BeFWLW8Uz/9tPkfUYABIWJN2GH1+bFQAlQhjST9NRUeh brPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version; bh=dmjkxVRskQEsLQoRnTF8V/JT1CMGrtmJ1iX0qIzc/tM=; b=GPWVnTKqSty600WftHdeRTCuyzlMVEzC/Ogk+TRYwkuvg8VOwPhcTt9lHk7a3smKWU mGaGEcYBZAKaKQNxRTTrVTIrzRDehEdr3ELh4qjj18jbbSlrWUMF5kZEpQKahJb7N+9D jqp+N6QNN8W74FO8u7+bxiRUBMEnRw6a11lRXs2vTGcoDK7inmn5eyxZfr/rCZx8XHt+ IYAihdBNjQhI6bWYfKTWHzjWbLLaDDiW/cO7CcZc/DFIBtdTAPNu08Xvdb7+CBj77jnx j93YHktbqoUNlZseopbUQEQKqTPrtpaUUaoV0fgUemRXsPUjWLLvQOYvNUx82lLrxU71 XSOA== X-Gm-Message-State: AOAM533TJtv6DNeJ7wWFKcAli3eFfuaLRD+GMmOQ+M0QBYGn1HLRTo4N CIZsN55ARJesPO8YdaxHBme5pW5pt7EvHcZP X-Google-Smtp-Source: ABdhPJy4uISFcRKkUk0xQv94gvPt4JkGRuaST4csXUkVOObC9zxHOzWJ7qeUu4vLLik+m7JCMB13CA== X-Received: by 2002:a05:600c:4f56:: with SMTP id m22mr8081770wmq.16.1624706188017; Sat, 26 Jun 2021 04:16:28 -0700 (PDT) Received: from [192.168.178.50] ([79.132.252.54]) by smtp.gmail.com with ESMTPSA id s23sm8455837wmh.5.2021.06.26.04.16.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 26 Jun 2021 04:16:27 -0700 (PDT) Message-ID: <e8199a1f66fa65f2cf493c88d0ef2c25980e2a64.camel@odiso.com> From: aderumier@odiso.com To: Proxmox VE user list <pve-user@lists.proxmox.com>, "pve-user@pve.proxmox.com" <pve-user@pve.proxmox.com> Date: Sat, 26 Jun 2021 13:16:27 +0200 In-Reply-To: <mailman.16.1624545042.464.pve-user@lists.proxmox.com> References: <mailman.16.1624545042.464.pve-user@lists.proxmox.com> User-Agent: Evolution 3.40.2 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.995 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature HTML_MESSAGE 0.001 HTML included in message KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [maas.io] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] BIG cluster questions X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> List-Post: <mailto:pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Sat, 26 Jun 2021 11:16:35 -0000 Le jeudi 24 juin 2021 à 16:30 +0200, Eneko Lacunza via pve-user a écrit : > Now for more general questions; do you think a 88-node Proxmox VE > cluster is feasible? Well, corosync is not really done to this amount of node. (I think the hardcoded limit is around 100), but in practice, I have seen a lot of users having problem with corosync communication starting around 30 nodes (Maybe with low-latency switches + fast frequencies cpu, it's possible to keep latency enough low to get it working) > Those 88 nodes will host about 14.000 VMs. Will HA manager be able to > manage them, or are they too many? (HA for those VMs doesn't seem to > be > a requirement right know). I have cluster with 2500vms, it's working fine. (on a 15 nodes cluster with around 200vms on each node). I don't known with 15000vms, maybe the main pve-crm loop will take more time, I'm not sure about the timeouts. At work, I'm doing nodes 20 nodes cluster (1 by rack) to avoid to have big cluster. Multi-cluster central management is not yet on the roadmap, but they are some tool like https://maas.io/ which allow to manage multi-cluster in a central way.