How an AXOL server cluster is built: design and at least 24 hours of stress testing

From workload analysis to final validation: how AXOL designs, assembles, and puts every server cluster through at least 24 hours of stress testing.
Category:
Proxmox and Ceph
AXOL Server
texture axol newspaper tiny
How an AXOL server cluster is built: design and at least 24 hours of stress testing

A node stops responding. Alerts appear on the screen. Meanwhile, applications, databases, and users continue demanding resources. At that moment, the person responsible for the infrastructure isn’t thinking about the CPU model or the numbers on a spec sheet. They want to know whether services will remain available and how much headroom they have before a failure turns into downtime.

It’s a significant responsibility. Because behind a cluster, there are more than just virtual machines. There are employees who need to do their jobs, customers waiting for answers, production departments, and services that cannot be interrupted without consequences. And there’s an IT team that needs to have answers ready when something goes wrong. They can’t afford to improvise under pressure.

That’s why an AXOL Server cluster doesn’t start with a product catalog. It starts with the failures you need to withstand, the actual workloads the system must handle, and the growth it needs to accommodate. Only then come the hardware, Proxmox VE, Ceph, and assembly. Before delivery comes the longest test: at least 24 hours of stress testing, during which the cluster must demonstrate how it performs under sustained load.

At a glance

  • The project starts with workloads, expected growth, and the services that must remain available even during a failure.
  • CPU, memory, storage, and networking are sized as parts of a single Proxmox VE and Ceph system.
  • The cluster is assembled at AXOL’s facilities and undergoes at least 24 hours of stress testing before delivery.
  • The goal isn’t to promise that nothing will ever fail, but to reduce uncertainty and prepare the system to handle unexpected events.

Before building the cluster, understand what’s at stake

The first question isn’t “How many nodes do you need?” It’s “What needs to keep running if a host becomes unavailable?” The number of virtual machines matters, but it doesn’t tell the whole story. You need to understand which applications take priority, which depend on databases or external services, when peak loads occur, and how long an outage can last before it causes real damage.

This information isn’t always readily available in a structured form. A company may know how much storage space is being used but not its monthly growth rate. It may have a complete VM inventory but no reliable historical I/O data. That’s normal. The first step isn’t to pretend the data is more precise than it is. It’s to distinguish measured data from estimates, make assumptions explicit, and identify what’s missing.

This is where AXOL acts as a technical advisor. The customer brings their knowledge of the existing environment, constraints, and priorities. Our systems engineers turn that information into testable scenarios. This helps avoid two costly mistakes: an undersized cluster that runs out of headroom precisely when a failure occurs, or an oversized system designed to account for every uncertainty.

To develop an initial architecture, we gather at least the following information:

  • actual CPU and memory usage, including peaks, rather than just the resources allocated to VMs;
  • used storage capacity, expected growth, and workloads sensitive to latency or I/O performance;
  • services that must remain available during maintenance or a failure;
  • recovery time objectives and acceptable data loss;
  • application dependencies, networking, backups, and access requirements;
  • planned new workloads and the level of autonomy the internal IT team wants to maintain.

The design starts with the failures you need to withstand

When all nodes are available, the cluster’s total capacity may seem more than sufficient. The real sizing challenge becomes clear when part of the cluster is unavailable. With an N-1 requirement, critical workloads must be able to run with one node down. With N-2, the design must support two nodes being unavailable at the same time.

The difference becomes apparent at the worst possible moment. If the remaining capacity isn’t enough, someone will have to quickly decide which VMs to slow down or shut down. That decision shouldn’t have to be made during an incident. It needs to be addressed beforehand by defining priorities, minimum acceptable performance levels, and expected growth.

These requirements determine the number of nodes, CPU, RAM, storage, and networking. We don’t start with three or five servers and then look for a technical justification. The operating scenario determines the architecture.

A cluster is more than the sum of its components

Once the architecture is defined, the bill of materials takes shape. But this isn’t a competition to select the most powerful components. In a Proxmox VE and Ceph cluster, every part depends on the others. A fast CPU can’t compensate for an inadequate network, and a large amount of raw storage doesn’t guarantee the usable capacity or performance needed during recovery and rebalancing.

CPU and memory: performance even with one node down
Core counts, clock speeds, NUMA, overcommitment, and ECC memory are evaluated against actual workloads and the capacity that must remain available. The goal isn’t to fill a spec sheet with impressive numbers. It’s to prevent maintenance or a failure from pushing an otherwise stable environment to its limits.

Storage: terabytes don’t tell the whole story
Usable storage capacity, replication, failure domains, latency, throughput, and the space required for recovery operations all need to be considered together. Storage must support everyday workloads, but also the more demanding situations: rebuilding data, replacing a drive, or bringing data back into sync.

Networking: when Ceph is busy, it needs headroom
In a Ceph cluster, the network is part of the storage system. VM traffic, replication, migration, recovery, and management operations can overlap. Network topology, redundancy, and capacity must therefore be designed for what happens under load, not just normal operating conditions.

Power and remote access: small details until you need them
Redundant power supplies, UPS systems, power distribution, and out-of-band management become essential when the operating system stops responding or physical access to the rack isn’t possible. These features may not stand out in a demo, but they make a real difference when technical intervention is required.

On the workbench, design assumptions become real-world responsibilities

AXOL clusters are assembled at our facilities in Ivrea and Lomazzo, Italy. On the workbench, we verify that the system we’ve built matches the approved design. Components, cabling, power supply, airflow, and management interfaces must all be consistent with the agreed architecture.

Every node must also start from a consistent baseline. Firmware, settings, and configurations can’t depend on the memory of the person who assembled the system. An undocumented difference may seem insignificant today, only to make troubleshooting more difficult months later, precisely when time matters most.

Next, Proxmox VE and Ceph are configured, including networking, quorum, storage, high availability, and administrative access. Backups and disaster recovery remain separate layers of protection. High availability can handle certain cluster failures, but it doesn’t replace an independent backup copy or a verified restore.

Why does every AXOL cluster undergo at least 24 hours of stress testing?

When everything appears ready, the approach to testing changes. We’re no longer looking for confirmation that everything works. We’re looking for problems. A cluster that powers on and shows all nodes online has passed only the most basic check. Before running business-critical services, it needs to be observed under load, with enough time for potential instabilities to emerge.

That’s why every AXOL cluster undergoes at least 24 hours of stress testing. Extended testing is designed to uncover instabilities that shorter tests might miss. Throughout the process, the cluster is monitored as a complete system, under load, following AXOL’s internal technical procedures.

If an anomaly is detected or a component needs to be replaced, we address the cause and restart the stress test from scratch. The cluster then undergoes another full testing cycle of at least 24 hours. The goal is to increase the likelihood that a problem surfaces on AXOL’s workbench, where it can be addressed, rather than in the customer’s rack while production services are running.

From testing to delivery

Once testing is complete, the cluster moves on to the delivery activities agreed upon with the customer.

Installation, data center hosting, implementation support, and migration depend on the agreed scope of work. Moving the system into production requires its own plan. Workloads, dependencies, maintenance windows, backups, and rollback procedures cannot be improvised on delivery day.

For those responsible for the infrastructure, confidence doesn’t come from a promise that nothing will ever fail. No one can realistically make that promise. It comes from knowing the system has been sized for real-world scenarios, monitored under stress for at least 24 hours, and delivered with capacity headroom and technical experts to rely on.

You don’t need perfect data to get started

For an initial discussion, an inventory of hosts and VMs, available utilization data, used storage capacity, expected growth, and key business continuity requirements are enough. From there, we can identify missing measurements, scenarios that need validation, and assumptions that require a cautious approach.

The AXOL configurator generates an initial configuration and a technical and cost summary. Based on those results, you can request one hour of free consultation with AXOL’s systems engineers to validate the data, examine workloads in more detail, and develop a tailored technical and commercial proposal. This is the step that turns an initial estimate into a project discussed with both the people who will build it and those who will manage it.

AXOL Server

LINKEDIN | All rights reserved – DkR s.r.l. | VAT No. 07459820960 | Share Capital €105,000 fully paid up | Cookie Policy | Privacy Policy | Accessibility Statement | Made with ❤ by DkR