Performance

Benchmark Tests were conducted on Amazon Web Services EC2 servers with the following results :

  • Results using Turbo without Caching feature :
    • Turbo Agents on 3 m6a.xlarge EC2 instances (18 Slots) gave 31.7% improvement
    • Turbo Agents on 4 m6a.xlarge EC2 instances (24 Slots) gave 46.0% improvement
    • Turbo Agents on 6 m6a.xlarge EC2 instances (36 Slots) gave 55.0% improvement
    • Turbo Agents on 8 m6a.xlarge EC2 instances (48 Slots) gave 57.0% improvement
  • Results using Turbo with Caching feature :
    • Turbo Agents on 3 m6a.xlarge EC2 instances (18 Slots) gave 69.9% improvement
    • Turbo Agents on 4 m6a.xlarge EC2 instances (24 Slots) gave 69.5% improvement
    • Turbo Agents on 6 m6a.xlarge EC2 instances (36 Slots) gave 69.3% improvement
    • Turbo Agents on 8 m6a.xlarge EC2 instances (48 Slots) gave 68.7% improvement
  • Compilation times were an average from at least 3 results
  • Points of note:
    • Up to 2.3 times faster without caching
    • Up to 3.5 times faster with caching
    • Even just 3 machines in the Cluster gave significant build-time improvement
    • Build times with caching (and 100% hit ratio) remained constant regardless of cluster size because all jobs were scheduled to the local Agent. In a more realistic environment, other Agents may have the cache result and will be scheduled. This would give marginally slower build-time due to the transfer of the object file across the network
    • Build-time gains diminished as cluster size increased to 8 machines, due to the local machine having the bottleneck of the pre-processing workload. It cannot feed the cluster quickly enough to take full advantage of its resources. However in a real world scenario, the other machines will also be running builds and the spare cluster resources will be utilised

The details :

  • The EC2 instances used were m6a.xlarge (4 vCPU, 16GB RAM) using Amazons AMI for Windows 2019 Server
  • Compilation time was measured for the Foundation solution from within the Poco Framework v1.13.2
  • Compilation was on Visual Studio 2019, under "Release" mode and in x64 architecture using msbuild.exe
  • Each EC2 instance had a Turbo Agent installed with 6 CPU Slots allocated
  • The Turbo Scheduler was installed on a separate m6a.large EC2 instance (no Agent installed on this machine)
  • Some folders used by Turbo were "excluded from realtime scanning" from Microsoft Windows Defender to prevent it affecting results (c:\ProgramData\Turbo\logs, c:\ProgramData\Turbo\work-temp, and the Poco Foundation source code folder)
  • Only a single build was running on the Build Cluster
  • Consistent, repeatable testing was achieved using a Packer built EC2 image and deployed using Terraform (files available on request)

Some background on factors that affect compilation times :

  • The number of c/c++ files within each VS Project. Fewer and larger Projects will see more improvement
  • The size of the c/c++ files
  • Compilation mode. Release mode will see more improvement than Debug
  • Size of the Build Cluster
  • The current workload on the Build Cluster
  • Caching of compiled object files and the number of cache 'hits' within the build
  • Maximizing build delegation throughout the cluster by having matching build environments on all machines

See here for an overview of the Turbo Build Accelerator