Shootout 0001

ToC

1. Introduction
   1. Motivation
   2. Ring Problem
   3. Fan Problem
2. Observations
   1. Observation Conditions
   2. Platform 1
   3. Platform 2
   4. Names
      1. Columns Explained
      2. Some Program Names Explained
3. Reactions
4. Future Work
   1. Machinate
   2. Shootout
      1. More Platforms
      2. More Problems
      3. More Programs

Introduction

Motivation

This shootout is a collection of programs that can be run and the performance observed in some way. The purpose of the shootout is to provide a mechanism for comparing machinate's performance to itself over time, and also to compare against other similar libraries.

The programs are grouped into different implementation of the "same" problem.

Ring Problem

The ring problem involves setting up some number of threads/logical threads/virtual threads/etc to copy from an input channel/stream/queue to an output, arranged to form a ring. The performance measured is the number of laps around the ring that are made in a given fixed time period.

Fan Problem

The fan problem is a sort of "fan out" followed by a "fan in". A thread writes N messages to a channel, and N threads read a message each from that output and then write that message to a single channel, and the original thread reads N messages from that channel and then loops. The performance measurement is again how many iterations of that process happen in a fixed time period.

Observations

Observation Conditions

This iteration of the shootout was run on two different platforms. Platform 1 is a dedicated computer with a very modest 4 cores. Platform 2 is a cloud vm with 48 "vcpus".

The shootout code attempts to determine how many virtual threads the jvm allows to execute concurrently (how many can be mounted at once) and sets core.async's dispatch thread pool (used by go) to be that size. This is an attempt to provide something of an apples to apples comparison, but in some cases, like using core.async's blocking api with virtual threads, may cause some contention for resources between virtual threads and core.async's thread pool.

Platform 1 (4 cores)

tag	name	min(measure)	avg(measure)	max(measure)	count(*)	version	range
fan	machinate-fan	1.16125	1.23849077453366	1.29788333333333	822	0.0.79-35-g7f85ba3-dirty	0.136633333333333
fan	core-async-go-loop-fan	0.175966666666667	0.208628200692042	0.2257	867	0.0.79-35-g7f85ba3-dirty	0.0497333333333334
fan	machinate0-0-79-fan	0.00483333333333333	0.0824708084428514	0.169633333333333	837	0.0.79-35-g7f85ba3-dirty	0.1648
ring	transfer-queue-ring	0.729683333333333	1.05268213317847	1.49736666666667	861	0.0.79-35-g7f85ba3-dirty	0.767683333333333
ring	core-async-go-loop-ring	0.812433333333333	0.910689824360443	1.13861666666667	873	0.0.79-35-g7f85ba3-dirty	0.326183333333333
ring	machinate-ring	0.5916	0.883672948717949	0.9794	780	0.0.79-35-g7f85ba3-dirty	0.3878
ring	thread-parking-locking-strawman-ring	0.5606	0.843746034482759	0.944816666666667	870	0.0.79-35-g7f85ba3-dirty	0.384216666666667
ring	core-async-virtual-thread-ring	0.395116666666667	0.817322968490879	1.07066666666667	804	0.0.79-35-g7f85ba3-dirty	0.67555
ring	thread-parking-single-queue-ring	0.591833333333333	0.815828427895981	1.05895	846	0.0.79-35-g7f85ba3-dirty	0.467116666666667
ring	thread-parking-strawman-ring	0.599983333333333	0.761403076582559	0.874516666666667	753	0.0.79-35-g7f85ba3-dirty	0.274533333333333
ring	machinate-go-loop-ring	0.213	0.237484157832744	0.270433333333333	849	0.0.79-35-g7f85ba3-dirty	0.0574333333333334
ring	machinate0-0-79-ring	0.0399833333333333	0.0832531461569096	0.114733333333333	837	0.0.79-35-g7f85ba3-dirty	0.07475

Platform 2 (48 vcpus)

tag	name	min(measure)	avg(measure)	max(measure)	count(*)	version	range
fan	machinate-fan	0.45685	0.469560222222222	0.482666666666667	75	0.0.79-35-g7f85ba3-dirty	0.0258166666666667
fan	machinate0-0-79-fan	0.136983333333333	0.152629111111111	0.168433333333333	75	0.0.79-35-g7f85ba3-dirty	0.03145
fan	core-async-go-loop-fan	0.0835	0.14226994949495	0.165816666666667	66	0.0.79-35-g7f85ba3-dirty	0.0823166666666667
ring	core-async-go-loop-ring	1.16768333333333	1.26121047008547	1.34861666666667	78	0.0.79-35-g7f85ba3-dirty	0.180933333333333
ring	transfer-queue-ring	0.4999	0.956492831541219	1.03095	93	0.0.79-35-g7f85ba3-dirty	0.53105
ring	thread-parking-single-queue-ring	0.653516666666667	0.702432592592593	0.7236	90	0.0.79-35-g7f85ba3-dirty	0.0700833333333334
ring	core-async-virtual-thread-ring	0.581516666666667	0.610411428571428	0.62765	105	0.0.79-35-g7f85ba3-dirty	0.0461333333333334
ring	thread-parking-strawman-ring	0.423933333333333	0.599186257309941	0.6382	114	0.0.79-35-g7f85ba3-dirty	0.214266666666667
ring	thread-parking-locking-strawman-ring	0.463966666666667	0.586507026143791	0.60825	102	0.0.79-35-g7f85ba3-dirty	0.144283333333333
ring	machinate-ring	0.401783333333333	0.483170899470899	0.499083333333333	63	0.0.79-35-g7f85ba3-dirty	0.0973
ring	machinate0-0-79-ring	0.118783333333333	0.132741555555556	0.145783333333333	75	0.0.79-35-g7f85ba3-dirty	0.027
ring	machinate-go-loop-ring	0.0755333333333333	0.0997511494252874	0.139383333333333	87	0.0.79-35-g7f85ba3-dirty	0.06385

Names

Columns Explained

• tag indicates which problem a program belongs to
• name is the name of the program
• count is the number of scores (measure) recorded for the given program
• version is a the output of git describe --tags --dirty in the git tree where the shootout was run.

Some Program Names Explained

• machinate-fan is the current version of machinate running the fan program using virtual threads
• machinate0-0-79-fan is machinate version 0.0.79 running the fan program
• transfer-queue-ring is the ring problem implemented using java's TransferQueue
• thread-parking-single-queue-ring, thread-parking-strawman-ring, and thread-parking-locking-strawman-ring are channel implementations that exist just as part of the shootout suite as explorations of the channel implementation space.
• except where indicated in the program name virtual threads are used as the light weight thread mechanism.

Reactions

1. Machinate performance has improved a fair bit between 0.0.79 and the upcoming release
2. Machinate performance still trails core.async's channels.
   • On the 4 core machine performance appears to be fairly close
   • On the 48 vcpu vm core.async is smoking machinate in the ring benchmark
   • Why?
     1. machinate may still be allocating more
     2. core.async elides some locks
     3. machinate may be overly aggressive looping in the channel event implementation causing contention.
     4. virtual threads are new :/
3. These programs are large enough and complicated enough that it doesn't take that many runs to get a fairly stable average score.

Future Work

machinate

Performance is not the only goal of machinate, and machinate provides more than just message passing over channels. But the next bit of work on the performance of message passing over channels will likely look at limiting any looping the channel event does while trying to ensure anything queued is processed.

shootout

More platforms

1. A thing a single core/vcpu platform might be interesting
2. It was very disappointing to see lots of improvement in machinates standing on the 4 core platform over the course of development, only to get a smaller improvement on the 48 vcpu platform. I should figure out a way to more regularly test on a beefier machine.

More Problems

1. core.async and machinate both provide a pubsub implementation. Add a problem based on that.
2. The fan problem is implemented kind of weird, maybe it should be using mults or the equivalent

More Programs

1. manifold is glaringly absent from the shootout.
2. it would be interesting to see how a transfer queue based fan implementation does.