Verisimilitude and Decision-Making

Last week IVX presented “Speed Mazer IVX” at the Waikato University Open Day 2016. The creation of Speed Mazer IVX was a good time to reflect on how verisimilitude forms perhaps the key criteria for our production decisions.

Verisimilitude is the appearance of being true or real. Verisimilitude is the seduction that teases the audience willingly into the world of the work. Things present or things-expected-but-missing will affect verisimilitude. Verisimilitude is the macro-clothing in which the work dresses. It is an overall sensory experience that emerges from the aesthetics of individual elements.

There are always pragmatic considerations when creating installation works. No concept proceeds cleanly from plan to execution intact. In fact, the sign of a great concept is that there are more ideas for how to improve the work then there are available resources to do them all. Why is that?

The obvious answer is that not all ideas are good ideas so having more ideas means that creators do not become wedded to bad ideas just because they are the only ideas. Comparing ideas against each other forces greater articulation of the desired verisimilitude.

Speed Mazer IVX is a player vs. player maze racer controlled by guitar hero instruments. The simplicity of the concept meant quick buy-in by the audience, a sense of competition and an element of mastery. Having randomly generated mazes meant that players could not develop muscle memory to complete the task.

Initially, we had cosmos inspired backgrounds. IVX loves cosmos pictures – indeed, our last year’s Open Day installation was “Write Your Name Among the Starts”. However, during testing, it was apparent that the cosmos background images distracted from the competitive feeling of racing through a maze. We went with black backgrounds to give an 8-bit arcade feel.

We tried to add sound – which always goes a long way to elevating verisimilitude. However, there were problems with the sound library that were going to take considerable time to resolve. We instead elected to put the time into visual aspects.

The maze is bumped when a player hits a wall. The bump reaction signals to the player that they have hit a wall. The maze feels more real because bumps behave with believable physics. The bump reactions happen in 3D, but the 3D renderer limited what we could do with the walls (without more work). The positive verisimilitude effect of the bumps in 3D was greater than the verisimilitude of nicer looking walls, so we kept the bump effects.

Particles are spat out opposite to the direction of player movement. Some of particles are added to the opposing player’s maze, and each player spits their colour. When players are concentrating on their own maze then still get feedback that the other player is nearby. Particles also help build up a visual intensity proportional to the speed at which the player moves through the maze. Getting the right feel to the particle fields were the subject of much experimentation.

Initially, we wanted to have the player’s represented by a shape that would change direction as the player traversed the maze. While this might have been a great idea, it was judged as having a lesser effect on the verisimilitude so was not attempted as the deadline approached. Other ideas; random maze wall art, generative backgrounds, power-ups, a cheat mode, and more visual prompts similarly prioritised.

The functional aspects - making a heads-up maze racer than one side could win - did not take long to program. However, the bulk (probably 75% or more) of the coding time went into making tuning the verisimilitude to create the right experience for the audience.

Reflecting on our decision-making process during the creation of Speed Mazer IVX made me realise the central role of verisimilitude is to our work. We dropped ideas that broke the verisimilitude and prioritised our features against our limited resources (time) according to verisimilitude. I look forward to advancing this thinking more in future IVX work.


Evicting and Flushing from Ceph Cache Tier/Cache Pools

Disaster! The OSDs backing my cache pools were reporting full. This occurred because the node carrying most of the backing pools crashed leaving insufficient replicas of the backing pool. Even when that node was brought back online, recovery operations going to take a long time. Here's what I did. The first thing was to set an absolute maximum size to the cache tier pool:
ceph osd pool set cachedata target_max_bytes ....

The next thing was to start manually evicting objects from the pool. (Flushing writes dirty objects back to the backing pool, evicting boots out clean objects). Flushing would need the backing pool up and able to accept new writes - but evicting would not. Evicting would free up space in the cache tier without the backing pool having stabilised.

The standard command to evict objects is:

rados -p cachepool cache-flush-evict-all

I found that locked up on me, complaining that some objects were locked. I also tried another variant, but that was not shrinking the pool either.

rados -p cachepool cache-try-flush-evict-all

My next trick was to use parallel (apt-get install parallel if you don't have it!) to try evicting objects one by one. I'd run this script until satisfied that the cache pool had shrunk to a reasonable size and then Ctrl-C to terminate the evictions.

rados -p cachepool ls | parallel -j16 rados -p cachepool cache-try-evict {}

What this command does is list the contents of cachepool and take each entry and spawn an instance of rados to try to evict each object sepearately. The -j16 means to spawn 16 rados processes at a time.

For completeness, the other cache flushing or evicting commands that rados recognises are (from here):


I believe that variants with "try" in the name are non-blocking while the rest will block.

Soon the SSD OSDs that back my cache tier were back under warning levels. My cluster continued recovering overnight and all the data lived happily ever after (at least until next time).

Ceph Cluster Diary January 2016

Another node is added to my Ceph cluster. This is a second-hand dual Pentium-D with 8GB of RAM and a 250GG SATA HDD. Removed from the box a dual-head Quadro graphics card and a RAID controller plus SCSI2 drives that were not compatible with Linux.

Cluster speed is a concern, so three 220GB drives were installed in the box and a writeback cache tier was created around the cephfs data pool. The online examples were very useful.

At first, the metadata pool was also cached. Though I begun to get cluster thrashing problems. I mitigated this, and some data-risk, by removing the metadata cache, but adding a crush rule to have two replicas of the metadata on SSDs and the final copy on HDDs.

There were also numerous page faults, so I gave up some space on one of the SSDs for a Linux swap partition and most of the page faults disappeared.

Most file operations are about three times faster than before. When the metadata was also cached there was about a 7x speed up, but the cluster was less reliable. My backing storage devices are mainly external USB hard drives running on old USBv1 hardware so any speed up is welcome.

The result is a much more reliable cluster that gives consistent enough speed to run virtual hard-drive files for some Virtual Machines that I occasionally run on my main desktop. Previously, those Virtual Machines had a tendency to crash when run from cephfs.

Early on, I did have a problem with the cache filling up, but fixed that by applying more aggressive cache sizing policies. In particular, I set the target_max_bytes to 85% of my SSD size.

ceph osd pool set cachedata target_max_bytes .....

I very pleased with the setup now. One or two more tweaks and I might be ready to begin retiring my dedicated NAS box and switch all my network storage to ceph.