Showing posts with label howto. Show all posts
Showing posts with label howto. Show all posts

2016-05-16

Verisimilitude and Decision-Making

Last week IVX presented “Speed Mazer IVX” at the Waikato University Open Day 2016. The creation of Speed Mazer IVX was a good time to reflect on how verisimilitude forms perhaps the key criteria for our production decisions.

Verisimilitude is the appearance of being true or real. Verisimilitude is the seduction that teases the audience willingly into the world of the work. Things present or things-expected-but-missing will affect verisimilitude. Verisimilitude is the macro-clothing in which the work dresses. It is an overall sensory experience that emerges from the aesthetics of individual elements.

There are always pragmatic considerations when creating installation works. No concept proceeds cleanly from plan to execution intact. In fact, the sign of a great concept is that there are more ideas for how to improve the work then there are available resources to do them all. Why is that?

The obvious answer is that not all ideas are good ideas so having more ideas means that creators do not become wedded to bad ideas just because they are the only ideas. Comparing ideas against each other forces greater articulation of the desired verisimilitude.

Speed Mazer IVX is a player vs. player maze racer controlled by guitar hero instruments. The simplicity of the concept meant quick buy-in by the audience, a sense of competition and an element of mastery. Having randomly generated mazes meant that players could not develop muscle memory to complete the task.

Initially, we had cosmos inspired backgrounds. IVX loves cosmos pictures – indeed, our last year’s Open Day installation was “Write Your Name Among the Starts”. However, during testing, it was apparent that the cosmos background images distracted from the competitive feeling of racing through a maze. We went with black backgrounds to give an 8-bit arcade feel.

We tried to add sound – which always goes a long way to elevating verisimilitude. However, there were problems with the sound library that were going to take considerable time to resolve. We instead elected to put the time into visual aspects.

The maze is bumped when a player hits a wall. The bump reaction signals to the player that they have hit a wall. The maze feels more real because bumps behave with believable physics. The bump reactions happen in 3D, but the 3D renderer limited what we could do with the walls (without more work). The positive verisimilitude effect of the bumps in 3D was greater than the verisimilitude of nicer looking walls, so we kept the bump effects.

Particles are spat out opposite to the direction of player movement. Some of particles are added to the opposing player’s maze, and each player spits their colour. When players are concentrating on their own maze then still get feedback that the other player is nearby. Particles also help build up a visual intensity proportional to the speed at which the player moves through the maze. Getting the right feel to the particle fields were the subject of much experimentation.

Initially, we wanted to have the player’s represented by a shape that would change direction as the player traversed the maze. While this might have been a great idea, it was judged as having a lesser effect on the verisimilitude so was not attempted as the deadline approached. Other ideas; random maze wall art, generative backgrounds, power-ups, a cheat mode, and more visual prompts similarly prioritised.

The functional aspects - making a heads-up maze racer than one side could win - did not take long to program. However, the bulk (probably 75% or more) of the coding time went into making tuning the verisimilitude to create the right experience for the audience.

Reflecting on our decision-making process during the creation of Speed Mazer IVX made me realise the central role of verisimilitude is to our work. We dropped ideas that broke the verisimilitude and prioritised our features against our limited resources (time) according to verisimilitude. I look forward to advancing this thinking more in future IVX work.

2015-08-30

Ceph Cluster Thrash and Rebooting Nodes

I have a home cluster with low traffic volumes but terabytes of data - mostly photos. Ceph provides peace of mind that my data is resilient against failure, but my nodes are made with recycled equipment, so when a node reboot places considerable stress on the cluster to bring it back in.

Cluster thrash occurs when activity on a Ceph cluster causes it to start timing out OSDs. It's bad because recovery processes will begin - further stressing the cluster and potentially also causing problems. Then some of those outed OSDs will attempt to rejoin causing yet more problems. The usual way to avoid cluster thrash is to properly resource the cluster in the first place to handle the loads. In my case that's not a justifiable expense - my nodes are recycled (HDs are new) and my normal load doesn't stress the cluster much until a node reboots.

Here's how to deal with cluster thrash.

Tell ceph to not rebalance the cluster due to OSDs leaving. If possible, do this before rebooting your storage node, but it can be done at any time.
ceph osd set nodown
ceph osd set noout

Temporarily disable access to the cluster. My cluster predominantly serves files so stopping the network file system daemon does the job. Also unmount cephfs.
sudo service samba stop
sudo umount -lf /mnt/ceph
Your file system might not be windows networking (so not samba). Run this on the node that serves the files - which is not necessarily the same as the ceph mds or monitor nodes.

Also, shutdown the MDS service on each node that runs it. MDS is the process the oversees cephfs.
sudo service ceph stop mds 

The reason for stopping file access is to prevent load put on the cluster while it is recovering and the avoid potentially causing different versions of data to exist on the cluster. Ceph handles the latter situation very well on its own but consumes some effort in doing so. If disabling cluster access is not an option then see the tips at the end of the article.

Temporarily disable scrubbing since that takes resources:
ceph osd set noscrub
ceph osd set nodeep-scrub

Add the OSDs back into the cluster one or two at a time and allow the cluster to stabilise as much as it can in between. Newly added OSDs will go through a process of peering. Wait until all placement groups have finishing peering before adding another OSD to the cluster.
sudo service ceph start osd.<X>

This guide wouldn't be complete without listing the actions to re-enable normal operations on the cluster.
Re-start the MDS service. I prefer to do this first because it takes some time before it's ready to serve my cephfs again. I watch the monitor until the IO ceases - or I keep attempting to mount cephfs until it eventually succeeds.
sudo service ceph start MDS

Remount cephfs
sudo mount /mnt/ceph
Restart the samba service
sudo service samba start

At this point the cluster is again serving files, but we still need to re-enable scrubbing and allow the cluster to re-balance when there's errors. Never run a cluster without deep-scrubbing because that's your defense against data corruption.
ceph osd unset noscrub
ceph osd unset nodeep-scrub
ceph osd unset noout
ceph osd unset nodown
And you're done.

If you cannot afford to disable file access then all of the other tips might still be useful. In addition, if you have three or more replicas then you can also temporarily lower the minimum number of placement group replicas the cluster requires to be available. You should not put this lower than (number_of_replicas  / 2) + 1 or you risk data inconsistency. Check the number of replicas by getting the size of the pools (in my case 3), record the usual value for min_size and then use set to change the min_size temporarily.
ceph osd pool get <poolname> size
ceph osd pool get <poolname> min_size
ceph osd pool set <poolname> min_size 2
If you're using a fairly standard cephfs setup then there are actually two pools called: data and metadata. Change the min_size on both of them but always check the size of each pool first because they might be different. I run my more replicas of my metadata pool. 
Don't forget to set the min_size back to whatever value you normally have it set too once the cluster stabilises.

It does look like a lot of work to reboot a node. In practice I don't reboot nodes all that often and usually I don't need even half of the above tips to bring my current cluster back without cluster thrash. Since going away from USB flash drives into HD-spinners, I no longer have the cluster thrash problems I once did. Though, following the above tips does bring OSDs back into the cluster much quicker than letting it all happen by itself.

Happy cephing.

2015-08-22

Bulletproof Processing: try…catch

Processing is stable and crashes are infrequent. However, my experience doing performances and exhibitions for IVX has shown me it is handy to add a bit of extra resiliency to sketches. This is especially true when I expect to run for extended periods of time or when the sketch runs largely unattended. The first tip I have is to use a Java try…catch structure inside the draw() method.

Since Processing is based in Java then why not take advantage of Java’s native error handling. A try…catch structure attempts to execute all code within the try part of the structure and only executes the catch block if there is a problem. This particular patterns helps if the errors are transient, meaning that the error will generally go away on its own.

void draw() {
    try {

        // Normal drawing code goes here


    } catch (Exception e) {
        println(“draw(): “ + e.getMessage());
        // Pause for a quarter second. Hope problem goes away
        try {
            Thread.currentThread().sleep(250);
        } catch (Exception e1) {
        }
    }
}

This particular skeleton will catch any exceptions that occur in the draw() method, print them and then pause for 250 milliseconds. That quarter second is usually enough for a transient error to correct itself – perhaps a file loading or some memory becoming available. Either way, the sketch will pause for a short time and then attempt to resume itself. Without this exception handing then any Exception will cause the sketch to stop running.

Note that the sleeping function itself can throw an exception so Java forces us to declare a try…catch block – even though the catch block is empty.

Using try…catch in the setup() method might be useful if the code will be run by others, but in that case make extra sure that error messages are informative. I would prefer that setup() fails outright since most setup() errors are not transient.

So, you’ll see that it does not take much extra code to add a good amount of resiliency to a Processing sketch. Give it a try. Java's website has good resources that take a more in-depth look at exception handling.

2015-07-26

Play The Maryland Extra Credit Problem

A screenshot of a class problem from the University of Maryland has been doing the rounds. The teacher invites the students to vote to receive extra credit:

Here you have the opportunity to earn some extra credit on your final paper grade. Select whether you want 2 point or 6 points added onto your final paper grade. But there's a small catch: if more than 10% of the class selects 6 points, then nobody gets any extra points. Your responses will be anonymous to the rest of the class, only I will see the responses.

This situation is a little different than the Prisoner's Dilemma made famous in Game Theory because nobody stands to lose anything. All outcomes are either neutral or a gain. From that vantage point the best course of action is to always vote for six points. However, I think there are some political dynamics at play that might alter your decision to decrease the likelihood of the neutral outcome.

Depending on your expected final grade, here's my advice on how to play:

If you're a troll then go for six points. #YOLO

If you're a high scoring student (A / A+) then select 2 points. You don't need the extra marks. You're doing so well that you're above all this competitive stuff. Give the other people a chance for a few extra points. Unless, you reckon that people should have to earn their place and you think the mountain top has room for only you... then go for 6 points.

Jo Beeplus: Always go 6 points. You stand to lose nothing if it ends up nobody gets any bonus points and you might just get the six points to your grade into A territory. You work hard, you deserve a shot at an A right?

Jo SeePlus: Go for the 2 points. You might need the bonus marks to ensure you pass so you don't want to risk getting zero bonus points.

Jo "NearFail": Go for six points. You might get them and two points or zero points won't make a difference.

Jo "TotalFailure" You're so far behind you should give others a shot to shine: go for 2 points. Unless you're spiteful.

Bonus evilness karma if you vote for 6 points while talking up a big game regarding the virtues of choosing 2 points.

Go play!

2015-02-14

Forcing AIO on Ceph OSD journals

My Ceph cluster doesn't run all that quick. On reads it was about 20% slower than my RAID5 NAS and writes were 4x slower! Ouch. A good part of that is probably down to using USB flash keys but...

Upon starting the OSDs I see this message:

journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
My OSDs are XFS backed which supports async writing to the journal, so let's set that up.

First, ssh into the ceph-deploy node and get the running version of the .conf file, replacing {headnode] with the hostname of the main monitor:

ceph-deploy --overwrite-conf config pull {headnode}
You can skip this step if your .conf is up to date.

Next edit the .conf file and add the following. If you already have an [OSD] section then update accordingly.

[OSD]
journal aio = true
journal dio = true
journal block align = true
journal force aio = true
This will try to apply this setting to all OSDs. You can control this on a per OSD basis by adding sections named after the OSD. E.g.
[OSD.4]
journal aio = true
journal dio = true
journal block align = true
journal force aio = true
And here's the official documentation.

Next, push the config back out to all the ceph nodes and restart ceph your osds. Separate hostnames with spaces.

ceph-deploy --overwrite-conf config push {headnode} {cephhost1} {cephhost2} ....
A word of warning: the --overwrite-conf flag is destructive. I'll leave it to you to take backups.
Then SSH into the various nodes restarting the ceph service as you go. I just restarted all ceph services,
sudo service ceph restart
But it's probably okay to just restart the OSDs
sudo service ceph restart osd

I experienced an almost 2 times speed increase on writes until the journals fill. Still slow, but getting much better. My fault for not having better hardware!

Read more about my Ceph Cluster.

2015-02-05

Howto Deep-Scrub on All Ceph Placement Groups

Ceph automatically takes care of deep-scrubbing all placement groups periodically. The exact timing of that is tunable but you're probably here because you want to force deep-scrubs.

The basic command for deep-scrubbing is:

ceph pg deep-scrub <pg-id>

and you can find the placement group ID using:
ceph pg dump

And if you want to instruct all placement groups to deep-scrub, use the same script from repairing inconsistent PGs. Basically loop over all the active PGs, instructing each to deep-scrub:

ceph pg dump | grep -i active | cut -f 1 | while read i; do ceph pg deep-scrub ${i}; done

The repair article explains how this line of script works.

You can be more specific about which PGs are deep-scrubbed by altering the

grep
part of the script. For example, to only scrub active+clean PGs:
ceph pg dump | grep -i active+clean | cut -f 1 | while read i; do ceph pg deep-scrub ${i}; done

Some general caveats are in order. Repair your PGs before attempting to deep-scrub; it's safer to only scrub PGs that are active and clean. You can use

ceph pg dump_stuck
and
ceph health detail
to help find out what's going on. Here's a link to Ceph placement group statuses.

Good luck!

2015-02-02

Bringing back an LVM backed volume

What can you do when an LVM backed logical volume goes offline? This happens on my slower netbook on an LVM logical volume spanning about 20 USB flash drives. Sometimes those PVs go missing and the filesystem stops! Here's the steps I take to fix this problem without a reboot. My volume group is called "usb" and my logical volume is called "osd.2".

Since my volume is part of a ceph cluster, I should ensure that the ceph osd is stopped. service ceph stop osd.2. You probably don't need to do this since the OSD probably exited once it saw errors on the filesystem.

Next, unmount the filesystem and mark the logical volume as inactive. We use the -f -l switches to force the dismount and lazily deal with the dismount in the background. Without those switches the umount might freeze.
umount -f -l /dev/mapper/usb-osd.2
Marking the logical volume as inactive can be done in two ways. Prefer the first method since it is more specific. The second method will mark inactive all dismounted logical volumes and that might be overkill.
lvchange -a n usb/osd.2 -or- vgchange -a n

At this point I unplug all the USB drives and check the hubs. Plug in the USB keys a few at a time and use pvscan as you go to ensure that each USB key is being recognised. If you have a dead USB key then try again in another port. If that doesn't work then check the hubs have power - even replug the hub. Failing that try a reboot. Failing that... attempt to repair the LVM volume some other way. Since ceph already replicates data I don't bother running the LVM backed logical volumes on RAID - I just overwrite the LV and make a new one from the remaining USB flash drives.

Once all the PVs have come back then pvscan one last time then vgscan. Now you should see your volume groups have all their PVs in place. Now it's time to reactivate the logical volumes. Both methods will work but again I prefer the first once since it is more specific.
lvchange -a y usb/osd.2 -or- vgchange -a y

All things going well and the Logical Volume is now active. It's a good idea to do a filesystem consistency check before you remount the drive. Since I use XFS I'll carry on with the steps for that. You should use whatever tools work for your filesystem.
mount /dev/mapper/usb-osd.2 mounting the drive allows the journal to replay. That usually fixes any file inconsistency problems.
umount /dev/mapper/usb-osd.2 unmount the drive before checking.
xfs_check /dev/mapper/usb-osd.2 to check the drive and use xfs_repair /dev/mapper/usb-osd.2 if there are any errors.

Now we're ready to mount the logical volume again: mount /dev/mapper/usb-osd.2

And since I'm running ceph I want to restart the OSD process: service ceph osd restart osd.2

Done!

Read more about my ceph cluster running on USB drives.

2014-12-06

Display text at end of passage macro for Twine

I was helping Harry Giles out with some modifications to his IFCOMP2014 entry Raik. It became useful to set text early in the passage that would be shown only at the end of the passage.

An easy way to achieve this in Twine with custom macros. They are <<atend>> ... <<atendd>>.

::PassageTitle
<<atend>>

This text appears at the end of the current passage.<<atendd>>And this text appears in the usual place.

How do you use this in your own projects? Add the following line to your StoryIncludes:

http://raw.githubusercontent.com/tweecode/TwineQuest/master/macros/9999%20Macro%20atend.twee

Why might you want this macro? Here are some examples. You be inside an <<if>> macro from which you want to set a choice to appear at the end of the passage. You might be using the excellent ReplaceMacros from Glorious Tranwrecks and want to have text that always appear at the end of the passage no matter what the state of the rest of the passage.
Please share / comment / like - your actions guide me on what to write about.

2014-10-08

Ceph repair inconsistent pg placement groups

If you run Ceph for any length of time you may find some placement groups become inconsistent. The Ceph website has a handy list of placement groups statuses. The entry for "inconsistent" is what you'd expect; there's a difference between replicas of an object.

ceph pg dump | grep -i incons | cut -f 1 | while read i; do ceph pg repair ${i} ; done

(from here)

Get the cluster as healthy as you can before attempt this. Ideally the inconsistent placement groups should be at "active+clean+inconsistent". That means first resolving any missing OSDs and allowing them time to heal. If the OSDs don't seem to cooperate try restarting them and then retry the above command.

Explanation of the command:
ceph pg dump
gives a list of all pgs and their current status.
| grep -i incons
find only the lines containing "incons" - short for inconsistent
| cut -f 1
we only want the first field from the output
| while read i; do
loop through each line (one per pg), storing the pg number in a shell variable called i
ceph pg repair ${i}
instructs ceph to repair the pg
; done
signals the closing of the loop

The above command has always worked for me, but there are things you can try if this command doesn't work.

The ceph website says that inconsistent placement groups can happen as an error during scrubbing or if there are media errors. Check your other system logs to rule out media-errors because they may indicate a failing storage device.

Good luck!

2014-09-10

Ceph on Thumbdrive Update: BTRFS and one more node.

A few things have happened to my Ceph cluster. The AspireOne netbook was really not up to the job. It is fine for just a few OSD processes but anything more was caused slow-downs resulting in cluster thrash. Amalgamating thumb drives using LVM was helpful… until I wanted to run CephFS. I was time to add another node to the mix.

Read about earlier stories about the Ceph on USB thumb drive cluster here:
Adding another node to Ceph is trivial. This was an old, but much more powerful laptop in every way. I’ve moved the mon and mds functions onto this laptop so now the AspireOne only runs OSD processes. When I add another node then I’ll also run a mon process on the AspireOne so that there is an odd number for quorum building.

The new node uses faster and larger USB keys. These were 32GB – which was both the fastest and cheapest price per GB available at my local PBTech. The new node currently runs two of these sticks in an OSD process each.

I also moved the cluster away from XFS to BTRFS. This was trivial and involved zero cluster downtime. Yes: zero. First ensure the cluster is reasonably healthy, then drop the weight of the OSD using:
ceph osd reweight OSDID 0.1
. Actually I got bored waiting – the cluster was healthy and the pools all had size 3 with min_size 2… so I just stopped the OSD process and removed it from the ceph. Don’t do that on a live cluster, especially where pools have few replicas. But, this was just for testing. Then...
sudo service ceph stop osd.X
ceph osd crush rm osd.X
ceph osd rm osd.X
ceph auth del osd.X


Then umount the backing storage and format it using BTRFS. Then I followed the instruction in my previous tutorial to add the storage back into Ceph. Wait for the cluster to heal before migrating another OSD from XFS to BTRFS.

The AspireOne node has three groups of 8GB keys, federated by USB hub. BTRFS is capable of spanning physical drives without LVM so I removed LVM once all groups had been migrated. Do read about the options for BTRFS stores because the choices matter. I went with RAID10 for the metadata and RAID0 for the data. RAID0 maybe gives better parallel IO performance because the extents are scattered among the drives but it does mean that all block devices in the FS effectively operate at the size of the smallest one. I can live with that.

Three BTRFS OSDs on the AspireOne is sometimes a bit much for that machine. Though, one cool thing about BTRFS was I extended a mounted BTRFS volume with a few more thumb drives, then restarted the osd process. Use
ceph osd crush reweight osd.X Y
to tell Ceph to reallocate space. That was instant storage expansion without downtime on the OSD long enough to trigger a recovery process. I did the whole process in less than ten minutes – and most of that was googling to find the correct BTRFS commands.

The cluster happily serves files to my desktop machine over CIFS. While it’s not a setup I’d recommend for production use it is kinda fun.

2014-08-27

While Loop Macro for Twine

One of the often requested features for Twine are proper loops. It has been possible to simulate loops with a recursion hack but that was ugly. Before we continue go and see the possibilities.

Yes. Those are nested while loops. Here's the code that makes it all happen:
<<set $j = 1>><<while $j lte 10>>
<<print $j>> countdown: <<set $i = 5>><<while $i gt 0>>
<<print $i>>... <<set $i = $i - 1>>
<<endwhile>> BOOM! 
<<set $j = $j + 1>>
<<endwhile>>

You've probably worked out the basic syntax (clever bunny) which is:
<<while $condition eq true>>
Do some stuff. 
NB: update the $condition or the loop will run to infinity.
<<endwhile>>

How do you use this in your own projects? Add the following line to your StoryIncludes:

http://github.com/tweecode/TwineQuest/raw/master/macros/9999%20Macro%20while.twee

You can also download the demo project and .twee file as a .zip archive.

Enjoy!

Please share / comment / like - your actions guide me on what to write about.

2014-08-25

Twine Game Story Authoring 1: Structuring Game stories with Pain and Progress

Twine is a versatile tool from writing game stories. It is freed from the imposition of somebody else's RighWayToDoThings framework and allows the author to focus on what is important to them. The downside is that the author must program everything they want themselves. A previous article discusses how Twine can be thought of from a programmers perspective but don't worry it that article makes little sense. This article is specifically about structuring game stories.

For our purposes, a game story has a mainloop because the story is driven by data. That data might be a map, character stats, an inventory or something else you can dream up. This style of story does not suit a hypertext branching narrative.

The over all structure of the story goes:
  1. Introduction Text
  2. Initialise Variables
  3. MainLoop Passage
  4. Actions Passages
  5. Check Conditions Passages
  6. End of Game Passages

I like examples, so here is a link to the Pleasure and Pain v1 (HTML | Twee) files. Try out the playable HTML version first then take a look at the Twee code in your favourite text editor. In twee new passages start on lines beginning with double colons ::passagename. Pain And Progress is a simple demonstration with two conditional variables. Let's examine the passages and their intent.

Start, RealStart, Instructions
These passages deal with beginning the story, giving background and the option for instructions if the reader so chooses. Here you might add extended about information and links to information about the story and the author.

InitGame
The game re-runs this passage whenever the game is (re)started. Initialise all the variables that your game uses in here. Twine itself does not require variables to be declared and initialised but that can cause awkward side effects if a game story is re-run. Imagine the reader picks up an axe in one play-through, then the $hasAxe variable is not reset and they suddenly have an axe on the next play-thru. Not good - but completely avoidable if ALL variables are initialised here.
Enclose the variable initialisations in a <<silently>> ... <<endsilently>> block so that you can add free-form comments to the variables that will not be seen by the user.
Once this passage has ended then control is passed to the MainEventLoop.

MainEventLoop
This is where the major action occurs. Typically the story might perform any engine initiated actions (e.g. random weather events, monster encounters), display status and provide a menu of actions.

MainStatus
Consider separating status displays so that they can be re-used.

MainActionPain, MainActionProgress
These passages are the entry points from user actions. They start by performing any action related things and then checking the game state. A flag variable $gameendflag lets the game story know if they should print user actions or not. The reason for this is that <<display>> will always return to the passage that invoked it and nothing further from these passages should be displayed if the game should end.

CheckPain, CheckProgress
Perform constraint checks in their own passages so that they can be re-used through the game story.

GameEndLose, GameEndWin, PlayAgain, PlayAgainNo
These passages deal with the game ending conditions. Game stories could expand this list for different ending conditions. The PlayAgain passage ensures that replays will begin again from the InitGame passage.

The Pain and Progress, the passage names are typically prefixed by their function, though prefixing by variable name is also valid. The idea is to make things as obvious as possible.

Hopefully this example will serve as a guide to structuring gamestories and encourage more Twine authors to try this type of story. If you found this guide useful then Share / Comment / Like - because it encourages me to write more on this topic.

Twine Thinking for Programmers

Programmers find Twine's hypertext way of doing things a little strange at first. Twine works well for branching hypertexts but needs some thinking for stories driven by variables. I've done a few game stories in twine - mostly conversions of early BASIC programs - so this is a lessons learned type of blog. You might find this article for useful for any Game Story; usually adventure games, RPGs.

Twine's basic unit is the passage. Passages work like procedure calls. Passages are similar to GOSUB in BASIC. By default passages print their content to the screen and you use tweecode macros to execute game logic.

Passages are "called" by the Twine engine in a few ways:
  • The Start passage is called to begin the story
  • Readers activating [[link]] or <<CHOICE>>
  • <<DISPLAY>> macro within passages.
The <<DISPLAY>> macro is the programmers GOSUB / procedure call. Twine has no GOTO equivalent: <<DISPLAY>> always RETURNs to the passage that called it. Anything after the <<DISPLAY>> macro will still be processed (and output if appropriate) by the Twine Engine. Use <<DISPLAY>> generously - it is the workhouse of programmer-like Twine and the basic unit of code re-use within a story.

Twine's tweecode has only global variables. This means that variables cannot be directly passed to passages - they are <<SET>> in global variable space before calling the <<DISPLAY>> function. Twine allows long variable names so use generous prefixes to distinguish variables.

Twine has no inbuilt loop constructs (for, do, while). Loop constructs can be built using <<IF>><<ELSE>><<ENDIF>> and passages. Here's a helpful article: How to Simulate for, while or do loops in Twine.
UPDATE: I've just released <<while>> macros.

Once your passage count gets higher Twine's diagram based UI can get unwieldy. Programmers used to text-based programming will find it more natural to write in TWEECODE and use the StoryIncludes feature to import files into a Twine Story. There is only a global namespace for passages so ensure that passage names are unique. Passages can be moved between the main Twine story file and included files as needed. Use StoryIncludes and tweecode files as the basic unit of code reuse between different stories.

Tweecode files are text files (UTF-8) that add a few extra things to the Twine macros already used. It's easiest to think of Tweecode files as a bucket of passages - for that most part Twine does not care about the order of passages. New passages begin with a single line passage header and end either when a new passage header begins or the file ends. A passage header begins a line with double colon (::) followed by the passage name. Optionally tags can be added space delimited with square brackets. Here's a brief example:
::Passage Title 1 [tag1 anothertag yet_another_tag]
This is part of passage one.

::Passage Title 2
more content

Twine allows complete access to JavaScript though consider keeping Javascript use to a minimum so that your story has fewer dependancies. If you do use javascript then consider placing custom scripts into their own tweecode files; both for your own reuse and to provide an easy way to find code if it must be later rewritten by future generations. Here are some useful articles: Once you get used to tracking state variables globally and how the <<DISPLAY>> macro always returns then it is only small extension to create event loop based games. I find it easier to work with example code so here's some classic game conversions with full source available: Please share / like comment: your actions influence what I decide to write about.

2014-08-15

How I added my LVM volumes as OSDs in Ceph

This article expands on how I added an LVM logical volume based OSD to my ceph cluster. It might be useful to somebody else who is having trouble getting
ceph-deploy osd create ... 
or
ceph-deploy osd prepare ...
to work nicely.

Here's how Ceph likes to have its OSDs setup. Ceph OSDs are mounted by OSD.id in
/var/lib/ceph/ceph-*
. Within that folder should be a file called
journal
. The journal file can either live on that drive or be a symlink. That symlink should be to another raw partition (e.g. partition one on an SSD) though it does work with a symlink to a regular file too.

Here's a run-down of the steps that worked for me:

First
mkfs
the file system on the intended OSD data volume. I use XFS because BTRFS would add to the strain on my netbook but YMMV. After the mkfs is complete you'll have a drive with an empty filesystem.

Then issue
ceph osd create
which will return a single number: this is your OSDNUM. Mount your OSD data drive to
/var/lib/ceph/osd/ceph-{OSDNUM}
remembering to substitute in your actual OSDNUM. Update your
/etc/fstab
to automount the drive to the same folder on reboot. (Not quite true for LVM on USB keys: I have noauto in fstab and a script that mounts the LVM logical volumes later in the boot sequence).

Now prepare the drive for Ceph with
ceph-osd -i {OSDNUM} --mkfs --mkkey
. Once this is done you'll have a newly minted but inactive OSD complete with a shiny new authenication key. There will be a bunch of files in the filesystem. You can now go ahead and symlink the journal if you want. Everything up to this point is somewhat similar to what
ceph-deploy osd prepare ..
does.

Doing the next steps manually can be a bit tedious so I use ceph-deploy.
ceph-deploy osd activate hostname:/var/lib/ceph/osd/ceph-{OSDNUM}



There's a few things that might go wrong.

If you've removed OSDs from your cluster then
ceph osd create
might give you a OSDNUM that is free in the CRUSH map but still has an old
ceph auth
entry. That's why you should
ceph auth del osd.{OSDNUM}
when you delete an OSD. Another useful command is
ceph auth list
so you can see if there's any entries that need cleaning up. The key in the
ceph auth list
should match the key in
/var/lib/ceph/osd/ceph-{OSDNUM}
. If it doesn't then delete the auth entry with
ceph auth del osd.{OSDNUM}
. The
ceph-deploy osd activate ... 
command will take care of adding correct keys for you but will not overwrite an existing [old] key.

Check that the new OSD is up and in the CRUSH map using
ceph osd tree
. If the OSD is down then try restarting it with
/etc/init.d/ceph restart osd.{OSDNUM}
. Also check that the weight and reweight columns are not zero. If they are then get the CRUSHID from
ceph osd tree
. Change the weight with
ceph osd crush reweight {CRUSHID} 
. If the reweight column is not 1 then set it using
ceph osd reweight {CRUSHID} 1.0


(Here is more general information about how OSDs can be removed from a cluster, the drives joined using LVM and then added back to the cluster).

2014-08-14

Going to LVM for performance and graceful failures.

A few things happened in the world of the 12 USB drive netbook ceph node. Basically the netbook wasn't up to the job. Under any kind of reasonable stress (such fifteen parallel untars of the kernel sources to and from ceph-filesystem) the node would spiral into cluster thrush. The major problem appeared to be OSDs being swapped out to virtual memory and timing out.

Aside: my install of debian (Wheezy) came with a 3.2 kernel. Ceph likes a kernel version 3.16 or greater. I complied a kernel that was 3.14 since it was marked as longterm supported. My tip is to do this before you install ceph. Doing it afterwards resulted in kernel feature mismatches with some of my OSDs.

Back to the main problem. My USB configuration had introduced a new failure and performance domain. The AspireOne netbook has three USB ports - each of which I attached a hub and each hub has four usb keys: three hubs times four USB drives is 12 drives total. Ideally I'd like to alter the crush map so that PGs don't replicate on the same USB hub. This looked easy enough in ceph ... edit the crushmap and introduce a bucket type called "bus" that sat between "osd" and "host" then change the default chooseleaf type to bus.

It turns out there was an easier way to solve both my problems: LVM. The Linux Volume Manager joins block devices together into a single logical volume. LVM can also stripe data from logical volumes across multiple devices. However, it does mean that if a single USB key fails then the whole logical volume fails too ... and that OSD goes down. I can live with that.

Identical looking flash drives are impossible to match with linux block devices in the /dev folder. I am running ceph so it was just easier to pull a USB key, see what OSD died and find what device was associated with it. I let the cluster heal in between each pull of a USB drive until I have a hub's worth of flash-keys pulled. I then worked a USB-hub at a time: bringing the new LVM-backed OSD into ceph before working on the next hub. Details follow.

Bring down the devices and remove them from ceph. Use
ceph osd crush remove id
then
ceph osd down id
,
ceph osd rm id
. Then stop the OSD process with
/etc/init.d/ceph stop osd.id
. It pays also to tidy up the authentication keys with
ceph auth del osd.id
or you'll have problems later. You can then safely unmount the device and then get hacking with your favourite partition editor.

There are good resources for LVM online. The basics are: setup an LVM partition on your devices. Use
pvcreate
on each LVM partition to let LVM know this is a physical volume. Then create volume groups using
vgcreate
- I made a different volume group per USB hub. Then you can make the logical volume (i.e. the thing used by the OSD) from space on a volume group
lvcreate
. The hierachy is: pv-physical volumes, vg-volume groups, lv-logical volumes. I used the
-i
option on
lvcreate
to have LVM stripe data across the USB keys because parallelism. If you've noticed a pattern in the create commands then bonus: the list commands follow the same pattern
pvs
,
vgs
,
lvs
. Format the logical volume using your favourite filesystem, though ceph prefers XFS (maybe BTRFS).

Once the logical volume is formatted then it's time to bring it back into ceph. I tried to do things the hard way and then gave up and used ceph-deploy instead. The commands used are described here.

A disadvantage with this setup is that LVM tends to scan for volumes before the USB drives are visible so the drives would not automount. I solved this with a custom init.d script. While in /etc I also changed inittab to load
ceph -w
onto tty1 so that the machine boots directly into a status console.

The performance is much faster with the new 4_LVMx3_OSD configuration compared with the 12_OSD cluster. Write speeds are almost double with RADOS puts of multi-megabyte objects. There is almost zero swapfile activity too.

I hope to soon test ceph filesystem performance on this setup before adding another node or two. I've glossed over many steps so let me know in the comments if you'd like details on any part of the process.

(I also wrote about the 12 USB drive OSD cluster with a particular focus on the ceph.conf settings)

2014-08-11

Ceph on USB thumb drives

Ceph is an open source distributed object store mean to work at huge scales on COTS (common off the shelf) hardware. It works in huge datacenters, so why not dust off an old netbook, plug in 12 USB flash drives and have at it.
The netbook is an Acer AspireOne (Intel Atom N270 1.6 GHz, 1 Gig RAM, 160Gig HD). What follows are the config changes made in ceph.conf before running the ceph-deploy command. The ceph version is Firefly 0.81. Since this is a one machine cluster I needed to tell ceph to replicate across OSDs and not across hosts.
[default]
osd crush chooseleaf type = 0 

I messed up setting the default journal size. At first I thought: Pfft. Journal, make it tiny – it just robs space. And my 4MB (yes four megabytes) journal made the cluster unworkable. With the tiny journals and default settings I could never reliably keep more than two OSDs up and data throughput was terrible. I rebuilt with 512MB journals instead.
[osd]
osd journal size = 512 

The machine was way underpowered. So I tuned a few other things. The authentication cephx was turned off. There are risks to this but this is a hobbyist project on a secured subnet.
[default]
auth cluster required = none
auth service required = none
auth client required = none

The cluster uses a ton of memory and CPU when recovering objects. It helps to limit this activity somewhat.
[osd]
osd max backfills = 1
osd recovery max active = 2 

And since things could get a bit slow I increased a few timeouts:
[osd]
osd op thread timeout = 180
osd op complaint time = 300
osd default notify timeout = 240
osd command thread timeout = 180

I was not able to get 2G flash keys to come up. Given the price of 8G sticks is only five bucks this isn’t much of a limitation. I suppose I could use LVM striping to join a bunch of 2G sticks together into a larger unit.

The speed is not all that quick. Ceph –w reports the write speed about 3 megabytes per second. That doesn’t sound like much except the data pool I was testing on writes three copies of the data – six if you count journaling. A lot of things could affect speed: tiny memory, slow CPU, slow USB sticks and/or the USB bus being saturated.

This config uses XFS on the USB sticks where BTRFS might perform better. While the speeds look poor, remember that ceph OSDs don’t report that a write is successful until the object is written to both the media and the journal. I could probably mitigate this double write by: having fewer OSDs by joining up groups of USB sticks with LVM stripes and/or moving the journal to a different device – right now ceph is using the USB sticks both for data and journal.

I stress tested RADOS by adding objects until the store filled up. It’s robust and not a single OSD timed out of the pool. As of writing this blog I am currently testing untarring linux kernel sources to the ceph filesystem – I’ll keep you posted.

My future plans are to expand the cluster utilising old hardware I have lying about. I’d like to add at least two more nodes – but they won’t necessarily be USB thumb drive backed.

2013-08-19

Twine: Multiple files in one story using StoryIncludes

List your .tws or .twee files in a special passage titled "StoryIncludes" and recent v1.3.5 betas will import the passages from those files when your story's .html file is built.

Newest Twine alphas/betas are here.

2013-01-27

How to Insert Images into Twine Without Linking


UPDATE: Twine 1.4 makes this process much easier. Go to the "Story" menu and select "Import Image". The techniques in this article are still valid too.


Twine has enjoyed some recent popularity and it is great to see the diverse and wonderful things people do. Inserting images into Twine is something that has become popular but having to transport the image as a file separate to the Twine output .html introduces some additional complications. I like the single output file nature of Twine.

Look at this Twine demo:
http://eturnerx.com/files/twinehowto/ImageDataURI.html (source .tws)
There is an image stored in the .html file itself using a Data URI (RFC 2397). That image is not a separate linked file. You can check it yourself, I'll wait.

Data URIs contains the data for the image in an encoded string that is embedded into the html <img> element. This works fine in most recent browsers (IE 8+) but is only suitable for smallish files. For IE8 the limit is not much bigger than 20 kilobytes but other browsers (including IE9) work just fine with much larger file sizes.

It is surprising how much image can fit within 20 kilobytes just by playing with filetype, compression settings and reducing pixel dimensions. Most flat line drawings in a few colours will generally compress well as PNG or GIF. Scanned line drawings should have the contrast boosted and/or posterized to flatten the colours in the image. Photographs or scanned images generally work better as JPEGs.

Once you have a small image file then it will need converting to a Data URL. I like to use:
http://websemantics.co.uk/online_tools/image_to_data_uri_convertor/
They also have links to online tools that can further compress your image.

Here is the code that goes into your Twine passage:

<html><img width="11" height="14" src="data:image/gif;base64,R0lGOD
lhCwAOAMQfAP////7+/vj4+Hh4eHd3d/v7+/Dw8HV1dfLy8ubm5vX19e3t7fr
6+nl5edra2nZ2dnx8fMHBwYODg/b29np6eujo6JGRkeHh4eTk5LCwsN3d3dfX
13Jycp2dnevr6////yH5BAEAAB8ALAAAAAALAA4AAAVq4NFw1DNAX/o9imAsB
tKpxKRd1+YEWUoIiUoiEWEAApIDMLGoRCyWiKThenkwDgeGMiggDLEXQkDoTh
CKNLpQDgjeAsY7MHgECgx8YR8oHwNHfwADBACGh4EDA4iGAYAEBAcQIg0Dk
gcEIQA7" alt="File Icon"></html>

(data URI example from: http://dataurl.net/#about)

The <html>...</html> tells Twine to treat the code inside as raw HTML. The <img ...> is the HTML code for placing an image. The width="11" and height="14" code give the width and height of the image - you should change these to suit your image. The alt="..." code improves the accessibility of your file by giving a text alternative to your image. Most likely the code will work without a width, height or alt specified but it is considered good practise to include them if you can. The real magic happens inside src="...". That src is an abbreviation for "source" and normally that would link to an external file. Here we give it the data URI instead.

TIP: These data URIs can produce large walls of text that are annoying for you as an author to scroll past. What I suggest is to put all the image code into a different passage then use the <<display>> macro to.... ummm... yeah... display the image. This makes the image reusable anywhere in your story and keeps your text passages nice and tidy. Consider this abbreviated example... (:: Double colon is the passage title)

::Start
<<display "Image:Document Icon">>
This is my text and there is an image above.

::Image:Document Icon
<html><img ...></html>

Your Twine story can use both data URI images and normal "linked" images. And data URIs work just fine in CSS too; great for setting background patterns. (The thing in square brackets [] is a tag applied underneath the passage title).

::Stylesheet1 [stylesheet]
body {
  background-image:url(data:image/gif;base64,R0lGOD... etc );
}

You have the tool, go make something cool.

2012-12-19

How to use Twine/Tweecode Variables and Macros from Custom Macro

This tutorial is aimed at people wanting to use JavaScript in their Twine stories. This is advanced topic and is not necessary for writing most stories. Twine's ease in integrating JavaScript was a key factor when I was evaluating interaction fiction systems. Please read the tutorial on creating a custom macro in Twine first.

Twine/Tweecode has the ability to use macros and variables. In the Responsive, Sugarcane and Jonah StoryFormats macros and variables are stored in a place where they are accessible by custom macros for both reading and writing.

All custom macros are stored in a global object called "macros". They can be referred to by name. The actual macro code is in the "handler" property.

macros["custommacro"].handler(place, macroName, params, parser);
macros["display"].handler(place, "display", [ "SomePassage" ], parser);

In most cases it is sufficient to pass the place and parser arguments given to your custom macro into the macro you are calling.

Now, onto Tweecode variables. In tweecode variables are prefixed with a dollar sign so that the parser knows to lookup the variablename.

<<set $variablename = 1>><<print $variablename>>
<<if $variablename eq 1>>Success<<endif>>

Tweecode stores variables in the JavaScript object: state.history[0].variables. They are stored without the $ dollar sign "sigil".

state.history[0].variables["customvariables"] = 1;
if(state.history[0].variables["customvariables"] == 1) { 
  //... 
}

Knowing how to reuse Twine's macros and working with variables set in Tweecode from within JavaScript custom macros (and vice versa) opens many possibilities.

2012-12-14

How to use Parameters and Variables in Custom Twine Macros

The last article looked at basic macros in Twine. We'll expand on that article by looking at parameters and using an internal variable. First the code example:

try {
  version.extensions['recallMacro'] = { major:1, minor:0, revision:0 };
  macros['recall'] = {
    handler: function(place, macroName, params, parser) {
      if(params.length > 0) {
        this.m = params[0];
      }
      new Wikifier(place, this.m);
    },
    init: function() {
      this.m = "A blank slate";
    },
    m: null,
  };
} catch(e) {
  throwError(place,"recall Setup Error: "+e.message); 
}

Let's go through the code a piece at a time (not necessarily in written order).

try {
  version.extensions['recallMacro'] = { major:1, minor:0, revision:0 };
  macros['recall'] = {
    ...
  };
} catch(e) {
  throwError(place,"recall Setup Error: "+e.message); 
}

This is boilerplate code that is explained in the last article.

    init: function() {
      this.m = "A blank slate";
    },
    m: null,

This code declares a variable as part of the macro called "m". Throughout the macro we refer to this variable using this.m. The variable is declared within the macro to avoid polluting the global namespace and to avoid variable name clashes there. Note that m is declared with an initial value of null and the value of m is set within the init() method. This practice makes the macro work better if the story is restarted without refreshing the page.

    handler: function(place, macroName, params, parser) {
      if(params.length > 0) {
        this.m = params[0];
      }
      new Wikifier(place, this.m);
    },

This code checks the length of the params array. If there is something in the params array then the first parameter is saved into this.m. In all cases the contents of this.m are printed.

Once you have added this macro to your story then try adding the following tweecode to a passage in your story:

Initial state: <<recall>>
Store John: <<recall John>>
Store Ben: <<recall Ben>>
What is stored?: <<recall>>

Using parameters and variables in Twine macros is a great way to increase the power of the macro. Storing variables as properties inside the macro object avoids any name clashes with other macros.