The 2011 CiSRA Puzzle Competition is in its last few days. It’s hard making new and interesting puzzles, but what’s even harder is figuring out how difficult they are for other people to solve.

We test all of our new puzzles on each other, to make sure that it’s possible for someone to actually solve them. Then we provide feedback and suggestions to tweak the puzzle, make it more elegant, make difficult steps more compelling, provide extra hints within the structure of the puzzle where we think they’re needed. And once we’re done, before we publish the puzzles, we assign a difficulty rating in three relative levels: Easy, Medium, or Hard.

This is a very subjective thing. It’s really, really tough. Of course some people will solve some puzzles faster than others, while other people will find them to be of opposite difficulty. We try to factor in the amount of work, the amount of knowledge, the level of logic needed to progress, and – hardest of all – the difficulty and non-obviousness of any intuitive leaps that you need to make. We come up with a rating, stick it on the puzzle, and then it goes out in the competition.

In Group 4, we published 4 puzzles, three of which we rated Medium, and one Hard. After 24 hours, the Hard puzzle had more teams solve it than all the others put together, and one of the Mediums – in fact the one that several of us puzzle creators thought was the easiest of the group – had no teams manage to solve it. In five years of running this competition (that’s 100 puzzles), we’ve previously only ever had one puzzle go unsolved after 24 hours before, and that was a puzzle we all knew was the hardest thing we’d ever published until then.

But this one, this year… we’re flabbergasted that it proved so hard for the competition teams. Assessing puzzle difficulty… is Very Hard.