The system me and notveryclever used for judging the item contest was extremely effective and sped up the process by a huge amount.
If their had been more then two judges it would have gone even faster.
And we didn't lose any quality in our judging because we were both able to express our differing opinions.
Basically, we each judged 6/12 items(because their was only two judges), then we sent out reviews to each other so we could review our reviews. Adding in comments where we disagreed.
For example, in his review, Notveryclever wrote
Originally Posted by notveryclever
Is the item easy to learn how to use/understand? As hard to use as Ghost Scepter. But then, not overly difficult.
And I had a differing opinion, so I wrote
The item is extremely easy to learn. Banish is a ladder skill, and the passive has such a simple concept.
Averaged the scores came out to be 8.
We each went through each others reviews, and added in our comments were we did not agree with each other. Instead of having us both write 12 different but very similar reviews, we simply added in our opinions into each others reviews, and average the score.
I think that this method, especially with heroes, would speed the process up enough to where it could get done in 2-3 days. I believe it to be equally as effective in judging, but I know it to be much faster, and MUCH more practical, especially when dealing with heroes.