It is very difficult to benchmark recommender systems, not only because getting good datasets is hard, but different methods and algorithms have different advantages and disadvantages that are difficult to expose.
Here is a list of some benchmarking tools:
- TagRec Tag Recommender Benchmarking Framework
- RiVaL an open source toolkit for recommender system evaluation. Some results are posted here.
- Idomaar is a reference framework for recommender algorithm testing. It is developed in the framework of the CrowdRec project.