Evolutionary Algorithms have been applied successfully for solving real-world multi-objective problems which explains the influx of newly proposed Multi-Objective Evolutionary Algorithms (MOEAs). In order to determine their performance, comparison with existing algorithms must be conducted. However, conducting a comparison is not a trivial task. Benchmark functions must be selected and the results have to be analyzed using a statistical method. In addition, the results of MOEAs can be evaluated with different Quality Indicators (QIs), which aggravates the comparison additionally. In this paper, we present a chess rating system which was adapted for ranking MOEAs with a Quality Indicator ensemble. The ensemble ensures that different aspects of quality are evaluated of the resulting approximation sets. The chess rating system is compared with an existing method which uses a double-elimination tournament and a quality indicator ensemble. Experimental results show that the chess rating system achieved similar rankings with fewer runs of MOEAs.
