This paper presents a systematic procedure to fairly compare experimental performance scores for machine learning methods for fault diagnosis based on vibration signals. In the vast majority of related scientific publications, the estimated accuracy and similar performance criteria are the sole quality parameter presented. However, the experimental design giving rise to these results is mostly biased, based on unacceptably simple validation methods and on recycling identical patterns in test data sets, previously used for training. Moreover, the methods in general overfit their hyperparameters, introducing additional overoptimistic results. In order to remedy this defect, we critically analyse the usual training-validation-test division and propose an algorithmic guideline in the form of a validation framework. This allows a well defined comparison of experimental results. In order to illustrate the ideas of the paper, the Case Western Reserve University Bearing Data benchmark is used as a case study. Four distinct classifiers are experimentally compared, under gradually more difficult generalization tasks using the proposed evaluation framework: K-Nearest-Neighbor, Support Vector Machine, Random Forest and One-Dimensional Convolutional Neural Network. An extensive literature review suggests that most vibration based research papers, particularly for the Case Western Reserve University Bearing Data, use similar patterns for training and testing, making their classification an easy task.