In this study, we combine the expansion and the growth data in order to investigate the ability of the three most popular holographic dark energy models, namely event future horizon, Ricci scale, and Granda–Oliveros IR cutoffs, to fit the data. Using a standard χ2 minimization method, we place tight constraints on the free parameters of the models. Based on the values of the Akaike and Bayesian information criteria, we find that two out of three holographic dark energy models are disfavoured by the data, because they predict a non-negligible amount of fractional dark energy density at early enough times. Although the growth rate data are relatively consistent with the holographic dark energy models which are based on Ricci scale and Granda–Oliveros IR cutoffs, the combined analysis provides strong indications against these models. Finally, we find that the model for which the holographic dark energy is related with the future horizon is consistent with the combined observational data.