Age estimation from facial images is increasingly receiving attention to solve age-based access control, age-adaptive targeted marketing, amongst other applications. Since even humans can be induced in error due to the complex biological processes involved, finding a robust method remains a research challenge today. In this work, we propose a new framework for the integration of Active Appearance Models (AAM), Local Binary Patterns (LBP), Gabor wavelets (GW) and Local Phase Quantization (LPQ) in order to obtain a highly discriminative feature representation which is able to model shape and appearance as well as wrinkles and skin spots. In addition, this work proposes a novel hierarchical age estimation approach consisting of a multi-class Support Vector Machine (SVM) to obtain the age group while the final age estimation is made using Support Vector Regression (SVR). The errors in the classification step, caused by the hard boundaries between age classes, are compensated in the detailed age estimation by overlapping flexibly the age ranges of each age function. The performance of the proposed approach was evaluated on FG-NET Aging and MORPH Album 2 datasets and a mean absolute error ($MAE$) of 4.50 and 5.86 years was achieved respectively. Furthermore, a combination of both datasets was evaluated to verify the robustness of the proposed framework and a $MAE$ of 5.20 years was achieved. The proposed approach is competitive with current state-of-the-art and these results are achieved with an increased robustness to blur, lighting and expression variance through local phase features. We also compared the human age estimation to the proposed method and it has shown that the machine outperforms humans.