The presence of Antinuclear Autoantibodies (ANA) in human serum is connected with several autoimmune diseases. Indirect Immunofluorescence (IIF) imaging of human epithelial type-2 cells (HEp-2) is the dominant protocol used for the identification of ANA. However, due to limitations in the processes, several attempts have been made to automate the procedure of HEp-2 cell classification. In this work, we focus on the task of HEp-2 cell classification and we propose a novel method for local feature encoding that allows us to generalize the concept of residual encoding in sparse vectors. More specifically, our method hierarchically aggregates the residual of the feature vectors’ sparse representation leading to a Vector of Hierarchically Aggregated Residuals (VHAR). Using SIFT descriptors computed on a dense grid and multiple scales, as well as considering spatial information our method achieves 78.0% classification and 82.3% mean class rate (MCR) on ICPR2012 and ICPR2014 HEp-2 cell contest datasets respectively.