Machine fault diagnosis is essentially an issue of pattern recognition, which heavily depends on suitable unsupervised learning method. The Self-Organizing Map (SOM), a popular unsupervised neural network, has been used for failure detection but with two limitations: needing predefined static architecture and lacking ability for the representation of hierarchical relations in the data. This paper presents a novel study on failure detection of gearbox using the Growing Hierarchical Self-Organizing Map (GHSOM), an artificial neural network model with hierarchical architecture composed of independent growing SOMs. The GHSOM can adapt its architecture during unsupervised training process and provide a global orientation in the individual layers of the hierarchy; hence the original data structure can be described correctly for machine faults diagnosis. Gearbox vibration signals measured under different operating conditions are analyzed using the proposed technique. The results prove that the hierarchical relations in the gearbox failure data can be intuitively represented, and inherent structure can be unfolded. Then gearbox operating conditions including normal, tooth cracked and tooth broken are classified and recognized clearly. The study confirms that GHSOM is very useful and effective for pattern recognition in mechanical fault diagnosis, and provides a good potential for application in practice.