A general-purpose useful parameter in data analysis is the intrinsic dimension of a data set, corresponding to the minimum number of variables necessary to describe the data without significant loss of information. Feature extraction, including linear or nonlinear mapping technique, is efficient to estimate the intrinsic dimension of the data set, which is a key issue to machine fault diagnosis. This paper presents a novel application of feature extraction using the nonlinear mapping technique called curvilinear component analysis (CCA) for gear failure detection. In the approach high-dimensional data are nonlinearly projected toward an output space with dimension equal to the intrinsic dimension. Hence, enough information is remained to describe correctly the original data structure, and feature extraction based on CCA reduces dimensionality of the raw feature space for machine failure detection. Gearbox vibration signals measured under different operating conditions are analyzed using the technique. The results indicate that the intrinsic dimension of the data set is estimated and a 2-D subspace is extracted by the CCA technique, then the high-dimensional original feature data are projected into the 2-D space and form several clustering regions, each indicative of a specific gear condition, respectively. Thus, the gear operating conditions including normal, one cracked tooth, and one broken tooth are classified and detected clearly. It confirms that feature extraction based on the nonlinear mapping is very useful and effective for pattern recognition in mechanical fault diagnosis, and provides a good potential for applications in practice.