One of the most challenging problems in real-time operation of power system is the prediction of transient stability. Fast and accurate techniques are imperative to achieve on-line transient stability assessment (TSA). This problem has been approached by various machine learning algorithms, however they find a class decision estimate rather than a probabilistic confidence of the class distribution. To counter the shortcoming of common machine learning methods, a novel machine learning technique, i.e. ‘relevance vector machine’ (RVM), for TSA is presented in this paper. RVM is based on a probabilistic Bayesian learning framework, and as a feature it can yield a decision function that depends on only a very fewer number of so-called relevance vectors. The proposed method is tested on New England power system, and compared with a state-of-the-art ‘support vector machine’ (SVM) classifier. The classification performance is evaluated using false discriminate rate (FDR). It is demonstrated that the RVM classifier can yield a decision function that is much sparser than the SVM classifier while providing higher classification accuracy. Consequently, the RVM classifier greatly reduces the computational complexity, making it more suitable for real-time implementation.