GMM with modified weight is proposed, and it is applied into the audio segment classification. Since GMM is training only by intra-class data, during classification, there may be some components in different GMM overlapping each other. That means the likelihoods of the overlap components are similar, which will induce the confusion during classification. To avoid this, the weight in GMM is modified according to the distance of components. For that component easily to confuse, the weight is deduced to decrease the effect of the component in whole result. Instead, for that component with well separability, the weight is enhanced. Based on the ratio of sub-band energy and the proposed model, an optimized likelihood is further put forward, in which the length of clip can be adjusted. From experiments, it can be drawn that GMM with modified weight has better performance than traditional GMM, and combined with optimized likelihood, the performance can be further enhanced.