In this paper, the new detection algorithm of scene detection based on mutual information has been presented. Our approach first detect shot boundary using mutual information. Then key frames are selected from different shots. To cluster similar shots into groups, we present a novel clustering method based on mutual information. Experiments on test video indicate that the proposed scene detection method accurately detects most of the scene boundaries while preserving a good tradeoff between recall and precision.