For the purpose of reduce the complicated process of H.264/AVC standard, the overall coding flow is studied, and some of the key technolies are discussed in the front part of this thesis. An algorithm for determining the distortion for different data partitions in H.264/AVC is proposed. A low-pass filter is implemented for each macroblock; after the macroblock is smoothed by this filter we compare the result of the filtered macroblock with the original data and generate the difference between them. Then the basic candidate-mode class is determined based on the difference, and the further mode decision can be made within this sub-class. So that the number of candidate modes is reduced, and the complexity of I-frame mode decision process is reduced.