In a high-speed rotor system, vibration control is necessary to ensure steady operation of the system. Magnetic damper is one of the effective means to control the vibration. This paper is concerned with the optimal design of a passive magnetic damper’s system parameter, including its geometric and electrical parameters. The relation between the damper’s system parameter and its damping coefficient is derived. A mathematical model that maximizes the damping coefficient is built, with the constraints of the geometric and electrical parameters. The optimization program is based on the algorithm of sequential quadratic programming method. The program yields the maximum damping coefficient under the given condition of the rotor’s diameter.