The paper proposes a new variable friction system, of which the friction force can increase linearly with the displacement of system. This new system can be used in TMD to avoid the disadvantage of Coulomb friction TMD. Using first order harmonic balance method, the equivalent damping ratio and frequency of SDOF variable friction system is deduced and analyzed. The frequency response characteristics of SDOF variable friction system is discussed. The control effectiveness of variable friction TMD under harmonic excitation is analyzed theoretically. The results demonstrate that the frequency response curves of variable friction TMD and classically damped TMD are similar and both can effectively reduce structural response under harmonic excitation.