The optical properties of composites, formed by filling the high density polyethylene (HDPE) with carbon black (CB), multi-walled carbon nanotubes (MWNTs) and fullerene (C60), respectively, in the frequence region from 0.3 to 2.0 terahertz (THz) were characterized with THz time-domain spectroscopy (THz-TDS). It is found that the optical parameters and the details of their variation with frequence and filler concentration are significantly different for different kinds of carbon materials. The MWNTs filled composites have the biggest absorption coefficient value, refractive index and real conductivity. The C60/HDPE composite shows very little changes in absorption and refractive index compared to that of the HDPE even at a filler-content of 12.3% in volume fraction. These phenomena are related to the special properties of the fillers as well as their particulate structures, such as aspect ratio, particle size, and aggregate structure, etc. The results are analyzed by using Cole-Cole theory of dipole relaxation under the assumption that carbon particles dispersed in the matrix behave like dipoles.