In this paper, a calibration method is proposed for fish-eye panoramic camera. We adopt the unifying spherical projection model to represent the optical nature of fisheye camera. Traditional corner detection works unreliably in highly distorted image. Therefore, edge-based corner detection is proposed based on Hough transform. The line segments within corner detection window and their intersection corner points are extracted accurately and reliably due to the robustness of Hough transform. We use the nonlinear optimization technique to estimate the camera parameters. Experimental results demonstrate the performance of our method.