It is important to accurately inspect road pavement conditions for maintaining the safety and comfort of drivers. In recent years, Mobile Mapping System (MMS) mounted with a laser scanner has been widely used and pavement defect extraction using MMS point clouds is expected. This paper presents an extraction method of region-type defects such as rutting and potholes and line-type defects such as cracks of pavement surfaces using point clouds acquired by a Mobile Mapping System. The features of the proposed method compared with related works are the ability to extract displacements of the region and line-type defects using a single framework and the ability to extract defects with different scales and complex geometries. The proposed method first estimates an ideal pavement surface based on smooth curve fitting to scanlines using an index that represents the impact level of defects at each point. Then, vector data of region and line-type defects are extracted using displacements from the ideal pavement surface at each point. In the experiments, the proposed method achieved an IoU (Intersection over Union) of 80.2% for extracted regions and ground truth of region-type defects, and a recall of 90.7%, a precision of 97.0%, and an RMS of 10.0mm by Heipke's proposed method in the extraction of line-type defects.