Although fisheye cameras are in high demand in many application areas due to their large field of view, many image and video sig- nal processing tasks such as motion compensation suffer from the introduced strong radial distortions. A recently proposed projection- based approach takes the fisheye projection into account to improve fisheye motion compensation. However, the approach does not con- sider the large field of view of fisheye lenses that requires the consid- eration of different motion planes in 3D space. We propose a novel viewport-adaptive motion compensation technique that applies the motion vectors in different perspective viewports in order to realize these motion planes. Thereby, some pixels are mapped to so-called virtual image planes and require special treatment to obtain reliable mappings between the perspective viewports and the original fisheye image. While the state-of-the-art ultra wide-angle compensation is sufficiently accurate, we propose a virtual image plane compensa- tion that leads to perfect mappings. All in all, we achieve average gains of +2.40 dB in terms of PSNR compared to the state of the art in fisheye motion compensation.