CRS-based Video Compression Enhancement (VCE). ↓ (in red) is for bicubic down-sampling; E, D, and B represent video encoder, decoder, and decoded picture buffer (DPB) respectively. The CRS Model is comprised of three modular components: a motion alignment and aggregation network (MAN), a texture compensation network (TCN), and a similarity-driven fusion.

Abstract

This paper proposes a decoder-side Cross Resolution Synthesis (CRS) module to pursue better compression efficiency beyond the latest Versatile Video Coding (VVC), where we encode intra frames at original high resolution (HR), compress inter frames at a lower resolution (LR), and then super-resolve decoded LR inter frames with the help from preceding HR intra and neighboring LR inter frames. For a LR inter frame, a motion alignment and aggregation network (MAN) is devised to produce temporally aggregated motion representation to best guarantee the temporal smoothness; Another texture compensation network (TCN) is utilized to generate texture representation from decoded HR intra frame for better augmenting spatial details; Finally, a similarity-driven fusion engine synthesizes motion and texture representations to upscale LR inter frames for the removal of compression and resolution re-sampling noises. We enhance the VVC using proposed CRS, showing averaged 8.76% and 11.93% Bjøntegaard Delta Rate (BD-Rate) gains against the latest VVC anchor in Random Access (RA) and Low-delay P (LDP) settings respectively. In addition, experimental comparisons to the state-of-the-art super-resolution (SR) based VVC enhancement methods, and ablation studies are conducted to further report superior efficiency and generalization of the proposed algorithm. All materials will be made to public at \url{https://njuvision.github.io/CRS} for reproducible research.

Additional Experimental Results

BD-rate and BD-PSNR of Proposed CRS against HEVC anchor (1-second GOP used in LDP common test).

Averaged BD-rate and BD-PSNR of Proposed CRS against HEVC anchor at various GOP size.

BD-rate of Proposed CRS against AV1 anchor using LDP profile.

Averaged BD-rate of Proposed CRS against VVC anchor with VMAF as the distortion metric.

Preprint Paper


Code (Github)


Paper 

            @article{lu2022decoder,
                      title={Decoder-side Cross Resolution Synthesis for Video Compression Enhancement},
                      author={Lu, Ming and Chen, Tong and Dai, Zhenyu and Wang, Dong and Ding, Dandan and Ma, Zhan},
                      journal={IEEE Transactions on Multimedia},
                      year={2022},
                      publisher={IEEE}
                    }