PatchFlow: A Two-Stage Patch-Based Approach for Lightweight Optical Flow Estimation
Alhawwary, Ahmed; Mustaniemi, Janne; Heikkilä, Janne (2023-03-11)
Springer
11.03.2023
Alhawwary, A., Mustaniemi, J., Heikkilä, J. (2023). PatchFlow: A Two-Stage Patch-Based Approach for Lightweight Optical Flow Estimation. In: Wang, L., Gall, J., Chin, TJ., Sato, I., Chellappa, R. (eds) Computer Vision – ACCV 2022. ACCV 2022. Lecture Notes in Computer Science, vol 13847. Springer, Cham. https://doi.org/10.1007/978-3-031-26293-7_32
https://rightsstatements.org/vocab/InC/1.0/
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
https://rightsstatements.org/vocab/InC/1.0/
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
https://rightsstatements.org/vocab/InC/1.0/
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:oulu-202404192850
https://urn.fi/URN:NBN:fi:oulu-202404192850
Tiivistelmä
Abstract
The deep learning-based optical flow methods have shown noticeable advancements in flow estimation. The dense optical flow map offers high flexibility and quality for aligning neighbouring video frames. However, they are computationally expensive, and the memory requirements for processing high-resolution images such as 2K, 4K and 8K on resources-limited devices such as mobile phones can be prohibitive.
We propose a patch-based approach for optical flow estimation. We redistribute the regular CNN-based optical flow regression into a two-stage pipeline, where the first stage estimates an optical flow for a low-resolution image version. The pre-flow is input to the second stage, where the high-resolution image is partitioned into small patches for optical flow refinement. With such a strategy, it becomes possible to process high-resolution images when the memory requirements are not sufficient. On the other hand, this solution also offers the ability to parallelize the optical flow estimation when possible. Furthermore, we show that such a pipeline can additionally allow for utilizing a lighter and shallower model in the two stages. It can perform on par with FastFlowNet (FFN) while being 1.7x faster computationally and with almost a half of the parameters. Against the state-of-the-art optical flow methods, the proposed solution can show a reasonable accuracy trade-off for running time and memory requirements. Code is available at: https://github.com/ahmad-hammad/PatchFlow.
The deep learning-based optical flow methods have shown noticeable advancements in flow estimation. The dense optical flow map offers high flexibility and quality for aligning neighbouring video frames. However, they are computationally expensive, and the memory requirements for processing high-resolution images such as 2K, 4K and 8K on resources-limited devices such as mobile phones can be prohibitive.
We propose a patch-based approach for optical flow estimation. We redistribute the regular CNN-based optical flow regression into a two-stage pipeline, where the first stage estimates an optical flow for a low-resolution image version. The pre-flow is input to the second stage, where the high-resolution image is partitioned into small patches for optical flow refinement. With such a strategy, it becomes possible to process high-resolution images when the memory requirements are not sufficient. On the other hand, this solution also offers the ability to parallelize the optical flow estimation when possible. Furthermore, we show that such a pipeline can additionally allow for utilizing a lighter and shallower model in the two stages. It can perform on par with FastFlowNet (FFN) while being 1.7x faster computationally and with almost a half of the parameters. Against the state-of-the-art optical flow methods, the proposed solution can show a reasonable accuracy trade-off for running time and memory requirements. Code is available at: https://github.com/ahmad-hammad/PatchFlow.
Kokoelmat
- Avoin saatavuus [38697]