I am implementing a video processing project in real time which comes from an HDMI input. The video input is going to have a green background, which will be replaced by an image stored in the FPGA in order to generate a new video with a different background. I am using PYNQ-Z2 board.

So far, I have tried the following:

1. Storing the whole image in BRAM is not possible because there is not enough space

2. Using a second stream for the image and then try to mix the 2 streams (video + image). Cannot synchronize the 2 streams.

3. Store the image in RAM and use a double buffering scheme to load part of the image in BRAM. The first buffer is used for the processing 1 row of the image. The second one is used for loading the next row from DDR memory via the DMA (DMA is controlled by the CPU). When a row is done, then an interrupt is sent from the FPGA to the CPU so that the next line can be sent from DDR memory. Also, I switch the buffers so that new data starts loading. This solution has too much latency in the DMA transfer and the image in the video output is broken.