I was trying to render captions on top of a video (to generate a video with captions on top).
Captions is a simple <p/> tag they are filled fully in width and height with transparent background on top of a video <video/> tag. They are being displayed by 3 word. As the video plays they change accordingly to the words being told in a video.
I needed to get the video of the ongoing Captions with the same length as the video. Because then captionsVideo is being overlayed on top of inputVideo with FFmpeg and i get the video with captions.
But capturing is the problem. I wrote a simple function to capture them html2canvas.
// Export video
const exportVideo = async () => {
const video = videoRef.current; // Define the video element
const fps = 60; // frames per second
const interval = 1 / fps; // interval of capturing in seconds
const frames = []; // array to store the frames (captured screenshots)
const captureFrame = async () => {
// If the video has reached the end, remove the event listener and send the frames to the server to convert them to a video
if (video.currentTime >= video.duration) {
video.removeEventListener('seeked', captureFrame);
// Send the frames and video URL to the server
const response = await fetch('http://127.0.0.1:5000/api/convert', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({ frames, width: video.videoWidth, height: video.videoHeight }),
});
// If the server returns an error, throw an error
if (!response.ok) {throw new Error(`Server error: ${response.statusText}`);}
// If the server returns a success response, download the video
const blob = await response.blob();
const url = window.URL.createObjectURL(blob);
const link = document.createElement('a');
link.href = url;
link.download = 'output.mp4';
document.body.appendChild(link);
link.click(); // Click the link to download the end video
document.body.removeChild(link); // Clean up and remove the link
}
// Capture the captions using html2canvas with increased scale and transparent background
const canvas = await html2canvas(document.querySelector('#captionsContainer'), { scale: 5, backgroundColor: null });
const dataUrl = canvas.toDataURL(); // Convert the canvas to a data URL
frames.push(dataUrl); // Push the data URL to the frames array
video.currentTime += interval; // Move the video's current time forward by the interval
};
video.addEventListener('seeked', captureFrame); // Add the event listener to the video
video.currentTime = 0; // start at the beginning of the video
}
They care capturing good but animations (there is a simple popup text animation made with motion.div by framer motion) are rough
<motion.div
key={currentLine}
initial={{ opacity: 0, scale: 0.5 }}
animate={{ opacity: 1, scale: 1 }}
transition={{
duration: 0.3,
ease: [0, 0.71, 0.2, 1.01],
scale: {
type: "spring",
damping: 5,
stiffness: 100,
restDelta: 0.001
}
}}
>
<p id="popupText" style={{fontFamily: fontMode, fontWeight: myWeight, textTransform: caseMode,textShadow: `${offsetX}px ${offsetY}px ${blur}px ${shadowColor}`,textAlign: "center"}} className='captions-words' dangerouslySetInnerHTML={{ __html: currentLine }} />
</motion.div>
, because when animation fires it lasts for a certain period of time, and capturer can’t be on time and is capturing only the remains of animation.[enter image description here](https://i.stack.imgur.com/RwlD8.png)
I have tried to use not only html2canvas but also domtoimage, rasterizehtml, canvas2image.
Domtoimage, rasterizehtml, canvas2image provide bad quality during capturing and reset fonts.
Html2canvas resets the position of the captions container.
I tried to use GSAP to sync video and caption’s animation to not hurry the capturer and capture all animations on time. But there is no solution in gsap for that as well as in FRAMER MOTION (in which i am creating all animations)
I have tried to use SCREENCAPTURER API but it requires user’s permission to record an actual video. Hence im taking screenshots.
I expected to get the captions captured with animations perfectly but animations are still rough.
