ffmpeg remove parts without motion

What I'd do, is do exactly like you said, and save a big file of the stream, then use the select FFMPEG filter with the scene expression, that compares the similarity of consecutive frames: select=gt(scene\,0.003) for instance. The higher the number, the more change between frames is ignored, in quick testing you might need to go as low as 0.00001-0.00005 depending on the kind of footage you're dealing with.

Combine that with the setpts filter, that modifies the "start time" of video frames, and you'd end up with something like (for a 25fps video):

ffmpeg -i input.mp4 -vf "select=gt(scene\,0.003),setpts=N/(25*TB)" output.mp4

Mind you, you won't be able to use -vcodec copy when you use video filters.