Very bad memory management

[Vadim2S]

I am try this project. I am like idea and good description. But I am can not do anything valuable. In many places here one pattern used: load all data into memory, calculate all results, save all results. All. Not pieces in cycle. It has advantage of clear and simple code ... but I am try train "in the wild" and can not do it even with 12 seconds video! My computer have 24GB RAM. I am create docker image, run command and see as memory consumption raised to 80-90% and swap, swap, swap... My main SSD has usage 90% and command progress almost freeze. At least split_video.py, preprocess.py and what most unfortunate train.py has this flaw.

P.S. How many RAM you have? At first I am rewrite split_video.py try 7 minutes video and preprocess.py drop down to swap at 23% of progress. At second try I am use 12 seconds video, preprocess.py OK but train.py drop down to swap even before any progress bar appeared.

[g-milis]

Hi @Vadim2S, are you using a GPU? I have ran this code on RTX 3060 and RTX 2080 Ti.

[Vadim2S]

Yes. RTX 3050. But this is irrelevant. This it not GPU related. I am clarify situation with small example:

Lets see at file /misc/split_video.py line 24. You use code "video = skvideo.io.vread(input_mp4)" for reading video file in memory.
Lets use it with 25fps 1980x720 6m30sec video:

As you see - there no GPU yet. We simply can not create buffer for all frames of videofile.

P.S. I am rewrite split_video.py with something like this. Sorry this is only for experoments and not good ether (slow - read video for each segment):

# linux only
# ffmpeg and libs must be installed
# torchvision must be compiled from source
torchvision.set_video_backend('video_reader')

# Process each segment
for segment in tqdm(segments):
    # start, end - time in seconds

    # Extract segment
    start_sec = segment["start"]
    end_sec = segment["end"]
    video_segment, audio_segment, info_segment = torchvision.io.read_video(filename=input_mp4, start_pts=start_sec, end_pts=end_sec, pts_unit='sec')

    # read_video read audio_segments as [T, C]
    # torchaudio work with [C, T]
    audio_segment = torch.swapaxes(audio_segment, 0, 1)

    audio_segment_sr = int(info_segment['audio_fps'])
    if audio_segment_sr != desired_sr:
        audio_segment = torchaudio.functional.resample(audio_segment, audio_segment_sr, desired_sr)
    if audio_segment.size(0) > 1:
        audio_segment = torch.mean(audio_segment, dim=0, keepdim=True)

    Path(f"{output_path}/tts/{subject}").mkdir(parents=True, exist_ok=True)

    # Save audio segment
    output_wav = f"{output_path}/tts/{subject}/{segment['id']:03d}.wav"
    torchaudio.save(output_wav, audio_segment, desired_sr, bits_per_sample=16)

    Path(f"{output_path}/videos/").mkdir(parents=True, exist_ok=True)

    # Save video segment
    torchvision.io.write_video(f"{output_path}/videos/{segment['id']:03d}.mp4", video_segment, fps=desired_fps)

    # Save text to a corresponding text file
    output_txt = f"{output_path}/tts/{subject}/{segment['id']:03d}.lab"
    with open(output_txt, "w") as txt_file:
        txt_file.write(segment["text"])

P.P.S. Sorry I am not remember exactly what with code in preprocess.py and train.py. I am just see what my docker container eat ~10GB RAM (not GPU RAM!) and start swapping.

[g-milis]

I understand. Loading the entire video is unnecessary. One solution would be to use OpenCV and load only the desired frames, as shown here. Please try that and let me know if it works so I can update the code as soon as I have the time.

[Vadim2S]

Big thanks in advance. Meantime I am try you project on more powerfull computer