make long-file CUDA OOM recoverable per file

- replace permanent _CUDA_BROKEN flag with CudaOOMError and retry helper
- retry WhisperX with configured settings, then batch=1 without VAD, then once more
- keep the loaded model cached between retries instead of reloading it
- actually pass vad_filter through transcribe() and use it in aligned ASR
- skip only the OOM file in sequential, staged and LRC pipelines
- mark OOM files with oom_skipped / lrc+oom for the summary
- fix staged vocal separation writing input_dir/vocals.wav when convert is off
- route original inputs and resumed files through safe per-file separation
- validate ffmpeg before starting the LRC task
This commit is contained in:
jadssss
2026-10-27 23:03:29 +03:00
parent 999934f174
commit f88ca913ce
4 changed files with 86 additions and 47 deletions

View File

@@ -14,7 +14,7 @@ from .metadata import read_metadata
from .pipeline import (
FileResult, _artifact_belongs_to, _save_unrecognized, collect_audio_files,
)
from .transcribe import transcribe_aligned
from .transcribe import CudaOOMError, transcribe_aligned
LogFn = Callable[[str], None]
@@ -155,6 +155,10 @@ def run_lrc_pipeline(
metadata=read_metadata(f),
)
results.append(FileResult(f, None, track_out, None, "lrc"))
except CudaOOMError as e:
error = f"CudaOOMError: {e}"
log(f" ⊘ {f.name}: OOM — пропущен, идём дальше")
results.append(FileResult(f, None, None, error, "lrc+oom"))
except Exception as e: # noqa: BLE001
error = f"{type(e).__name__}: {e}"
log(f" ✗ Ошибка: {error}")