file-validator

What is it?

A command-line program to generate XML (or jsonlines) reports about the validity of files. It scans directories and tests every file for errors. This is useful to find corrupt/broken files on the HDD.

How do I use it?

• Install Python 3.8+
• py -m pip install . or py -m pip install .[html,images,orc,parquet,pdf,xml] (including all plugin dependencies)
• py -m filevalidator.validator --help

CLI usage

usage: validator.py [-h] [-d REPORTDIR] [-x XSLFILE] [-r] [-v] [--only EXT [EXT ...]] [-i EXT [EXT ...]] [--relative]
                    [--recall] [--resume RESUME] [--out {xml,json,stdout}]
                    DIRECTORY [DIRECTORY ...]

FileValidator

positional arguments:
  DIRECTORY             directories to create report for

optional arguments:
  -h, --help            show this help message and exit
  -d REPORTDIR, --reportdir REPORTDIR
                        set output directory for reports (default: reports)
  -x XSLFILE, --xsl XSLFILE
                        set XSL style sheet file (default: report.xsl)
  -r, --recursive       scan directories recursively (default: False)
  -v, --verbose         output debug info (default: False)
  --only EXT [EXT ...]  only include these extensions (default: None)
  -i EXT [EXT ...], --ignore EXT [EXT ...]
                        extensions to ignore (default: None)
  --relative            Output relative paths (default: False)
  --recall              Download files which are currently only available online (on OneDrive for example), otherwise
                        they are skipped. (default: False)
  --resume RESUME       Resume validation using a previous XML report (default: None)
  --out {xml,json,stdout}
                        Output method. xml: write to xml file, json: write to json file, stdout: simple format written
                        to stdout (default: xml)

Example

• py -m filevalidator.validator "C:\PUBLIC"

Tests

• python -m pytest
• The video plugin tests use the FFMPEG_BINARY environment variable if set, otherwise they fall back to ffmpeg on PATH. If no FFmpeg executable is available, those tests are skipped.

Processing reports

Reports record each file's size and, when available, nanosecond modification time from immediately before validation. validator-process always checks the size and also checks the modification time when it was recorded. --ignore-mtime disables only the modification-time check; the file size must still match.

--relative requires exactly one input directory. XML reports store that directory as their base, and validator-process resolves relative entries against it. Pass --base DIRECTORY to override the stored base, for example after moving the scanned directory. Relative reports without stored base metadata require --base before they can be processed.

Non-Python dependencies

• Plugins:
  • Archives: rar and 7z executable, must be in path, or set path in plugin config
  • Videos: ffmpeg executable, must be in path, or set path in plugin config
  • FLAC: flac executable, must be in path, or set path in plugin config
  • PDF: qpdf executable is used when available, or set its path in plugin config; pypdf remains the fallback

Plugin configs are JSON files loaded from C:\Users\<username>\AppData\Local\Dobatymo\file-validator\config (on Windows). See config-example for examples JSON files.

How does it work?

It loads plugins which handle one or more file extensions. Then for every file the right plugin is used to validate it and output the results to a XML or jsonlines file.

It doesn't support file type .xxx

Plugins can be written by everybody. For an example of a trivial plugin see filevalidator/plugins/txt.py.