UnicodeDecodeError when loading OpenNeuro ds002842

[m0lenaar]

When loading ds002842 from OpenNeuro, ancpbids crashes with a UnicodeDecodeError. BIDS validation on OpenNeuro assesses the dataset as valid albeit with two warnings.

Steps to reproduce

My platform:

• Linux
• fresh virtual environment created with uv
• Python version: 3.12
• ancpbids 0.3.0 from PyPI

Download dataset from OpenNeuro: https://openneuro.org/datasets/ds002842/versions/1.0.1

Run this minimal example:

import ancpbids


def main():
    ds = ancpbids.load_dataset("data/ds002842")


if __name__ == "__main__":
    main()

Output

Traceback (most recent call last):
  File "/home/toby/Projects/ancpbidstest/main.py", line 9, in <module>
    main()
  File "/home/toby/Projects/ancpbidstest/main.py", line 5, in main
    ds = ancpbids.load_dataset("data/ds002842")
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toby/Projects/ancpbidstest/.venv/lib/python3.12/site-packages/ancpbids/__init__.py", line 81, in load_dataset
    dsplugin.execute(ds, schema)
  File "/home/toby/Projects/ancpbidstest/.venv/lib/python3.12/site-packages/ancpbids/plugins/plugin_dsloader.py", line 60, in execute
    self._handle_tsv_files(dataset)
  File "/home/toby/Projects/ancpbidstest/.venv/lib/python3.12/site-packages/ancpbids/plugins/plugin_dsloader.py", line 141, in _handle_tsv_files
    self._handle_tsv_files(child)
  File "/home/toby/Projects/ancpbidstest/.venv/lib/python3.12/site-packages/ancpbids/plugins/plugin_dsloader.py", line 141, in _handle_tsv_files
    self._handle_tsv_files(child)
  File "/home/toby/Projects/ancpbidstest/.venv/lib/python3.12/site-packages/ancpbids/plugins/plugin_dsloader.py", line 135, in _handle_tsv_files
    newfile.contents = newfile.load_contents()
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toby/Projects/ancpbidstest/.venv/lib/python3.12/site-packages/ancpbids/plugins/plugin_schema_patches.py", line 55, in load_contents
    contents = utils.load_contents(file_path)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toby/Projects/ancpbidstest/.venv/lib/python3.12/site-packages/ancpbids/utils.py", line 95, in load_contents
    return reader(file_path, return_type=return_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toby/Projects/ancpbidstest/.venv/lib/python3.12/site-packages/ancpbids/plugins/plugin_files_handlers.py", line 45, in read_tsv
    return list(csv.DictReader(f, dialect="excel-tab"))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toby/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/csv.py", line 115, in __next__
    self.fieldnames
  File "/home/toby/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/csv.py", line 102, in fieldnames
    self._fieldnames = next(self.reader)
                       ^^^^^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9a in position 157: invalid start byte

[erdalkaraca]

@m0lenaar I cannot reproduce this behavior with Python 3.9 and 3.12: it seems ancpbids tries to load a corrupt TSV file.
Can you download the dataset again and/or make sure the files are not corrupt?