Skip to content

SameFileError when processing with StartMaja #96

Description

@BartoszJaroszkowski

Given a workplan

               DATE |       TILE |     MODE |                                                             L1-PRODUCT | ADDITIONAL INFO
2021-10-10 10:09:41 |      32UQU | BACKWARD |      S2A_MSIL1C_20211010T100941_N0301_R022_T32UQU_20211010T122329.SAFE | Backward of 8 products
2021-10-10 10:09:41 |      32UQU |  NOMINAL |      S2A_MSIL1C_20211010T100941_N0301_R022_T32UQU_20211010T110450.SAFE | L2 from previous
2021-10-15 10:10:29 |      32UQU |  NOMINAL |      S2B_MSIL1C_20211015T101029_N0301_R022_T32UQU_20211015T121549.SAFE | L2 from previous
2021-10-23 10:21:01 |      32UQU |  NOMINAL |      S2A_MSIL1C_20211023T102101_N0301_R065_T32UQU_20211023T110926.SAFE | L2 from previous
2021-10-30 10:11:41 |      32UQU |  NOMINAL |      S2A_MSIL1C_20211030T101141_N0301_R022_T32UQU_20211030T110921.SAFE | L2 from previous
2021-11-04 10:11:09 |      32UQU |  NOMINAL |      S2B_MSIL1C_20211104T101109_N0301_R022_T32UQU_20211104T121548.SAFE | L2 from previous
2021-11-14 10:11:59 |      32UQU |  NOMINAL |      S2B_MSIL1C_20211114T101159_N0301_R022_T32UQU_20211114T105348.SAFE | L2 from previous
2021-11-19 10:13:21 |      32UQU |  NOMINAL |      S2A_MSIL1C_20211119T101321_N0301_R022_T32UQU_20211119T110331.SAFE | L2 from previous

I got the following error

Traceback (most recent call last):
  File "/maja/lib/python/orchestrator/launcher/maja.py", line 100, in <module>
    main()
  File "/maja/lib/python/orchestrator/launcher/maja.py", line 81, in main
    lprocessor.scientific_processing()
  File "/maja/lib/python/orchestrator/processor/l2_nominal_processor.py", line 216, in scientific_processing
    l2_image_file_writer, checking_conditional_clouds)
  File "/maja/lib/python/orchestrator/processor/l2_processor.py", line 828, in process_one_product
    self._cams_status, l_IgnoreCurrentLTC)
  File "/maja/lib/python/orchestrator/processor/l2_processor_header_writer_setup.py", line 157, in setup_l2_header_writer
    l_L2HeaderFileWriter.write()
  File "/maja/lib/python/orchestrator/plugins/common/muscate/maja_muscate_l2_header_writer.py", line 75, in write
    self.pre_processing()
  File "/maja/lib/python/orchestrator/plugins/common/base/maja_l2_header_writer_base.py", line 124, in pre_processing
    self.outputl2privateimagefilenamesprovider.get_ltc_image_dir_filename())
  File "/maja/lib/python/orchestrator/common/file_utils.py", line 129, in copy_tree
    copy_file(src_filename, dst_filename)
  File "/maja/lib/python/orchestrator/common/file_utils.py", line 112, in copy_file
    shutil.copyfile(source, dest)
  File "/maja/lib/python3.7/shutil.py", line 104, in copyfile
    raise SameFileError("{!r} and {!r} are the same file".format(src, dst))
shutil.SameFileError: '/mnt/vol/tmp/Start_maja_e73b1f2b4319aeef75c481cf23529f75/SENTINEL2A_20211010-101729-643_L2A_T32UQU_C_V1-0/DATA/SENTINEL2A_20211010-101729-643_L2A_T32UQU_C_V1
-0_PVD_ALL/LTC/20211104.mha' and '/mnt/vol/input/L2A-Products/32UQU/SENTINEL2A_20211010-101729-643_L2A_T32UQU_C_V1-0/DATA/SENTINEL2A_20211010-101729-643_L2A_T32UQU_C_V1-0_PVD_ALL/L
TC/20211104.mha' are the same file

It seems to be happening rather randomly - sometimes reprocessing with the same exact inputs works as intended, sometimes it fails again. Looking at the traceback it seems pretty easy to mitigate (either with try/except block or by checking if the file exists beforehand) - the question is whether it is not a symptom of some underlying issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions