perf!: lazily process each 64 tasks by KSXGitHub · Pull Request #287 · KSXGitHub/parallel-disk-usage

KSXGitHub · 2025-07-01T13:28:09Z

Breaking Change

AccessEntry has been removed from error_report::Operation.
Generic parameter Name of tree_builder::Info has been replaced with NameIter.
Generic parameter Name of TreeBuilder has been replaced with NameIter.

github-actions · 2025-07-01T14:01:32Z

Performance Regression Reports

commit: a945d7f

--quantity=block-size --max-depth=10 --min-ratio=0.01 --progress

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`pdu`	162.2 ± 50.6	104.3	206.0	1.49 ± 0.54
`pdu-0.12.0`	108.5 ± 19.0	104.2	205.4	1.00

Logs

Benchmark 1: pdu
  Time (mean ± σ):     162.2 ms ±  50.6 ms    [User: 102.9 ms, System: 303.2 ms]
  Range (min … max):   104.3 ms … 206.0 ms    28 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Benchmark 2: pdu-0.12.0
  Time (mean ± σ):     108.5 ms ±  19.0 ms    [User: 61.2 ms, System: 294.8 ms]
  Range (min … max):   104.2 ms … 205.4 ms    28 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Summary
  pdu-0.12.0 ran
    1.49 ± 0.54 times faster than pdu

JSON

{
  "results": [
    {
      "command": "pdu",
      "mean": 0.16215835784285712,
      "stddev": 0.050640641644162376,
      "median": 0.2049639802,
      "user": 0.10286151285714283,
      "system": 0.3032453985714286,
      "min": 0.1043387872,
      "max": 0.2059967572,
      "times": [
        0.10457305620000001,
        0.10469708720000001,
        0.2052052832,
        0.10468553420000001,
        0.2052557172,
        0.2052809742,
        0.2050240372,
        0.2049699042,
        0.10458410920000001,
        0.2051666862,
        0.20547213620000002,
        0.10500078420000002,
        0.2059967572,
        0.2053176182,
        0.10501935020000001,
        0.2051086492,
        0.2043641892,
        0.10499140620000001,
        0.10462858920000001,
        0.10476125720000001,
        0.2054094962,
        0.10479276520000001,
        0.1043387872,
        0.20495805620000002,
        0.2051966092,
        0.2053681242,
        0.2054817712,
        0.10478528420000001
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    },
    {
      "command": "pdu-0.12.0",
      "mean": 0.10848854091428574,
      "stddev": 0.01900390819176906,
      "median": 0.10492308320000002,
      "user": 0.06118758428571429,
      "system": 0.29481568428571425,
      "min": 0.10420847720000001,
      "max": 0.2054476952,
      "times": [
        0.1051728432,
        0.10492722520000002,
        0.10480735320000001,
        0.10492542220000001,
        0.10478168720000001,
        0.10420847720000001,
        0.10499968920000001,
        0.10491395220000001,
        0.10486026120000001,
        0.10491596620000002,
        0.2054476952,
        0.10502302620000001,
        0.10510462820000001,
        0.10510692220000001,
        0.10482083720000002,
        0.10455723720000001,
        0.10513642420000001,
        0.10496997620000001,
        0.10492074420000001,
        0.10462528120000002,
        0.10515767520000001,
        0.10480289620000001,
        0.10500926120000001,
        0.10449680220000002,
        0.10556143720000001,
        0.10489813220000001,
        0.1049518722,
        0.1045754202
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    }
  ]
}

--quantity=block-size --max-depth=1 --min-ratio=0 --no-sort

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`pdu`	99.9 ± 0.4	99.5	101.0	1.14 ± 0.01
`pdu-0.12.0`	87.7 ± 0.5	87.2	89.7	1.00

Logs

Benchmark 1: pdu
  Time (mean ± σ):      99.9 ms ±   0.4 ms    [User: 98.8 ms, System: 292.3 ms]
  Range (min … max):    99.5 ms … 101.0 ms    29 runs
 
Benchmark 2: pdu-0.12.0
  Time (mean ± σ):      87.7 ms ±   0.5 ms    [User: 56.7 ms, System: 285.8 ms]
  Range (min … max):    87.2 ms …  89.7 ms    33 runs
 
Summary
  pdu-0.12.0 ran
    1.14 ± 0.01 times faster than pdu

JSON

{
  "results": [
    {
      "command": "pdu",
      "mean": 0.09994349445034484,
      "stddev": 0.00037230308742456725,
      "median": 0.09990830614000001,
      "user": 0.09884482275862069,
      "system": 0.2923172848275862,
      "min": 0.09948520414,
      "max": 0.10102603014000001,
      "times": [
        0.10072030914,
        0.10037904814000001,
        0.10004297914000002,
        0.09949855414000001,
        0.09971037914000001,
        0.10041290814,
        0.10035672114000001,
        0.09955444114,
        0.09960735514,
        0.10007871514000001,
        0.09995129814,
        0.10009007714000001,
        0.09956384914000001,
        0.09986972414,
        0.09965006714,
        0.10102603014000001,
        0.09998829914000001,
        0.09999302614000001,
        0.09976118314000001,
        0.09991251814,
        0.09990830614000001,
        0.09960209814000001,
        0.09961998814,
        0.09964359514000001,
        0.09948520414,
        0.10026832714000002,
        0.10005416214000001,
        0.09975011914000001,
        0.09986205614000002
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    },
    {
      "command": "pdu-0.12.0",
      "mean": 0.08773859086727273,
      "stddev": 0.0004775574611217126,
      "median": 0.08767264014000001,
      "user": 0.056660188484848474,
      "system": 0.28581345515151513,
      "min": 0.08721482714,
      "max": 0.08971072614,
      "times": [
        0.08764498814,
        0.08837957514000001,
        0.08768676414000001,
        0.08772949314,
        0.08800716914000001,
        0.08790097414,
        0.08742091914000001,
        0.08767264014000001,
        0.08731131514000001,
        0.08748090014000001,
        0.08726912414000002,
        0.08729669714,
        0.08736606614,
        0.08766647914,
        0.08829453714,
        0.08740488914000001,
        0.08788168714000001,
        0.08836028714000001,
        0.08722545414,
        0.08753024414,
        0.08801523614000001,
        0.08799398514000001,
        0.08776054214000001,
        0.08785287414000001,
        0.08749762214000001,
        0.08731805914000002,
        0.08771037614,
        0.08721482714,
        0.08971072614,
        0.08781757814,
        0.08755254214000001,
        0.08744404214000001,
        0.08795488414000001
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    }
  ]
}

--quantity=block-count --max-depth=10 --min-ratio=0.01

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`pdu`	103.6 ± 0.8	102.2	105.3	1.13 ± 0.01
`pdu-0.12.0`	91.8 ± 1.0	91.0	96.4	1.00

Logs

Benchmark 1: pdu
  Time (mean ± σ):     103.6 ms ±   0.8 ms    [User: 102.4 ms, System: 298.4 ms]
  Range (min … max):   102.2 ms … 105.3 ms    28 runs
 
Benchmark 2: pdu-0.12.0
  Time (mean ± σ):      91.8 ms ±   1.0 ms    [User: 58.6 ms, System: 294.8 ms]
  Range (min … max):    91.0 ms …  96.4 ms    32 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Summary
  pdu-0.12.0 ran
    1.13 ± 0.01 times faster than pdu

JSON

{
  "results": [
    {
      "command": "pdu",
      "mean": 0.10361257668428572,
      "stddev": 0.0008085087096941004,
      "median": 0.10347839772,
      "user": 0.10235372142857144,
      "system": 0.29836967999999997,
      "min": 0.10223210022000001,
      "max": 0.10532970122,
      "times": [
        0.10293081122,
        0.10340761122,
        0.10359806322000001,
        0.10530881322,
        0.10308201422,
        0.10396379622,
        0.10306299822000001,
        0.10319622822,
        0.10312708822000001,
        0.10489452922,
        0.10354665022000001,
        0.10223210022000001,
        0.10291188622000001,
        0.10372948722,
        0.10288104922,
        0.10522735022,
        0.10369815222,
        0.10297165822000001,
        0.10470131322000001,
        0.10341014522,
        0.10381308322,
        0.10291699822,
        0.10306348622,
        0.10362434422000001,
        0.10372914922000001,
        0.10385688722,
        0.10532970122,
        0.10293675222000001
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    },
    {
      "command": "pdu-0.12.0",
      "mean": 0.09181036837625003,
      "stddev": 0.0009519240820882141,
      "median": 0.09164630222,
      "user": 0.05855065000000001,
      "system": 0.29481243,
      "min": 0.09098223822000001,
      "max": 0.09640179922,
      "times": [
        0.09146426822,
        0.09099624622000001,
        0.09189490322,
        0.09098223822000001,
        0.09122462322000001,
        0.09146224222,
        0.09136987322000001,
        0.09114441422000001,
        0.09205871322,
        0.09157035622000001,
        0.09257661022000001,
        0.09187597522,
        0.09127124122000001,
        0.09162068922000001,
        0.09167191522000001,
        0.09129255822,
        0.09130714322000001,
        0.09640179922,
        0.09122812822000001,
        0.09139020822,
        0.09124348922,
        0.09181019122,
        0.09170507622,
        0.09188556622,
        0.09189014922000001,
        0.09197146322000001,
        0.09189043422000001,
        0.09241946522000001,
        0.09180138622,
        0.09230274022,
        0.09133691122000001,
        0.09287076922000001
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    }
  ]
}

--quantity=apparent-size --max-depth=1 --min-ratio=0

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`pdu`	99.7 ± 0.5	99.2	101.2	1.14 ± 0.01
`pdu-0.12.0`	87.6 ± 1.0	86.8	92.1	1.00

Logs

Benchmark 1: pdu
  Time (mean ± σ):      99.7 ms ±   0.5 ms    [User: 96.0 ms, System: 294.3 ms]
  Range (min … max):    99.2 ms … 101.2 ms    29 runs
 
Benchmark 2: pdu-0.12.0
  Time (mean ± σ):      87.6 ms ±   1.0 ms    [User: 56.4 ms, System: 285.1 ms]
  Range (min … max):    86.8 ms …  92.1 ms    34 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Summary
  pdu-0.12.0 ran
    1.14 ± 0.01 times faster than pdu

JSON

{
  "results": [
    {
      "command": "pdu",
      "mean": 0.09971818267034481,
      "stddev": 0.0004951283179918479,
      "median": 0.09956195736000001,
      "user": 0.0959786875862069,
      "system": 0.294286112413793,
      "min": 0.09923121936000001,
      "max": 0.10119096136000001,
      "times": [
        0.09941131436,
        0.09956195736000001,
        0.09934432636000001,
        0.09927185336000001,
        0.09969463636,
        0.10008146536000001,
        0.09930238536000001,
        0.09923121936000001,
        0.10020460436,
        0.09949145536000001,
        0.09926938136,
        0.09938736336000001,
        0.09949389036,
        0.09945638036000001,
        0.09977689836,
        0.10114176636000001,
        0.09981016336000001,
        0.09942261536000001,
        0.09940953336000001,
        0.09982975636000001,
        0.09957920136000001,
        0.10035603636000001,
        0.09944808336000001,
        0.09985470236,
        0.09992422236000001,
        0.10119096136000001,
        0.09993058336,
        0.09937417636000001,
        0.09957636436
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    },
    {
      "command": "pdu-0.12.0",
      "mean": 0.08758734638941176,
      "stddev": 0.0009697717349952648,
      "median": 0.08725914686000001,
      "user": 0.056392359999999975,
      "system": 0.28509626352941175,
      "min": 0.08682321536000001,
      "max": 0.09208533836,
      "times": [
        0.08718457736000002,
        0.08717239336,
        0.08768512536,
        0.08714555136,
        0.08688212436000001,
        0.08733414036,
        0.08836732236000001,
        0.08724389836,
        0.08845494236000001,
        0.09208533836,
        0.08953574836000001,
        0.08682321536000001,
        0.08749323536,
        0.08720944436000001,
        0.08736671536000001,
        0.08726149136000001,
        0.08712834536000001,
        0.08820488936000001,
        0.08720147936000001,
        0.08725680236000001,
        0.08773886136,
        0.08712794836000001,
        0.08687700536000001,
        0.08739108236000001,
        0.08745750636,
        0.08745575736000001,
        0.08715373636000001,
        0.08732768436,
        0.08706317636000001,
        0.08841726236000001,
        0.08753250436000001,
        0.08703271536000001,
        0.08725656436000001,
        0.08710119236000001
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    }
  ]
}

--quantity=block-size --max-depth=10 --min-ratio=0.01

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`pdu`	104.6 ± 1.0	103.2	108.4	1.13 ± 0.05
`pdu-0.12.0`	92.5 ± 4.0	90.9	113.2	1.00

Logs

Benchmark 1: pdu
  Time (mean ± σ):     104.6 ms ±   1.0 ms    [User: 103.4 ms, System: 301.0 ms]
  Range (min … max):   103.2 ms … 108.4 ms    28 runs
 
Benchmark 2: pdu-0.12.0
  Time (mean ± σ):      92.5 ms ±   4.0 ms    [User: 60.6 ms, System: 293.1 ms]
  Range (min … max):    90.9 ms … 113.2 ms    31 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Summary
  pdu-0.12.0 ran
    1.13 ± 0.05 times faster than pdu

JSON

{
  "results": [
    {
      "command": "pdu",
      "mean": 0.10460512452000001,
      "stddev": 0.000988936317591879,
      "median": 0.10434516952,
      "user": 0.10341372999999997,
      "system": 0.30103066857142846,
      "min": 0.10320989402000001,
      "max": 0.10839704702000001,
      "times": [
        0.10406975602000001,
        0.10337252702000001,
        0.10397776602000001,
        0.10425058202000001,
        0.10428685102000002,
        0.10517522502000001,
        0.10511703502000001,
        0.10419387702,
        0.10417288002000001,
        0.10583399002,
        0.10320989402000001,
        0.10465704202000001,
        0.10522695902000001,
        0.10472108102000001,
        0.10493400602000001,
        0.10410071402000001,
        0.10440348802,
        0.10401596502000002,
        0.10463477602000001,
        0.10608175502000002,
        0.10839704702000001,
        0.10505640202000001,
        0.10399564902000001,
        0.10443484602000001,
        0.10424144402,
        0.10365474502000001,
        0.10412893402000001,
        0.10459825002
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    },
    {
      "command": "pdu-0.12.0",
      "mean": 0.09247895447161288,
      "stddev": 0.0039759326286093,
      "median": 0.09152819202000001,
      "user": 0.06063499612903226,
      "system": 0.29307759483870965,
      "min": 0.09085642102000001,
      "max": 0.11318201302000001,
      "times": [
        0.09361901502000002,
        0.09176885302000001,
        0.09233677802000001,
        0.09289815402000001,
        0.09165521702000001,
        0.09131096902000001,
        0.09131937302000001,
        0.09092741102000002,
        0.09168062702000002,
        0.09109085602000001,
        0.09178270602000001,
        0.09140287902000001,
        0.09170097602,
        0.09159039802,
        0.09122057702000001,
        0.09085642102000001,
        0.09152819202000001,
        0.09604320302000001,
        0.11318201302000001,
        0.09305864102000001,
        0.09261720102000001,
        0.09097680502000001,
        0.09100911502,
        0.09148941002000001,
        0.09126234702000001,
        0.09116417602000002,
        0.09162097302000001,
        0.09106651302,
        0.09149719202,
        0.09186116202000001,
        0.09130943502000001
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    }
  ]
}

KSXGitHub · 2025-07-01T14:13:15Z

Performance is consistently horrible. Closed.

KSXGitHub added 6 commits July 1, 2025 13:15

feat(api)!: stop reporting dir entry errors

d2719ec

perf!: do not create copies of dir entries

84b2321

perf: eagerly process each 64 tasks

d81be04

perf: lazily process each 64 tasks

bf5a38f

fix(perf): actually make it different

52e4071

chore(git): merge from master

93bbea3

KSXGitHub closed this Jul 1, 2025

KSXGitHub deleted the lazily-process-chunks-of-tasks branch July 1, 2025 14:13

KSXGitHub added the wontfix This will not be worked on label Jul 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf!: lazily process each 64 tasks#287

perf!: lazily process each 64 tasks#287
KSXGitHub wants to merge 6 commits intomasterfrom
lazily-process-chunks-of-tasks

KSXGitHub commented Jul 1, 2025

Uh oh!

github-actions Bot commented Jul 1, 2025 •

edited

Loading

Uh oh!

KSXGitHub commented Jul 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

KSXGitHub commented Jul 1, 2025

Breaking Change

Uh oh!

github-actions Bot commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Regression Reports

Uh oh!

KSXGitHub commented Jul 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Jul 1, 2025 •

edited

Loading