Question: par_iter slower than shell version; threads not 100% busy

[bdrewery]

If there's a better place to ask this please close and direct me there but this seems like it is a lack of Documentation or a bug. It could simply be that Rayon's parallelism model is not the right one for my task.

My main problem is that iter_par seems to start up with NCPU threads (16) and then eventually
backs off to 3-4 at a time while the rest just sleep. The same functionality in (quite complex) shell runs in 190 seconds, but 250 seconds in rust (--release). The code being ran in parallel is very expensive since it does a fork+exec.

I've tried .with_max_len(1) and other combinations like using std::usize::MAX but nothing seems to help.

What I was looking for was a way to have a shared work queue that keeps all threads busy, sending work to the next available thread rather than splitting the work up "evenly". That already sounds like it's not what Rayon is intended for, but with the work stealing feature I figured it should be a decent library for prototyping, but was surprised by the results.

Details

I'm also on day 3 of learning Rust.

Using rayon 1.0.0.

let mut portvars: FnvHashMap<&String, FnvHashMap<String, String>> =
    FnvHashMap::with_capacity_and_hasher(ports.len(), Default::default());
portvars.par_extend(ports.par_iter().with_max_len(1).map(|port|
    (port, port_var_fetch(&config, port, &fetchvars))
));

Here ports is a Vec<String> with 26000 items. I have 16 CPU.

port_var_fetch runs make to gather variables from Makefiles (on FreeBSD make -V can be used for this, I'm not sure what the equivalent in gmake is).

port_var_fetch

I've listed the port_var_fetch in case its implementation is relevant.

This is the fetchvars variable that is passed in.

let mut fetchvars = Vec::new();
fetchvars.push(("PKGNAME", "PKGNAME"));
fetchvars.push(("MAINTAINER", "MAINTAINER"));
fetchvars.push(("FLAVORS", "FLAVORS"));
fetchvars.push(("FLAVOR", "FLAVOR"));
fetchvars.push(("CATEGORIES", "CATEGORIES"));
fetchvars.push(("IGNORE", "IGNORE"));
fetchvars.push(("DEPEND_SPECIALS", "${_DEPEND_SPECIALS:C,^${PORTSDIR}/,,}"));
fetchvars.push(("SELECTED_OPTIONS", "SELECTED_OPTIONS:O"));
fetchvars.push(("LIB_DEPENDS", "LIB_DEPENDS"));
fetchvars.push(("RUN_DEPENDS", "RUN_DEPENDS"));

fn port_var_fetch(config: &HashMap<String, String>,
                  port: &str, vars: &Vec<(&str, &str)>) -> FnvHashMap<String, String> {
    let mut args = Vec::with_capacity(vars.len() * 2);
    for var in vars {
        args.push("-V");
        args.push(var.1);
    }
    let results = pmake(&config, port, args);
    let mut result: FnvHashMap<String, String> =
        FnvHashMap::with_capacity_and_hasher(vars.len(), Default::default());
    result.extend(vars.into_iter().zip(results.lines())
        .map(|(var, value)| ((*var).0.into(), value.into())));
    result
}

fn pmake(config: &HashMap<String, String>,
         origin: &str, args: Vec<&str>) -> String {
    make(&(config.get("PORTSDIR").unwrap().to_string() + "/" + origin), args)
}

fn make(dir: &str, args: Vec<&str>) -> String {
    let mut cmd = Command::new("/usr/bin/make");
    cmd.stdin(Stdio::inherit())
        .stderr(Stdio::inherit())
        .arg("-C").arg(dir)
        .args(args);
    let output = cmd.output().expect("Make failed");
    String::from_utf8_lossy(&output.stdout).trim().to_string()
}

[cuviper]

My main problem is that iter_par seems to start up with NCPU threads (16) and then eventually
backs off to 3-4 at a time while the rest just sleep.

There will be two kinds of sleeping here -- IO waiting in cmd.output() for the external process to finish, and idle sleeping because a thread can't find any work. The former is just the nature of what you're doing, and the latter is what we'd like to minimize.

Having with_max_len(1) means we should split all the way down to single items, so threads going fully idle would imply there's just nothing left to steal. If there's a significant period at the end with only 3-4 threads active, then my hunch would be that you just have a few items that take noticeably longer than the rest.

Can you monitor which make commands are running? Maybe even just print the thread id and directory as each command starts and finishes, so you get a sense of how they're running, especially at the end.

[bdrewery]

I think this is not a Rayon problem as I switched to Threadpool with a Channel and am seeing the same results (and slower). Will try your suggestion.

[bdrewery]

My problem turned out to be just how slow constructing the HashMap is in my code. Removing that and just returning the full string got it down to 150 seconds, and 145 seconds with with_max_len(1), beating the shell time of 190 as expected. Using channels I only get down to 182 seconds.

So Rayon is working very well. Thanks!

[bdrewery]

Digging into this more with dtrace and gdb and other profiling has been interesting.

The main thread using par_extend to collect the results adds significant time, 140 seconds to 250 seconds. I am not yet sure how to resolve this.

The forking in the threads to run make ends up contending on jemalloc locks which is why I am so often seeing <NCPU make running and too much sleeping in the threads.

I think in my case I will need to have a process worker pool rather than thread worker pool, to avoid jemalloc lock contention; a wrapper program that can proxy my make requests to avoid forking in a thread.

[bdrewery]

Really this is a consequence of Command using fork rather than vfork or posix_spawn.

[cuviper]

A lot of the actions in Rust's do_exec could be handled by posix_spawn, but not cases like the arbitrary closures from CommandExt::before_exec. It might be interesting to dynamically choose how to spawn based on which actions are requested.

There shouldn't be any allocation in the forked pre-exec child though, so I'm not sure why fork/vfork/posix_spawn would make a difference to jemalloc locking.

[bdrewery]

On FreeBSD a fork in a thread reaches into jemalloc pre/post hooks that cause some locking [thr_fork][_malloc_prefork]. This isn't done for vfork and posix_spawn.

I'm playing around with Command to see about getting vfork or posix_spawn support in the cases that could work, even optionally. Changing directory is another that would not be allowed. I think that's one big holdup.
There's more discussion at rust-lang/rust#14232 and rust-lang/rust#17605

[cuviper]

Oh right, like pthread_atfork -- OK, that makes sense.

[bdrewery]

Switching to vfork gives me up to 55% improvement on FreeBSD, bringing the total time from 326 to 147 seconds. I noticed a scaling problem as well before; if I loaded up an idle ThreadPool then the time was worse, but now that problem is gone.
I'm working on a portable patch for Command to upstream.
(I'm updating this bug in case other people run into similar use cases and problems.)

[bdrewery]

posix_spawn support added to Rust at rust-lang/rust#48624 for FreeBSD. Other platforms will surely follow.