How to use ProcessPoolExecutor?

[antlobach]

Hello,

I'm trying to run a task using ProcessPoolExecutor, but I'm encountering an AttributeError, I believe, related to pickling. I've tried several approaches but haven't been able to get any of them to work.
Below is a minimal example.

(defn foo [x] x)

(binding [*executor-pool* (basilisp.lang.futures/ProcessPoolExecutor)]  
    (def fut (future (foo 1))))
    
@fut

The error:

AttributeError: Can't get local object 'bound_fn__STAR__.<locals>.__bound_fn__STAR____lisp_fn_2730'

I've tried to call .submit directly, but it didn't work either.

[chrisrink10]

I tried this locally and can confirm I'm getting the same error. I admit when I added the process executor, I didn't really test it -- it was just something Python supported so I added it without much further though.

I think given the limitations of what can be pickled that it won't be possible to use future, future-call, bound-fn*, or bound-fn with a process pool executor. They create anonymous functions which contain the state of the current thread bindings, so my guess is there's not really any way of recreating that functionality using pickle (which is how concurrent.futures.ProcessPoolExecutor works).

[chrisrink10]

I will need to update the documentation on this as well. Sorry about this.

[antlobach]

Ah, I see.

Hm, I wonder if there is any potential workaround for doing multicore in Basilisp. Maybe for future releases.

I will need to update the documentation on this as well. Sorry about this.

No worries at all, you've already done so much incredible work for it.

[antlobach]

I've experimented with Python 3.14+freethreaded build and it works with ThreadPoolExecutor

(defn foo [x] x)

(binding [*executor-pool* (basilisp.lang.futures/ThreadPoolExecutor)]  
    (def fut (future (foo 1))))
    
@fut ;; => 1

I had to start the repl with PYTHON_GIL=0 because otherwise I was getting the following message:

<frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'immutables._map', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

Not sure what the implications of disabling PYTHON_GIL are though.

[antlobach]

I've also tested some heavy tasks running in parallel and it worked across multiple cores.