How to include tasks from a directory not in or below a set of tasks?

[rberger]

I have a mono repo and I want to have tasks at different levels (in the directory tree). When I run invoke at a deeper level, I want to have access to the tasks in the higher levels + the tasks in the current level. They can (should) be in different namespaces.

i.e.

repo_top-|
         |_ tasks-|
                  |_ install
         |_apps-|
                |_ tasks-|
                         |_ lambdas

If I am in the repo_top/apps directory (or lower) I should be able to say

inv --list

and see a listing of the tasks in repo_top/tasks AND repo_top/apps/tasks

I have it so I see the top level stuff if I am in the top level or any sub level that does not have its own tasks module and I see the apps tasks if I am in the apps directory or lower.

I've tried a few things but so far have not figured out how to make it so I see all the tasks if I am in the apps directory or lower.

[rberger]

I figured a way to do this but it requires one slight hack:
Need to make a symbolic link to repo_top/tasks so we can refer to the top level tasks package with another name.
In the repo_top directory I did:

ln -s tasks top_tasks

Once you do that, you can still call invoke from the project directory on down into the project subdirectories and it will continue to work like normal using the repo_top/tasks.

But if you add a tasks package in any subdirectory, those tasks packages can now reference the repo_top/tasks as top_tasks in import statements, once you add top_tasks to the sys.path as shown below.

Set up the sub task to access to top_tasks

In repo_top/apps/tasks/__init__.py:

## Import tasks from the top_tasks directory from the project root
project_top_dir = Path(__file__).parents[2]
sys.path.append(str(project_top_dir))
import top_tasks

At that point you can include any tasks from top_tasks into the ns/collection in the apps/tasks

ns = Collection()
ns.add_collection(top_tasks.install)

Of course you can also import and use the tasks that are defined in apps/tasks as usual. So a complete apps/tasks/__init__.py would look like:

import sys
from pathlib import Path
from invoke import Collection

##
## Import tasks from the top_tasks directory from the project root
##

# Get the absolute path to `repo_top`
project_top_dir = Path(__file__).parents[2] 

 # Add it to the Python sys.path
sys.path.append(str(project_top_dir))

# Import the top_tasks (which is the same as `repo_top/tasks`, but now we can refer to it in the import statement)
import top_tasks 

## Import any tasks we want from the tasks directory in the apps/tasks directory as normally
from . import lambdas

# Create the collections / namespace
ns = Collection()
# Add the `apps/tasks`
ns.add_collection(lambdas)
# Add the `top_tasks` tasks
ns.add_collection(top_tasks.install)

Sub Tasks Accessing functions/variables from modules in top_tasks

If you needed to import something like a function or a variable from a module in top_tasks such as top_tasks/utils into a module in apps/tasks (like apps/tasks/lambdas you would do something like the following in apps/tasks/lambdas to access env_name that was defined in top_tasks/utils:

In apps/tasks/lambdas.py (it inherits the path append from apps/tasks/__init__.py):

from top_tasks.utils import *

print(f"env_name from top_tasks/utils: {env_name}")

Caveats

I just got this to work and haven't excercised it fully, but so far it looks like a pretty good solution other than the ugliness of having to have an extra "file" (the symlink) to the tasks directory for any level where you want to allow sub tasks directories.