BDD development mega issue

[tk3369]

This is a summary issue that keeps track of the work involved for implementing the BDD test suite for GraknClient.jl (for background, see issue #22). Note that ExecutableSpecifications.jl needs to be enhanced to support this project, and that's being tracked separately.

We should be able to largely model the work from Grakn's python-client. The links below point to that repo for reference.

• BDD: Implement "background" code #37 Implement background code with hooks before/after the run and each feature test
• BDD: Implement "context" code #38 Implement context code for initializing context object
• BDD: Implement common parsing functions #39 Implement common parsing functions
• BDD: Implement concept steps #40 Implement concept steps
• BDD: Implement connection steps #41 Implement connection steps
• BDD: Implement graql steps #42 Implement graql steps
• BDD: Configure CI to run BDD tests #43 Configure CI to run BDD process (pull feature files, install Grakn, and execute the tests)

[alexjpwalker]

This looks like a great summary of the various parts of the Grakn Python client's BDD test framework, @tk3369 !

The only notable omission is an automated workflow for executing the tests (say, in CI). In our Python client we have a CI job that clones the behaviour repo, downloads, unzips and runs Grakn, executes the tests storing the result to a variable, closes Grakn and returns the test result. I don't know if that would be applicable to this Julia project?

[tk3369]

That's a good point. Julia projects typically uses GitHub actions for CI. Is there any pre-made actions for Grakn? If not, we can probably set that up.

I was also a little curious about that because I see you use Bazel everywhere. We don't need to follow that, do we?

[alexjpwalker]

We do use Bazel everywhere, however, protocol has been designed specifically with non-Bazel users in mind. It deploys a pip artifact for Python, and an npm artifact for JavaScript. As for behaviour, the build system is irrelevant because all a Grakn client needs to do is clone the behaviour repo and read the .feature files verbatim.

We don't use GitHub actions, so we don't have any "prebuilt" ones. For inspiration, here's the Bazel rule we're currently using to run BDD in client-python - which is mostly just a shell script (Bazel's syntax is based on Python!):

    # Store the path of the first feature file. It is recommended to only have one feature file.
    feats_dir = ctx.files.feats[0].dirname

    # behave requires a 'steps' folder to exist in the test root directory.
    steps_out_dir = ctx.files.feats[0].dirname + "/steps"

    grakn_distro = str(ctx.files.native_grakn_artifact[0].short_path)

    cmd = "set -e && GRAKN_DISTRO=%s" % grakn_distro
    cmd += """
           if test -d grakn_distribution; then
             echo Existing distribution detected. Cleaning.
             rm -rf grakn_distribution
           fi
           mkdir grakn_distribution
           echo Attempting to unarchive Grakn distribution from $GRAKN_DISTRO
           if [[ ${GRAKN_DISTRO: -7} == ".tar.gz" ]]; then
             tar -xf $GRAKN_DISTRO -C ./grakn_distribution
           else
             if [[ ${GRAKN_DISTRO: -4} == ".zip" ]]; then
               unzip -q $GRAKN_DISTRO -d ./grakn_distribution
             else
               echo Supplied artifact file was not in a recognised format. Only .tar.gz and .zip artifacts are acceptable.
               exit 1
             fi
           fi
           DIRECTORY=$(ls ./grakn_distribution)
           if [[ $GRAKN_DISTRO == *"cluster"* ]]; then
             PRODUCT=Cluster
           else
             PRODUCT=Core
           fi
           echo Successfully unarchived Grakn $PRODUCT distribution.
           RND=20001
           while [ $RND -gt 20000 ]  # Guarantee fair distribution of random ports
           do
             RND=$RANDOM
           done
           PORT=$((40000 + $RND))
           echo Starting Grakn $PRODUCT Server.
           mkdir ./grakn_distribution/"$DIRECTORY"/grakn_test
           if [[ $PRODUCT == "Core" ]]; then
             ./grakn_distribution/"$DIRECTORY"/grakn server --port $PORT --data grakn_test &
           else
             ./grakn_distribution/"$DIRECTORY"/grakn server --address "127.0.0.1:$PORT:$(($PORT+1))" --data grakn_test &
           fi
           POLL_INTERVAL_SECS=0.5
           MAX_RETRIES=60
           RETRY_NUM=0
           while [[ $RETRY_NUM -lt $MAX_RETRIES ]]; do
             RETRY_NUM=$(($RETRY_NUM + 1))
             if [[ $(($RETRY_NUM % 4)) -eq 0 ]]; then
               echo Waiting for Grakn $PRODUCT server to start \($(($RETRY_NUM / 2))s\)...
             fi
             lsof -i :$PORT && STARTED=1 || STARTED=0
             if [[ $STARTED -eq 1 ]]; then
               break
             fi
             sleep $POLL_INTERVAL_SECS
           done
           if [[ $STARTED -eq 0 ]]; then
             echo Failed to start Grakn $PRODUCT server
             exit 1
           fi
           echo Grakn $PRODUCT database server started
           """
    # TODO: If two step files have the same name, we should rename the second one to prevent conflict
    cmd += "cp %s %s" % (ctx.files.background[0].path, feats_dir)
    cmd += " && rm -rf " + steps_out_dir
    cmd += " && mkdir " + steps_out_dir + " && "
    cmd += " && ".join(["cp %s %s" % (step_file.path, steps_out_dir) for step_file in ctx.files.steps])
    cmd += " && behave %s -D port=$PORT && export RESULT=0 || export RESULT=1" % feats_dir
    cmd += """
           echo Tests concluded with exit value $RESULT
           echo Stopping server.
           kill $(lsof -i :$PORT | awk '/java/ {print $2}')
           exit $RESULT
           """

Hopefully, it should be fairly straightforward to port this to a GitHub action! The following lines:

    cmd += "cp %s %s" % (ctx.files.background[0].path, feats_dir)
    cmd += " && rm -rf " + steps_out_dir
    cmd += " && mkdir " + steps_out_dir + " && "
    cmd += " && ".join(["cp %s %s" % (step_file.path, steps_out_dir) for step_file in ctx.files.steps])
    cmd += " && behave %s -D port=$PORT && export RESULT=0 || export RESULT=1" % feats_dir

are specific to the behave test runner for Python, and are likely to work differently in Julia.