Skip to content

Grove CLI

Use cases

  1. Generate deployment pipelines
  2. Wrap Tutor
  3. Prepare dynamic environment for wrapped CLI commands
  4. Increase testability
  5. Increase reusability

Current state

Right now there are several different scripts in the tools-container/scripts directory. Some are written in bash, some are in python. All those scripts have dependencies with environment variables and all those checking logic is everywhere. All the logic in these scripts is not tested. Scripts are getting messy and hard to manage.

Proposed solution

We propose a new python CLI package - grove-cli. It will be created under the tools-container/grove-cli folder. This single cli will be responsible for doing all Grove-related things. We can use typer to create API interfaces for this new CLI tool.

Project structure

The project structure for the proposed package will be following:

grove-cli
|-- bin
    |-- cli.py
|-- grove
|   |-- templates
|   |   |-- pipelines
|   |       |-- base_image.yml
|   |       |-- common.yml
|   |       |-- dummy.yml
|   |       |-- openedx.yml
|   |       |-- terraform.yml
|   |-- instance.py
|   |-- const.py
|   |-- terraform.py
|   |-- type_defs.py
|   |-- handlers.py
|   |-- pipeline.py
|   |-- utils.py
|   |-- exceptions.py
|-- tests
  • cli.py file will be the entry point of this package and holds all CLI commands.
  • pipeline.py file holds the logic for generating different pipeline jobs.
  • utils.py file will contain some utility functions like opening Grove configuration file, parsing commit message format, etc.
  • const.py file will contain consts like workspace path, script dir path, template path, etc.
  • instance.py file will contain logic for creating new instances, pre, and post-deployment activities, syncing tutor env directory, etc.
  • terraform.py file will contain a terraform wrapper.
  • tests directory will contain tests.

We can create a grove.sh file (contains logic to run this cli without installing using pip) in scripts and put it in /usr/local/bin for easy access. This approach will help us during development as well and we don't need to run pip install every time we change something in the package files.

Create a new Open edX instance

grove new $INSTANCE_NAME

This command will copy the default configuration and create a new instance $INSTANCE_NAME in the instances/$INSTANCE_NAME directory. This command should not have any dependency on Tutor or Terraform. This raises an error if the instance is already present.

grove prepare $INSTANCE_NAME

This command is similar to the new command. Only difference is that it creates the instance if the instance isn't exist and updates the existing one. We will use this from GitLab CI mostly. This command will use Tutor to generate the initial configuration. It doesn't require Terraform but it will use it if the infrastructure is ready for the instance.

Generate Pipelines

grove pipeline $COMMIT_TITLE $GENERATED_FILE_PATH

Given a $COMMIT_TITLE, Grove CLI will generate the pipeline for it and write it on $GENERATED_FILE_PATH. This command will be used by GitLab CI to generate a child pipeline based on commit messages. Check commit based pipeline for more info.

Generate pipeline on GitLab or GitHub webhook call

The logic for handling webhook triggers will be inside Grove CLI as well.

grove webhookpipeline --output $GENERATED_FILE_PATH

By handling triggers via the CLI itself will help us create a dynamic pipeline without any Commit. Since we are generating a pipeline using the CLI, we can reuse that logic without creating a commit message and create a deployment pipeline directly from a trigger without any commits.

Handling triggers via the CLI itself will help us create a dynamic pipeline without pushing empty commits. This is going to simplify handling batch redeployments and redeploying instances without any configuration changes.

Wrap Tutor

Sync Tutor Env directory

Following command will pull env from s3, render env-overrides dir, runs tutor config save, and pushes changes to s3.

grove tutor sync $INSTANCE_NAME

Pre deploy

grove predeploy $INSTANCE_NAME

This will do the preparation work for $INSTANCE_NAME before deployment, ex: clone template, override SCSS, etc.

Deploy an Open edX instance

grove deploy $INSTANCE_NAME

This will perform deployment for $INSTANCE_NAME.

Post Deploy

grove postdeploy $INSTANCE_NAME

This will perform post-deployment steps for $INSTANCE_NAME, ex: enable theme, force pull updated images in k8s, etc.

Build images

grove buildimage $INSTANCE_NAME $IMAGE_NAME --cache_from $CACHE_FROM

This wraps thetutor images build command with an easy interface.

Run any Tutor command

grove tutor exec $INSTANCE_NAME $ANY_COMMAND_AS_STRING

It would be useful to wrap Tutor commands in Grove CLI itself. We can take benefit of the same environment creation logic for the pipeline here as well. Right now tutor.sh is getting larger and larger with all those S3 syncs, env-overrides logic. Which should be decoupled and tested properly.

Testability

We should properly test if Grove CLI generates the correct pipeline for different situations. We can test that and other aspects by mocking external services (Tutor, Terraform, Kubectl), etc. We should create a test pipeline to maintain better code quality.

For testing and code quality following tools can be used -

  • pytest as the unit testing framework
  • black for formatting python code
  • isort for sorting imports

Since we will be moving most of the logic to the new grove-cli and bash scripts will just wrap external commands, we can keep the scope of testing limited to grove-cli.