- Generate deployment pipelines
- Wrap Tutor
- Prepare dynamic environment for wrapped CLI commands
- Increase testability
- Increase reusability
Right now there are several different scripts in the
tools-container/scripts directory. Some are written in bash, some are in python. All those scripts have dependencies with environment variables and all those checking logic is everywhere. All the logic in these scripts is not tested. Scripts are getting messy and hard to manage.
We propose a new python CLI package -
grove-cli. It will be created under the
tools-container/grove-cli folder. This single cli will be responsible for doing all Grove-related things. We can use
typer to create API interfaces for this new CLI tool.
The project structure for the proposed package will be following:
| |-- templates
| | |-- pipelines
| | |-- base_image.yml
| | |-- common.yml
| | |-- dummy.yml
| | |-- openedx.yml
| | |-- terraform.yml
| |-- instance.py
| |-- const.py
| |-- terraform.py
| |-- type_defs.py
| |-- handlers.py
| |-- pipeline.py
| |-- utils.py
| |-- exceptions.py
cli.pyfile will be the entry point of this package and holds all CLI commands.
pipeline.pyfile holds the logic for generating different pipeline jobs.
utils.pyfile will contain some utility functions like opening Grove configuration file, parsing commit message format, etc.
const.pyfile will contain consts like workspace path, script dir path, template path, etc.
instance.pyfile will contain logic for creating new instances, pre, and post-deployment activities, syncing tutor env directory, etc.
terraform.pyfile will contain a terraform wrapper.
testsdirectory will contain tests.
We can create a
grove.sh file (contains logic to run this cli without installing using pip) in scripts and put it in
/usr/local/bin for easy access. This approach will help us during development as well and we don't need to run
pip install every time we change something in the package files.
Create a new Open edX instance¶
grove new $INSTANCE_NAME
This command will copy the default configuration and create a new instance
$INSTANCE_NAME in the
instances/$INSTANCE_NAME directory. This command should not have any dependency on Tutor or Terraform. This raises an error if the instance is already present.
grove prepare $INSTANCE_NAME
This command is similar to the
new command. Only difference is that it creates the instance if the instance isn't exist and updates the existing one. We will use this from GitLab CI mostly. This command will use Tutor to generate the initial configuration. It doesn't require Terraform but it will use it if the infrastructure is ready for the instance.
grove pipeline $COMMIT_TITLE $GENERATED_FILE_PATH
$COMMIT_TITLE, Grove CLI will generate the pipeline for it and write it on
$GENERATED_FILE_PATH. This command will be used by GitLab CI to generate a child pipeline based on commit messages. Check commit based pipeline for more info.
Generate pipeline on GitLab or GitHub webhook call¶
The logic for handling webhook triggers will be inside Grove CLI as well.
grove webhookpipeline --output $GENERATED_FILE_PATH
By handling triggers via the CLI itself will help us create a dynamic pipeline without any Commit. Since we are generating a pipeline using the CLI, we can reuse that logic without creating a commit message and create a deployment pipeline directly from a trigger without any commits.
Handling triggers via the CLI itself will help us create a dynamic pipeline without pushing empty commits. This is going to simplify handling batch redeployments and redeploying instances without any configuration changes.
Sync Tutor Env directory¶
Following command will pull env from s3, render env-overrides dir, runs
tutor config save, and pushes changes to s3.
grove tutor sync $INSTANCE_NAME
grove predeploy $INSTANCE_NAME
This will do the preparation work for
$INSTANCE_NAME before deployment, ex: clone template, override SCSS, etc.
Deploy an Open edX instance¶
grove deploy $INSTANCE_NAME
This will perform deployment for
grove postdeploy $INSTANCE_NAME
This will perform post-deployment steps for
$INSTANCE_NAME, ex: enable theme, force pull updated images in k8s, etc.
grove buildimage $INSTANCE_NAME $IMAGE_NAME --cache_from $CACHE_FROM
This wraps the
tutor images build command with an easy interface.
Run any Tutor command¶
grove tutor exec $INSTANCE_NAME $ANY_COMMAND_AS_STRING
It would be useful to wrap Tutor commands in Grove CLI itself. We can take benefit of the same environment creation logic for the pipeline here as well. Right now
tutor.sh is getting larger and larger with all those S3 syncs,
env-overrides logic. Which should be decoupled and tested properly.
We should properly test if Grove CLI generates the correct pipeline for different situations. We can test that and other aspects by mocking external services (Tutor, Terraform, Kubectl), etc. We should create a test pipeline to maintain better code quality.
For testing and code quality following tools can be used -
pytestas the unit testing framework
blackfor formatting python code
isortfor sorting imports
Since we will be moving most of the logic to the new
grove-cli and bash scripts will just wrap external commands, we can keep the scope of testing limited to