Skip to content

Installing tutor plugins per instance

In this document we consider the effort and implications required for each Grove instance to have it's own set of tutor plugins.

Rationale

At OpenCraft we expect to deploy multiple instances on a single cluster. These instances will have different requirements for their tutor configuration. The plugin list could be different, while it might also be needed to run different versions of tutor (especially in the case of upgrading between releases).

Prior art

eduNEXT created called Tutor Version Manager for managing tutor and Python environments. It is well suited to the task.

Unfortunately, there are architectural decisions that preclude it's inclusion into Grove.

  1. It has to be installed globally. While unlikely, it's possible that different versions will be needed per instance, which is not possible.
  2. The app state is stored globally under a .tvm directory. This means that deleting an instance will be more complicated as Grove will need to run the requisite tvm commands when deleting an instance. In the unlikely event that the .tvm directory is corrupted, all instances are affected.
  3. It's an extra dependency and dependencies are bad (not really, but we try to limit them where possible).

Suggested solution

Python3 has built in virtual environments via the venv module. By utilising this module to store virtual environments in each instance directory, it becomes possible to have completely separate Grove instances from a Python point of view.

Within an instance's directory, we'll create a .venv directory which will house the virtual environment. By default, we will install tutor, it's related plugins and the tutor-grove plugin as they are required for building an instance.

Any command within Grove that calls tutor will need to be changed to point to the instance specific directory. Since the .venv directory needs to be managed, it is required to create grove-cli commands to manage it. The .venv folder will not be committed to git.

Since each instance, will have it's own list of requirements, we envision that a grove-requirements.txt will be created within each instance directory as well. This file will be tracked in git.

Technical Details: How to implement this

Add Grove CLI commands

We will create commands to manage the virtual environment under the venv namespace.

  • ./grove venv update [instance-name] => Updates the virtual environment and installs the packages in grove-requirements.txt. It the virtual environmenth doesn't exist, it is created.
  • ./grove venv reset [instance-name] => Delete and recreates the virtual environment. In cases where the virtual environment becomes invalid somehow (eg. the user installed incompatible packages), this command will recreate it.
  • ./grove venv activate [instance-name] => Starts a shell in docker with the relevant environment activated.

Replace all tutor invocations to use the virtual environment

tutor is invoked in various places in Grove. We will need to replace them with the equivalent virtual environment command.

Please note that the above is not an exhaustive list. It will be up to the implementor to find all instances.

Update pipelines to build and use the virtual environment

Now that tutor's running within a virtual environment, we will need to make sure this virtual environment exists when running deployment pipelines. For the create, update and delete pipelines we will need to build the virtual environment.

Since multiple images can be built in a single deploy, creating the virtual environment at the time of building the image isn't efficient. For this reason, we suggest adding an extra pipeline step to build the virtual environment, which will be shared to any later pipelines.

Remove tutor and it's plugins from Grove

At this point, tutor can be remove from Grove as their is no longer a direct dependency. Remove all tutor-related, requirements from the tools-container itself and add them to the defaults directory in grove-template.