Development

The following instructions have been tested on Fedora 28 with Python 3.6. Instructions for other Linux distributions should be similar.

All software components have dependencies defined in pyproject.toml and pyproject.lock files (see PEP 518). Poetry is used to work with these files, to automatically create virtual environments and to build and deploy the resulting Python packages.

System Dependencies

Curious Containers requires a Linux distribution of your choice and the latest stable version of Docker to be installed. It is recommended to install Docker-CE from the official external Docker repository (see instructions), because Docker packages shipped by Linux distributions are often outdated.

In addition to Docker-CE, install the following system packages.

sudo dnf install git python3-pip python3-venv 

Python Dependencies

Use pip3 to install poetry, which will handle all the Python dependencies of each package.

pip3 install --user --upgrade poetry

Git Repositories

Clone the git repostories of the software components and place them in a folder next to each other.

git clone https://github.com/curious-containers/cc-core.git
git clone https://github.com/curious-containers/cc-faice.git
git clone https://github.com/curious-containers/cc-agency.git

CC Core

Use poetry to install Python dependencies in a virtual environment.

cd cc-core
poetry install

Run CLI modules.

poetry run python3 -m cc_core.agent --help

Most CLI modules in the Curious Containers ecosystem provide subcommands, which have their own --help flags for detailed usage information.

poetry run python3 -m cc_core.agent red --help

CC FAICE

Use poetry to install Python dependencies in a virtual environment.

cd cc-faice
poetry install

Run CLI modules.

PYTHONPATH=../cc-core poetry run python3 -m cc_faice --help

CC Agency

Install additional system packages.

sudo dnf install uwsgi uwsgi-plugin-python3 docker-compose

Use poetry to install Python dependencies in a virtual environment.

cd cc-agency
poetry install

Run the following components in separate terminals.

Terminal 1 - MongoDB

Run MongoDB in a Docker container.

docker-compose -f dev/docker-compose.yml up

The created database is stored in the local filesystem (see dev/docker-compose.yml for details) and will persist if the container is deleted.

A MongoDB admin user account is created automatically by a mongo-seed container defined in dev/docker-compose.yml. The credentials are read from dev/cc-agency.yml.

Terminal 2 - CC-Agency Controller

You can only run one instance of CC-Agency Controller at a time. It provides the central scheduling component, which connectes to a cluster of docker-engines.

PYTHONPATH=../cc-core poetry run python3 -m cc_agency.controller -c dev/cc-agency.yml

Terminal 3 - CC-Agency Broker

CC-Agency Broker provides a REST API, to schedule RED experiments, receive agent callbacks and to query information. It informs the Controller about changes via a ZMQ socket. Edit dev/uwsgi.ini to increase the number of Broker processes or threads.

PYTHONPATH=../cc-core poetry run uwsgi --ini dev/uwsgi.ini

Create Users

Create users to authenticate with the CC-Agency Broker REST API, with or without admin privileges. Admin privileges can change the behaviour of certain API endpoints.

PYTHONPATH=../cc-core poetry run python3 -m cc_agency.tools create-broker-user -c dev/cc-agency.yml

Create CC-Core Image

The conf file dev/cc-agency.yml references a cc-core Docker image. You can build it locally.

docker build -t cc-core ../cc-core

Run a container based on this image, to check if everything is working.

docker run -u 1000:1000 cc-core ccagent --help

Reset Database

If you need to reset the database during development, run the following command and specify the collections to be dropped.

COLLECTIONS="experiments batches users tokens block_entries callback_tokens"
PYTHONPATH=../cc-core poetry run python3 -m cc_agency.tools drop-db-collections -c dev/cc-agency.yml ${COLLECTIONS}