CC-Agency API

This is a documentation of the available REST API endpoints of CC-Agency.

Before using the CC-Agency API, you should read the Introduction to CC-Agency, especially the section about data security.

Usage examples are provided as Python code. Install and import the following Python modules to prepare for the examples.

pip3 install --user --upgrade requests ruamel.yaml

import requests
from ruamel.yaml import YAML
yaml = YAML(typ='safe')

Endpoint	Methods
/	GET
/token	GET
/version	GET
/red	POST
/experiments/count	GET
/experiments	GET
/experiments/EXPERIMENT_ID	GET
/batches/count	GET
/batches	GET
/batches/BATCH_ID	GET, DELETE
/batches/BATCH_ID/FILENAME	GET
/nodes	GET

GET /

Hello World.

Request (Python):

requests.get(
    'https://example.com/'
)

Response (JSON):

{"Hello": "World"}

GET /token

Request temporary password token, which replaces the actual user password for basic auth.

Hint: This endpoints requires authorization with username and password credentials, not another token.

Request (Python):

requests.get(
    'https://example.com/token',
    auth=('guest', 'guest')
)

Response (JSON):

{
    "token": "d474a3d73a57e5636b070b03eee8ac2a0d8c302798ffd38f",
    "validForSeconds": 86400
}

Token can be used for authorization with other endpoints as follows.

Request (Python):

requests.get(
    'https://example.com/version',
    auth=('guest', 'd474a3d73a57e5636b070b03eee8ac2a0d8c302798ffd38f')
)

GET /version

Installed CC-Core and CC-Agency versions.

Request (Python):

requests.get(
    'https://example.com/version',
    auth=('guest', 'guest')
)

Response (JSON):

{
    "agencyVersion": "9.0.0",
    "coreVersion": "9.0.0"
}

POST /red

Send RED data to CC-Agency, which will result in one database entry in the experiments collection and one or more entries in the batches collection. The agency’s scheduler will trigger the parallel processing of the batches in the distributed cluster of docker-engines.

Additionally required fields:

outputs
container.settings.image.url
container.settings.image.ram

File red.yml:

redVersion: "9"
cli:
  cwlVersion: "v1.0"
  class: "CommandLineTool"
  baseCommand: "grepwrap"
  doc: "Search for query terms in text files."

  inputs:
    query_term:
      type: "string"
      inputBinding:
        position: 0
      doc: "Search for QUERY_TERM in TEXT_FILE."
    text_file:
      type: "File"
      inputBinding:
        position: 1
      doc: "TEXT_FILE containing plain text."
    after_context:
      type: "int?"
      inputBinding:
        prefix: "-A"
      doc: "Print NUM lines of trailing context after matching lines."
    before_context:
      type: "int?"
      inputBinding:
        prefix: "-B"
      doc: "Print NUM lines of leading context before matching lines."

  outputs:
    out_file:
      type: "File"
      outputBinding:
        glob: "out.txt"
      doc: "Query results."

container:
  engine: "docker"
  settings:
    image:
      url: "docker.io/copla/grepwrap"
    ram: 256

inputs:
  query_term: "QU"
  text_file:
    class: "File"
    connector:
      command: "red-connector-http"
      access:
        url: "https://raw.githubusercontent.com/curious-containers/vagrant-quickstart/master/in.txt"
  before_context: 1

outputs:
  out_file:
    class: "File"
    connector:
      command: "red-connector-http"
      access:
        url: "https://example.com/out.txt"

execution:
  engine: "ccagency"
  settings:
    batchConcurrencyLimit: 8

Request (Python):

with open('red.yml') as f:
    red = yaml.load(f)

requests.post(
    'https://example.com/red',
    auth=('guest', 'guest'),
    json=red
)

Response (JSON):

{"experimentId": "5b7b30f2aafce97767a50a95"}

URL arguments:

Argument	Type	Optional	Default	Description
disableRetry	boolean	yes	0 (false)	Debug option to not retry the execution of a failed batch

requests.post(
    'https://example.com/red?disableRetry=1',
    auth=('guest', 'guest'),
    json=red
)

GET /experiments/count

Count experiments stored in database.

Hint: Users without admin priviliges will only receive information about their own database entries.

URL parameters:

Parameter	Type	Optional	Default	Description
username	string	yes		Filter by username.

Request (Python):

requests.get(
    'https://example.com/experiments/count',
    auth=('guest', 'guest')
)

Response (JSON):

{"count": 1}

GET /experiments

List experiments stored in database sorted by registration time in descending order.

Hint: Users without admin priviliges will only receive information about their own database entries.

URL parameters:

Parameter	Type	Optional	Default	Description
skip	integer >= 0	yes	0	Skip a certain number of experiments.
limit	integer >= 1	yes		Limit the amount of experiments returned.
ascending	boolean	yes	False	Sort results by registrationTime in ascending order. Order is descending (newest first) by default.
username	string	yes		Filter by username.

Request (Python):

requests.get(
    'https://example.com/experiments?ascending=true&limit=10',
    auth=('guest', 'guest')
)

GET /experiments/EXPERIMENT_ID

Get full experiment stored in database by its experimentId.

Hint: Users without admin priviliges will only receive information about their own database entries.

Request (Python):

requests.get(
    'https://example.com/experiments/5b7b2c98aafce967d9e2709b',
    auth=('guest', 'guest')
)

GET /batches/count

Count batches stored in database.

Hint: Users without admin priviliges will only receive information about their own database entries.

URL parameters:

Parameter	Type	Optional	Description
username	string	yes	Filter by username.
node	string	yes	Filter by node.
state	enum: registered, scheduled, processing, succeeded, failed, cancelled	yes	Filter by state.
experimentId	string	yes	Filter by experimentId.

Request (Python):

requests.get(
    'https://example.com/batches/count',
    auth=('guest', 'guest')
)

Response (JSON):

{"count": 1}

GET /batches

List batches stored in database sorted by registration time in descending order.

Hint: Users without admin priviliges will only receive information about their own database entries.

URL parameters:

Parameter	Type	Optional	Default	Description
skip	integer >= 0	yes	0	Skip a certain number of experiments.
limit	integer >= 1	yes		Limit the amount of experiments returned.
ascending	boolean	yes	False	Sort results by registrationTime in ascending order. Order is descending (newest first) by default.
username	string	yes		Filter by username.
node	string	yes		Filter by node.
state	enum: registered, scheduled, processing, succeeded, failed, cancelled	yes		Filter by state.
experimentId	string	yes		Filter by experimentId.

Request (Python):

requests.get(
    'https://example.com/batches?ascending=true&limit=10',
    auth=('guest', 'guest')
)

GET /batches/BATCH_ID

Get full batch stored in database by its batchId.

Hint: Users without admin priviliges will only receive information about their own database entries.

Request (Python):

requests.get(
    'https://example.com/batches/5b7b2c98aafce967d9e2709c',
    auth=('guest', 'guest')
)

DELETE /batches/BATCH_ID

Cancel unfinished batch by its batchId. This will not delete their database entries.

Hint: Users without admin priviliges can only cancel their own batches.

Request (Python):

requests.delete(
    'https://example.com/batches/5b7b2c98aafce967d9e2709c',
    auth=('guest', 'guest')
)

GET /batches/BATCH_ID/FILENAME

Get the stdout/stderr of the user process of the batch with the given batchId.

The stdout/stderr files are only provided, if the given batch failed or if the user specified the cli.stdout/cli.stderr files in the RED experiment.

To get the stdout of the user process use /batches/BATCH_ID/stdout. To get the stderr of the user process use /batches/BATCH_ID/stderr. Using the filename(s) specified in the RED file will not work.

Hint: Users without admin priviliges will only receive information about their own database entries.

requests.get(
    'https://example.com/batches/5b7b2c98aafce967d9e2709c/stdout',
    auth=('guest', 'guest')
)

GET /nodes

List docker-engine nodes available in the cluster.

Request (Python):

requests.get(
    'https://example.com/nodes',
    auth=(guest, guest)
)