MyTardis Client API

MyTardis Client provides various Python classes for interacting with MyTardis’s REST API.

mtclient.models.api

The api module contains model classes for MyTardis API v1’s endpoints.

This following lists all of the supported endpoints: /api/v1/?format=json

API functionality available for a particular model can be retrieved with:

/api/v1/facility/schema/?format=json

The ‘schema’ request above requires authentication.

class mtclient.models.api.ApiEndpoint(model, endpoint_dict)

Bases: object

Model class for MyTardis API v1’s endpoints.

static list()

Retrieve a list of API endpoints, encapsulated in an ApiEndpoints object.

The ApiEndpoints object encapsulates the entire JSON response from the /api/v1/ query.

Returns

A list of API endpoints, encapsulated in an ApiEndpoints object.

class mtclient.models.api.ApiEndpoints(response_dict)

Bases: object

Dictionary of API endpoints (list_endpoint and schema) with model names as keys.

class mtclient.models.api.ApiSchema(model, response_dict)

Bases: object

Model class for MyTardis API v1’s schemas.

static get(model)

Get a list of API-accessible functionality for a particular model.

Parameters

model – The name of an API-accessible model, e.g. ‘dataset_file’.

Returns

An ApiSchema object, which encapsulates the list of API-accessible functionality for a particular model.

mtclient.models.config

Model class for the configuration, usually stored in ~/.config/mytardisclient/mytardisclient.cfg

class mtclient.models.config.Config(path='/home/docs/.config/mytardisclient/mytardisclient.cfg')

Bases: object

Model class for the minimal MyTardis server configuration (MyTardis URL, username and API key), usually stored in ~/.config/mytardisclient/mytardisclient.cfg

apikey

The MyTardis API key, e.g. ‘644be179cc6773c30fc471bad61b50c90897146c’

property datasets_path

Location to create symlinks to dataset folders. Default: ~/.config/mytardisclient/servers/[mytardis_hostname]/

property default_headers

Default headers to use for API queries (including API key authorization).

property hostname

Determine the MyTardis hostname from the MyTardis URL

load(path=None)

Sets some default values for settings fields, then loads a config file.

Parameters

path – The path to the config file, usually ~/.config/mytardisclient/mytardisclient.cfg

logging_config_path

The logging config path. Default: ~/.config/mytardisclient/logging.cfg

path

The config file’s location. Default: ~/.config/mytardisclient/mytardisclient.cfg

save(path=None)

Saves the configuration to disk.

Parameters

path – The path to save to, usually ~/.config/mytardisclient/mytardisclient.cfg

url

The MyTardis URL, e.g. ‘http://mytardisdemo.erc.monash.edu.au’

username

The MyTardis username, e.g. ‘demofacility”

validate()

Ensure that the config contains a non-empty username, API key and MyTardis URL.

mtclient.models.facility

Model class for MyTardis API v1’s FacilityResource.

class mtclient.models.facility.Facility(response_dict)

Bases: mtclient.models.model.Model

Model class for MyTardis API v1’s FacilityResource.

static get(**kwargs)

Get facility by ID

Parameters

**kwargs – See below

Keyword Arguments

  • id (int) – ID of the Facility to retrieve

Returns

A Facility record.

Raises

requests.exceptions.HTTPError –

static list(filters=None, limit=None, offset=None, order_by=None)

Retrieve a list of facilities.

Parameters

  • filters – Filters, e.g. “id=123” or “name=Test Facility”

  • limit – Maximum number of results to return.

  • offset – Skip this many records from the start of the result set.

  • order_by – Order by this field.

Returns

A list of Facility records, encapsulated in a ResultSet object`.

mtclient.models.instrument

Model class for MyTardis API v1’s InstrumentResource.

class mtclient.models.instrument.Instrument(response_dict)

Bases: mtclient.models.model.Model

Model class for MyTardis API v1’s InstrumentResource.

static create(facility_id, name)

Create an instrument record.

Parameters

  • facility_id – The ID of the facility to create the instrument in.

  • name – The name of the instrument.

Returns

A new Instrument record.

static get(**kwargs)

Retrieve a single instrument record

Parameters

**kwargs – See below

Keyword Arguments

  • id (int) – ID of the Instrument to retrieve

Returns

A Instrument record.

Raises

requests.exceptions.HTTPError –

static list(filters=None, limit=None, offset=None, order_by=None)

Retrieve a list of instruments

Parameters

  • filters – Filters, e.g. “id=123” or “facility__id=45”

  • limit – Maximum number of results to return.

  • offset – Skip this many records from the start of the result set.

  • order_by – Order by this field.

Returns

A list of Instrument records, encapsulated in a ResultSet object.

static update(instrument_id, name)

Update an instrument record.

Parameters

  • instrument_id – The ID of the instrument record to update.

  • name – The new name of the instrument.

Returns

An updated Instrument record.

mtclient.models.experiment

Model class for MyTardis API v1’s ExperimentResource.

class mtclient.models.experiment.Experiment(response_dict, include_metadata=False)

Bases: mtclient.models.model.Model

Model class for MyTardis API v1’s ExperimentResource.

static create(title, description='', institution=None, params_file_json=None)

Create an experiment record.

Parameters

  • title – The title of the experiment.

  • description – The description of the experiment.

  • institution – The institution of the experiment.

  • params_file_json – Path to a JSON file with experiment parameters.

Returns

A new Dataset record.

static get(**kwargs)

Retrieve a single experiment record

Parameters

**kwargs – See below

Keyword Arguments

  • id (int) – ID of the Experiment to retrieve

Returns

An Experiment record.

Raises

requests.exceptions.HTTPError –

static list(filters=None, limit=None, offset=None, order_by='-created_time')

Retrieve a list of experiments.

Parameters

  • filters – Filters, e.g. “title=Exp Title”

  • limit – Maximum number of results to return.

  • offset – Skip this many records from the start of the result set.

  • order_by – Order by this field.

Returns

A list of Experiment records, encapsulated in a ResultSet object.

static update(experiment_id, title, description)

Update an experiment record.

class mtclient.models.experiment.ExperimentParameter(response_dict)

Bases: object

Model class for MyTardis API v1’s ExperimentParameterResource.

static list(filters=None, limit=None, offset=None, order_by=None)

List experiment parameter records in parameter set.

class mtclient.models.experiment.ExperimentParameterSet(response_dict)

Bases: object

Model class for MyTardis API v1’s ExperimentParameterSetResource.

static list(filters=None, limit=None, offset=None, order_by=None)

Retrieve a list of experiment parameters.

Parameters

filters – Filters, e.g. “experiments__id=123”

Returns

A list of ExperimentParameterSet records, encapsulated in a ResultSet object`.

mtclient.models.dataset

Model class for MyTardis API v1’s DatasetResource.

class mtclient.models.dataset.Dataset(response_dict=None, include_metadata=False)

Bases: mtclient.models.model.Model

Model class for MyTardis API v1’s DatasetResource.

static create(experiment_id, description, instrument_id=None, params_file_json=None)

Create a dataset record.

Parameters

  • experiment_id – The ID of the experiment to create the dataset in.

  • description – The description of the dataset.

  • instrument_id – The instrument the data was collected on.

  • params_file_json – Path to a JSON file with dataset parameters.

Returns

A new Dataset record.

static download(dataset_id)

Download a dataset

static get(**kwargs)

Retrieve a single dataset record

Parameters

**kwargs – See below

Keyword Arguments

  • id (int) – ID of the Dataset to retrieve

Returns

A Dataset record.

Raises

requests.exceptions.HTTPError –

static list(filters=None, limit=None, offset=None, order_by=None)

Retrieve a list of datasets.

Parameters

  • filters – Filters, e.g. “experiments__id=123&description=Dataset Description”

  • limit – Maximum number of results to return.

  • offset – Skip this many records from the start of the result set.

  • order_by – Order by this field.

Returns

A list of Dataset records, encapsulated in a ResultSet object.

static update(dataset_id, description)

Update an dataset record.

class mtclient.models.dataset.DatasetParameter(response_dict)

Bases: object

Model class for MyTardis API v1’s DatasetParameterResource.

static list(filters=None, limit=None, offset=None, order_by=None)

List dataset parameter records in parameter set.

class mtclient.models.dataset.DatasetParameterSet(response_dict)

Bases: object

Model class for MyTardis API v1’s DatasetParameterSetResource.

static list(filters=None, limit=None, offset=None, order_by=None)

Retrieve a list of dataset parameters.

Parameters

filters – Filters, e.g. “datasets__id=123”

Returns

A list of DatasetParameterSet records, encapsulated in a ResultSet object`.

mtclient.models.datafile

Model class for MyTardis API v1’s DataFileResource.

class mtclient.models.datafile.DataFile(response_dict, include_metadata=False)

Bases: mtclient.models.model.Model

Model class for MyTardis API v1’s DataFileResource.

static create(dataset_id, storagebox, dataset_path, path, check_local_paths=True, create_dataset_symlink=True)

Create one or more DataFile records, depending on whether the supplied path is a single file or a directory.

Parameters

  • dataset_id – The ID of the dataset to create the datafile record(s) in.

  • storagebox – The storage box containing the datafile(s).

  • dataset_path – The path to the directory which is to be mapped to a MyTardis dataset, meaning that files immediately inside this directory will have DataFile records created with the “directory” field set to “”, and files within dataset_path/subdir1/ will have DataFile records creatd with the “directory” field set to “subdir1” etc. The MyTardis Client will create a symlink to dataset_path in ~/.config/mytardisclient/servers/[mytardis_hostname]/ which will enable MyTardis to verify and ingest the file if the filesystem running mytardisclient is accessible to MyTardis, e.g. by SSHFS.

  • path – The path to a file to be represented in the DataFile record or to a directory containing the files to create records for. The subdirectory (‘subdir1’) to be recorded in the DataFile record(s) will be determined automatically by compareing the dataset_path with the path.

static create_datafile(dataset_id, storagebox, dataset_path, file_path, return_new_datafile=True, check_local_paths=True, create_dataset_symlink=True, size=None, md5sum=None, mimetype=None)

Create a DataFile record.

Parameters

  • dataset_id – The ID of the dataset to create the datafile in.

  • storagebox – The storage box containing the datafile.

  • dataset_path – The path to the directory which is to be mapped to a MyTardis dataset, meaning that files immediately inside this directory will have DataFile records created with the “directory” field set to “”, and files within dataset_path/subdir1/ will have DataFile records creatd with the “directory” field set to “subdir1” etc. The MyTardis Client will create a symlink to dataset_path in ~/.config/mytardisclient/servers/[mytardis_hostname]/ which will enable MyTardis to verify and ingest the file if the filesystem running mytardisclient is accessible to MyTardis, e.g. by SSHFS.

  • file_path – The local path to the file to be represented in the DataFile record. The subdirectory (‘subdir1’) to be recorded in the DataFile record(s) will be determined automatically by compareing the dataset_path with the file_path.

Returns

A new DataFile record.

See also: mtclient.models.datafile.DataFile.upload()

Suppose someone with username james generates a file called “results.dat” on a data analysis server called analyzer.example.com in the directory ~james/analysis/dataset1/. User james could grant the MyTardis server temporary access to his account on analyzer.example.com and then have MyTardis copy the file(s) into a more permanent location.

If james agrees to allow the MyTardis server to do so, it could SSHFS-mount james@analyzer.example.com:/home/james/analysis/, e.g. at /mnt/sshfs/james-anaylzer/

Then user james doesn’t need to upload results.dat, he just needs to tell MyTardis how to access it, and tell MyTardis that it is not yet in a permanent location.

MyTardis’s default storage box model generates datafile object identifiers which include a dataset description (e.g. ‘dataset1’) and a unique ID, resulting in path like ‘dataset1-123/results.dat’ for the datafile object. Because user james doesn’t want to have to create the ‘dataset1-123’ folder himself, he could entrust the MyTardis Client to do it for him.

The MyTardis administrator can create a storage box for james called “james-analyzer” which is of type “receiving”, meaning that it is a temporary location. The storage box record (which only needs to be accessed by the MyTardis administrator) would include a StorageBoxOption with key ‘location’ and value ‘/mnt/sshfs/james-analyzer’.

Once james knows the dataset ID of the dataset he wants to upload to (123 in this case), he can create a DataFile record as follows:

mytardis datafile create 123 –storagebox=james-analyzer ~/analysis/dataset1/results.dat

The file_path argument (set to ~/analysis/dataset1/results.dat) specifies the location of ‘results.dat’ on the analysis server.

To enable the MyTardis server to access (and verify) the file via SSHFS / SFTP, a symbolic link can be created in ~james/.mytardisclient/servers/[mytardis_hostname]/, named “dataset1-123” pointing to the location of ‘results.dat’, i.e. ~james/analysis/dataset1/.

static create_datafiles(dataset_id, storagebox, dataset_path, dir_path)

Create a DataFile record for each file within the dir_path directory.

Parameters

  • dataset_id – The ID of the dataset to create the datafile record(s) in.

  • storagebox – The storage box containing the datafile(s).

  • dataset_path – The path to the directory which is to be mapped to a MyTardis dataset, meaning that files immediately inside this directory will have DataFile records created with the “directory” field set to “”, and files within dataset_path/subdir1/ will have DataFile records creatd with the “directory” field set to “subdir1” etc. The MyTardis Client will create a symlink to dataset_path in ~/.config/mytardisclient/servers/[mytardis_hostname]/ which will enable MyTardis to verify and ingest the file if the filesystem running mytardisclient is accessible to MyTardis, e.g. by SSHFS.

  • dir_path – The path to a directory containing file(s) to create DataFile records for. If the dir_path is relative (not absolute) path, e.g. ‘dataset1/subdir1’, then the MyTardis Client will assume that the first component of the path (e.g. ‘dataset1/’) is the local dataset path, and create a symlink to this path in ~/.config/mytardisclient/servers/[mytardis_hostname]/ which will enable MyTardis to verify and ingest its file(s). If dir_path is an absolute path, e.g. ‘/home/james/dataset1/subdir1/’, then the dataset_path argument must be used to specified the dataset path, e.g. ‘/home/james/dataset1’.

static download(datafile_id, basedir=None, overwrite=False, force_overwrite=False)

Download datafile with id datafile_id

Parameters

  • datafile_id – The ID of a datafile to download.

  • basedir – If specified, the datafile will be downloaded to the path obtained by joining basedir with the DataFile’s directory field.

  • overwrite – If set to True, existing files will be re-downloaded and overwritten without asking for confirmation if their file size is wrong.

  • force_overwrite – If set to True, existing files will be re-downloaded and overwritten without asking for confirmation, even if their file size is correct.

static exists(dataset_id, directory, filename)

If MyTardis is running with DEBUG=False, then we won’t be able detect duplicate key errors easily, we will just receive a generic HTTP 500 from the MyTardis API. This method checks whether a DataFile record already exists for the supplied dataset_id, directory and filename.

Parameters

  • dataset_id – The ID of the dataset to check existence in.

  • directory – The directory within the dataset to check existence in.

  • filename – The filename to check for existence.

Returns

True if a matching DataFile record already exists.

static get(**kwargs)

Retrieve a single datafile record

Parameters

**kwargs – See below

Keyword Arguments

  • id (int) – ID of the DataFile to retrieve

Returns

A DataFile record.

Raises

requests.exceptions.HTTPError –

static list(filters=None, limit=None, offset=None, order_by=None)

Retrieve a list of datafiles.

Parameters

  • filters – Filters, e.g. “dataset__id=123&filename=file1.txt”

  • limit – Maximum number of results to return.

  • offset – Skip this many records from the start of the result set.

  • order_by – Order by this field.

Returns

A list of DataFile records.

static update(datafile_id, md5sum)

Update a DataFile record.

Parameters

  • datafile_id – The ID of a datafile to be updated.

  • md5sum – The new MD5 sum value.

This method is not usable yet, because the MyTardis API doesn’t yet allow update_detail to be performed on DataFile records.

For a large file, its upload can commence before the local MD5 sum calculation is complete, i.e. the DataFile record can be initially created with a bogus checksum which is later updated using this method.

static upload(dataset_id, storagebox, dataset_path, file_path)

Upload datafile to dataset with ID dataset_id, using HTTP POST.

Parameters

  • dataset_id – The ID of the dataset to create the datafile in.

  • dataset_path – The path to the directory which is to be mapped to a MyTardis dataset, meaning that files immediately inside this directory will have DataFile records created with the “directory” field set to “”, and files within dataset_path/subdir1/ will have DataFile records creatd with the “directory” field set to “subdir1” etc. The MyTardis Client will create a symlink to dataset_path in ~/.config/mytardisclient/servers/[mytardis_hostname]/ which will enable MyTardis to verify and ingest the file if the filesystem running mytardisclient is accessible to MyTardis, e.g. by SSHFS.

  • file_path – The local path to the file to be represented in the DataFile record. If dataset_path is not specified, file_path must be a relative (not absolute) path, e.g. ‘dataset1/subdir1/datafile1.txt’.

property verified

All replicas (DFOs) must be verified and there must be at least one replica (DFO).

static verify(datafile_id)

Ask MyTardis to verify a datafile with id datafile_id

Parameters

datafile_id – The ID of a datafile to be verified.

class mtclient.models.datafile.DataFileParameter(response_dict)

Bases: object

Model class for MyTardis API v1’s DataFileParameterResource.

static list(filters=None, limit=None, offset=None, order_by=None)

List datafile parameter records in parameter set.

Parameters

datafile_param_set – The datafile parameter set to list parameters for.

class mtclient.models.datafile.DataFileParameterSet(response_dict)

Bases: object

Model class for MyTardis API v1’s DataFileParameterSetResource.

static list(filters=None, limit=None, offset=None, order_by=None)

Retrieve a list of datafile parameter sets

Parameters

  • filters – Filters, e.g. “datafiles__id=12345”

  • limit – Maximum number of results to return.

  • offset – Skip this many records from the start of the result set.

  • order_by – Order by this field.

Returns

A list of DatasetParameterSet records, encapsulated in a ResultSet object`.

mtclient.models.datafile.md5_sum(file_path, blocksize=65536)

Calculate MD5 checksum without reading the whole file into memory.

mtclient.models.schema

Model class for MyTardis API v1’s SchemaResource.

class mtclient.models.schema.ParameterName(response_dict)

Bases: mtclient.models.model.Model

Model class for MyTardis API v1’s ParameterNameResource.

static get(**kwargs)

Retrieve a single parameter name record

Parameters

**kwargs – See below

Keyword Arguments

  • id (int) – ID of the ParameterName to retrieve

Returns

A ParameterName record.

static list(filters=None, limit=None, offset=None, order_by=None)

Retrieve a list of parameter names.

Parameters

  • filters – Filters, e.g. “schema__id=123” or “name=ParamName”

  • limit – Maximum number of results to return.

  • offset – Skip this many records from the start of the result set.

  • order_by – Order by this field.

Returns

A list of ParameterName records, encapsulated in a ResultSet object`.

class mtclient.models.schema.Schema(response_dict, param_names=False)

Bases: mtclient.models.model.Model

Model class for MyTardis API v1’s SchemaResource.

static get(**kwargs)

Retrieve a single schema record

Parameters

**kwargs – See below

Keyword Arguments

  • id (int) – ID of the Schema to retrieve

Returns

A Schema record.

Raises

requests.exceptions.HTTPError –

static list(filters=None, limit=None, offset=None, order_by=None)

Retrieve a list of schemas.

Parameters

  • filters – Filters, e.g. “id=123” or “namespace=NameSpace”

  • limit – Maximum number of results to return.

  • offset – Skip this many records from the start of the result set.

  • order_by – Order by this field.

Returns

A list of Schema records, encapsulated in a ResultSet object`.

mtclient.models.storagebox

Model class for MyTardis API v1’s StorageBoxResource.

class mtclient.models.storagebox.StorageBox(response_dict)

Bases: mtclient.models.model.Model

Model class for MyTardis API v1’s StorageBoxResource.

static get(**kwargs)

Retrieve a single storage box record

Parameters

**kwargs – See below

Keyword Arguments

  • id (int) – ID of the StorageBox to retrieve

Returns

A StorageBox record.

Raises

requests.exceptions.HTTPError –

static list(filters=None, limit=None, offset=None, order_by=None)

Retrieve a list of storage boxes.

Parameters

  • filters – Filters, e.g. “id=123” or “name=box1”

  • limit – Maximum number of results to return.

  • offset – Skip this many records from the start of the result set.

  • order_by – Order by this field.

Returns

A list of StorageBox records, encapsulated in a ResultSet object`.

class mtclient.models.storagebox.StorageBoxAttribute(response_dict)

Bases: object

Model class for MyTardis API v1’s StorageBoxAttributeResource.

class mtclient.models.storagebox.StorageBoxOption(response_dict)

Bases: object

Model class for MyTardis API v1’s StorageBoxOptionResource.

mtclient.models.resultset

This module contains the ResultSet class, an abstraction to represent the JSON returned by the MyTardis API, particularly for queries which return multiple records and could be subject to pagination.

class mtclient.models.resultset.ResultSet(model, url, response_dict)

Bases: object

Abstraction to represent JSON returned by MyTardis API which includes a list of records and some meta information e.g. whether there are additional pages of records to retrieve.

static empty(model)

Return an empty result set

mtclient.views

Views for MyTardis records

mtclient.views.render(data, render_format='table', display_heading=True)

Generic render function.

Calls a more specific render function depending on the data type to display (render) the data in the desired format.

Parameters

  • data – The data to be displayed. An instance of a model class (e.g. mtclient.models.dataset.Dataset) or an instance of mtclient.models.resultset.ResultSet or an instance of mtclient.models.api.ApiEndpoints.

  • render_format – The format to display the data in (‘table’ or ‘json’).

  • display_heading – When using the ‘table’ render format for a ResultSet containing multiple records, setting display_heading to True ensures that the meta information returned by the query is summarized in a ‘heading’ before displaying the table. This meta information can be used to determine whether the query results have been truncated due to pagination.

mtclient.views.render_result_set(result_set, render_format, display_heading=True)

Render result set.

Calls a more specific render function depending on the type of data stored within the ResultSet to display (render) the data in the desired format.

Parameters

  • result_set (mtclient.models.resultset.ResultSet) – The result set to be rendered.

  • render_format – The format to display the data in (‘table’ or ‘json’).

  • display_heading – When using the ‘table’ render format for a ResultSet containing multiple records, setting display_heading to True ensures that the meta information returned by the query is summarized in a ‘heading’ before displaying the table. This meta information can be used to determine whether the query results have been truncated due to pagination.

mtclient.views.render_single_record(data, render_format)

Render single record.

Calls a more specific render function depending on the data type to display (render) the data in the desired format.

Parameters

  • data – The data to be displayed. An instance of a model class (e.g. mtclient.models.dataset.Dataset).

  • render_format – The format to display the data in (‘table’ or ‘json’).

mtclient.conf

Defines the singleton configuration instance of mtclient.models.config.Config.

It can be imported as follows:

from mtclient.conf import config