Enhancements and Cleanup:

ISA MODEL

• The model.py file has been split into the model module. Each class has been split into a dedicated file.
• Added a method from_dict() to the ISA Investigation class to let users load objects from a dictionary.
• Added a method to_dict() to each ISA model class that let users dump ISA objects to dictionaries. (also see ISA JSONLD section)
• Added a new GraphQL interface to ISA investigation (see below querying isa for more information)

ISA TAB

• The isatab.py file has been split into the isatab module.
  Functions and classes have been split into three submodules: load (to read isatab files), dump (to write isafiles) and validate (to run validation rules against an isatab file). Usage remains the same.
• The isatab validation pipeline have been reworked to allow the rules set to be extended with user custom rules. Please contact us if you would like to implement your own validation rules.

ISA JSON

• the isajson.py file has been split into the isajson module.
  Functions and classes have been split into three files: load.py, dump.py and validate.py. Usage remains the same but there is now an alternative using dicts instead of files (see below)
• load.py and dump.py are now wrappers. The code of the isa json serializer/deserializer has been moved to be part of the ISA model. Users can still use the old load() and dump() methods but are now able to do it directly from an investigation object. Example

from json import load
from isatools.model import Investigation

isa_json_file = 'path/to/isa.json'  # the json file path
with open(isa_json_file, 'r') as f:
    isa_json_input = load(f)  # load the dict from the file
investigation = Investigation()  # create en investigation
investigation.from_dict(isa_json_input)  # feed in the dictionary, will generated missing identifiers make inputs and outputs different
isa_json_output = investigation.to_dict()  # Dump it back to a dict
another_investigation = Investigation()  # create en investigation
another_investigation.from_dict(isa_json_output)  # feed in the dictionary
assert investigation == another_investigation  # assert equality between the created investigations.
assert investigation.to_dict() == another_investigation.to-dict()  # compare to_dict() methods

Note: identifiers may be missing from the json file and autogenerated by the ISA objects. In this case, output dictionnaries won't match which is why the investigation is being compared to another investigation.

ISA JSON-LD

• Using the new to_dict() function from model.py, it is possible to produce JSON-LD.
  The The method takes an optional boolean parameter ld (default to False) that serializes to json-ld dictionaries by injecting @context, @id and @type attributes. See isa json below for more information.
• Customisation of the JSON-LD serialization is achieved through the set_context() function of the model module.
• Added support for SIO, WD, OBO and SDO annotations (see https://github.com/ISA-tools/isa-api/tree/master/isatools/resources/json-context).
• Added support for generating URL identifiers (necessary for hosting triple stores).
• Check https://github.com/ISA-tools/isa-api/blob/master/isa-cookbook/content/notebooks/querying-isa-with-graphql-and-sparql.ipynb for examples.

SPARQL querying over ISA:

• The outputs of the ISA JSON LD serializer have been tested as a valid RDF representation. Using MTBLS instances with rdflib and settting the contexts with set_context(vocab='wd', all_in_one=False, local=False, include_contexts=False) we were able to produce RDF triples and store them into a virtuosoDB and a berkeleyDB with SPARQL support.
• Check https://github.com/ISA-tools/isa-api/blob/master/isa-cookbook/content/notebooks/querying-isa-with-graphql-and-sparql.ipynb for examples.

GRAPHQL querying over ISA:

• Each ISA Investigation now has a graphQL interface bound to it.
  A full documentation of the graphQL interface is available at https://github.com/ISA-tools/isa-api/tree/master/isatools/graphQL/README.md

MTBLS, the EMBL-EBI MetaboLights Client:

• Refactored the MTBLS client but kept the same functions (and signature) exposed. You can now use MTBLInvestigation instead of the old functions. Example:

from isatools.net.mtbls import MTBLSInvestigation

investigation =  MTBLSInvestigation('MTBLS1')
# This creates a temp dir for the investigation that gets deleted when the object is destroyed through the automatic garbage collector. You can pass a directory if you want to persist the investigation.
investigation =  MTBLSInvestigation(mtbls_id='MTBLS1', output_directory="path/to/output/dir", output_format='tab')

# You can then run all the old function through methods with the same name and chain methods without downloading the whole investigation again.
print(investigation.get_characteristics_summary())
print(investigation.get_factor_names())

ISA SQL:

• Created a relational SQL Alchemy model for ISA objects tested with PSQL and SQLite. All model files are available here: https://github.com/ISA-tools/isa-api/tree/master/isatools/database/models
  

Other

Versions update:

• Dropped python 3.6 and 3.7
• Added compatibility for python 3.10 (production ready)
• Added compatibility for python 3.11 (test only)
• Bump numpy, pandas and mzml.

Cookbook

• Added a cookbook for querying isa with sparql and grapql. See https://github.com/ISA-tools/isa-api/blob/master/isa-cookbook/content/notebooks/querying-isa-with-graphql-and-sparql.ipynb

Testing:

• The application has been further tested with CI, especially the model. Global coverage increased from 75% to 84%.
• Build time was reduced due to mocking some network operations.

ISA-API performance profiler:

• Added a small command line interface using cProfile to generate performance reports of different ISA features (in particular serialization and deserialization to tab and json).

Known issues:

• There is a know performance issue with the coveralls/coverage modules on python 3.11 making the build very slow (see nedbat/coveragepy#1287). Performances for end users shouldn't be affected.

Contributors:

• Batista Dominique, University of Oxford (https://orcid.org/0000-0002-2109-489X)
• Philipoe Rocca Serra, University of Oxford (https://orcid.org/0000-0001-9853-5668)

Release v0.13.0 - Release Candidate v2

Enhancements and Cleanup:

• significant optimisation of isatab.dump() via reworking the study/assay graph implementation. This can improve up to 3 or 4 orders of magnitude the writing to ISA-tab for datasets with thousands of samples (PR #403). An additional tweak has been implemented to compute only once the study/assay process graph (8c2d09d)
• All the major objects of the ISA models are now Commentable (PR #395)
• isatools.create.connectors now supports both strings and Ontology Annotations for Subject Type and Sample Type (PR #405)
• CI is now migrated from Travis CI to GitHub CI (#404)

Breaking changes:

• Changes to the isatools.create.connectors module to align it with the Datascriptor study JSON config. Now the Datascriptor study config has a "design" property holding all the study design information (PR #405)

Bug Fixes:

• all names for protocols in studies generated by isatools.create.connectors from Datascriptor study configs are unique (issue #398)
• identifiers for studies and investigations are coerced to strings (commit 7e9f468)

Release v0.13.0 - Release Candidate

Enhancements and Cleanup:

• significant optimisation of isatab.dump() via reworking the study/assay graph implementation. This can improve up to 3 or 4 orders of magnitude the writing to ISA-tab for datasets with thousands of samples (PR #403)
• All the major objects of the ISA models are now Commentable (PR #395)
• isatools.create.connectors now supports both strings and Ontology Annotations for Subject Type and Sample Type (PR #405)
• CI is now migrated from Travis CI to GitHub CI (#404)

Breaking changes:

• Changes to the isatools.create.connectors module to align it with the Datascriptor study JSON config. Now the Datascriptor study config has a "design" property holding all the study design information (PR #405)

Bug Fixes:

• all names for protocols in studies generated by isatools.create.connectors from Datascriptor study configs are unique (issue #398)
• identifiers for studies and investigations are coerced to strings (commit 7e9f468)

This Bugfix release fixes the broken networkx dependency in setup.py.

Update to fix:

• missing Factor Values from isatools.create serialisation (PR #391)

Release v0.12.0

Features:

• Added the isatools.create.connectors module. This module contains the generate_study_design_from_config function to generate an isatools.create.StudyDesign from a JSON configuration document containing metadata about the study design, including sampling and assay plan. These JSON documents can be generated by the Datascriptor application.(issues #355, #356, #377)
• Added support for observational variables to isatools.create.model.StudyArm (with the parameter source_characteristics) (PR #367)
• StudyDesign.generate_isa_study()now generates one Assay per assay type (via StudyDesign._generate_samples()). (issue #369, #373)
• The names of the assay nodes (extract, labelled, extract, protocol node, and data file) (issue #370, #373)
• isatools.create.model.ProductNode now has an attribute extension (e.g. a file extension fo a DataFile node) (issue #378)

Enhancements and Cleanup:

• Introduced support for (built-in, non-configurable) synonyms in protocol types. Protocol types have also been made case insensitive. (issue #382)
• Refactored constants and error messages for the create mode, introducing the modules isatools.create.constants and isatools.create.errors (issue #342, #368)
• Removed asterisk imports (*) from all the core modules (issue #368)
• Removed leftover print() statements (issue #374)
• overall test coverage increased from 67% to 75%. The increase is due both to an increase in the number of tests and the removal of some deprecated, experimental or spurious modules.

Breaking changes:

• isatools.create.models has been renamed to isatools.create.model (issue #368)
• IsaModelAttributeError has been replaced with AttributeError. We will favour standard Exceptions whenever applicable.

Bug Fixes:

• Fixed typo in NAME_PROPERTY_ASSIGNMENT_ERROR (issue #328)
• Silent expansion of Characteristic categories from str to OntologyAnnotation now supported. You can assign a string to a characteristic category and it will be automatically wrapped into an ontology annotation. (issue #332)
• Fixed duplicated in OntologyAnnotation ids (issue #381)
• Fixed raising of an error when correctly setting a comment value (issue #360)
• Assigned correct header "Data Transformation Name" for protocol type "sequence analysis data transformation" (commit b8f2f55)

Release v0.12.0 - Release Candidate

Features:

• Added the isatools.create.connectors module. This module contains the generate_study_design_from_config function to generate an isatools.create.StudyDesign from a JSON configuration document containing metadata about the study design, including sampling and assay plan. These JSON documents can be generated by the Datascriptor application.(issues #355, #356, #377)
• Added support for observational variables to isatools.create.model.StudyArm (with the parameter source_characteristics) (PR #367)
• StudyDesign.generate_isa_study()now generates one Assay per assay type (via StudyDesign._generate_samples()). (issue #369, #373)
• The names of the assay nodes (extract, labelled, extract, protocol node, and data file) (issue #370, #373)
• isatools.create.model.ProductNode now has an attribute extension (e.g. a file extension fo a DataFile node) (issue #378)

Enhancements and Cleanup:

• Introduced support for (built-in, non-configurable) synonyms in protocol types. Protocol types have also been made case insensitive. (issue #382)
• Refactored constants and error messages for the create mode, introducing the modules isatools.create.constants and isatools.create.errors (issue #342, #368)
• Removed asterisk imports (*) from all the core modules (issue #368)
• Removed leftover print() statements (issue #374)
• overall test coverage increased from 67% to 75%. The increase is due both to an increase in the number of tests and the removal of some deprecated, experimental or spurious modules.

Breaking changes:

• isatools.create.models has been renamed to isatools.create.model (issue #368)
• IsaModelAttributeError has been replaced with AttributeError. We will favour standard Exceptions whenever applicable.

Bug Fixes:

• Fixed typo in NAME_PROPERTY_ASSIGNMENT_ERROR (issue #328)
• Silent expansion of Characteristic categories from str to OntologyAnnotation now supported. You can assign a string to a characteristic category and it will be automatically wrapped into an ontology annotation. (issue #332)
• Fixed duplicated in OntologyAnnotation ids (issue #381)
• Fixed raising of an error when correctly setting a comment value (issue #360)
• Assigned correct header "Data Transformation Name" for protocol type "sequence analysis data transformation" (commit b8f2f55)

Release v0.12.0 - Release Candidate

Features:

• Added the isatools.create.connectors module. This module contains the generate_study_design_from_config function to generate an isatools.create.StudyDesign from a JSON configuration document containing metadata about the study design, including sampling and assay plan. These JSON documents can be generated by the Datascriptor application.(issues #355, #356, #377)
• Added support for observational variables to isatools.create.model.StudyArm (with the parameter source_characteristics) (PR #367)
• StudyDesign.generate_isa_study()now generates one Assay per assay type (via StudyDesign._generate_samples()). (issue #369, #373)
• The names of the assay nodes (extract, labelled, extract, protocol node, and data file) (issue #370, #373)
• isatools.create.model.ProductNode now has an attribute extension (e.g. a file extension fo a DataFile node) (issue #378)

Enhancements and Cleanup:

• Introduced support for (built-in, non-configurable) synonyms in protocol types. Protocol types have also been made case insensitive. (issue #382)
• Refactored constants and error messages for the create mode, introducing the modules isatools.create.constants and isatools.create.errors (issue #342, #368)
• Removed asterisk imports (*) from all the core modules (issue #368)
• Removed leftover print() statements (issue #374)
• overall test coverage increased from 67% to 75%. The increase is due both to an increase in the number of tests and the removal of some deprecated, experimental or spurious modules.

Breaking changes:

• isatools.create.models has been renamed to isatools.create.model (issue #368)
• IsaModelAttributeError has been replaced with AttributeError. We will favour standard Exceptions whenever applicable.

Bug Fixes:

• Fixed typo in NAME_PROPERTY_ASSIGNMENT_ERROR (issue #328)
• Silent expansion of Characteristic categories from str to OntologyAnnotation now supported. You can assign a string to a characteristic category and it will be automatically wrapped into an ontology annotation. (issue #332)
• Fixed duplicated in OntologyAnnotation ids (issue #381)
• Fixed raising of an error when correctly setting a comment value (issue #360)
• Assigned correct header "Data Transformation Name" for protocol type "sequence analysis data transformation"

Release v0.12.0 - Release Candidate

Features:

• Added the isatools.create.connectors module. This module contains the generate_study_design_from_config function to generate an isatools.create.StudyDesign from a JSON configuration document containing metadata about the study design, including sampling and assay plan. These JSON documents can be generated by the Datascriptor application.(issues #355, #356, #377)
• Added support for observational variables to isatools.create.model.StudyArm (with the parameter source_characteristics) (PR #367)
• StudyDesign.generate_isa_study()now generates one Assay per assay type (via StudyDesign._generate_samples()). (issue #369, #373)
• The names of the assay nodes (extract, labelled, extract, protocol node, and data file) (issue #370, #373)
• isatools.create.model.ProductNode now has an attribute extension (e.g. a file extension fo a DataFile node) (issue #378)

Enhancements and Cleanup:

• Introduced support for (built-in, non-configurable) synonyms in protocol types. Protocol types have also been made case insensitive. (issue #382)
• Refactored constants and error messages for the create mode, introducing the modules isatools.create.constants and isatools.create.errors (issue #342, #368)
• Removed asterisk imports (*) from all the core modules (issue #368)
• Removed leftover print() statements (issue #374)
• overall test coverage increased from 67% to 76%. The increase is due both to an increase in the number of tests and the removal of some deprecated, experimental or spurious modules.

Breaking changes:

• isatools.create.models has been renamed to isatools.create.model (issue #368)
• IsaModelAttributeError has been replaced with AttributeError. We will favour standard Exceptions whenever applicable.

Bug Fixes:

• Fixed typo in NAME_PROPERTY_ASSIGNMENT_ERROR (issue #328)
• Silent expansion of Characteristic categories from str to OntologyAnnotation now supported. You can assign a string to a characteristic category and it will be automatically wrapped into an ontology annotation. (issue #332)
• Fixed duplicated in OntologyAnnotation ids (issue #381)
• Fixed raising of an error when correctly setting a comment value (issue #360)

This release of isatools introduces a major overhaul of the pre-existing create mode for the generation of ISA documents from study design metadata. We have also introduced some (mostly) minor fixes to the core ISA API, enforcing a stricter validation on some objects.

Among the most noteworthy changes there are:

• Support of complex longitudinal study designs with the introduction of Element, Treatment, StudyCell and StudyArm classes #318, #321
• Increased flexibility of the sample and assay plans, with the introduction of the AssayGraph and the overhaul of the SampleAssayPlan class, now renamed SampleAndAssayPlan #329, #333, #296
• Increased flexibility for Quality Control support through the QualityControlService utility class
• Reworked serialization to JSON/deserialization from JSON of all the study design objects #313
• Check added when submitting quantitative Parameter Values: only numerical values are allowed #347

For more information on the known issues with this release, please check the documentation at http://isatools.readthedocs.org