Python scripts for automatically evaluating NLU services (API.ai, IBM Watson Conversation, Microsoft LUIS, RASA) based on the format used by NLU-Evaluation-Corpora.
You can use the converters to create .json- or .zip-files with training data which can be imported into the respective NLU service using the web interface.
#luis (also works for rasa)
luis_converter = LuisConverter()
luis_converter.import_corpus("WebApplicationsCorpus.json")
luis_converter.export("WebApplicationsTraining_Luis.json")You can also use the Luis file format to train Rasa, however, we recommend using the Dialogflow data format for training Rasa.
#watson
watson_converter = WatsonConverter()
watson_converter.import_corpus("WebApplicationsCorpus.json")
watson_converter.export("WebApplicationsTraining_Watson.json")#dialogflow (also works for rasa)
dialogflow_converter = DialogflowConverter()
dialogflow_converter.import_corpus("WebApplicationsCorpus.json")
dialogflow_converter.export("WebApplicationsTraining_Dialogflow.zip")You can use the analysers to annotate the test data and generate a .json-file with an evaluation of the annotations.
#luis
luis_analyser = LuisAnalyser("application_id", "subscription_key")
luis_analyser.get_annotations("WebApplicationsCorpus.json", "WebApplicationsAnnotations_Luis.json")
luis_analyser.analyse_annotations("WebApplicationsAnnotations_Luis.json", "WebApplicationsCorpus.json", "WebApplicationsAnalysis_Luis.json")#watson
watson_analyser = WatsonAnalyser("workspace_id", "user", "password")
watson_analyser.get_annotations("WebApplicationsCorpus.json", "WebApplicationsAnnotations_Watson.json")
watson_analyser.analyse_annotations("WebApplicationsAnnotations_Watson.json", "WebApplicationsCorpus.json", "WebApplicationsAnalysis_Watson.json")#dialogflow
dialogflow_analyser = DialogflowAnalyser("project_id")
dialogflow_analyser.get_annotations("WebApplicationsCorpus.json", "WebApplicationsAnnotations_Dialogflow.json")
dialogflow_analyser.analyse_annotations("WebApplicationsAnnotations_Dialogflow.json", "WebApplicationsCorpus.json", "WebApplicationsAnalysis_Dialogflow.json")In order to evaluate the annotations from Rasa, you have to start the Rasa server with the option "-e luis".
#rasa
rasa_analyser = RasaAnalyser("http://localhost:5000/parse")
rasa_analyser.get_annotations("WebApplicationsCorpus.json", "WebApplicationsAnnotations_Rasa.json")
rasa_analyser.analyse_annotations("WebApplicationsAnnotations_Rasa.json", "WebApplicationsCorpus.json", "WebApplicationsAnalysis_Rasa.json")If you have any questions, please contact:
Daniel Braun (Technical University of Munich) daniel.braun@tum.de