Article

A process for evaluating legal knowledge-based systems based upon the context criteria contingency-guidelines framework

Authors:

Maria Jean J. Hall,

John ZeleznikowAuthors Info & Claims

ICAIL '03: Proceedings of the 9th international conference on Artificial intelligence and law

Pages 274 - 283

https://doi.org/10.1145/1047788.1047843

Published: 24 June 2003 Publication History

Abstract

In an ideal world, a legal knowledge-based system would be evaluated by an evaluator with expertise in both the legal domain and software engineering evaluation processes. However in the real world, this task is typically undertaken by legal professionals, who may lack software engineering expertise. Where software engineers do have this responsibility, they may lack legal domain knowledge. We extend the ISO 14598 evaluation process with a novel evaluation framework which satisfies three important requirements: elements of existing software engineering evaluation methodologies are integrated and subsumed, making them more readily accessible to the evaluator; requirements specific to the legal domain are included; and the intended users do not necessarily require extensive software engineering expertise. The framework emphasises the importance of the evaluation context and goals and integrates these with system properties and contingency-guidelines to suggest appropriate evaluation criteria. The evaluation process supports the selection of criteria by manual, semi-automated or automated methods and a design of an architecture to support this choice of appropriate criteria, is presented. Two evaluations are discussed that were conducted using the process and its associated framework. With ongoing research and development in the field of Artificial intelligence and Law, the need for an easily accessible and specialized evaluation methodology is apparent. Such a method would assist legal professionals frame an evaluation of legal knowledge-based systems and help software engineers understand the evaluation requirements of legal professionals.

References

[1]

ACM Code of Ethics and Professional Conduct. http://www.acm.org/constitution/code.html.

[2]

Adelman, L. Integrating evaluation methods into the DSS development process. Information & Decision Technologies, 1989, 15(4), 227--41.

[3]

Adelman, L., Gualtieri, J., and Riedel, S. L. A multifaceted approach to evaluating expert systems. Artificial Intelligence for Engineering Design, Analysis & Manufacturing, 1994, 8(4), 289--306.

[4]

Avison, D. E., Horton, J., Powell, P., Nandhakumar, J. Incorporating evaluation in the information systems development process. In Proceedings of Second European Conference on Information Technology Investment Evaluation., (Henley on Thames, UK., 1995) Oper. Res. Soc.Birmingham, UK, 310--19, 319/1.

[5]

Bellucci, E. and Zeleznikow, J. Representations of Decision-Making Support in Negotiation. Journal of Decision Systems, 2001, 10(3--4).

[6]

Boehm, B. W. Software Engineering Economics, Englewood Cliffs, NJ Prentice Hall, 1981.

Digital Library

[7]

Boloix, G. and Robillard, P. N. A software system evaluation framework. Computer, 1995, 28(12), 17--26.

Digital Library

[8]

Borenstein, D. Towards a practical method to validate decision support systems. Decision Support Systems, 1998, 23(3), 227--239.

Digital Library

[9]

Bromby, M. C. and Hall, M.J.J. The Development and Rapid Evolution of the Knowledge Model of ADVOKATE: An Advisory System to Assess the Credibility of Eyewitness Testimony. in Proceedings of JURIX 2002: Fifteenth International Conference on Legal Knowledge Based Systems, (London, 2002) OS Press, Amsterdam, Netherlands, 143--152.

[10]

Brown, A. W. and Wallnau, K. C. A framework for evaluating software technology. IEEE Software, 1996, 13(5), 39--49.

Digital Library

[11]

Chelimsky, E. The coming transformations in evaluation, in Evaluation for the 21st Century: A handbook, E. Chelimsky and W. R. Shadish, Editors. 1997, SAGE, Thousand Oaks, CAL. 1--26.

[12]

DeLone, W. H., McLean, E. R. Information Success: The quest for the dependent variable. Informations Systems Research, 1992, 3(1), 60--95.

Digital Library

[13]

Farbey, B., Land, F., Targett, D. Moving IS evaluation forward: learning themes and research issues. Journal of Strategic Information Systems, 1999, 8(2), 189--207.

[14]

Fenton, N., Pfleeger, S. L. Software Metrics a rigorous and practical approach. 2nd ed, London International Thompson Computer Press, 1997.

Digital Library

[15]

Ginsberg, M. J. and Zmud, R. W. Evolving criteria for Information Systems assessment. in Proceedings of Information Systems Assessment: Issues and Challenges: proceedings of the IFIP WG 8.2 Working Conference on Information Systems Assessment., (Noordwijkerhout, The Netherlands, 1988) Oxford, North Holland, 41--52.

[16]

Gregory, A. Which evaluation methodology when? A contingency approach to evaluation., in Systems Thinking in Europe, M. C. Jackson, et al., Editors. 1991, Plenum, New York. 435--441.

[17]

Hall, M.J.J. and Zeleznikow, J. Acknowledging insufficiency in the evaluation of legal knowledge-based systems: strategies towards a broad-based evaluation model. in Proceedings of 8th International Conference on Artificial Intelligence and Law, (St Louis, MO USA, 2001) ACM, 147--156.

Digital Library

[18]

Hall, M.J.J., Stranieri, A., and Zeleznikow, J. A strategy for evaluating web based discretionary decision support systems. in Proceedings of ADBIS 2002 6th East European Conference on Advances in Databases and Information Systems, (Slovak University of Technology, Bratislava, Slovak Republic, 2002), 108--120.

[19]

Hall, M.J.J. and Zeleznikow, J. The Context, Criteria, Contingency evaluation framework for legal knowledge-base systems. in Proceedings of BIS 2002 5th International Conference on Business Information Systems, (Poznan, Poland, 2002) ACM,

[20]

ISO/IEC IS 14598-5: Information Technology - Software Product Evaluation - Part 5: Process for Evaluators. 1998.

[21]

ISO/IEC IS 14598-1: Information Technology - Software Product Evaluation - Part 1: General Overview. 1998.

[22]

ISO/IEC IS 9126-1 Information Technology - Software quality characteristics and metrics - Part 1: Quality characteristics and subcharacteristics. 1998.

[23]

Jeanrenaud, A. and Romanazzi, P. Software product evaluation metrics: a methodological approach. in Proceedings of Software Quality Management II. Building Quality into Software. Vol.2. Comput. Mech. Publications, (Southampton, UK, 1994), 59--69.

[24]

Jung Ho-Won. and Min-Suk., Y. A software product quality evaluation and resource allocation model. in Proceedings of Software Quality and Business Opportunities. Fifth European Conference on Software Quality, (Dublin, Ireland, 1996) Irish Quality Assoc., 286--94.

[25]

Juristo, N. and Morant, J. L. Common framework for the evaluation process of KBS and conventional software. Knowledge Based systems, 1998, 11(2), 145--159.

[26]

Kitchenham, B. A. Evaluating software engineering methods and tool. 2. Selecting an appropriate evaluation method-technical criteria. Sigsoft Software Engineering Notes, 1996, 21(2), 11--15.

Digital Library

[27]

Knauf, R., Philippow, I., and Gonzalez, A. J. Towards validation and refinement of rule-based systems. Journal of Experimental & Theoretical Artificial Intelligence, 2000, 12(4), 421--31.

[28]

Koers, A. W., et al. Criteria for the classification of legal knowledge systems advisory systems on legal questions, in Legal knowledge Based Systems, D. Kracht, C.N.J. De Vey Mestdagh, and J. S. Svensson, Editors. 1990, Koninlijke Vermande, Lelystad, Netherlands. 23--35.

[29]

Kolodner, J. Case-Based Reasoning, San Mateo, CA Morgan Kaufman, 1993.

Digital Library

[30]

Linstone, H. A. and Turoff, M. The Delphi Method Techniques and Applications, Reading, Mas Addison Wesley, 1975.

[31]

O'Keeffe, R. M. and O'Leary, D. E. Expert System verification and validation: a survey and tutorial. Artificial Intelligence Review, 1993, 7(1), 3--42.

[32]

Oppermann, R. and Reiterer, H. Software evaluation using the 9241 evaluator. Behaviour & Information Technology, 1997, 16(4--5), 232--245.

[33]

Oskamp, A. and Tragter, M. W. Automated legal decision systems in practice: the mirror of reality. Artificial Intelligence & Law, 1997, 5(4), 291--322.

Digital Library

[34]

Paulk, M. C., et al. Capability Maturity Model for Software version 1.1. 1993, Software Engineering Institute.

[35]

Rae, A., Robert, P., and Hausen, H. L. Software Evaluation for Certification: Principles, Practice and Legal Liability. McGraw Hill international software assurance series, Maidenhead mcGraw Hill, 1995.

Digital Library

[36]

Serafeimidis, V. and Smithson, S. Rethinking the approaches to information systems investment evaluation. Logistics Information Management, 1999, 12(1/2), 94--107.

[37]

Spreeuwenberg, S., Van Engers, T., and Gerrits, I. R. The Role of verification in improving the quality of legal decision-making. in Proceedings of Jurix 2001: the Fourteenth Annual International Conference on Legal Knowledge and Information Systems. Vol 70. Frontiers in Artificial Intelligence and Applications., (Amsterdam, Netherlands, 2001) IOS Press.

[38]

Stranieri, A. and Zeleznikow, J. The evaluation of legal knowledge based systems. in Proceedings of Seventh International Conference on Artificial Intelligence and Law., (Oslo, Norway, 1999) ACM Press, New York, NY, 18--24.

Digital Library

[39]

Stranieri, A., Yearwood, J., and Zeleznikow, J. Tools for placing legal decision support systems on the World Wide Web. in Proceedings of 8th International conference on Artificial Intelligence and Law (ICAIL), (Washington University, St. Louis, MO, 2001) ACM Press, 206--214.

Digital Library

[40]

Susskind, R. Expert systems in law: A jurisprudential approach to artificial intelligence and legal reasoning Modern Law Review, 1986, 49(2), 168--194.

[41]

Susskind, R. E. Expert systems in law. Out of the research laboratory and into the marketplace. in Proceedings of ICAIL-1987, (Boston. MA, 1987) ACM Press, 1--8.

Digital Library

[42]

Svensson, J. S., et al. ExpertiSZe, a Tool for Determining the Effects of Social Security Legislation. in Proceedings of JURIX'92: Legal Knowledge Based Systems: Information Technology & Law, (Lelystad Netherlands, 1992) Koninklijke Vermande.

[43]

Toulmin, S. The Uses of Argument, Cambridge, UK Cambridge University press, 1958.

[44]

Touma, C. A Web based analysis of an evaluation framework for (Legal) decision support systems, in Unpublished Honours thesis in the Department of Computer Science and Engineering. 2003, La Trobe University, Melbourne.

[45]

Wexler, D. B. and Winik, B. J. Law in Therapeutic Key: Developments in Therapeutic Jurisprudence xvii, 1996.

[46]

Zeleznikow, J. Building judicial decision support systems in discretionary legal domains. International Review of Law, Computers and Technology, 2000, 14(3), 341--356.

Cited By

Novotná TLibal T(2022)An Evaluation of Methodologies for Legal FormalizationExplainable and Transparent AI and Multi-Agent Systems10.1007/978-3-031-15565-9_12(189-203)Online publication date: 23-Sep-2022
https://doi.org/10.1007/978-3-031-15565-9_12
Bartolini CLenzini GSantos C(2019)An Agile Approach to Validate a Formal Representation of the GDPRNew Frontiers in Artificial Intelligence10.1007/978-3-030-31605-1_13(160-176)Online publication date: 11-Oct-2019
https://doi.org/10.1007/978-3-030-31605-1_13
Conrad JZeleznikow JFrancesconi EVerheij B(2013)The significance of evaluation in AI and lawProceedings of the Fourteenth International Conference on Artificial Intelligence and Law10.1145/2514601.2514624(186-191)Online publication date: 10-Jun-2013
https://dl.acm.org/doi/10.1145/2514601.2514624
Show More Cited By

Index Terms

A process for evaluating legal knowledge-based systems based upon the context criteria contingency-guidelines framework
1. General and reference
  1. Cross-computing tools and techniques
    1. Validation
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Empirical software validation
      2. Process validation

Recommendations

Acknowledging insufficiency in the evaluation of legal knowledge-based systems: strategies towards a broadbased evaluation model
ICAIL '01: Proceedings of the 8th international conference on Artificial intelligence and law

This paper considers the need for evaluation of knowledge-based systems in general and legal knowledge-based systems in particular. Some special features of legal knowledge-based systems pertinent to their evaluation are presented. The expected benefits ...
Common framework for the evaluation process of KBS and conventional software

Conventional software (CS) and knowledge-based software (KBS) validation and verification have been traditionally accepted to be radically different disciplines with nothing in common. However, the differences between CS and KBS should not stop efforts ...
Design of Knowledge-Based Systems with a Knowledge-Based Assistant
Special Issue on Artificial Intelligence in Software Applications

The authors propose a model for an intelligent assistant to aid in building knowledge-based systems (KBSs) and discuss a preliminary implementation. The assistant participates in KBS construction, including acquisition of an initial model of a problem ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICAIL '03: Proceedings of the 9th international conference on Artificial intelligence and law

June 2003

304 pages

ISBN:1581137478

DOI:10.1145/1047788

Conference Chair:
John Zeleznikow
The Edinburgh Law School, University of Edinburgh, Scotland, UK
,
Program Chair:
Giovanni Sartor
CIRSFID, University of Bologna, Italy

Copyright © 2003 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
West Group, Thomson Legal & Regulatory
The International Association for Artificial Intelligence and Law
The Joseph Bell Centre for Forensic Statistics and Legal Reasoning
The University of Edinburgh School of Law

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 June 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 69 of 169 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
736
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Novotná TLibal T(2022)An Evaluation of Methodologies for Legal FormalizationExplainable and Transparent AI and Multi-Agent Systems10.1007/978-3-031-15565-9_12(189-203)Online publication date: 23-Sep-2022
https://doi.org/10.1007/978-3-031-15565-9_12
Bartolini CLenzini GSantos C(2019)An Agile Approach to Validate a Formal Representation of the GDPRNew Frontiers in Artificial Intelligence10.1007/978-3-030-31605-1_13(160-176)Online publication date: 11-Oct-2019
https://doi.org/10.1007/978-3-030-31605-1_13
Conrad JZeleznikow JFrancesconi EVerheij B(2013)The significance of evaluation in AI and lawProceedings of the Fourteenth International Conference on Artificial Intelligence and Law10.1145/2514601.2514624(186-191)Online publication date: 10-Jun-2013
https://dl.acm.org/doi/10.1145/2514601.2514624
Yearwood JStranieri A(2012)Tools and Applications for Reasoning CommunitiesApproaches for Community Decision Making and Collective Reasoning10.4018/978-1-4666-1818-3.ch009(210-259)Online publication date: 2012
https://doi.org/10.4018/978-1-4666-1818-3.ch009
Bellucci EMacfarlane DZeleznikow J(2010)How Information Technology Can Support Family Law and MediationBusiness Information Systems Workshops10.1007/978-3-642-15402-7_31(243-255)Online publication date: 2010
https://doi.org/10.1007/978-3-642-15402-7_31
Bellucci E(2008)AssetDividerProceedings of the 1st International Working Conference on Human Factors and Computational Models in Negotiation10.1145/1609170.1609172(11-18)Online publication date: 8-Dec-2008
https://dl.acm.org/doi/10.1145/1609170.1609172
Zeleznikow JBellucci ESchild UMackenzie GGardner AWinkels R(2007)Bargaining in the shadow of the law - using utility functions to support legal negotiationProceedings of the 11th international conference on Artificial intelligence and law10.1145/1276318.1276365(237-246)Online publication date: 4-Jun-2007
https://dl.acm.org/doi/10.1145/1276318.1276365
Bellucci EZeleznikow J(2005)Developing negotiation decision support systems that support mediatorsArtificial Intelligence and Law10.1007/s10506-006-9013-113:2(233-271)Online publication date: 1-Jun-2005
https://dl.acm.org/doi/10.1007/s10506-006-9013-1
Zeleznikow J(2004)Building Intelligent Legal Decision Support Systems: Past Practice and Future ChallengesApplied Intelligent Systems10.1007/978-3-540-39972-8_7(201-254)Online publication date: 2004
https://doi.org/10.1007/978-3-540-39972-8_7

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents