article

An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models

Authors:

Johan Huysmans,

Karel Dejaeger,

Christophe Mues,

Jan Vanthienen,

Bart BaesensAuthors Info & Claims

Decision Support Systems, Volume 51, Issue 1

Pages 141 - 154

https://doi.org/10.1016/j.dss.2010.12.003

Published: 01 April 2011 Publication History

Abstract

An important objective of data mining is the development of predictive models. Based on a number of observations, a model is constructed that allows the analysts to provide classifications or predictions for new observations. Currently, most research focuses on improving the accuracy or precision of these models and comparatively little research has been undertaken to increase their comprehensibility to the analyst or end-user. This is mainly due to the subjective nature of 'comprehensibility', which depends on many factors outside the model, such as the user's experience and his/her prior knowledge. Despite this influence of the observer, some representation formats are generally considered to be more easily interpretable than others. In this paper, an empirical study is presented which investigates the suitability of a number of alternative representation formats for classification when interpretability is a key requirement. The formats under consideration are decision tables, (binary) decision trees, propositional rules, and oblique rules. An end-user experiment was designed to test the accuracy, response time, and answer confidence for a set of problem-solving tasks involving the former representations. Analysis of the results reveals that decision tables perform significantly better on all three criteria, while post-test voting also reveals a clear preference of users for decision tables in terms of ease of use.

References

[1]

Andrews, R., Diederich, J. and Tickle, A., Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowledge Based Systems. v8 i6. 373-389.

[2]

Baesens, B., Van Gestel, T., Viaene, S., Stepanova, M., Suykens, J. and Vanthienen, J., Benchmarking state of the art classification algorithms for credit scoring. Journal of the Operational Research Society. v54 i6. 627-635.

[3]

Benbasat, I. and Taylor, R.N., Behavioral aspects of information processing for the design of management information systems. IEEE Transactions on Systems, Man, and Cybernetics. v12 i4. 439-450.

[4]

Benbasat, I., Dexter, A.S. and Todd, P., An experimental program investigating color-enhanced and graphical information presentation: An integration of the findings. Communications of the ACM. v29 i11. 1094-1105.

[5]

Breiman, L., Friedman, J., Olsen, R. and Stone, C., Classification and Regression Trees. 1984. Wadsworth and Brooks.

[6]

Campbell, D., Task Complexity: A Review and Analysis. Academy of Management Journal. v13 i1. 40-52.

[7]

Cantú-Paz, E. and Kamath, C., Inducing oblique decision trees with evolutionary algorithms. IEEE Transactions on Evolutionary Computation. v7 i1. 54-68.

[8]

Chandra, A. and Krovi, R., Representational congruence and information retrieval: Towards an extended model of cognitive fit. Decision Support Systems. v25. 271-288.

Digital Library

[9]

Chen, F., Learning accurate and understandable rules from SVM classifiers. 2004. Simon Fraser University, Master's thesis.

[10]

Clark, P. and Niblett, T., The CN2 induction algorithm. Machine Learning. v3 i4. 261-283.

[11]

Cohen, W., Fast effective rule induction. In: Prieditis, A., Russell, S. (Eds.), Proc. the 12th International Conference on Machine Learning, Morgan Kaufmann, Tahoe City. pp. 115-123.

[12]

Coll, R.A., Coll, J.H. and Thakur, G., Graphs and tables: A four-factor experiment. Communications of the ACM. v37 i4. 76-86.

[13]

Colomb, R., Representation of propositional expert systems as partial functions. Artificial Intelligence. v109 i1-2. 187-209.

[14]

Dennis, A. and Carte, T., Using geographic information systems for decision making: extending cognitive fit theory to map-based presentations. Information Systems Research. v9 i2. 194-203.

[15]

DeSanctis, G., Computer grapics as decision aids: directions for research. Desicion Sciences. v15 i4. 463-487.

[16]

Dickson, G.W., DeSanctis, G. and McBride, D.J., Understanding the effectiveness of computer graphics for decision support: A cumulative experimental approach. Communications of the ACM. v29 i1. 40-47.

[17]

Equal Credit Opportunity Act. United States Code, 1974.

[18]

Feelders, A., Daniels, H. and Holsheimer, M., Methodological and practical aspects of data mining. Information & Management. v37. 271-281.

[19]

Freitas, A.A., Are we really discovering "interesting" knowledge from data? expert update. The BCS-SGAI Magazine. v9 i1. 41-47.

[20]

Frownfelter-Lohrke, C., The effects of differing information presentations of general purpose financial statements on users' decisions. Journal of Information Systems. v12 i4. 99-107.

[21]

Gilmore, D. and Green, T., Comprehension and recall of miniature programs. International Journal of Man-Machine Studies. v21 i1. 31-48.

[22]

Goodhue, D. and Thompson, R., Task-technology fit and individual performance. MIS Quarterly. v19 i2. 213-236.

[23]

Halverson, R., An empirical investigation comparing if-then rules and decision tables for programming rule-based expert systems. In: Proceedings of 26th Hawaii International Conference on System Sciences, pp. 316-323.

[24]

Huysmans, J., Baesens, B. and Vanthienen, J., ITER: an algorithm for predictive regression rule extraction. In: DaWaK 2006, 4081. Springer Verlag. pp. 270-279.

[25]

Huysmans, J., Baesens, B. and Vanthienen, J., Using rule extraction to improve the comprehensibility of predictive models, FETEW research report KBI 0612. 2006. Katholieke Universiteit Leuven.

[26]

Jarvenpaa, S. and Dickson, G., Graphics and managerial decisionmaking: Research based guidelines. Communications of the ACM. v31 i6. 764-774.

[27]

Johansson, U., König, R. and Niklasson, L., Automatically balancing accuracy and comprehensibility in predictive modeling. In: Proceedings of the 8th International Conference on Information Fusion,

[28]

Kohavi, R., The power of decision tables. In: Lavrac, N., Wrobel, S. (Eds.), Lecture Notes in Artificial Intelligence, 914. Springer Verlag, Berlin, Heidelberg, New York. pp. 174-189.

[29]

Larkin, J.H. and Simon, H.A., Why a diagram is (sometimes) worth ten thousand words. Cognitive Sience. v11. 65-99.

[30]

Lee, C.-C., Cheng, H. and Cheng, H.-H., An empirical study of mobile commerce in insurance industry: Task-technology fit and individual differences. Decision Support Systems. v43. 95-110.

Digital Library

[31]

Lucas, H., An experimental investigation of the use of computer based graphics in decision-making. Management Science. v27 i7. 757-768.

[32]

Martens, D., Baesens, B., Van Gestel, T. and Vanthienen, J., Comprehensible credit scoring models using rule extraction from support vector machines. European Journal of Operational Research. v183 i3. 1466-1476.

[33]

Martens, D., Baesens, B. and Van Gestel, T., Decompositional rule extraction from support vector machines by active learning. IEEE Transactions on Knowledge and Data Engineering. v21 i2. 178-191.

[34]

McGrath, J., Groups: Interaction and Performance. 1984. Prentice Hall, Englewood Cliffs, NY.

[35]

Pazzani, M., Knowledge discovery from data?. IEEE Intelligent Systems. v15 i2. 10-13.

[36]

Quinlan, J., Simplifying decision trees. International journal of man-machine studies. v27 i3. 221-234.

[37]

Quinlan, J., C4.5: Programs for Machine Learning. 1993. Morgan Kaufmann Publishers Inc, San Francisco, CA, USA.

[38]

Remus, W., A study of graphical and tabular displays and their interaction with environmental complexity. Management Science. v33 i9. 1200-1204.

[39]

Santos-Gomez, L. and Darnell, M., Empirical evaluation of decision tables for constructing and comprehending expert systems. Knowledge Acquisition. v4 i4. 427-444.

[40]

Setiono, R. and Leow, W.K., Pruned neural networks for regression. In: Miziguchi, R., Slaney, J. (Eds.), PRICAI 2000, Springer, Melbourne, Australia. pp. 500-509.

[41]

Setiono, R. and Liu, H., Neurolinear: From neural networks to oblique decision rules. Neural Computing. v17 i1. 1-24.

[42]

Setiono, R. and Thong, J., An approach to generate rules from neural networks for regression problems. European Journal of Operational Research. v155 i1. 239-250.

[43]

Shaft, T. and Vessey, I., The role of cognitive fit in the relationship between software comprehension and modification. MIS Quarterly. v30 i1. 29-55.

[44]

Sheskin, D.J., Handbook of Parametric and Nonparametric Statistical Procedures. 1997. CRC Press.

[45]

Sinha, A.P. and Zhao, H., Incorporating domain knowledge into data mining classifiers: An application in indirect lending. Decision Support Systems. v46. 287-299.

Digital Library

[46]

Speier, C., The influence of information presentation formats on complex task decision-making performance. International Journal of Human-Computer Studies. v64 i11. 1115-1131.

[47]

Subramanian, G., Nosek, J., Raghunathan, S. and Kanitkar, S., A comparison of the decision table and tree. Communications of the ACM. v34 i1. 89-94.

[48]

Swait, J. and Adamowicz, W., The influence of task complexity on consumer choice: a latent class model of decision strategy switching. Journal of Consumer Research. v28. 135-148.

[49]

Sweller, J., Cognitive load during problem solving: Effects on learning. Cognitive Science. v12 i2. 257-285.

[50]

Taha, I. and Ghosh, J., Symbolic interpretation of artificial neural networks. IEEE Transactions on Knowledge and Data Engineering. v11 i3. 448-463.

[51]

Tegarden, D., Business information visualisation. Communications of AIS. v1 i4. 1-37.

[52]

Tufte, E., The Visual Display of Quantitative Information. 2001. Graphics Press, Chesire, CT.

[53]

Umanath, N. and Vessey, I., Multiattribute data presentation and human judgement: a cognitive fit perspective. Decision Sciences. v25 i5. 795-825.

[54]

Vanthienen, J., A more general comparison of the decision table and tree: A response. Communications of the ACM. v37 i2. 109-113.

[55]

Vanthienen, J. and Wets, G., From decision tables to expert system shells. Data and Knowledge Engineering. v13 i3. 265-282.

[56]

Vanthienen, J., Mues, C. and Aerts, A., An illustration of verification and validation in the modelling phase of KBS development. Data and Knowledge Engineering. v27 i3. 337-352.

[57]

Vessey, I., Cognitive fit: A theory-based analysis of the graphs versus tables literature. Decision Sciences. v22 i2. 219-240.

[58]

Vessey, I., The effect of information presentation on decision making: a cost-benefit analysis. Information & Management. v27. 103-119.

[59]

Vessey, I. and Weber, R., Structured tools and conditional logic: An empirical investigation. Communications of the ACM. v29 i1. 48-57.

[60]

Viaene, S., Derrig, R., Baesens, B. and Dedene, G., A comparison of state-of-the-art classification techniques for expert automobile insurance fraud detection. Journal of Risk and Insurance (Special Issue on Fraud Detection). v69 i3. 433-443.

[61]

Ware, C., Information Visualisation: Perception for Design. 2000. Academic Press, San Diego, CA.

[62]

Wood, R., Task complexity: definition of the construct. Organizational Behavior and Human Decision Processes. v37. 60-82.

[63]

Zhou, Z.-H., Jiang, Y. and Chen, S.-F., Extracting symbolic rules from trained neural network ensembles. AI Communications. v16 i1. 3-15.

Cited By

Li PChang YWei MHsieh H(2024)Estimating Future Financial Development of Urban Areas for Deploying Bank Branches: A Local-Regional Interpretable ModelACM Transactions on Management Information Systems10.1145/365647915:2(1-26)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1145/3656479
Ciatto GSabbatini FAgiollo AMagnini MOmicini A(2024)Symbolic Knowledge Extraction and Injection with Sub-symbolic Predictors: A Systematic Literature ReviewACM Computing Surveys10.1145/364510356:6(1-35)Online publication date: 8-Feb-2024
https://dl.acm.org/doi/10.1145/3645103
Fleiß JBäck EThalmann S(2024)Mitigating Algorithm Aversion in Recruiting: A Study on Explainable AI for Conversational AgentsACM SIGMIS Database: the DATABASE for Advances in Information Systems10.1145/3645057.364506255:1(56-87)Online publication date: 6-Feb-2024
https://dl.acm.org/doi/10.1145/3645057.3645062
Show More Cited By

Recommendations

Comparison of some classification algorithms based on deterministic and nondeterministic decision rules
Transactions on rough sets XII

We discuss two, in a sense extreme, kinds of nondeterministic rules in decision tables. The first kind of rules, called as inhibitory rules, are blocking only one decision value (i.e., they have all but one decisions from all possible decisions on their ...
Improving the interpretability of classification rules discovered by an ant colony algorithm: Extended results

Most ant colony optimization ACO algorithms for inducing classification rules use a ACO-based procedure to create a rule in a one-at-a-time fashion. An improved search strategy has been proposed in the cAnt-Miner<inline-formula><inline-graphic xlink="...
Building comprehensible customer churn prediction models with advanced rule induction techniques

Customer churn prediction models aim to detect customers with a high propensity to attrite. Predictive accuracy, comprehensibility, and justifiability are three key aspects of a churn prediction model. An accurate model permits to correctly target ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Decision Support Systems

Decision Support Systems Volume 51, Issue 1

April, 2011

246 pages

ISSN:0167-9236

Issue’s Table of Contents

Copyright © Elsevier B.V. © 2010.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 April 2011

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

104
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li PChang YWei MHsieh H(2024)Estimating Future Financial Development of Urban Areas for Deploying Bank Branches: A Local-Regional Interpretable ModelACM Transactions on Management Information Systems10.1145/365647915:2(1-26)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1145/3656479
Ciatto GSabbatini FAgiollo AMagnini MOmicini A(2024)Symbolic Knowledge Extraction and Injection with Sub-symbolic Predictors: A Systematic Literature ReviewACM Computing Surveys10.1145/364510356:6(1-35)Online publication date: 8-Feb-2024
https://dl.acm.org/doi/10.1145/3645103
Fleiß JBäck EThalmann S(2024)Mitigating Algorithm Aversion in Recruiting: A Study on Explainable AI for Conversational AgentsACM SIGMIS Database: the DATABASE for Advances in Information Systems10.1145/3645057.364506255:1(56-87)Online publication date: 6-Feb-2024
https://dl.acm.org/doi/10.1145/3645057.3645062
Nadizar GRovito LDe Lorenzo AMedvet EVirgolin M(2024)An Analysis of the Ingredients for Learning Interpretable Symbolic Regression Models with Human-in-the-loop and Genetic ProgrammingACM Transactions on Evolutionary Learning and Optimization10.1145/36436884:1(1-30)Online publication date: 23-Feb-2024
https://dl.acm.org/doi/10.1145/3643688
Nishtala SRavindran B(2024)Cost-Sensitive Trees for Interpretable Reinforcement LearningProceedings of the 7th Joint International Conference on Data Science & Management of Data (11th ACM IKDD CODS and 29th COMAD)10.1145/3632410.3632443(91-99)Online publication date: 4-Jan-2024
https://dl.acm.org/doi/10.1145/3632410.3632443
Zhang YTang CYu LZhou JLi LCui QFan FJiang LZhao XAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)Domain Level Interpretability: Interpreting Black-box Model with Domain-specific EmbeddingProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635688(1102-1105)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635688
Sokol KVogt J(2024)What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building BlocksExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651047(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651047
Ramon YMartens DEvgeniou TPraet S(2024)Can metafeatures help improve explanations of prediction models when using behavioral and textual data?Machine Language10.1007/s10994-021-05981-0113:7(4245-4284)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s10994-021-05981-0
Speith TCrook BMann SSchomäcker ALanger M(2024)Conceptualizing understanding in explainable artificial intelligence (XAI): an abilities-based approachEthics and Information Technology10.1007/s10676-024-09769-326:2Online publication date: 15-Jun-2024
https://dl.acm.org/doi/10.1007/s10676-024-09769-3
Atzmueller MFürnkranz JKliegr TSchmid U(2024)Explainable and interpretable machine learning and data miningData Mining and Knowledge Discovery10.1007/s10618-024-01041-y38:5(2571-2595)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s10618-024-01041-y
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents