Abstract
Besides increasing transparency and demonstrating awareness of the author, self-reported limitations enable other researchers to effectively learn from, build on, validate, and extend the original work. However, this topic is understudied in information systems design science research (IS DSR). The study has assessed 243 IS DSR papers published in the period 2013–2022 and built a typology of the 19 most relevant limitations, organized into four categories: (1) Input Knowledge and Technology, (2) Research Process, (3) Resulting Artifact, and (4) Design Knowledge. Further, the contribution suggests actions to mitigate each type of limitation throughout the entire IS DSR project lifecycle. The authors have also created guidelines to report the limitations in a useful way for knowledge accumulation. The proposed typology and guidelines enable reviewers and editors to better frame self-reported limitations, assess rigor and relevance more systematically, and provide more precise feedback. Moreover, the contribution may help design researchers identify, mitigate, and effectively communicate the uncertainties inherent to all scientific advances.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aguinis H, Lawal SO (2012) Conducting field experiments using eLancing’s natural environment. J Bus Ventur 27:493–505. https://doi.org/10.1016/j.jbusvent.2012.01.002
Aguinis H, Solarino AM (2019) Transparency and replicability in qualitative research: the case of interviews with elite informants. Strateg Manag J 40:1291–1315. https://doi.org/10.1002/smj.3015
Aguinis H, Ramani RS, Cascio WF (2020) Methodological practices in international business research: an after-action review of challenges and solutions. J Int Bus Stud 51:1593–1608. https://doi.org/10.1057/s41267-020-00353-7
Baskerville R, Pries-Heje J (2019) Projectability in design science research. J Inf Technol Theory Appl 20:53–76
Baskerville R, Baiyere A, Gregor S et al (2018) Design science research contributions: finding a balance between artifact and theory. J Assoc Inf Syst 19:358–376. https://doi.org/10.17705/1jais.00495
Beller EM, Glasziou PP, Altman DG et al (2013) PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts. PLoS Med. https://doi.org/10.1371/journal.pmed.1001419
Brereton P, Kitchenham BA, Budgen D et al (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80:571–583. https://doi.org/10.1016/j.jss.2006.07.009
Briggs RO, Nunamaker J, Sprague R (2011) Applied science research in information systems: the last research mile. J Manag Inf Syst 28:13–16
Brutus S, Duniewicz K (2012) The many heels of Achilles: an analysis of self-reported limitations in leadership research. Leadersh Q 23:202–212. https://doi.org/10.1016/j.leaqua.2011.11.015
Brutus S, Gill H, Duniewicz K (2010) State of science in industrial and organizational psychology: a review of self-reported limitations. Pers Psychol 63:907–936. https://doi.org/10.1111/j.1744-6570.2010.01192.x
Brutus S, Aguinis H, Wassmer U (2013) Self-reported limitations and future directions in scholarly reports: analysis and recommendations. J Manag 39:48–75. https://doi.org/10.1177/0149206312455245
Buhl HU (2011) Business & information systems engineering (BISE) ist AIS affiliated journal. Bus Inf Syst Eng 53:3. https://doi.org/10.1007/s11576-010-0259-6
Cater-Steel A, Toleman M, Rajaeian MM (2019) Design science research in doctoral projects: an analysis of Australian theses. J Assoc Inf Syst 20:1844–1869. https://doi.org/10.17705/1jais.00587
Chandra Kruse L, Seidel S, vom Brocke J (2019) Design archaeology: generating design knowledge from real-world artifact design. International conference on design science research in information systems and technology. Springer, Cham, pp 32–45. https://doi.org/10.1007/978-3-030-19504-5_3
Chanson M, Bogner A, Bilgeri D et al (2019) Blockchain for the IoT: privacy-preserving protection of sensor data. J Assoc Inf Syst 20:1271–1307. https://doi.org/10.17705/1jais.00567
Chatterjee S, Byun J, Dutta K et al (2018) Designing an Internet-of-Things (IoT) and sensor-based in-home monitoring system for assisting diabetes patients: iterative learning from two case studies. Eur J Inf Syst 27:670–685. https://doi.org/10.1080/0960085X.2018.1485619
Coenen T, Coertjens L, Vlerick P et al (2018) An information system design theory for the comparative judgement of competences. Eur J Inf Syst 27:248–261. https://doi.org/10.1080/0960085X.2018.1445461
Connelly LM (2013) Limitation section. MEDSURG Nurs 22:2013
De Leoz G, Petter S (2018) Considering the social impacts of artefacts in information systems design science research. Eur J Inf Syst 27:154–170. https://doi.org/10.1080/0960085X.2018.1445462
Del-Río-Ortega A, Resinas M, Durán A et al (2019) Visual ppinot: a graphical notation for process performance indicators. Bus Inf Syst Eng 61:137–161. https://doi.org/10.1007/s12599-017-0483-3
Denyer D, Tranfield D, van Aken JE (2008) Developing design propositions through research synthesis. Organ Stud 29:393–413. https://doi.org/10.1177/0170840607088020
Diederich S, Lichtenberg S, Brendel AB, Trang S (2020) Promoting sustainable mobility beliefs with persuasive and anthropomorphic design: insights from an experiment with a conversational agent. In: 40th international conference on information systems ICIS 2019. pp 1–17
Dincelli E, Chengalur-Smith I (2020) Choose your own training adventure: designing a gamified SETA artefact for improving information security and privacy through interactive storytelling. Eur J Inf Syst 29:669–687. https://doi.org/10.1080/0960085X.2020.1797546
Dinter B, Krawatzeck R (2015) Towards a configurative publication schema for design science research. In: International conference on information systems ICIS 2015. pp 1–13
Feine J, Morana S, Maedche A (2019) Designing a chatbot social cue configuration system. In: 40th international conference on information systems ICIS 2019. pp1–17
French J (1953) Experiments in field settings. In: Festinger L, Katz D (eds) Research methods in behavioral sciences. Dryden Press, Hinsdale, p 101
Gray P, Hovav A (2011) Methods for studying the information systems future. In: Chiasson M et al (eds) Researching the future in information systems IFIP. Advances in information and communication technology, vol 356. Springer, Heidelberg, pp 299–316. https://doi.org/10.1007/978-3-642-21364-9_21
Gregor S (2006) The nature of theory in information systems. MIS Q 30:611–642
Gregor S, Hevner AR (2013) Positioning and presenting design science research for maximum impact. MIS Q 37:337–355
Gregor S, Jones D (2007) The anatomy of a design theory. J Assoc Inf Syst 8:312–335. https://doi.org/10.17705/1jais.00129
Gregor S, ChandraKruse L, Seidel S (2020) Research perspectives: the anatomy of a design principle. J Assoc Inf Syst 21:1622–1652. https://doi.org/10.17705/1jais.00649
Grotherr C, Semmann M, Böhmann T (2018) Engaging users to co-create - implications for service systems design by evaluating an engagement platform. In: Hawaii international conference on system sciences
Guyatt GH, Oxman AD, Vist G et al (2011) GRADE guidelines: 4. Rating the quality of evidence – study limitations (risk of bias). J Clin Epidemiol 64:407–415. https://doi.org/10.1016/j.jclinepi.2010.07.017
Heinrich P, Schwabe G (2014) Communicating nascent design theories on innovative information systems through multi-grounded design principles. Lect Notes Comput Sci 8463:148–163. https://doi.org/10.1007/978-3-319-06701-8_10
Hevner AR, March ST, Park J, Ram S (2004) Design science in information systems research. MIS Q 28:75–105
Hobert S (2019) Say hello to ’Coding Tutor’! Design and evaluation of a chatbot-based learning system supporting students to learn to program. In: 40th international conference on information systems. pp 1–17
Hovav A (2014) Using scenarios to understand the frontiers of IS: fifteen years later (a postscript). Inf Syst Front 16:347–352
Hovorka DS, Peter S (2019) How the future is done. In: 52nd Hawaii international conference on system science. Grand Wailea, pp 6290–6299
Huber RXR, Püschel LC, Röglinger M (2019) Capturing smart service systems: development of a domain-specific modelling language. Inf Syst J 29:1207–1255. https://doi.org/10.1111/isj.12269
Iivari J, Rotvit Perlt Hansen M, Haj-Bolouri A (2021) A proposal for minimum reusability evaluation of design principles. Eur J Inf Syst 30:286–303. https://doi.org/10.1080/0960085X.2020.1793697
Ioannidis JPA (2007) Limitations are not properly acknowledged in the scientific literature. J Clin Epidemiol 60:324–329. https://doi.org/10.1016/j.jclinepi.2006.09.011
Ioannidis JPA (2008) Why most discovered true associations are inflated. Epidemiology 19:640–648. https://doi.org/10.1097/EDE.0b013e31818131e7
Janiesch C, Rosenkranz C, Scholten U (2020) An information systems design theory for service network effects. J Assoc Inf Syst 21:1402–1460. https://doi.org/10.17705/1jais.00642
Kammler F, Brinker J, Vogel J, et al (2019) How do we support technical tasks in the age of augmented reality? Some evidence from prototyping in mechanical engineering. In: 40th international conference on information systems. pp 1–17
Kitchenham B (2004) Procedures for performing systematic reviews. Technical report TR/SE-0401 and NICTA technical report 0400011T.1. Staffordshire
Kolkowska E, Karlsson F, Hedström K (2017) Towards analysing the rationale of information security non-compliance: devising a value-based compliance analysis method. J Strateg Inf Syst 26:39–57. https://doi.org/10.1016/j.jsis.2016.08.005
Köster A, Bergert C, Gundlach J (2018) Information as a life vest: understanding the role of social networking sites for the social inclusion of Syrian refugees. In: 39th international conference on information systems. pp 1–9
Larsen KR, Lukyanenko R, Mueller RM, Storey VC (2020) Validity in design science research. In: International conference on design science research in information systems and technology. pp 1–15
Lee AS, Hubona G (2009) A scientific basis for rigor in information systems research. Manag Inf Syst Q 33:237–262
Legner C, Pentek T, Otto B (2020) Accumulating design knowledge with reference models: insights from 12 years’ research into data management. J Assoc Inf Syst 21:735–770. https://doi.org/10.17705/1jais.00618
Linhart A, Röglinger M, Stelzl K (2020) A project portfolio management approach to tackling the exploration/exploitation trade-off. Bus Inf Syst Eng 62:103–119. https://doi.org/10.1007/s12599-018-0564-y
Lukyanenko R, Parsons J (2020) Design theory indeterminacy: what is it, how can it be reduced, and why did the polar bear drown? J Assoc Inf Syst 21:1343–1369. https://doi.org/10.17705/1jais.00639
March ST, Smith GF (1995) Design and natural science research on information technology. Decis Support Syst 15:251–266
Meth H, Mueller B, Maedche A (2015) Designing a requirement mining system. J Assoc Inf Syst 16:799–837. https://doi.org/10.17705/1jais.00408
Miah SJ, Gammack JG, McKay J (2019) A metadesign theory for tailorable decision support. J Assoc Inf Syst 20:570–603. https://doi.org/10.17705/1jais.00544
Mingers J, Standing C (2020) A framework for validating information systems research based on a pluralist account of truth and correctness. J Assoc Inf Syst 21:117–151. https://doi.org/10.17705/1jais.00594
Morana S, Kroenung J, Maedche A, Schacht S (2019) Designing process guidance systems. J Assoc Inf Syst 20:499–535. https://doi.org/10.17705/1jais.00542
Nalchigar S, Yu E (2020) Designing business analytics solutions: a model-driven approach. Bus Inf Syst Eng 62:61–75. https://doi.org/10.1007/s12599-018-0555-z
Neville K, Pope A, O’Riordan S, Ó’Lionáird M (2018) Evaluating an emergency management decision support system with practitioner-driven scenarios: action design research. In: 39th international conference on information systems. pp 1–17
Niemöller C, Metzger D, Berkemeier L et al (2019) Mobile service support based on smart glasses. J Inf Technol Theory Appl 20:4
Olbrich S, Frank U, Gregor S et al (2017) On the merits and limits of replication and negation for IS research. AIS Trans Replication Res 3:1–19
Palvia P, Kakhki MD, Ghoshal T et al (2015) Methodological and topic trends in information systems research: a meta-analysis of IS journals. Commun Assoc Inf Syst 37:630–650. https://doi.org/10.17705/1cais.03730
Peffers K, Tuunanen T, Rothenberger MA, Chatterjee S (2007) A design science research methodology for information systems research. J Manag Inf Syst 24:45–78
Piel J-H, Hamann JFH, Koukal A, Breitner MH (2017) Promoting the system integration of renewable energies: toward a decision support system for incentivizing spatially diversified deployment. J Manag Inf Syst 34:994–1022. https://doi.org/10.1080/07421222.2017.1394044
Popper K (1982) Unended quest: an intellectual autobiography. Open Court Publishing, La Salle
Prat N, Comyn-Wattiau I, Akoka J (2015) A taxonomy of evaluation methods for information systems artifacts. J Manag Inf Syst 32:229–267. https://doi.org/10.1080/07421222.2015.1099390
Price JH, Murnan J (2004) Research limitations and the necessity of reporting them. Am J Heal Educ 35:66–67. https://doi.org/10.1080/19325037.2004.10603611
Puhan MA, Akl EA, Bryant D et al (2012) Discussing study limitations in reports of biomedical studies – the need for more transparency. Health Qual Life Outcomes 10:2–5. https://doi.org/10.1186/1477-7525-10-23
Rose J, Holgersson J, Söderström E (2019) Designing innovative digital services for government: a business model canvas adaptation. In: European conference on information systems. pp 1–13
Ross PT, Bibler Zaidi NL (2019) Limited by our limitations. Perspect Med Educ 8:261–264. https://doi.org/10.1007/s40037-019-00530-x
Schoormann T, Behrens D, Knackstedt R (2018) The noblest way to learn wisdom is by reflection: designing software tools for reflecting sustainability in business models. In: 39th international conference on information systems. pp 1–17
Schuster R, Wagner G, Schryen G (2018) Information systems design science research and cumulative knowledge development: an exploratory study. In: International conference on information systems, pp 1–17
Sein MK, Henfridsson O, Purao S et al (2011) Action design research. MIS Q 35:37–56
Shrestha A, Cater-Steel A, Toleman M (2014) How to communicate evaluation work in design science research? An exemplar case study. ACIS
Simon H (1996) The sciences of the artificial, 3rd edn. MIT Press, Cambridge
Simon BMK, Goes J (2013) Assumptions, limitations, delimitations, and scope of the study. Retrieved from Diss.com
Slack MK, Draugalis JLR (2001) Establishing the internal and external validity of experimental studies. Am J Heal Pharm 58:2173–2184. https://doi.org/10.1093/ajhp/58.22.2173
Storey MA, Engstrom E, Host M, et al (2017) Using a visual abstract as a lens for communicating and promoting design science research in software engineering. In: International symposium on empirical software engineering and measurement. pp 181–186. https://doi.org/10.1109/ESEM.2017.28
Theofanidis D, Fountouki A (2018) Limitations and delimitations in the research process. Perioper Nurs 7:155–162. https://doi.org/10.5281/zenodo.2552022
Thuan NH, Drechsler A, Antunes P (2019) Construction of design science research questions. Commun Assoc Inf Syst 44:332–363. https://doi.org/10.17705/1CAIS.04420
Vaishnavi VK, Kuechler W (2015) Design science research methods and patterns: innovating information and communication technology. CRC Press, Florida
Van Aken JE, Romme AGL (2012) A design science approach to evidence-based management. Oxford Handb Evid-Based Manag. https://doi.org/10.1093/oxfordhb/9780199763986.013.0003
Van Aken J, Chandrasekaran A, Halman J (2016) Conducting and publishing design science research. J Oper Manag 47–48:1–8
Venable J, Pries-Heje J, Baskerville R (2016) FEDS: a framework for evaluation in design science research. Eur J Inf Syst 25:77–89
Venable JR (2015) Five and ten years on: have DSR standards changed? In: DESRIST 2015: new horizons in design science: broadening the research Agenda. pp 264–279
Vom Brocke J, Hevner A, Maedche A et al (2020) Special issue editorial – accumulation and evolution of design knowledge in design science research: a journey through time and space. J Assoc Inf Syst 21:520–544. https://doi.org/10.17705/1jais.00611
Vom Brocke J, Maedche A (2019) The DSR grid: six core dimensions for effectively planning and communicating design science research projects. Electron Mark 29:379–385
Wang MTM, Bolland MJ, Grey A (2015) Reporting of limitations of observational research. JAMA Intern Med 175:1571. https://doi.org/10.1001/jamainternmed.2015.2147
Webster J, Watson RT (2002) Analyzing the past to prepare the future. MIS Q 26:xiii–xxiii
Weyns D, Iftikhar MU, Malek S, Andersson J (2012) Claims and supporting evidence for self-adaptive systems: a literature study. ICSE Work Softw Eng Adapt Self-Manag Syst. https://doi.org/10.1109/SEAMS.2012.6224395
Wickboldt C (2019) Blockchain for workshop event certificates – a proof of concept in the aviation industry. In: Eur Conf Inf Syst. pp 0–15
Wieringa R (2014) Design science methodology for information systems and software engineering. Springer, Heidelberg
Wieringa R, Daneva M (2015) Six strategies for generalizing software engineering theories. Sci Comput Program 101:136–152. https://doi.org/10.1016/j.scico.2014.11.013
Wu J, Huang L, Zhao JL (2019) Operationalizing regulatory focus in the digital age: evidence from an e-commerce context. MIS Q 43:745–764. https://doi.org/10.25300/MISQ/2019/14420
Yavchitz A, Ravaud P, Hopewell S et al (2014) Impact of adding a limitations section to abstracts of systematic reviews on readers’ interpretation: a randomized controlled trial. BMC Med Res Methodol 14:1–9. https://doi.org/10.1186/1471-2288-14-123
Zhang R, Indulska M, Sadiq S (2019) Discovering data quality problems: the case of repurposed data. Bus Inf Syst Eng 61:575–593. https://doi.org/10.1007/s12599-019-00608-0
Zschech P, Horn R, Höschele D et al (2020) Intelligent user assistance for automated data mining method selection. Bus Inf Syst Eng 62:227–247. https://doi.org/10.1007/s12599-020-00642-3
Acknowledgements
This work is funded by national funds through the FCT-Foundation for Science and Technology, I.P., within the scope of the project CISUC-UID/CEC/00326/2020 and by European Social Fund, through the Regional Operational Program Centro 2020.
Author information
Authors and Affiliations
Corresponding author
Additional information
Accepted after three revisions by Óscar Pastor.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Barata, J., da Cunha, P.R. & de Figueiredo, A.D. Self-reporting Limitations in Information Systems Design Science Research. Bus Inf Syst Eng 65, 143–160 (2023). https://doi.org/10.1007/s12599-022-00782-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12599-022-00782-8