An Integrated Bibliometric Tool to Efficiently Conduct Quality Literature Reviews


  • Isabelle Walsh Skema Business School. Université Côte d'Azur.
  • Alexandre Renaud EM Normandie
  • Maximiliano Jeanneret- Medina HEG-ARC
  • Cedric Baudet HEG-ARC
  • Gaetan Mourmant IESEG


Bibliometrics, literature review, design science research, reference co-citation analysis, document bibliographic coupling analysis


In the current context of scientific information overload, researchers and practitioners alike could benefit from integrated bibliometric-based software tools to help them conduct reviews of existing literature. Using a design science research approach, and two bibliometric techniques (co-citation analysis of cited references and bibliographic coupling of citing documents) we propose a detailed workflow to conduct literature reviews and an artefact – a software tool we name ARTIREV (ARTificial Intelligence and literature REViews) that we evaluate in the management and medical fields. We show that ARTIREV addresses some issues identified in existing bibliometric software. These issues in existing tools are (1) the need for extensive bibliometric training to be able to effectively use them, (2) data cleaning that is insufficient to obtain reliable results, and (3) graphical representations, which are visually pleasing, but often difficult to interpret. The software tool resulting from our work could support the conduct of literature reviews for all prospective users: researchers and practitioners; bibliometric experts and neophytes.

Dans le contexte actuel de surcharge informationnelle scientifique, les chercheurs et les praticiens pourraient tirer profit d’un logiciel bibliométrique intégré pour les aider à conduire leurs revues de la littérature existante. En utilisant une approche de recherche ancrée dans les sciences de la conception ainsi que deux techniques bibliométriques (l’analyse de co-citation de références citées et l’analyse de couplage bibliographique de documents citant), nous proposons un workflow détaillé pour conduire des revues de littérature et un logiciel intégré nommé ARTIREV (Intelligence ARTIficielle et REVues de littérature) que nous évaluons dans les champs du management et de la médecine. Nous montrons qu’ARTIREV résout trois problèmes identifiés dans les outils existants. Ces problèmes dans les outils existants sont : (1) la nécessité d’avoir des connaissances bibliométriques approfondies pour pouvoir effectivement les utiliser ; (2) le nettoyage des données bibliographiques qu’ils proposent n’est pas suffisant pour obtenir des résultats fiables ; et (3) les représentations graphiques fournies sont visuellement plaisantes, mais souvent difficiles à interpréter. Le logiciel résultant de notre travail pourrait aider la conduite de revues de littérature pour tout type d’utilisateurs potentiels : chercheurs et praticiens, experts et néophytes en bibliométrie.

Author Biography

Isabelle Walsh, Skema Business School. Université Côte d'Azur.

Distinguished Emerita Professor, PhD, HDR.

Digitalization Academy. 


Abbasi, A., Dobolyi, D., Vance, A., & Zahedi, F. M. (2021). The phishing funnel model: A design artifact to predict user susceptibility to phishing websites. Information Systems Research, 32(2), 410-436.

Adamopoulos, P., Todri, V., & Ghose, A. (2020). Demand effects of the internet-of-things sales channel: Evidence from automating the purchase process. Information Systems Research, 32(1), 238-267. .

Alter, S. (2015). The concept of ‘IT artifact has outlived its usefulness and should be retired now. Information Systems Journal, 25(1), 47-60.

Antunes, P., Thuan, N. H., & Johnstone, D. (2022). Nature and purpose of visual artifacts in design science research. Information Systems and e-Business Management, 1-36.

Antons, D., Breidbach, C. F., Joshi, A. M., & Salge, T. O. (2021). Computational literature reviews: Method, algorithms, and roadmap. Organizational Research Methods, 1094428121991230.

Aria, M. & Cuccurullo, C. (2017). Bibliometrix: An R-tool for comprehensive science mapping analysis. Journal of Informetrics, 11(4), 959–975.

Arnott, D. & Pervan, G. (2012). Design science in decision support systems research: An assessment using the Hevner, March, Park, and Ram guidelines. Journal of the Association for Information Systems, 13(11), 923–949.

Authors (2017). Reference not communicated for anonymity purposes.

Baird, A., & Maruping, L. M. (2021). The Next Generation of Research on IS Use: A Theoretical Framework of Delegation to and from Agentic IS Artifacts. MIS Quarterly, 45(1). DOI: 10.25300/MISQ/2021/15882.

Bales, M. E., Wright, D. N., Oxley, P. R., & Wheeler, T. R. (2020). Bibliometric visualization and analysis software: State of the art, workflows, and best practices. Working paper, Cornell University.

Baskerville, R. (2008). What design science is not. European Journal of Information Systems, 17(5), 441–443.

Batistič, S. & van der Laken, P. (2019). History, evolution and future of big data and analytics: A bibliometric analysis of its relationship to performance in organizations. British Journal of Management, 30(2), 229–251.

Belussi, F., Orsi, L., & Savarese, M. (2019). Mapping business model research: A document bibliometric analysis. Scandinavian Journal of Management, 35(3), 101048.

Benbya, H., Pachidi, S., & Jarvenpaa, S. (2021). Special Issue Editorial: Artificial Intelligence in Organizations: Implications for Information Systems Research. Journal of the Association for Information Systems, 22(2). DOI: 10. 10.17705/1jais.00662.

Biggi, G. & Giuliani, E. (2021). The noxious consequences of innovation: What do we know. Industry and Innovation, 28(1), 19–41.

Bonina, C., Koskinen, K., Eaton, B., & Gawer, A. (2021). Digital platforms for development: Foundations and research agenda. Information Systems Journal.

Borgman, C. L. & Furner, J. (2002). Scholarly communication and bibliometrics. Annual Review of Information Science and Technology, 36, 3–72.

Bornmann, L. & Mutz, R. (2015). Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. Journal of the Association for Information Science and Technology, 66(11), 2215–2222.

Burton-Jones, A. & Grange, C. (2013). From use to effective use: A representation theory perspective. Information Systems Research, 24(3), 632–658.

Casprini, E., Dabic, M., Kotlar, J., & Pucci, T. (2020). A bibliometric analysis of family firm internationalization research: Current themes, theoretical roots, and ways forward. International Business Review, 29(5), 101715.

Chen, C. (2017). Science mapping: A systematic review of the literature. Journal of Data and Information Science, 2(2), 1–40.

Chen, T., Zhu, J., Zhao, Y., Li, H., Li, P., Fan, J., & Wei, X. (2021). The global state of research in pain management of osteoarthritis (2000–2019): A 20-year visualized analysis. Medicine, 100(2), e23944. doi: 10.1097/MD.0000000000023944.

Cheng, X., Su, L., Luo, X., Benitez, J., & Cai, S. (2021). The good, the bad, and the ugly: impact of analytics and artificial intelligence-enabled personal information collection on privacy and participation in ridesharing. European Journal of Information Systems, 1-25.

Cobo, M. J., López‐Herrera, A. G., Herrera‐Viedma, E., & Herrera, F. (2012). SciMAT: A new science mapping analysis software tool. Journal of the American Society for Information Science and Technology, 63(8), 1609-1630.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340.

Davison, R. M., Martinsons, M. G., & Malaurent, J. (2021). Research Perspectives: Improving Action Research by Integrating Methods. Journal of the Association for Information Systems, 22(3), 1. DOI: 10.17705/1jais.00682.

Diez-Vial, I. & Montoro-Sanchez, A. (2017). Research evolution in science parks and incubators: Foundations and new trends. Scientometrics, 110(3), 1243–1272.

Dong, J. Q., Karhade, P. P., Rai, A., & Xu, S. X. (2021). How firms make information technology investment decisions: Toward a behavioral agency theory. Journal of Management Information Systems, 38(1), 29-58.

Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., & Lim, W. M. (2021). How to conduct a bibliometric analysis: An overview and guidelines. Journal of Business Research, 133, 285-296.

Ferreira, F. A. F. (2018). Mapping the field of arts-based management: Bibliographic coupling and co-citation analyses. Journal of Business Research, 85, 348–357.

Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of marketing research, 18(1), 39-50.

Fu, R., Huang, Y., & Singh, P. V. (2021). Crowds, lending, machine, and bias. Information Systems Research, 32(1), 72-92.

Galvagno, M. & Pisano, V. (2021). Building the genealogy of family business internationalization: A bibliometric mixed-method approach. Scientometrics, 126(1), 757–783.

Gao, J., Xing, D., Dong, S., & Lin, J. (2020). The primary total knee arthroplasty: A global analysis. Journal of Orthopaedic Surgery and Research, 15, 1–12.

Gass, O., Koppenhagen, N., Biegel, H., Mädche, A., & Mueller, B. (2012). Anatomy of Knowledge Bases used in Design Science Research-A Literature Review: 7. International Conference on Design Science Research in Information Systems and Technology (DESRIST 2012).

Glaser, B. G. (1978). Theoretical sensitivity. Mill Valley, CA: Sociological Press.

Glaser, B. G. & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New York: Aldine.

Gomez-Jauregui, V., Gomez-Jauregui, C., Manchado, C., & Otero, C. (2014). Information management and improvement of citation indices. International Journal of Information Management, 34(2), 257-271.

Gregor, S. (2006). The nature of theory in information systems. MIS Quarterly, 611-642.

Gregor, S. & Hevner, A. R. (2013). Positioning and presenting design science research for maximum impact. MIS Quarterly, 37(2), 337–355.

Hevner, A. R. (2007). A three cycle view of design science research. Scandinavian Journal of Information Systems, 19(2), 87–92.

Hevner, A. R. & Chatterjee, S. (2010). Design research in information systems. In Design research in information systems. Integrated series in information systems (Vol. 22) (pp. 63–86). Berlin: Springer.

Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information system research. MIS Quarterly, 28(1), 75–105.

Hu, Y., Yu, Z., Chen, X., Luo, Y., & Wen, C. (2020). A bibliometric analysis and visualization of medical data mining research. Medicine, 99(22), e20338.

Jarneving, B. (2001). The cognitive structure of current cardiovascular research. Scientometrics, 50(3), 365–389.

Jung, J. J. (2015). Big bibliographic data analytics by random walk model. Mobile Networks and Applications, 20(4). 533–537.

Klein, H. K., & Myers, M. D. (1999). A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, 67-93.

Kovács, A., Van Looy, B., & Cassiman, B. (2015). Exploring the scope of open innovation: A bibliometric review of a decade of research. Scientometrics, 104(3), 951–983.

Kwark, Y., Lee, G. M., Pavlou, P. A., & Qiu, L. (2021). On the spillover effects of online product reviews on purchases: Evidence from clickstream data. Information Systems Research.

La Paz, A., Merigó, J. M., Powell, P., Ramaprasad, A., & Syn, T. (2020). Twenty‐five years of the Information Systems Journal: A bibliometric and ontological overview. Information Systems Journal, 30(3), 431–457.

Larsen, K. R., & Bong, C. H. (2016). A tool for addressing construct identity in literature reviews and meta-analyses. MIS Quarterly, 40(3), 529-551.

Larsen, K. R., Hovorka, D. S., Dennis, A. R., & West, J. D. (2019). Understanding the elephant: The discourse approach to boundary identification and corpus construction for theory review articles. Journal of the Association for Information Systems, 20(7), Article 15. .

Larsen, P. O. & von Ins, M. (2010). The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics 84(3), 575–603.

Lee, Z. W., Cheung, C. M., & Chan, T. K. (2021). Understanding massively multiplayer online role‐playing game addiction: A hedonic management perspective. Information Systems Journal, 31(1), 33-61.

Leidner, D. E., & Tona, O. (2021). The CARE Theory of Dignity Amid Personal Data Digitalization. MIS Quarterly, 45(1). DOI: 10.25300/MISQ/2021/15941.

Li, Z., Hong, Y., & Zhang, Z. (2021). The Empowering and Competition Effects of the Platform-Based Sharing Economy on the Supply and Demand Sides of the Labor Market. Journal of Management Information Systems, 38(1), 140-165.

Linnenluecke, M. K., Marrone, M., & Singh, A. K. (2020). Conducting systematic literature reviews and bibliometric analyses. Australian Journal of Management, 45(2), 175-194.

Lyytinen, K., Nickerson, J. V., & King, J. L. (2021). Metahuman systems= humans+ machines that learn. Journal of Information Technology, 36(4), 427-445.

Mandviwalla, M., & Flanagan, R. (2021). Small business digital transformation in the context of the pandemic. European Journal of Information Systems, 1-17.

Mao, X., Chen, C., Wang, B., Hou, J., & Xiang, C. (2020a). A global bibliometric and visualized analysis in the status and trends of subchondral bone research. Medicine, 99(22), e20406.

Mao, X., Guo, L., Fu, P., & Xiang, C. (2020b). The status and trends of coronavirus research: A global bibliometric and visualized analysis. Medicine, 99(22) e20137.

March, S. T., & Smith, G. F. (1995). Design and natural science research on information technology. Decision support systems, 15(4), 251-266.

Marchiori, D. M., Popadiuk, S., Mainardes, E. W., & Rodrigues, R. G. (2021). Innovativeness: A bibliometric vision of the conceptual and intellectual structures and the past and future research directions. Scientometrics, 126(1), 55–92.

Masiero, S., & Arvidsson, V. (2021). Degenerative outcomes of digital identity platforms for development. Information Systems Journal.

Meyer, M., Grant, K., Morlacchi, P., & Weckowska, D. (2014). Triple helix indicators as an emergent area of enquiry: A bibliometric perspective. Scientometrics, 99(1), 151–174

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Sage.

Moral-Muñoz, J. A., Herrera-Viedma, E., Santisteban-Espejo, A., & Cobo, M. J. (2020). Software tools for conducting bibliometric analysis in science: An up-to-date review. Profesional de La Informacion, 29(1), 1–20

Mousavi Baygi, R., Introna L.D., & Hultin, L.(2021), Everything flows: Studying continuous sociotechnological transformation in a fluid and dynamic digital world, MIS Quarterly, 45(1).

Nam, E. J., Han, Y., Mueller, K., Zelenyuk, A., & Imre, D. (2007). Clustersculptor: A visual analytics tool for high-dimensional data. 2007 IEEE Symposium on Visual Analytics Science and Technology (pp. 75–82). IEEE.

Noack, A. (2009). Modularity clustering is force-directed layout. Physical Review E, 79(2), 026102.

Noll, M., Fröhlich, D., & Schiebel, E. (2003). Knowledge maps of knowledge management tools – Information visualization with BibTechMon. Lecture Notes Computer Science. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), 2569, 14–27.

Olensky, M., Schmidt, M., & van Eck, N. J. (2015). Evaluation of the citation matching algorithms of CWTS and iFQ in comparison to the Web of Science. Journal of the Association for Information Science and Technology, 67(10), 2550–2564.

Pan, S. L., Li, M., Pee, L. G., & Sandeep, M. S. (2021). Sustainability design principles for a wildlife management analytics system: An action design research. European Journal of Information Systems, 30(4), 452-473.

Pee, L. G., Pan, S. L., Wang, J., & Wu, J. (2021). Designing for the future in the age of pandemics: a future-ready design research (FRDR) process. European Journal of Information Systems, 30(2), 157-175.

Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of Management Information Systems, 24(3), 45–77.

Pentland, B. T., Liu, P., Kremser, W., & Haerem, T. (2020). The dynamics of drift in digitized processes. MIS Quarterly, 44(1a), 19–47.

Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of applied psychology, 88(5), 879.

Prat, N., Comyn-Wattiau, I., & Akoka, J. (2015). A taxonomy of evaluation methods for information systems artifacts. Journal of Management Information Systems, 32(3), 229-267.

Prester, J., Wagner, G., Schryen, G. & Hassan, N. R., (2021). Classifying the ideational impact of Information Systems review articles: A content-enriched deep learning approach. Decision Support Systems, 140(1).

Renaud, A., Walsh, I., & Kalika, M. (2016). Is SAM still alive? A bibliometric and interpretive mapping of the strategic alignment research field. Journal of Strategic Information Systems, 25(2), 75–103.

Rivard, S. (2021). Theory building is neither an art nor a science. It is a craft. Journal of Information Technology, 36(3), 316-328.

Sánchez-Riofrío, A. M., Guerras-Martín, L. Á., & Forcadell, F. J. (2015). Business portfolio restructuring: A comprehensive bibliometric review. Scientometrics, 102(3), 1921–1950.

Shearer, C. (2000). The CRISP-DM model: the new blueprint for data mining. Journal of data warehousing, 5(4), 13-22.

Schiebel, E. (2012). Visualization of research fronts and knowledge bases by three-dimensional areal densities of bibliographically coupled publications and co-citations. Scientometrics, 91(2), 557–566.

Schuetz, S. W., Sykes, T. A., & Venkatesh, V. (2021). Combating COVID-19 fake news on social media through fact checking: antecedents and consequences. European Journal of Information Systems, 1-13.

Sein, M. K., Henfridsson, O., Purao, S., Rossi, M., & Lindgren, R. (2011). Action design research. MIS quarterly, 37-56.

Simon, H. A. (1996). Sciences of the artificial (3rd edtiion). Cambridge, MA: MIT Press."

Stelzer, B., Meyer-Brötz, F., Schiebel, E., & Brecht, L. (2015). Combining the scenario technique with bibliometrics for technology foresight: The case of personalized medicine. Technological Forecasting and Social Change, 98, 137–156.

Traag, V. A., Waltman, L., & van Eck, N. J. (2019). From Louvain to Leiden: Guaranteeing well-connected communities. Scientific Reports, 9(1), 1–12.

Trujillo, C. M. & Long, T. M. (2018). Document co-citation analysis to enhance transdisciplinary research. Science Advances, 4(1), 1–9.

van Eck, N. J. & Waltman, L. (2009). How to normalize co-occurrence data? An analysis of some well-known similarity measures. Science and Technology, 60(8), 1635–1651.

van Eck, N. J. & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, 84(2), 523–538.

van Eck, N. J. & Waltman, L. (2019). Accuracy of citation data in Web of Science and Scopus. arXiv preprint arXiv:1906.07011.

van Oorschot, J. A. W. H., Hofman, E., & Halman, J. I. M. (2018). A bibliometric review of the innovation adoption literature. Technological Forecasting and Social Change, 134, 1–21.

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 425-478.

Wagner, G., Lukyanenko, R., & Paré, G. (2021). Artificial intelligence and the conduct of literature reviews. Journal of Information Technology, 02683962211048201.

Walsh, I. (2015). Using quantitative data in mixed-design grounded theory studies: an enhanced path to formal grounded theory in information systems. European Journal of Information Systems, 24(5), 531-557.

Walsh, I. & Renaud, A. (2017). Reviewing the literature in the IS field: Two bibliometric techniques to guide readings and help the interpretation of the literature. Systèmes d’Information et Management, 22(3), 75–115.

Walsh, I., & Rowe, F. (2022). BIBGT: combining bibliometrics and grounded theory to conduct a literature review. European Journal of Information Systems, 1-22.

Wang, K., Xing, D., Dong, S., & Lin, J. (2019). The global state of research in nonsurgical treatment of knee osteoarthritis: A bibliometric and visualized study. BMC Musculoskeletal Disorders, 20(1), 1–10.

Wessel, L., Baiyere, A., Ologeanu-Taddei, R., Cha, J., & Blegind-Jensen, T. (2021). Unpacking the difference between digital transformation and IT-enabled organizational transformation. Journal of the Association for Information Systems, 22(1), 102-129.

Westmattelmann, D., Grotenhermen, J. G., Sprenger, M., & Schewe, G. (2021). The show must go on-virtualisation of sport events during the COVID-19 pandemic. European Journal of Information Systems, 30(2), 119-136.

Winter, R. (2008). Design science research in Europe. European Journal of Information Systems, 17(5), pp. 470–475.

Wirth, R., & Hipp, J. (2000). CRISP-DM: Towards a standard process model for data mining. Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining (pp. 29-40).

Xing, D., Zhao, Y., Dong, S., & Lin, J. (2018). Global research trends in stem cells for osteoarthritis: A bibliometric and visualized study. International Journal of Rheumatic Diseases, 21(7), 1372–1384.

Yin, R. K. (2009). Case study research: Design and methods (Vol. 5). Sage.

Yan, E. & Ding, Y. (2012). Scholarly network similarities: How bibliographic coupling networks, citation networks, cocitation networks, topical networks, coauthorship networks, and coword networks relate to each other. Journal of the American Society for Information Science and Technology, 63(7), 1313–1326.

Yoo, Y., Henfridsson, O., & Lyytinen, K. (2010). Research commentary—the new organizing logic of digital innovation: an agenda for information systems research. Information systems research, 21(4), 724-735.

Zupić, I. & Čater, T. (2015). Bibliometric methods in management and organization. Organizational Research Methods, 18(3), 429–472.





Methodology research