Research in Computer
Research in Computer
Research in Computer
www.elsevier.com/locate/jss
Abstract
In this paper, we examine the state of computer science (CS) research from the point of view of the following research questions:
To answer these questions, we examined 628 papers published between 1995 and 1999 in 13 leading research journals in the CS field.
Our results suggest that while CS research examines a variety of technical topics it is relatively focused in terms of the level at which
research is conducted as well as the research techniques used. Further, CS research seldom relies on work outside the discipline for
its theoretical foundations. We present our findings as an evaluation of the state of current research and as groundwork for future
CS research efforts.
2003 Elsevier Inc. All rights reserved.
Keywords: Topic ¼ Computing research; Research Approach ¼ Evaluative-Other; Research Method ¼ Literature analysis; Reference Disci-
pline ¼ Not applicable; Level of Analysis ¼ Profession
0164-1212/$ - see front matter 2003 Elsevier Inc. All rights reserved.
doi:10.1016/S0164-1212(03)00015-3
166 V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
rison and GeorgeÕs descriptive category and is used to and Wallace (1997) and Harrison and Wells (2000)
capture papers whose primary focus is describing a proposed a number of research methods similar to those
system. Descriptive-other (DO) was added to capture identified in the information systems studies cited above.
those papers that used a descriptive approach for de- In addition, we are aware of two papers that address
scribing something other than a system, for example, an research methods in both computer science and software
opinion piece. We added descriptive-review (DR) as a engineering. Glass (1995), for example, suggested a
subcategory into which we categorized papers whose fairly simplistic approach, derived from prior literature,
primary content was a review of the literature. which categorized methods as scientific, engineering,
The formulative research approach was subcatego- empirical, and analytical. Tichy et al. (1995) conducted a
rized into a rich set of possible entities being formulated, more general survey of articles in CS journal and con-
including processes/procedures/methods/algorithms (all ferences and found that CS research was lacking in its
categorized under FP), and frameworks and guidelines/ use of experimental methods.
standards (FF and FG, respectively). In all, there are six To assist in the categorization of the CS component
subcategories of the formulative research approach. of computing research, we added the following catego-
Our evaluative categories are based on the three ries to the above list: conceptual analysis/mathematical
alternative ‘‘evaluative’’ epistemologies identified by (CA/M) and mathematical proof to facilitate the clas-
Orlikowski and Baroudi (1991): positivist (evaluative- sification of papers that utilize mathematical techniques;
deductive in our system), interpretive (evaluative-inter- Simulation, to allow categorization of papers that uti-
pretive), and critical (evaluative-critical). We added an lized simulation as their primary research method; and
‘‘Other’’ category here to characterize those papers that concept implementation for papers whose prime re-
have an evaluative component but that did not use any search method was to demonstrate proof of a concept by
of the three approaches identified above. For example, building a prototype system. We also added the category
we classified papers that used opinion surveys to gather laboratory experiment (software) to characterize those
data (as opposed to questionnaires that used established papers that, for example, compare the performance of a
research instruments) under evaluative-other. newly-proposed system with other (existing) systems. It
is important to note that not all of the research methods
2.1.3. Classifying research method included in Table 5 are appropriate for computer science
Research method describes the specific technique research.
used in a given study. While the choice of research ap-
proach narrows the set of possible applicable research 2.1.4. Classifying unit/level of analysis
methods, there is typically a one-to-many relationship Level of analysis refers to the notion that research
between a given research approach and method. Hence, work may be conducted at one or more of several levels;
in addition to research approach, we also coded the for example, at a high level, the research may be tech-
detailed technique used by a study. nical or behavioral in nature. Example of technical re-
Unlike research approach, where there were few search would be focused on the computing system (CS),
candidate categories from which to choose, in the case computing element (CE, representing a program, com-
of research method, there were numerous classifications ponent, algorithm, or object) or abstract concept level
from which to choose. Recall that, while the objective of (AC, e.g., graph-based representations). An example of
this paper is to characterize the nature of research in behavioral research is the Watts Humphrey work on
computer science, the categories and taxonomies used in Team Software Process (http://www.sei.cmu.edu/tsp/
this paper were intended to cover the whole of the tsp.html), which would be categorized as GP for Group/
computing field, including computer science. Team, and his Personal Software Process work, which
Arguably, the computing discipline most concerned would be categorized as IN for individual (http://
with research method is Information Systems where www.sei.cmu.edu/tsp/psp.html). Some research work is
many prior publications have identified a number of done at the level of the profession (PRO), of which this
commonly used methods (see, for example, Alavi and paper is an example, as are those papers referenced in
Carlson, 1992; Farhoomand and Drury, 2000). These the introduction that address CS research in a particular
articles identify, for example, laboratory experiments country, while others may be conducted within an en-
(using human subjects), field studies, case studies, and terprise at the organizational (OC) level. Table 6 pre-
field experiment. Several other research methods have sents the levels of analysis used in this study.
also been identified; for example, conceptual analysis (or
conceptual study), literature review (Lai and Maha- 2.1.5. Classifying reference discipline
patra, 1997), instrument development (Alavi and Carl- By reference discipline, we mean any discipline out-
son, 1992), and exploratory survey (Cheon et al., 1993). side the CS field that CS researchers have relied upon for
Some studies have examined research methods spec- theory and/or concepts. Generally, a reference discipline
ific to a software engineering context. Both Zelkowitz is one that provides an important basis, such as theory,
168 V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
Table 2
Numbers of publications examined by journal and year
Overall COMP JACM KDE PAMI PDS TOCHI TODS TOG TOIS TOMCS TON TOPLAS VCG
1995 141 21 20 10 17 16 4 10 7 9 6 4 13 4
1996 128 20 16 10 17 14 7 3 7 7 6 8 9 4
1997 135 18 16 10 19 14 6 7 7 6 9 9 11 3
1998 122 17 15 8 19 12 5 5 6 7 8 6 11 3
1999 102 15 11 8 17 10 3 3 4 6 8 6 8 3
Totals 628 91 78 46 89 66 25 28 31 35 37 33 52 17
Table 8 presents the data by journal for each of the Transactions on Database Systems, and ACM Transac-
categories examined. While some of the results for the tions on Information Systems, it was data/information
discipline as a whole and the journals are somewhat concepts, and in IEEE Transactions on Pattern Analysis
predictable, some are fairly surprising. and Machine Intelligence, ACM Transactions on Graph-
ics, and IEEE Transactions on Visualization and
3.1. Findings for topic Computer Graphics, it was problem-domain-specific
concepts.
Table 3 shows that research in computer science is
spread evenly among the five categories: computer 3.2. Findings for research approach
concepts (28.67%), problem-domain-specific concepts
(21.50%), systems/software concepts (19.11%), data/ Table 4 shows the primary research approaches used
information concepts (15.45%), and problem-solving by CS researchers. Formulative was by far the dominant
concepts (14.65%). Two other categories, systems/soft- research approach representing 79.15% of the papers
ware management concepts, and organizational con- assessed, followed by evaluative and descriptive ap-
cepts, are represented minimally, while two categories, proaches, which were virtually equivalent at 10.98% and
societal concepts and disciplinary issues are not repre- 9.88%, respectively.
sented at all. Examination of the sub-categories of research ap-
The leading sub-category was computer graphics/ proach shows that FP, a multifaceted subcategory that
pattern analysis within the problem-domain-specific includes formulating processes, procedures, methods, or
concepts category. Twenty percent of articles were de- algorithms is the most important of the formulative sub-
voted to this category, while 17.68% were devoted to categories. Approximately half of computer science re-
inter-computer communication (part of computer con- search (50.55%) fell into this category. The next largest
cepts), which includes such topics as networking and category was FC (e.g., formulating a concept such as a
distributed systems. Other notable topics were com- data model), at 17.04%. Papers whose primary focus
puter/hardware principles/architecture at 10.19% (again was evaluation using techniques other than deductive,
part of computer concepts) and database/warehouse/ interpretive, or critical approaches (evaluative-other)
mart organization at 8.44% (part of data/information were third at 9.87%.
concepts), while papers focusing on mathematics/com- Table 8 (Panel B) shows the primary research ap-
putational science (part of problem-solving concepts) proaches by journal. The data shows that FP (formu-
were next at 6.69%. late-process, method, or algorithm) was the most
Table 8 (Panel A) presents the topics by journal. The important research approach in 12 of the 13 journals
results show that most journals tended to have a single examined while formulate-concept (FC) was the second
dominant topic as suggested by their title. These topics, most important approach (in 8 out of those 12 journals).
then, broadly define the sub-fields that make up the ACM Transactions on Computer–Human Interaction was
discipline of computer science. We found that 2 or 3 of the only journal in which the formulative research
the 13 journals typically focused on the same topic area. category did not dominate. Instead, 40% of the papers in
For example, the principal topic category in Journal of that journal were devoted to evaluative studies (evalu-
the ACM and ACM Transactions on Modeling and ative-deductive and evaluative-other at 20% each), with
Computer Simulation was problem-solving concepts; in a further 20% devoted to system descriptions (DS).
IEEE Transactions on Computers, IEEE Transactions on Other journals with significant numbers of evaluative
Parallel and Distributed Systems, and IEEE/ACM studies were ACM Transactions on Programming Lan-
Transactions on Networking it was computer concepts; guages and Systems (21.15%) and Journal of the ACM
in ACM Transactions on Computer–Human Interaction (20.51%).
and ACM Transactions on Programming Languages and Our results suggest, therefore, that the focus in most
Systems, it was systems/software concepts, in IEEE areas of computer science research is primarily on for-
Transactions on Knowledge and Data Engineering, ACM mulating things.
170 V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
Table 3
Findings for computing topics
1.0 Problem-solving concepts 14.65%
1.1 Algorithms 5.57%
1.2 Mathematics/computational science 6.69%
1.3 Methodologies (object, function/process, information/data, event, business rules,. . .) –
1.4 Artificial intelligence 2.39%
2.0 Computer concepts 28.67%
2.1 Computer/hardware principles/architecture 10.19%
2.2 Inter-computer communication (networks, distributed systems) 17.68%
2.3 Operating systems (as an augmentation of hardware) 0.80%
2.4 Machine/assembler-level data/instructions –
3.0 Systems/software concepts 19.11%
3.1 System architecture/engineering 0.48%
3.2 Software life-cycle/engineering (including requirements, design, coding, testing, maintenance) –
3.3 Programming languages 3.82%
3.4 Methods/techniques (including reuse, patterns, parallel processing, process models, data models. . .) 3.82%
3.5 Tools (including compilers, debuggers) 5.25%
3.6 Product quality (including performance, fault tolerance) 1.75%
3.7 Human–computer interaction 3.18%
3.8 System security 0.80%
4.0 Data/information concepts 15.45%
4.1 Data/file structures 1.91%
4.2 Data base/warehouse/mart organization 8.44%
4.3 Information retrieval 3.98%
4.4 Data analysis 0.64%
4.5 Data security 0.48%
5.0 Problem-domain-specific concepts (use as a secondary subject, if applicable, or as a primary subject if there is no other choice) 21.50%
5.1 Scientific/engineering (including bio-informatics) 0.48%
5.2 Information systems (including decision support, group support systems, expert systems) 0.64%
5.3 Systems programming –
5.4 Real-time (including robotics) 0.16%
5.5 Computer graphics/pattern analysis 20.22%
6.0 Systems/software management concepts 0.32%
6.1 Project/product management (including risk management) 0.32%
6.2 Process management –
6.3 Measurement/metrics (development and use) –
6.4 Personnel issues –
7.0 Organizational concepts 0.32%
7.1 Organizational structure –
7.2 Strategy –
7.3 Alignment (including business process reengineering) –
7.4 Organizational learning /knowledge management –
7.5 Technology transfer (including innovation, acceptance, adoption, diffusion) 0.16%
7.6 Change management –
7.7 Information technology implementation –
7.8 Information technology usage/operation –
7.9 Management of ‘‘computing’’ function 0.16%
7.11 IT impact –
7.11 Computing/information as a business –
7.12 Legal/ethical/cultural/political (organizational) implications –
8.0 Societal concepts –
8.1 Cultural implications –
8.2 Legal implications –
8.3 Ethical implications –
8.4 Political implications –
9.0 Disciplinary issues –
9.1 ‘‘Computing’’ research –
9.2 ‘‘Computing’’ curriculum/teaching –
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176 171
Table 4 Table 7
Findings for research approach Findings for reference discipline
Descriptive: 9.88% CP Cognitive psychology 0.80%
DS Descriptive-system 4.14% SB Social and behavioral science –
DO Descriptive-other 5.10% CS Computer science 89.33%
DR Review of literature 0.64% SC Science 0.96%
EN Engineering –
Evaluative: 10.98% EC Economics –
ED Evaluative-deductive 1.11%
LS Library science –
EI Evaluative-interpretive – MG Management –
EC Evaluative-critical – MS Management science –
EO Evaluative-other 9.87%
PA Public administration –
Formulative: 79.15% PS Political science –
FF Formulative-framework 2.39% MA Mathematics 8.60%
FG Formulative-guidelines/standards 0.64% OT Other 0.32%
FM Formulative-model 5.73%
FP Formulative-process, method, algorithm 52.55%
FT Formulative-classification/taxonomy 0.80%
FC Formulative-concept 17.04%
3.3. Findings for research method
11.76%
88.24%
82.35%
17.65%
organizational context, external business context, pro-
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
ject, and group/team were not represented.
Table 8 (Panel D) presents level of analysis by jour-
nal. The data shows that CE was the primary level of
7.69%
5.77%
55.77%
36.54%
94.23%
analysis in 8 of the 13 journals. The figures ranged from
a low of 51.69% in IEEE Transactions on Pattern
–
–
–
–
–
–
–
–
–
–
–
–
–
Analysis and Machine Intelligence to a high of 88.24%
in IEEE Transactions on Visualization and Computer
3.03%
36.36%
63.84%
96.97%
Graphics (VCG). Further, AC was the primary level of
–
–
–
–
–
–
–
–
–
–
–
–
–
analysis in four journals ranging from 42.86% to
56.04%, while Individual was the primary level of anal-
ysis in ACM Transactions on Computer–Human Inter-
45.95%
16.22%
37.84%
18.92%
81.08%
–
–
–
–
–
–
Transactions on Graphics (TOG) were the only journals
to publish articles that used a non-technical level of
analysis (i.e., levels of analysis other than AC, CS 6 or
42.86%
17.14%
40.00%
97.14%
2.86%
–
–
–
–
–
–
70.97%
90.32%
6.45%
6.45%
3.23%
–
–
–
–
–
100.00%
35.71%
60.71%
–
–
–
–
–
–
–
–
–
20.00%
72.00%
–
–
–
–
–
–
87.88%
–
–
–
–
–
–
–
–
–
2.25%
7.87%
44.94%
51.69%
92.13%
–
–
–
–
–
–
–
–
–
–
–
2.17%
34.78%
10.87%
52.17%
97.83%
–
–
–
–
–
–
–
–
–
–
–
–
–
57.69%
41.03%
57.69%
–
–
–
–
–
1.10%
56.04%
39.56%
98.90%
–
–
–
–
–
–
–
–
–
–
–
–
–
1.91%
5.57%
0.80%
8.60%
0.32%
0.96%
38.85%
53.34%
89.33%
–
–
–
–
–
–
–
–
Panel E:
PRO
EXT
SOC
MG
MA
MS
OC
AC
OT
GP
CE
EC
PR
CP
CS
CS
SC
SB
IN
IS
6
Computer System.
174 V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
CS research is fairly evenly distributed across five CE dominates AC by eight journals to four. It is,
major topic areas: problem-solving concepts, computer however, interesting to note that each of the four jour-
concepts, systems/software concepts, data/information nals in which AC is dominant focuses on one of the
concepts and problem-domain-specific concepts. The major topic categories; the only topic category that is
leading category is computer concepts, with problem- not the focus of one or more of the journals we studied is
domain-specific concepts (principally computer graphics problem-domain-specific concepts.
and pattern analysis) second. As would be expected, Note that we used our classification system to record
there is very little work in the area of systems/software the keywords describing this paper (following the ab-
management concepts (two papers), one paper on or- stract). The paper is classified as follows: (a) the topic is
ganizational concepts, and no papers that examined computing research (9.1); (b) the research approach is
societal concepts or disciplinary issues. EO (evaluative-other) because our paper is about eval-
In terms of both research approach and research uating CS research; (c) the research method is LR (lit-
method, CS research tends to be quite focused. The erature review/analysis); (d) the level of analysis is the
‘‘formulate’’ research approach category accounts for profession (PRO); and (e) the reference discipline is
almost 80% of the research with a majority of papers none because we did not use concepts from other dis-
being devoted to formulating a process, method, or al- ciplines in performing the study. We encourage authors
gorithm. The preferred research method is conceptual in the future to use our classification system not only to
analysis based on mathematical techniques. select keywords but also to write abstracts. Such a
With regard to levels of analysis, CS research falls practice would aid researchers to assess the relevance of
primarily into the CE or AC categories confirming that published research to their own endeavors.
the mission of CS is to conduct research that is focused A study of this nature is not without limitations. The
on technical levels of analysis. As would be expected, first limitation stems from the choice of journals. The
very little research focused on the society or profession results of our study reflect the nature of computer sci-
categories. ence research to the extent that these journals are rep-
With respect to reference disciplines, our data shows resentative of the field. While there are many other
that CS research seldom relies on research in other magazines, and high-quality research conferences that
disciplines and in the rare instances that it does, it relies publish CS research articles, we chose to analyze only
primarily on mathematics. articles published in journals because of the traditional
Table 9 presents a summary of the most important and enduring role that journals play in the development
research characteristics in each of the 13 journals. The of academic disciplines. A second potential limitation
data indicate that while CS research addresses a diverse arises from the fact that we coded only a sample of the
range of topics, there is a high degree of consistency in articles published in the selected journals. Given, how-
terms of the research approaches, research methods, and ever, that we used a systematic sampling procedure, we
levels of analysis used to study these topics. Further, have no reason to believe that the results are biased. A
across the 13 journals studied, ACM Transactions on final limitation arises from the subjective nature of the
Computer–Human Interaction is a clear outlier. It is, for coding process. We attempted to reduce the subjectivity
example, the only journal not to have FP (formulate by using two independent coders who revisited the
process, method or algorithm) as the predominant re- articles to resolve any disagreements. The relatively
search approach and CA/M as the predominant re- high-level of raw agreements suggests that articles were
search method. From the viewpoint of level of analysis, indeed coded in a consistent manner.
Table 9
Summary of characteristics of journals
Journal Principal topic Research approach Research method Level of analysis Reference discipline
TOMCS Problem-solving FP CA/M AC CS
JACM Problem-solving FP CA/M CE CS
COMP Computer FP CA/M AC CS
PDS Computer FP CA/M CE CS
TON Computer FP CA/M CE CS
TOIS Data/information FP CA/M AC CS
TODS Data/information FP CA/M CE CS
KDE Data/information FP CA/M CE CS
PAMI Problem-domain-specific FP CA/M CE CS
TOG Problem-domain-specific FP CA/M CE CS
VCG Problem-domain-specific FP CA/M CE CS
TOPLAS Systems/software FP CA/M AC CS
TOCHI Systems/software DS, ED, EO CA IN CS
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176 175
5. Conclusion Barki, H., Rivard, S., Talbot, J., 1988. An information systems
keyword classification scheme. MIS Quarterly 12 (2), 299–
322.
We examined 628 papers (over a 5-year period) in 13 Cheon, M.J., Grover, V., Sabherwal, R., 1993. The evolution of
leading research journals in the CS field from 1995 to empirical research in IS: a study in IS maturity. Information and
1999 to answer questions regarding the nature of CS Management 24 (5), 107–119.
research. We characterized CS research in terms of the Estivili-Castro, V., 1995. Computer Science research in Mexico. IEEE
topics, research approaches, research methods, levels of Computer 28 (8), 56–62.
Farhoomand, A.F., Drury, D.H., 2000. A Historiographical Examin-
analysis, and reference disciplines used. Our results ation of Information Systems. Communication of the AIS Vol. 1,
suggest that CS research focuses on a variety of tech- Article 19, 2000.
nical topics, using formulative approaches to study new Fletcher, P., 1995. The role of experiments in Computer Science.
entities that are either computing elements or abstract Journal of Systems and Software 30 (1–2), 61–163.
concepts, principally using mathematically-based re- Gallopoulous, E., Sameh, A., 1997. CSE: content and product. IEEE
Computational Science and Engineering 4 (2), 39–43.
search methods. Geist, R., Chetuparambil, M., Hedetniemi, S., Turner, A.J., 1996.
The results from our study should be of value to both Computing research programs in the US. Communications of the
researchers and doctoral students engaged in computer ACM 39 (12), 96–99.
science research. For example, our study provides a Glass, R.L., 1992. A comparative analysis of the topic areas of
characterization of the types of articles that computer Computer Science, Software Engineering, and Information Sys-
tems. Journal of Systems and Software (November), 277–289.
science journals publish. Researchers can use this Glass, R.L., 1995. A structure-based critique of contemporary
knowledge to make choices when deciding on a target computing research. Journal of Systems and Software (January).
journal for their research. Our results can also be used to Glass, R.L., Vessey, I., Ramesh, V., 2002. Research in software
provide insights into areas of CS research that are re- engineering: an analysis of the literature. Information and Software
ceiving little research attention. For example, in terms of Technology 44, 491–506.
Gruman, G., 1990. Conference tries to define computing-research
research approaches, our results clearly suggest that agenda. IEEE Software 7 (1), 127–132.
insufficient emphasis is being placed on the use of Harrison, R., Wells, M., 2000. A Meta-Analysis of Multidisciplinary
evaluative methodologies. However, while our results Research. In: Papers from The Conference on Empirical Assess-
clearly support Tichy et al.Õs (1995) claim regarding the ment in Software Engineering (EASE), pp. 1–15.
lack of focus on evaluation in CS research, Fletcher Hull, R., King, R., 1987. Semantic database modeling: survey,
applications and research issues. ACM Computing Surveys 19
(1995) cautions that the use of experimental methods (3), 201–260.
may not always be appropriate in computer science, a Lai, V.S., Mahapatra, R.K., 1997. Exploring the research in informa-
caveat that should be kept in mind. tion technology implementation. Information and Management 32,
Further, funding organizations such as NSF could 187–201.
use the findings of our research to provide focused calls Morrison, J., George, J.F., 1995. Exploring the Software Engineering
component in MIS research. Communications of the ACM 38 (7),
for proposals aimed at fostering research in particular 80–91.
areas or using particular approaches/methods. It is im- Orlikowski, W., Baroudi, J.J., 1991. Studying information technology
portant to note, however, that any interpretation of gaps in organizations: research approaches and assumptions. Informa-
represented in or findings must take into account the tion Systems Research 2 (1), 1–28.
fact the classification scheme was developed to cover a Ramamritham, K., 1997. Computer Science research in India. IEEE
Computer, 40–47.
broader scope than computer science, alone, by also Rice, J.R., 1995. Computational science and the future of computing
including the disciplines of software engineering, and research. IEEE Computational Science and Engineering 2 (4), 34–
information systems. Hence, for example, while our re- 41.
sults clearly show that there is a lack of emphasis on Sheth, A., Larson, J., 1990. Federated database systems for managing
organizational aspects of computing, that is the focus of distributed, heterogeneous, and autonomous databases. ACM
Computing Surveys 22 (3), 183–236.
IS researchers (see Vessey et al., 2002) and does not Swanson, E.B., Ramiller, N., 1993. Information systems research
necessarily represent opportunities for CS researchers. thematics: submissions to a new journal 1987–1992. Information
We hope that our evaluation of the state of current Systems Research 4 (4), 299–330.
CS research fosters future CS research efforts. Tichy, W.F., Lukowicz, P., Prechelt, L., Heinz, E., 1995. Experimental
evaluation in computer science: a quantitative study. Journal of
Systems and Software (January).
Vessey, I, Ramesh, V., Glass, R.L., 2001. Research in Computing
Disciplines: A Comprehensive Classification Scheme, Indiana
References University: Working Paper (TR 107-1), http://www.kelley.iu.edu/
ardennis/wp/ (accessed 02/08/02), 2001.
ACM CCS: ACM Computing Classification System [1998 version], Wegner, P., Doyle, J., 1996. Strategic directions in computing research.
http://www.acm.org/class/1998, 1998, accessed 11/06/01. ACM Computing Surveys 28 (4), 565–574.
Alavi, M., Carlson, P., 1992. A review of MIS research and Westin, S.M., Roy, M., Kim, C.K., 1994. Cross-fertilizations of
disciplinary development: implications for deans/administrators. knowledge: the case of MIS and its reference disciplines. Informa-
Journal of Management Information Systems 8, 45–62. tion Resources Management Journal 7 (2), 24–34.
176 V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
Zelkowitz, M.V., Wallace, D., 1997. Experimental validation in periodicals including Communications of the ACM (the ‘‘Practical
software engineering. Information and Software Technology 39, Programmer’’ column) and IEEE Software (‘‘The Loyal Opposition’’).
He was for 15 years a Lecturer for the ACM, and was named a Fellow
735–743. of the ACM in 1998. He received an honorary Ph.D. from Linkoping
University in Sweden in 1995. He describes himself by saying ‘‘my head
Ramesh Venkataraman, Ph.D., is an Assistant Professor of Informa- is in the academic area of computing, but my heart is in its practice.’’
tion Systems and Ford Motor Company Teaching Fellow in the De-
partment of Accounting and Information Systems at Kelley School of
Business, Indiana University. He has published over 25 papers in Iris Vessey is Professor of Information Systems at Indiana UniversityÕs
leading journals, books, and conferences. His areas of expertise are in Kelley School of Business, Bloomington. She received her M.Sc.,
database modeling and design, systems design and development, het- MBA, and Ph.D. in MIS from the University of Queensland, Aus-
erogeneous databases, and groupware systems. tralia. Her research into evaluating emerging information technologies
from both cognitive and analytical perspectives has been published in
journals such as Communications of the ACM, Information Systems
Robert L. Glass is president of Computing Trends, publishers of The Research, Journal of Management Information Systems, and MIS
Software Practitioner. He has been active in the field of computing and Quarterly. In recent years, her interests have focused on managerial
software for over 45 years, largely in industry (1954–1982 and 1988– issues associated with the management and implementation of enter-
present), but also as an academic (1982–1988). He is the author of over prise systems. She currently serves as Secretary of the Association for
20 books and 70 papers on computing subjects, Editor Emeritus of Information Systems and of the International Conference on Infor-
ElsevierÕs Journal of Systems and Software, and a columnist for several mation Systems and is an AIS Fellow.