An average behaviour and overall concept should comply to the best practices in model tuning

Most appropriate research design that implies satisfying behaviour.
================================================

Design Research in Information Systems


[Overview of Design Research] [Can Design be Research] [Outputs of Design Research]
[ An Example of Community Determined Outputs] The Philosophical Grounding of Design Research]
[ Design Research Methodology] [A Design Research Example]
[Design Research vs. Design] [Citatations On Design Research] [Resources For Design Research]

Welcome

Welcome to the page on Design Research in Information Systems (IS). The intent of the page is to provide design researchers in IS as well as others interested in design research with useful information on understanding, conducting, evaluating, and publishing design research.

Introduction

This page is dedicated to design research in Information Systems (IS). Design research is yet another "lens" or set of analytical techniques and perspectives (complementing the Positivist and Interpretive perspectives) for performing research in IS. Design research involves the analysis of the use and performance of designed artifacts to understand, explain and very frequently to improve on the behavior of aspects of Information Systems. Such artifacts include - but certainly are not limited to - algorithms (e.g. for information retrieval), human/computer interfaces and system design methodologies or languages. Design researchers can be found in many disciplines and fields, notably Engineering and Computer Science, using a variety of approaches, methods and techniques. In Information Systems, following a number of years of a general shift in IS research away from technological to managerial and organizational issues, an increasing number of observers are calling for a return to an exploration of the "IT" that underlies all IS research (Orlikowski and Iacono, 2001).

The page is organized as follows. We begin with a general overview of design research, provide its philosophical and epistemological underpinnings, and contrast design research in IS with traditional positivist and qualitative research in IS. This is followed by a section on design research methods that includes an extended discussion of a published example of design research in IS. Through the example we hope to make concrete all phases of the design research method: artifact design, construction, analysis and evaluation. This is then followed by a number of resource sub-sections that relate to design research in general as well as to design research in IS: citation lists, links to resources on the Internet for design researchers, calls for papers and links to conferences, workshops, journals and communities of practice for IS design research.

The goal is to provide the IS community with useful information on design research both in and outside of IS. The page contains numerous citations permitting the interested reader to easily access original material on and examples of this unique and dynamic IS research paradigm.

If you wish to cite this work, the complete citation information is included below. Please send suggestions for improvement to the Section Editors at: vvaishna@gsu.edu or kuechler@unr.edu
Overview of Design Research

Research

Drawing heavily from Kuhn (1996; first published in 1962) and Lakatos (1978), research can be very generally defined as an activity that contributes to the understanding of a phenomenon. In the case of design research, all or part of the phenomenon may be created as opposed to naturally occurring. The phenomenon is typically a set of behaviors of some entity(ies) that is found interesting by the researcher or by a group - a research community. Understanding in most western research communities is knowledge that allows prediction of the behavior of some aspect of the phenomenon. The set of activities a research community considers appropriate to the production of understanding (knowledge) are its research methods or techniques. Historically, some research communities have been observed to have nearly universal agreement on the phenomenon of interest and the research methods for investigating it; in this page we term these paradigmatic communities. Other research communities are bound into a nominal community by overlap in sets of phenomena of interest and/or overlap in methods of investigation. We term these pre-paradigmatic or multi-paradigmatic research communities. As of the writing of this page Information Systems is an excellent example of a multi-paradigmatic community.

Design

Design means "to invent and bring into being" [Webster's Dictionary and Thesaurus, 1992]. Thus, design deals with creating something new that does not exist in nature. The design of artifacts is an activity that has been carried out for centuries. This activity is also what distinguishes the professions from the sciences. "Schools of architecture, business, education, law, and medicine, are all centrally concerned with the process of design" [Simon, 1996; first published in 1969]. However, in this century natural sciences almost drove out the design from professional school curricula in all professions, including business, with exceptions for management science, computer science, and chemical engineering -- an activity that peaked two or three decades after the Second World War [Simon, 1996].

Simon sets out a prescription for schools of business (in which most IS departments are housed) that has motivated this page to a considerable degree: ". . . The professional schools will reassume their . . . responsibilities just to the degree that they can discover a science of design, a body of intellectually tough, analytic, partly formalizable, partly empirical teachable doctrine about the design process . . . "

To bring the design activity into focus at an intellectual level, Simon [1996] makes a clear distinction between "natural science" and "science of the artificial" (also known as design science):

A natural science is a body of knowledge about some class of things -- objects or phenomenon -- in the world (nature or society) that describes and explains how they behave and interact with each other. A science of the artificial, on the other hand, is a body of knowledge about artificial (man made) objects and phenomena designed to meet certain desired goals.

Simon further frames sciences of the artificial in terms of an inner environment, an outer environment, and the interface between the two that meets certain desired goals. The outer environment is the total set of external forces and effects that act on the artifact. The inner environment is the set of components that make up the artifact and their relationships – the organization – of the artifact. The behavior of the artifact is constrained by both its organization and its outer environment. The bringing-to-be of an artifact, components and their organization, which interfaces in a desired manner with its outer environment, is the design activity. The artifact is “structurally coupled” to its environment and many of the concepts of structural coupling that Valera (1988) and Maturana (1987) have developed for biological entities are applicable to designed artifacts.

In a perspective analogous to considering design as the crafting of an interface between inner and outer environment, design can be thought of as a mapping from function space - a functional requirement constituting a point in this multidimensional space - to attribute space, where an artifact satisfying the mapping constitutes a point in that space (Takeda, et al, 1990). Design then, is knowledge in the form of techniques and methods for performing this mapping – the know-how for implementing an artifact that satisfies a set of functional requirements.
A Short History of Design Research in Information Systems

Can Design Be Research

The question this page intends to answer in the affirmative is: can design (i.e. artifact construction) ever be considered an appropriate technique for conducting research into Information Systems? We will pursue the specific question in the next section. For the remainder of this section we discuss the question in the abstract – can design be research? – using as exemplars communities other than IS where the question of whether or not design is a valid research technique has for many years been a resounding Yes!




Owen (1997) discusses the relation of design to research with reference to a conceptual map of disciplines (Figure 1) with two axes: Symbolic/Real and Analytic/Synthetic. The horizontal axis of the map position disciplines according to their defining activities: disciplines on the left side of the map are more concerned with exploration and discovery. Disciplines on the right side of the map are characterized more by invention and making. The map’s vertical division, the symbolic/real axis, characterizes the nature of the subjects of interest to the disciplines – the nature of the phenomena that concerns the research community. Both axes are continua and no discipline is exclusively concerned with synthesis to the exclusion of analytic activities. Likewise, no activity is exclusively concerned with the real to the exclusion of the symbolic although the strong contrast along this axis between the physical science of chemistry (real) and the abstract discipline of mathematics (symbolic) is strongly and accurately indicated in the diagram.


The disciplines that lie predominantly on the synthetic side of the map are either design disciplines or the design components of multi-paradigmatic disciplines. Design disciplines have a long history of building their knowledge base through making – the construction of artifacts and the evaluation of artifact performance following construction. Architecture is a strongly construction-oriented discipline with a history extending over thousands of years. The architectural knowledge base consists of a pool of structural designs that effectively encourage the wide variety of human activities and has been accumulated largely through the post-hoc observation of successful constructions (Alexander, 1964). Aeronautical engineering provides a more recent example. From the Montigolfer balloon through WWI, the aeronautical engineering knowledge base was built almost exclusively by analyzing the results of intuitively guided designs – experimentation at essentially full scale.



Owen(1997) further presents a general model for generating and accumulating knowledge (above) that is helpful in understanding design disciplines and the design research process: “Knowledge is generated and accumulated through action. Doing something and judging the results is the general model . . . the process is shown as a cycle in which knowledge is used to create works, and works are evaluated to build knowledge." While knowledge building through construction is sometimes considered to lack rigor, the process is not unstructured. The channels in the diagram of the general model are the “systems of conventions and rules under which the discipline operates." They embody the measures and values that have been empirically developed as “ways of knowing” as the discipline has matured. They may borrow from or emulate aspects of other disciplines channels, but, in the end, they are special to the discipline and are products of its evolution."

Takeda, et al. (1990) have analyzed the reasoning that occurs in the course of a general design cycle illustrated in Figure 3. This diagram can be interpreted as an elaboration of the Knowledge Using Process arrow in Figure 2. In following the flow of creative effort through this diagram the types of new knowledge that arise from design activities and the reason that this knowledge is most readily found during a design effort will become apparent.



* An operational principle can be defined as “any technique or frame of reference about a class of artifacts or its
characteristics that facilitates creation, manipulation and modification of artifactual forms” (Dasgupta, 1996; Purao, 2002)


In this model all design begins with Awareness of a problem. Design research is sometimes called “Improvement Research” and this designation emphasizes the problem- solving/performance-improving nature of the activity. Suggestions for a problem solution are abductively drawn from the existing knowledge/theory base for the problem area (Pierce, 1931). An attempt at implementing an artifact according to the suggested solution is performed next. This stage is shown as Development in the diagram. Partially or fully successful implementations are then Evaluated (according to the functional specification implicit or explicit in the suggestion). Development, Evaluation and further Suggestion are frequently iteratively performed in the course of the research (design) effort. The basis of the iteration, the flow from partial completion of the cycle back to Awareness of the Problem, is indicated by the Circumscription arrow. Conclusion indicates termination of a specific design project.


New knowledge production is indicated in Figure 3 by the arrows labeled Circumscription and Operation and Goal Knowledge. The Circumscription process is especially important to understanding design research because it generates understanding that could only be gained from the specific act of construction. Circumscription is a formal logical method (McCarthy, 1980) that assumes that every fragment of knowledge is valid only in certain situations. Further, the applicability of knowledge can only be determined through the detection and analysis of contradictions – in common language, the design researcher learns or discovers when things don’t work “according to theory." This happens many times not due to a misunderstanding of the theory, but due to the necessarily incomplete nature of ANY knowledge base. The design process, when interrupted and forced back to Awareness of Problem in this way, contributes valuable constraint knowledge to the understanding of the always-incomplete-theories that abductively motivated the original design.


The Outputs of Design Research


Even within design research communities there is lack of consensus as to the precise objective – and therefore the desired outputs - of design research. In this page we present a broad perspective that explicates the types and levels of knowledge that can be derived from design research while reserving judgment on whether a narrower goal of design research should be held within any specific research community.


March and Smith (1995) in a widely cited paper contrasting design research with natural science research, propose four general outputs for design research: constructs, models, methods, and instantiations. Constructs are the conceptual vocabulary of a problem/solution domain. Constructs arise during the conceptualization of the problem and are refined throughout the design cycle. Since a working design (artifact) consists of a large number of entities and their relationships, the construct set for a design research experiment may be larger than the equivalent set for a descriptive (empirical) experiment.

A model is “a set of propositions or statements expressing relationships among constructs.” March and Smith identify models with problem and solution statements. They are proposals for how things are. Models differ from natural science theories primarily in intent: natural science has a traditional focus on truth whereas design research focuses more on (situated) utility. Thus a model is presented in terms of what it does and a theory described in terms of construct relationships. However a theory can always be extrapolated to what can be done with the implicit knowledge and a set of entities and proposed relationships can always be expressed as a theoretical statement of how or why the output occurs.


A method is a set of steps (an algorithm or guideline) used to perform a task. "Methods are goal directed plans for manipulating constructs so that the solution statement model is realized." Implicit in a design research method then is the problem and solution statement expressed in the construct vocabulary. In contrast to natural science research, a method may well be the object of the research program in design research. Since the axiology of design research stresses problem solving, a more effective way of accomplishing an end result – even or sometimes especially a familiar or previously achieved end result – is valued.

The final output from a design research effort in March and Smith’s explication is an instantiation which “operationalizes constructs, models and methods.” It is the realization of the artifact in an environment. Emphasizing the proactive nature of design research, they point out that an instantiation sometimes precedes a complete articulation of the conceptual vocabulary and the models (or theories) that it embodies. We emphasize this further by referring to the aeronautical engineering example given earlier in this page: aircraft flew decades before a full understanding of how such flight was accomplished. And, it is unlikely the understanding would ever have occurred in the absence of the working artifacts.

Rossi and Sein (2003) and Purao (2002) in an ongoing collaborative effort to promote design research have set forth their own list of design research outputs. All but one of these can be mapped directly to March and Smith’s list. Their fifth output, better theories, is highly significant and merits inclusion in our general list of design research outputs. Design research can contribute to better theories (or theory building) in at least two distinct ways, both of which may be interpreted as analogous to experimental scientific investigation in the natural science sense. First, since the methodological construction of an artifact is an object of theorizing for many communities (e.g. how to build more maintainable software), the construction phase of a design research effort can be an experimental proof of method or an experimental exploration of method or both.

Second, the artifact can expose relationships between its elements. It is tautological to say that an artifact functions as it does because the relationships between its elements enable certain behaviors and constrain others. However if the relationships between artifact (or system) elements are less than fully understood and if the relationship is made more visible than previously during either the construction or evaluation phase of the artifact, then the understanding of the elements has been increased, potentially falsifying or elaborating on previously theorized relationships. (Theoretical relationships enter the design effort during the abductive reasoning phase of Figure 3). For some types of research, artifact construction is highly valued precisely for its contribution to theory. Human-Computer Interface (HCI) researchers Carroll and Kellogg (1989) state that “. . . HCI artifacts themselves are perhaps the most effective medium for theory development in HCI.” Walls, Widmeyer and El Sawy (1992) elaborate the theory building potential of design and construction in the specific context of IS. Table 1 summarizes the outputs that can be obtained from a design research effort.
Output
Description

1
Constructs
The conceptual vocabulary of a domain

2
Models
A set of propositions or statements expressing relationships between constructs

3
Methods
A set of steps used to perform a task – how-to knowledge

4
Instantiations
The operationalization of constructs, models and methods.

5
Better theories
Artifact construction as analogous to experimental natural science
Table 1. The Outputs of Design Research




A different perspective on the output of design research is developed in Purao (2002) following Gregg, et al. (2001). In Figure 4 the multiple outputs of design research are classified by level of abstraction.




Explicitly the upper level of Figure 4 and implicitly the middle level, knowledge about operational principles, are theories about the emergent properties of the inner environment of the artifact (Simon, 1996). However, in any complex artifact, at either level of abstraction, multiple principles may be invoked simultaneously to explain aspects of the artifacts behavior. In this sense, the behavior of the artifact in any single design research project is overdetermined (Carroll and Kellogg, 1989). This inevitable aspect of design research has consequences discussed further in the section on Philosophical Grounding of Design Research.


An example of community determined outputs


Precisely what is obtained from a design research effort is determined by (1) the phase of research on which reflection and analysis focuses (from Figure 3) and (2) the level of abstraction to which the reflection and analysis generalize (from Figure 4). These factors in turn are strongly influenced by the community performing the research.


To illustrate the different outputs that are commonly seen as the desired result for design research, consider the same artifact development as carried out by different IS research sub-communities: database, software engineering, Human-Computer interface (HCI), decision sciences, and IS Cognitive Researchers (IS Cognitive Research Exchange -- IS CORE): the construction of a data visualization interface for complex queries against large relational databases. For all of the communities, the research is motivated by a common problem awareness: that a better interface can be developed that will allow users to more quickly and effectively obtain answers to questions about the performance of their business operations.

The theoretical impetus for the prospective improvement would vary between research communities. For the software engineering or database communities the motivation could be new knowledge of faster access techniques or visual rendering techniques. For the decision sciences community and the HCI and cognitive research communities the impetus could be new research in reference disciplines on visual impacts on cognition and/or on decision-making. The resulting artifact would be quite similar for all communities, as would the construction mechanics – the computer languages used in development, the deployment platforms, etc. However the stages of development on which observation and reflection centered and the measures used to evaluate the resultant artifact (cf. Figure 3) would be considerably different for each community. Table 2 lists the communities that might construct a data visualization artifact, the primary perspective with which they would view the artifact and the different knowledge that would emerge from the research effort as a result of the differing perspectives.
Community
Perspective
Knowledge Derived

HCI; IS CORE; Decision science
Artifact as experimental apparatus
What database visualization interfaces reveal about the cognition of complex data relationships

Database; Decision science Software engineering
Artifact as focused design principle exploration
Principles for the construction of data visualization interfaces

Database; Software engineering
Artifact as improved instance of tool.
A better data visualization interface for relational, business oriented databases.
Table 2. Design Research Perspectives and Outputs by Community



Some explications of design research in IS have stated that the primary focus is always on the finished artifact and how well it works rather than its component interactions i.e. why it works (Hevner, et al, 2004). Other writers and our example present a broader view. The apparent contradiction may simply be in how wide the net of IS Research is cast and the selection of sub-communities it is considered to contain.

General References on Design Research


The Philosophical Grounding of Design Research


Ontology is the study that describes the nature of reality: for example, what is real and what is not, what is fundamental and what is derivative?

Epistemology is the study that explores the nature of knowledge: for example, on what does knowledge depend and how can we be certain of what we know?

Axiology is the study of values: what values does an individual or group hold and why?

The definitions of these terms are worth reviewing because although assumptions about reality, knowledge and value underlie any intellectual endeavor, they are implicit most of the time for most people, including researchers. Indeed, as historians and philosophers of science have noted, in “tightly” paradigmatic communities, people may conduct research for an entire career without considering the philosophical implications of their passively received areas of interest and research methods (Kuhn, 1996; first published in 1962). It is typically only in multi-paradigmatic or pre-paradigmatic communities - such as IS - that researchers are forced to consider the most fundamental bases of the socially constructed realities (Berger and Luckman, 1966; Searle, 1995) in which they operate.

The contrasting ontological and epistemological assumptions implicit in natural science and social science research approaches have been authoritatively explicated in a number of widely cited works (Bunge, 1984; Guba and Lincoln, 1994). Gregg et al. (2001) add the meta-level assumptions of design research (which they term the Socio-technologist / developmentalist approach) to earlier work contrasting positivist and interpretive approaches to research. We have drawn from Gregg, et al. in compiling Table 3 which summarizes the philosophical assumptions of those three “ways of knowing” and have added several insights from our combined 40+ years of design research experience. Our first addition is the stress on iterative circumscription (cf. Figure 3) and how this essential part of the design research methodology iteratively determines (or reveals) the reality and the knowledge that emerge from the research effort. The second addition to Table 3 is the row labeled Axiology – the study of values. We believe it is the shared valuing of what researchers hope to find in the pursuit of their efforts that binds them into a community. Certainly the self and community valuation of their efforts and findings is a highly significant motivator for any researcher, and we were surprised to find how little stress this topic has received in the literature, especially given the significant differences in what each community values.

The metaphysical assumptions of design research are unique. First, neither the ontology, epistemology, nor axiology of the paradigm is derivable from any other. Second, ontological and epistemological viewpoints shift in design research as the project runs through circumscription cycles depicted in Figure 3. This iteration is similar to but more radical than the hermeneutic processes used in some interpretive research.

Design research by definition changes the state-of-the-world through the introduction of novel artifacts. Thus, design researchers are comfortable with alternative world-states. The obvious contrast is with positivist ontology where a single, given composite socio-technical system is the typical unit of analysis; even the problem statement is subject to revision as a design research effort proceeds. However, the multiple world-states of the design researcher are not the same as the multiple realities of the interpretive researcher: many if not most design researchers believe in a single, stable underlying physical reality that constrains the multiplicity of world-states. The abductive phase of design research (Figure 3) in which physical laws are tentatively composed into a configuration that will produce an artifact with the intended problem solving functionality virtually demands a natural-science-like belief in a single, fixed grounding reality.

Epistemologically, the design researcher knows that a piece of information is factual and knows further what that information means through the process of construction/circumscription. An artifact is constructed. Its behavior is the result of interactions between components. Descriptions of the interactions are information and to the degree the artifact behaves predictably the information is true. Its meaning is precisely the functionality it enables in the composite system (artifact and user). What it means is what it does. The design researcher is thus a pragmatist (Pierce, 1931). There is also a flavor of instrumentalism (Hendry, 2004) in design research. The dependence on a predictably functioning artifact (instrument) gives design research an epistemology that resembles that of natural-science research more closely than that of either positivist or interpretive research.

Axiologically, the design researcher values creative manipulation and control of the environment in addition to (if not over) more traditional research values such as the pursuit of truth or understanding. Certainly the design researcher must have a far higher tolerance for ambiguity than is generally acceptable in the positivist research stance. As many authors have pointed out, the end result of a design research effort may be very poorly understood and still be considered a success by the community (Hevner, et al, 2004). A practical or functional addition to an area body of knowledge, codified and transmitted to the community where it can provide the basis for further exploration, may be all that is required of a successful project. Indeed, it is precisely in the exploration of “wicked problems” for which conflicting or sparse theoretical bases exist that design research excels (March and Smith, 1995; Carroll and Kellogg, 1989).

Finally, the philosophical perspective of the design researcher changes as progress is iteratively made through the phases of Figure 3. In some sense it is as if the design researcher creates a reality through constructive intervention, then reflectively becomes a positivist observer, recording the behavior of the system and comparing it to the predictions (theory) set out during the abductive phase. The observations are interpreted, become the basis for new theorizing and a new abductive, interventionist cycle begins. In this sense design research is very similar to the action research methodology of the interpretive paradigm, however, the time frame of design research construction is enormously foreshortened relative to the social group interactions typical of action research.




Bunge (1984) implies that design research is most effective when its practitioners shift between pragmatic and critical realist perspectives, guided by a pragmatic assessment of progress in the design cycle. Purao (2002) presents a very rich elaboration on the perspective shifts that accompany any iterative design cycle. His analysis is grounded in semiotics and describes in detail how “the design researcher arrives at an interpretation (understanding) of the phenomenon and the design of the artifact simultaneously.”

References on Philosophical Grounding of Design Research


Design Research Methodology (by Example)


In this section of the page the general method underlying design research in its multiplicity of as-practiced variants is described, followed by a discussion of the method as used in a published example of IS design research.

The astute reader will recognize Figure 5, the general methodology for all design research, as a variant on Figure 3, reasoning in the design cycle. This is a logical and inevitable result of the fact that in design research knowing (Figure 3) is making (Figure 5). To better focus on the process as a research method, a column labeled Outputs has been substituted for the Logical Formalism column.


Note: there are many excellent descriptions (and diagrams) of the process of design research in IS, cf. Hevner, et al., 2004; Purao, 2002;
Gregg et al., 2001;. March and Smith, 1995; Nunamaker, et al., 1991. We chose this diagram because it emphasizes the knowledge generation
inherent in the method and because it originated in an analysis of the processes inherent in any design effort.

With reference to Figure 5 a typical design research effort proceeds as follows.

Awareness of Problem: An awareness of an interesting problem may come from multiple sources: new developments in industry or in a reference discipline. Reading in an allied discipline may also provide the opportunity for application of new findings to the researcher’s field. The output of this phase is a Proposal, formal or informal, for a new research effort.

Suggestion: The Suggestion phase follows immediately behind the proposal and is intimately connected with it as the dotted line around Proposal and Tentative Design (the output of the Suggestion phase) indicates. Indeed, in any formal proposal for design research such as one to be made to the NSF or an industry sponsor, a Tentative Design and likely the performance of a prototype based on that design would be an integral part of the Proposal. Moreover, if after consideration of an interesting problem a Tentative Design does not present itself to the researcher, the idea (Proposal) will be set aside. Suggestion is an essentially creative step wherein new functionality is envisioned based on a novel configuration of either existing or new and existing elements. The step has been criticized as introducing non-repeatability into the design research method; human creativity is still a poorly understood cognitive process. However the step has necessary analogues in all research methods; for example, in positivist research creativity is inherent in the leap from curiosity about an organizational phenomena to the development of appropriate constructs that operationalize the phenomena and an appropriate research design for their measurement.

Development: The Tentative Design is implemented in this phase. The techniques for implementation will of course vary depending on the artifact to be constructed. An algorithm may require construction of a formal proof. An expert system embodying novel assumptions about human cognition in an area of interest will require software development, probably using a high-level package or tool. The implementation itself can be very pedestrian and need not involve novelty beyond the state-of-practice for the given artifact; the novelty is primarily in the design, not the construction of the artifact.

Evaluation: Once constructed, the artifact is evaluated according to criteria that are always implicit and frequently made explicit in the Proposal (Awareness of Problem phase). Deviations from expectations, both quantitative and qualitative are carefully noted and must be tentatively explained. That is, the evaluation phase contains an analytic sub-phase in which hypotheses are made about the behavior of the artifact. This phase exposes an epistemic fluidity that is in stark contrast to a strict interpretation of the positivist stance. At an equivalent point in positivist research, analysis either confirms or contradicts a hypothesis. Essentially, save for some consideration of future work as may be indicated by experimental results, the research effort is over. For the design researcher, by contrast, things are just getting interesting! Rarely, in design research, are initial hypothesis concerning behavior completely borne out. Instead, the evaluation phase results and additional information gained in the construction and running of the artifact are brought together and fed back to another round of Suggestion (cf. the circumscription arrows of Figures 3 and 5). The explanatory hypotheses, which are quite broad, are rarely discarded, but rather are modified to be in accord with the new observations. This suggests a new design, frequently preceded by new library research in directions suggested by deviations from theoretical performance. (Design researchers seem to share Allen Newell’s conception (from Cognitive Science) of theories as complex, robust nomological networks. This conception has been observed by philosophers of science in many communities (Lakatos, 1978), and working from it Newell suggests that theories are not like clay pigeons, to be blasted to bits with the Popperian shotgun of falsification. Rather they should be treated like doctoral students. One corrects them when they err, and is hopeful they can emend their flawed behavior and go on to be ever more useful and productive (Newell, 1990).)

Conclusion: This phase is the finale of a specific research effort. Typically, it is the result of satisficing, that is, though there are still deviations in the behavior of the artifact from the (multiply) revised hypothetical predictions, the results are adjudged “good enough.” Not only are the results of the effort consolidated and “written up” at this phase, but the knowledge gained in the effort is frequently categorized as either “firm” - facts that have been learned and can be repeatably applied or behavior that can be repeatably invoked - or as “loose ends” – anomalous behavior that defies explanation and may well serve as the subject of further research.


An Example of IS Design Research

The example we have chosen to add detail and concreteness to the discussion of design research philosophy and method in Information Systems is one from the joint experience of the design research page authors. We make only two claims for this research: (1) it is a reasonable example as it comfortably encompasses all the points of the preceding discussion (2) since it is our research we are privy to and able to present a multitude of details that are rarely written up and available in journal publications. We describe the research, from conception to the first publication to be drawn from it, in phases corresponding to those in diagrams 3 and 5.

Smart Objects: A Design Research Project

Awareness of Problem

In the mid-1980’s one of the senior project participants, Vijay, began actively seeking to extend his research from designing efficient data and file structures (a primarily computer science topic) to software engineering (an area with a significant IS component). In the course of a discussion with one of his colleagues at Georgia State University (GSU) he became aware of a situation that showed research promise: development of a computerized decision support system for nuclear reactors. Three Mile Island had brought national awareness to the problems associated with safe operation of a nuclear power plant, rule based decision support systems were a current area of general IS interest, and the director of the research reactor at Georgia Tech was interested in developing a system to support its operations.

A doctoral student (Gary) was brought into the project to begin a preliminary support system development in the rule-based language Prolog. Within a few weeks it became apparent that a system to support the several thousand procedures found in a typical commercial power plant would be nearly impossible to develop in Prolog and if developed would be literally impossible to maintain. The higher-level expert system development packages available at the time (and currently) were more capable but still obviously inadequate. The difficulty of constructing and maintaining large expert systems was widely known at the time however, the Prolog pilot project gave the research group significant insights they would not otherwise have had into the root causes of the problem: continuously changing requirements and the complexity inherent in several thousand rule-based interlocking procedures. Out of detailed analysis of the failed pilot system emerged the first awareness of the problem on which the research would focus: how to construct and continuously maintain a support system for the operation of a complex, hierarchical, procedure driven environment.

Suggestion

There are many approaches to the problems of software system complexity and the research group discussed them over a period of months. Some of the alternatives that were discarded were: development of a new software development methodology specifically focused on operation support systems, automation of the maintenance function, and development of a high-level programming environment. New insights into the problem continued to emerge even as (and precisely because) potential solutions to the problem were considered. One key insight was that the system complexity resided primarily in control of the system, that is, although the individual procedures could be modeled straightforwardly, the procedure which should take precedence (control) over the others and where the results of that procedure should be routed depended in a highly complex fashion on past and present states of multiple procedures. Essential to the development of the system was the effective modeling of this complex control structure.

By this point Gary had decided to adopt the problem as his dissertation topic and under Vijay’s direction began extensive research into various mechanisms for modeling (describing in a precise, formal way) control. As the realization grew that they were in effect seeking to describe the semantics of the system, his reading began to focus especially on some of the techniques to emerge from the area of semantic modeling.

During the alternating cycles of discussion, reading and individual cogitation that characterize many design research efforts, several software engineering concepts were brought together with a final key insight to yield the ultimately successful direction for the development. During one discussion Vijay realized that the control information for the system was knowledge, identical in form to the domain knowledge in the procedures and could be modeled with rules, in the same way. However, since the execution of the individual procedures was independent of the control knowledge, the two types of rules could execute in different cycles, partitioning and greatly reducing the complexity of the overall system. Finally, the then relatively new concept of object orientation seemed the ideal approach to partitioning the total system knowledge into individual procedures. And if each object were further partitioned into a domain knowledge and an control knowledge component, and if the rules were stated in a high level English like syntax that was both executable and readable by domain experts . . .

Awareness of Problem Redoux

As noted in the general discussion of the design research method, any of its phases may be spontaneously revisited from any of the other phases. Especially in the early stages of a project, this results in a conceptual fluidity that can be disconcerting to practitioners of less dynamic paradigms. Though it is difficult in retrospect to pinpoint exactly where in the process the change occurred, by the inception of the development phase the problem statement had changed to a sub-goal implicit in the original problem statement: how to effectively model operations support systems for complex, hierarchical, procedure driven environments. [This sort of “drilling down” into the problem or re-scoping the research at a more basic level occurs frequently in all research, but is effectively part of the method in design research.]

Development

Although development of a design research artifact can be straightforward, that was not the case for smart objects. The construction was completely conceptual and involved the “discovery” through multiple thought and paper trials of the details of the novel entity that had been conceptualized at a high level in the Suggestion phase, the “smart object.”

For example: what (exactly) would the syntax be for the two types of rules, domain and control? How (exactly) should the two rule evaluation cycles for each type of knowledge interleave? Should the two types of knowledge be permitted to interact? If so, how? Should control rules have the ability to “write” or “rescind” domain rules, a la Lisp? Or vice versa?

In a conceptual development such as this, the suggestion and construction phases blur because a successful design decision is an output product. The final deliverable (from this initial development) was a conceptual model consisting of: (1) a set of meta-level rules for implementing domain knowledge and control knowledge separately, but within a single structure, the “smart object” and (2) another set of meta-rules that described how the domain and control knowledge, once “modeled” as smart objects, would be interpreted (a virtual machine for executing the smart objects.)

Evaluation

In a sense evaluation takes place continuously in a design process (research or otherwise) since a large number of “micro-evaluations” take place at every design detail decision. Each decision is followed by a “thought experiment” in which that part of the design is mentally exercised by the designer. However for the remainder of this section we will describe the “formal” evaluation that occurred after the design had stabilized.

In order to test the conceptual design, various operating environments were modeled and “hand-stepped” through the execution rules to determine that logically correct system behavior occurred at appropriate times in the simulation. The simulation that appeared in Gary’s dissertation, the first publication to result from the research, was a grocery bagging “robot.” This example had been popularized in a best-selling artificial intelligence textbook of the time and had the advantage of being a familiar logic test bed to many external evaluators of the artifact. Exponents of other IS research paradigms may find the evaluation criteria simplistic, and wonder why, for example, modeling of the nuclear power plant operating environment was not the obvious choice. The answer is: resources; the modeling and hand testing of even the grocery-bagging example occupied several man-months. During the evaluation minor redesign of the artifact (the smart object conceptual model) occurred on several occasions, a common occurrence in design research. By the end of the evaluation phase the smart object model had successfully completed simulation of numerous bagging exercises and was adjudged a success by the design team.

Conclusion

The finale for the first research effort involving Smart Objects was the codification of the problem development, design basis in prior work, the design itself, and the results of the evaluation effort in Gary’s dissertation (Buchanan, 1991) The successful defense of the dissertation at GSU required careful consideration and judgment of the artifact and its performance by a committee made up primarily of other design researchers. The core concepts were considered to have substantial merit, and Gary and Vijay produced several conference papers based on smart objects.

Epilogue

After Gary’s graduation Vijay and Gary collaborated on a paper based on the research project and submitted it to IEEE Transactions on Data and Knowledge Engineering (TDKE). The paper was returned for substantial revisions. At this point Gary’s interest in the project waned, however a recently admitted GSU CIS doctoral student (Bill) found the concepts interesting enough to enter into the research group and continue the development effort. After four years, four conference papers on smart objects and related topics and three major revisions the TKDE paper was finally published as “A Data/Knowledge Paradigm for the Modeling and Design of Operations Support Systems.” (Vaishnavi, Buchanan and Kuechler, 1997) By the time of acceptance, smart objects had been through several additional design research cycles, each focusing on the refinement of a different aspect of the original design, or a critical support function for its use-in-practice such as the methodology developed for partitioning workflow information into smart objects.

References on Design Research Methodology

Design Research vs. Design


A significant and valid question posed frequently to design researchers is: How is your research different from a design effort; what makes your work research and not simply state-of-practice design?

We propose that design research is distinguished from design by the production of interesting (to a community) new knowledge. In a typical industry design effort a new product (artifact) is produced, but in most cases, the more successful the project is considered to be, the less is learned. That is, it is generally desirable to produce a new product using state-of-practice application of state-of-practice techniques and readily available components. In fact, most product design efforts in industry are preceded by many meetings designed to “engineer the risk out of” the design effort. The risks that are identified in such meetings are the “we don’t know how to do this yet” areas that are precisely the targets of design research efforts. This is in no way meant to diminish the creativity that is essential to any design effort. We merely wish to point out that design is readily distinguished from design research (within its community of interest) by the intellectual risk, the number of unknowns in the proposed design which when successfully surmounted provide the new information that makes the effort research and assures its value.

References on Understanding Design Research in the Context of Information Systems Research

Citations on Design Research
General References on Design Research
References on Philosophical Grounding of Design Research
References on Design Research Methodology
References on Understanding Design Research in the Context of Information Systems


Resources for Design Researchers

Communities of Practice
Design Research Society (DRS) is a multi-disciplinary international learned society founded in 1967. Members of DRS are drawn from diverse backgrounds ranging from fine art to engineering to computing. The aims of DRS include advancing the theory and practice of design and understanding design research and its relationship with its education and practice. DRS is involved with such activities as organizing and sponsoring conferences, sponsoring e-mail discussion groups and a monthly e-mailed newsletter.
Design-Based Research Collective: The design research paradigm appears to have been pursued as well in education research (especially educational software and systems design). Although many of the early core readings are the same (Schon, Simon, Alexander) the later tradi



Sponsors & Academic Institutional Members