Instructional strategy considerations

What are the most important considerations when choosing a particular instructional strategy?

1. learning objectives - For example, if my learning objectives focus on learning a procedure, I would not choose Problem-solving instructional strategy. If learning objectives state learners are to learn skill to make something, I'm going to use something like Shank's Learning by Doing theory. As part of this, learning objectives need to be able to be categorized in a learning taxonomy like from Tripartite (Hilgard 1980), Gagne (1985), Bloom (1956), Anderson (1981), Merrill (1983), Reigeluth & Moore (1999), etc.

Is it important for students to drive the speed of the course, are experience necessary for the experience, are teacher-directed methods appropriate?

2. desired/prescribed learning environment - setting, use of simulation, importance of hands on experience, configurations
3. available resources - time, personnel; if there is limited time, consider applying teacher-directed strategies
4. learners/target audience - do learner characteristics really change the instructional strategy? 2c argues no. It might affect the delivery mode, though. Checklists are most appropriate for procedures with one answer. Higher order thinking with multiple correct answers are best to use with assessment rubrics. Novice versus expert - novices might need a more structured learning strategy where they have lots of scaffolding and have it slowly taken away.

Types of tests

Entry test - prerequisites Pretest - adapt instruction based on results Practice tests Final test

• normative referenced
• ISD doesn't do this because it doesn't measure skills; psychometrics
• critereon referenced
• conventional - m/c, t/f; good for assessing verbal outcomes, some concepts and principles
• performance/product checklist - procedural learning; one way to do something, one way of learning
• rubrics - multiple ways of doing something

Comparison of Alternative Instructional Design Models

by: Marlene Fauser, Kirk Henry, and David Kent Norman on February 4, 2006

An Instructional Design model gives structure and meaning to a problem, enabling the would-be designer to negotiate the design task using a process or systematic method. Models help us to visualize the problem, to break it down into discrete, manageable units. The true value of a model can only be determined within the context of use. A model should be judged by how it mediates the designer's intention, how well it can share a work load, and how effectively it shifts focus away from itself toward the object of the design activity (Ryder, 2006).

The purpose of this paper is to analyze three instructional design models, one from each of the classifications classroom, product, and system. The analysis should compare and contrast the key concepts associated with each model. Additionally this paper will identify and describe arguments against the use of ISD models in general.

Instructional Design models are classified into three types, classroom, product and system (Gustafson & Branch, 2002). Classroom models are of interest to, and are usually designed for, professional teachers from K-12, community colleges, vocational schools, and other related areas. These models take into consideration the environment of teachers. Their users may view them as a guide rather than a methodology. These models focus on the use of existing materials rather than the design of new. The output of these models is small, a unit or module of instruction used within the school year. The models also have less rigorous formative evaluation and revision than product or system models.

Product-Oriented models, prescriptive in nature, are primarily focused on creating instructional products. These instructional products may be self-study, self-paced computer based training, or other materials that can be used by a student with reduced guidance. The methodology of these models may cause them to be confused with system models, but product models are not used to create comprehensive systems as system models are. Product development models demonstrate four assumptions: the instructional product is needed, something needs to be produced rather than using or modifying existing materials, there will be considerable emphasis on tryout and revision, and the product must be usable by learners with only facilitators but not teachers. These models are drawn upon as computer-based instruction has become more often the instructional delivery method of choice (Gustafson & Branch, 2002).

Systems-oriented models are used to develop large amounts of instruction. This can be courses or curricula and may include the development of new materials or the repurposing of existing materials. Systems models align with the ADDIE methodology, emphasizing front-end analysis and design phases. Gustafson and Branch observed emphasis on front-end analysis may result in a proposal for a non-training solution (2002). Systems models, unlike product development models, typically assume a large scope of effort.

Classroom Oriented Model - Gerlach and Ely

The Gerlach and Ely Model is a prescriptive model that is well suited to K-12 and higher education. It is meant for novice instructional designers who have knowledge and expertise in a specific context (Braxton, Bronico, & Looms, 1995). Although drawn as a linear model, many of the steps are intended to occur simultaneously. The Gerlach and Ely Model recognizes most curriculum will be designed around the concepts to be taught in each subject matter and teachers, who are forced to adapt existing materials for use in their courses (Braxton, et al., 1995).

The first step in the Gerlach and Ely process is to simultaneously specify the content and objectives. With this model intended for K-12 teachers, these combined steps recognize that teachers know the content or think of content first, then define the objectives for that content. The objectives do need to be defined, however, for use in the next steps of the model.

The second step is to specify the entry point of learners, which for K-12 teachers may simply mean reviewing existing records. The third step is to perform five activities simultaneously: (1) determine strategy, (2) organize groups, (3) allocate time, (4) allocate space, and (5) select resources. The key is to determine which combinations will best allow the students to meet their objectives. The Gerlach and Ely model emphasizes using existing instructional materials rather than develop new materials.

After these simultaneous decisions are made, the next step is evaluation of performance. The teacher should determine what observable or measurable changes occurred in the students and if the observations can be attributed to the instruction. The model concludes with a feedback loop to determine the effectiveness of the instructions so changes can be made as necessary.

Product Oriented Model - Bates

In 1995, Tony Bates presented a model for developing open and distance learning. Bates realized digital communications were going to be used more frequently for distance education. Early in his research, he acknowledged the fact that there are inherent limitations of distance education. Bates created a framework for selecting learning technologies which is still relevant today. The organizing framework that Bates proposed is called ACTIONS, which stands for access, cost, teaching and learning implications, interaction, organizational issues, novelty and speed.

Bates's model has what he calls a front-end system design. A front-end system design has four phases: course outline development, selection of media, development/production of materials, and course delivery. Within each phase, Bates has properly identified the team roles and the actions or issues to be addressed. Bates readily admits he borrows heavily from the ADDIE model from which he derives many of his ideas. One of the reasons Bates may have felt compelled to create an additional model to ADDIE is because ADDIE has been widely and often criticized for being too systematic. ADDIE is often said to be too linear, too constraining, and even too time-consuming to implement with regards to e-learning. While Bates may have been making an attempt to simplify ADDIE, he cautions course design may take as much as two years.

One of his main concerns, prompting the development of ACTIONS, was the way face to face course material was adapted for web or other distance learning approaches. Bates feels the traditional remote instructor concept is nothing more than face-to-face instruction without direct interaction. Bates also states these specific scenarios often fail to take advantage of the unique benefits that are available through the specific technology being used.

Additionally, in his 1995 book, Technology: Open Learning and Distance Education, Bates details cost analysis when designing distance or e-learning solutions. If his design and evaluation methods are followed, one would easily be able to determine the effect each dollar spent on this e-learning solution has had on a given student. This can be particularly useful when dealing with fiscal management issues.

The Bates Model provides the user with a plethora of tools and ideas, each of which will be useful for designing e-learning environments. The model is broken down into four distinct groups. Each group is replete with projects and assignments.

System Oriented Model - Gentry IPDM

Castelle Gentry explains his model of Instructional Development in the course of a textbook. Gentry named his model the Instructional Project Development and Management (IPDM) Model. The IPDM model was born as a result of five conclusions Gentry made with regards to instructional design. The first conclusion admonishes introductory instructional design textbooks for not adequately linking relationships between instructional design processes and their supporting processes (Gentry, 1994, p. ix). The second of Gentry's conclusions says introductory instructional design textbooks should teach both what and how to "do", in other words "practical means for accomplishing specific tasks" (Gentry, 1994, p. ix). The third through fifth conclusions set a basis for Gentry to develop a generic Instructional Design model as a survey of instructional design processes, supporting processes, and techniques.

The result IPDM Model is has eight development components, defined by Gentry (1994) in a non-linear diagram.

1. Needs analysis - establish the validity of needs and goals for existing or proposed instruction
2. Adoption - establish acceptance of an innovation by those affected and obtain a commitment of resources.
3. Instructional Design - determine and specify objectives, strategies, techniques, and media for meeting instructional goals.
4. Production - construct elements of a project, as specified in a design and revision data.
5. Prototyping - assemble, pilot test, respecify, validate, and finalize an instructional unit.
6. Product Installation – establish the necessary conditions for effective operation of a new instructional product or process.
7. Ongoing Operation – maintain the continuing application of an instructional product and/or procedure.
8. Ongoing Instructional Unit Evaluation – collect and analyze data about an ongoing instructional unit to make decisions about future revision.

The development components have five supporting components, defined by Gentry (1994, p. 5):

1. Project Management – control, coordinate, and allocate resources.
2. Information Handling – select, collect, organize, store, retrieve, distribute, and assess information required by an ID project.
3. Resource Acquisition and Allocation – determine resource needs, formalize budgets, and manage resources.
4. ID Project Personnel – determine needs for staffing, training, assessment, motivation, counseling, censuring, and dismissing ID project members.
5. Facilities – organize and renovate spaces for design, implementation, and testing of elements of instruction.

The development and supporting components are linked by information sharing between the two clusters for the duration of the project. Gentry designed his model to show instructional design is not a linear process.

Model Comparison and Contrast

When doing the comparison between Bates, Gentry and Gerlach/Ely's models, it was noted that all three models allowed for some variation in implementation; it would not be incorrect to modify the order of the steps depending on the settings of the instructional situation. This is especially true for the Gerlach/Ely model. The three models we have chosen to review all were published between 1994 and 1995.

The Bates and Gentry models both rely heavily on front end work. This means that the model assumes that step one will involve an extensive needs analysis. The information learned from the analysis will be crucial in the next phases of the design. In particular, the Gentry IPDM model is best for large scale projects, as demonstrated by the communication core of the model. Both Bates and Gentry also provide a framework for helping the designers determine the overall cost of the solution. This may include the cost of the technology needed for implementation. Bates and Gentry are also very good at providing personnel assignments for each phase of the design. This allows the designer to determine, up front, the type of personnel that will be needed throughout the design and implementation.

The main strength of the Gerlach/Ely model is practicing classroom teachers can identify with the process it suggests. The Gerlach/Ely model allows for a novice instructional designer whereas Gentry and Bates need some design expertise to perform front-end analysis. As a result, classroom teachers might be more likely to apply Gerlach/Ely. The Gerlach/Ely model differs from the other two by emphasizing existing content as the basis for new instruction. Basing new instruction on old content by forgoing front-end analysis may unintentionally reinforce traditional learning teaching patterns rather than promoting a re-examination of best practices in classrooms.

Instructional Designers cannot be effective if they are familiar with only one model.  The designer must be able to fit the design to the situation and familiarity with various models will make that designer more successful.  Analyzing various models demonstrates that although the models had differences, combined steps in various ways, or used different vocabulary, they shared a fundamental principle of attempting to deliver effective learning or educational tools.

Arguments Against Instructional Design

In an article published in Training Magazine in April 2000, Jack Gordon and Ron Zemke lay out arguments against the use of ISD models. The article was the catalyst for a plethora of rebuttal articles, white papers, and seminars on why ISD is not dead and is still useful and relevant to the field (Clark, 2004). The arguments in the original article were:

• ISD is too slow and clumsy to meet today's training challenges
• There's no "there" there
• Used as directed, it produces bad solutions
• It clings to the wrong world view

Rebuttals concentrated on the question: Is it ISD that's flawed, or the manner in which it is applied that is the problem (Zemke & Rossett, 2002)? To address the argument of ISD being slow and clumsy, Sivasailam "Thiagi" Thiagarajan conducts workshops in "Rapid Instructional Design" which includes strategies and design elements to move more quickly through the ADDIE phases and to use partial processes where appropriate (Thiagi, 1999).

Argument two discusses the use of ISD as a "technology" of instruction. Project management for Information Technology (IT) projects is seen as a science. Yet IT project management is an art as is Instructional Design. IT project management has addressed similar flaws in waterfall or linear project management processes. Additional project life cycles have been introduced in this discipline. These models, selected based on project and environmental factors, include: incremental or prototype based methodology, Barry Boehm's Spiral Model, and the Rush to Base Evolutionary Model (BU, 2005). Options such as these applied to ISD would allow for the flexibility and rapidness desired for delivery of an instructional solution.

The third argument is that following ISD models will produce bad training. Clark (2004) discusses how the "A" of ADDIE was never meant to stand for Performance Analysis and that Instructional Design should only begin once it is determined training is the solution to the problem.

Charge 4 assumes ISD jobs have a set of best practice procedures that must be taught to students who are not master performers. Saul Carliner suggests we should remember ISD is a value system that must be applied appropriately and all ISD needs is re-tooling for all the new ways of learning we see since ISD was first proposed more than 50 years ago.

References

Boston University Corporate Education Center. (2005). Courseware MDP405: Managing Information Technology v4.0.

Braxton, S., Bronico, K., & Looms, T. (1995). Instructional design methodologies and techniques. Retrieved January 30, 2006, from University of Michigan, Educational Software Design and Authoring Web site: http://www.umich.edu/~ed626/Gerlach_Ely/ge_main.htm

Carliner, S. (2003). An instructional design framework for the twenty-first century. Unpublished manuscript, Concordia University. Retrieved January 30, 2006, from http://education.concordia.ca/~scarliner/idmodel.pdf

Clark, D. (2004, June 7). The Attack on ISD – 2000. Retrieved January 30, 2006, from http://www.nwlink.com/~donclark/history_isd/attack.html

Clark, D. (2004, June 13). A hard look at ISD - 2002. Retrieved January 30, 2006, from http://www.nwlink.com/~donclark/history_isd/look.html

Gentry, C.G. (1994). Introduction to instructional development: Process and technique. Belmont, CA: Wadsworth Publishing Company.

Gordon, J. & Zemke, R. (2000, April). The attack on ISD: Have we got Instructional Design all wrong?. Training Magazine.

Gustafson, K.L., & Branch, R.M. (1997). Survey of instructional development models: Fourth edition. Syracuse, NY: ERIC Clearing house of Information & Technology

Ryder, M. (2006). Instructional Design models. Retrieved January 29, 2006, from University of Colorado at Denver, School of Education Web site: http://carbon.cudenver.edu/~mryder/itc_data/idmodels.html#comparative

Thiagarajan, S. (1999). Rapid Instructional Design. Retrieved January 29, 2006, from http://www.thiagi.com/article-rid.html

Zemke, R. & Rossett, A. (2002, February 1). A hard look at ISD. Training Magazine

Analysis Techniques Portfolio

by: Marlene Fauser, Kirk Henry, and David Kent Norman on February 18, 2006

Task Analysis involves the systematic process of identifying specific tasks to be trained and a detailed analysis of each of those tasks (Ehrlich, 2000). To be able to design learning supported instruction, designers must understand the tasks the learners will be performing (Jonassen, Tessmer, & Hannum, 1999).

The purpose of this paper is to apply three analysis techniques from separate categories of methods as described by Jonassen, et al. to one instructional goal. This paper describes the key concepts, application, and experiences with a technique each from the Job, Procedural, and Skill Analysis Method, Instructional and Guided Learning Analysis Method, and Subject Mater/Content Analysis Method. A sample of each analysis technique is provided in the Appendices.

The Job, Procedural, and Skill Analysis Method emerged during WWII when most training being developed was technical training which emphasized procedural skill development. These methods describe in a procedural manner the way jobs are performed. Analyzing steps has become the most common conception of task analysis and there are many methods and approaches published. Alternatively, Instructional and Guided Learning Analysis Method describe tasks how they are best learned. This may or may not be consistent with the ways the job is performed. Subject Matter/Content Analysis Methods consider that designers elicit instructional information from SMEs. This information will usually come to the designer in an outline form, chunked by the subject matter expert. This organizational method can be effective, but is not always the best way to represent the instructional content. Subject Matter/Content Analysis Methods offer alternate methods for representing content structure.

Job, Procedural, and Skill Analysis Method – Job Task Analysis

The task analysis approach emerged from the field of systems analysis when systems analysis was applied to the world of work (Jonassen, et al., 1999). The purpose of creating task descriptions is to enhance the performance of the overall system or to enhance job or task performance.

In order to conduct a job analysis, you must describe and analyze each task. The description should include the stimulus situation and the proper response. One would typically start at a general level and then get more specific (Miller, 1962 as cited in Jonassen, et al., 1999). Jonassen, et al. would argue that there are several advantages to creating task descriptions:

• It is very job oriented
• It provides data to support training, personnel selection, and job design
• It describes tasks in sufficient detail to prevent misunderstanding
• It identifies the competencies that underlie job tasks

We must also consider the disadvantages of task descriptions as well. First, the process is generally limited to specific descriptive aspects of tasks. Secondly, the skill level of the task analyst must be high. Individuals with this level of skill may be difficult to find. Additionally, it has been found that the task description procedure as a whole does not help select specific tasks for training.

All of this information was considered when creating the analysis for the task of distance collaboration. The outline in Appendix A contains detailed information that follows the methods suggested by Jonassen, et al. (1999). There is additional information in Appendix B that further develops each solution and what it should be used for.

When going through this assignment, it was difficult for me to understand the level to which the analysis should be reported. I was torn between describing each button push and mouse click. For our chosen area, there are several categories where different applications could be used. I felt that it would be difficult to describe each button push for each individual software package as they did in the book for pre-flight checks.

I did notice, however, that putting this information on paper, provided me, an experienced technologist, with clear concise information which can and will be used to help others decide on platforms as well as conduct smooth, effortless collaboration sessions.

Instructional and Guided Learning Analysis Method - Learning Contingency Analysis

Learning contingency analysis differs from task analysis primarily in focusing on tasks in a learning environment rather than job performance (Jonassen, et al., 1999, p. 99). By focusing on identifying the behavioral components of tasks, the designer can discover the interdependencies among the tasks (Jonassen, et al., p. 99). Contingency analysis involves first sequencing the instructional content, followed by determining the most effective conditions for the instruction. For example, teaching algebra to a six-year-old during recess would be an extreme example of poor contingency analysis. First the child must have an understanding of basic arithmetic and a learning environment free of distractions.

The sequencing of instruction should reflect one behavior having dependence on another (Jonassen, et al., p. 100). To sort learning contingencies, Jonnassen cites Gropper's (1974) four behavioral components tasks can have:

• Superordinate/subordinate – a hierarchical relationship to show behavior X is a part of behavior Y.
• Coordinate relationship – behaviors at the same level in a hierarchical relationship, not necessarily performed at the same time or in the same order.
• Shared elements – behaviors that share some amount of the same concepts.
• No relationship – behaviors with no relationship to each other.

Appendix A references eight steps for conducting a learning contingency analysis, which includes Gropper's behavior components as part of steps three and four. The hierarchical learning analysis is similar to Dick, Carey, and Carey's (1999) chapter on subordinate skills analysis. Dick, et al, describes a method of drawing relationships between learning components in a pictorial form, versus a bulleted, tabular form in contingency analysis. The fifth step of contingency analysis skips a few chapters in the Dick and Carey model, but is analogous to developing instructional strategy.

In all, the learning contingency analysis is best for environments where learning is important, rather than performance. It is a time consuming analysis meant to draw a roadmap for the most effective sequence of instruction. The result instruction framework is rigid and may not be appropriate for learning tasks which can not be directly observed. A large instructional unit could make a confusing set of sequential dependencies between tasks in step 3 as compared to the pictorial method recommended by Dick, et al., so a highly skilled analyst is to perform contingency analyses.

Subject Mater/Content Analysis Method – Master Design Chart

The Master Design Chart approach to task analysis is used when the educational ends are the development of the behavioral capabilities of the students (Jonassen, et al., 1999, p. 207). This method has its basis in two foundations. The first foundational principle states that instructional outcomes can be stated as behavioral objectives. The second states the use of instructional taxonomies to describe the outcomes. In using this method, the instructional designer classifies content according to behaviors but is not bound to any taxonomy.

To conduct a Master Design Chart analysis the designer constructs the behavior axis using a taxonomy of instructional outcomes. The designer can use more specific behaviors as appropriate. Next the specific items of content are identified and listed as the vertical axis.

Each cell in the chart is now analyzed and the relative emphasis required in the course or curriculum is noted in the cell. This is done on a continuum with 0 representing no emphasis and a number such as 3 or 5 representing the most emphasis. Although the resulting chart does not represent relationships among the instructional outcomes, it is recommended a relationship step be taken.

A Master Design Chart is constructed for a curriculum or a whole course rather than for an individual lesson. Although it gives an overview of the entire course, no information is included regarding any instructional strategy. A strength of this method is that the risk of omitting important content is reduced. Weaknesses of this method are that it is dependant on the skills of the analyst, lacks basis in needs assessment and is time consuming to construct.

For our course, the matrix was created using document analysis techniques. The result is included as Appendix E. The content included information about being a "Distance Manager" and included content on what a leader should learn and know to successfully manage a geographically diverse team. Concentration from this SME was on higher level techniques as opposed to the detailed tasks analyzed in the Job Task Analysis approach.

While creating this matrix it was difficult to determine how the task list should be written. My habit is to include the action verb associated with the learning event. The matrix provides that association to the taxonomy, a challenge for this designer.

I selected to analyze this method since much of my work comes to me from Subject Matter Experts. I hoped I could learn to use a Subject Mater/Content Analysis Method in my job performance. I found it contrary to my method of thinking, which does helps me see things from a new perspective. However I also found it tedious and am not anxious to complete the Master Design Chart.

Overall Reflection

After reviewing the previous techniques, it has become clear that while some techniques can be used for the broadest of applications, others have been designed specifically for certain tasks. The more finely tuned the instructional designer's skill set is, the more they will be able to select the most specific tool for the task.

The task analysis method could be considered the broadest within the subset that we evaluated. It could be used in nearly every scenario. However, the resolution that it lacks may cause the end product to be deficient. Learning about and when to use the most refined tools available is the difference between being a novice or an expert.

References

Dick, W., Carey, L., & Carey, J. O. (2005). The systematic design of instruction: 6th edition. Boston, MA: Allyn & Bacon.

Ehrlich, D. (2000). Glossary of terms. Retrieved February 18, 2006, from the Northeastern Illinois University, Instructional Design II Web site: http://www.neiu.edu/~dbehrlic/hrd408/glossary.htm

Fazio, A., & Rieff, K. (2006, Jan). The distance manager. Presented at a meeting of the IT department of Walt Disney Parks & Resorts.

Jonassen, D. H., Tessmer, M., & Hannum, W. H. (1999). Task analysis methods for instructional design. Mahwah, NJ: Lawrence Erlbaum Associates

Writing up reliability results

Guiding Principle: No measure is ever reliable. Scores determined by a measure may be reliable depending on who completes the measure and the conditions under which the measure is completed (when and where).

Structure of the Reliability write-up

Sample paragraph

Responses for the various features of the agency website were judged to be moderately reliable for the customer service representatives who participated in the survey, with a reliability coefficient of 0.778. A review of the corrected item-total correlations suggests that the questions for the website as a critical tool and a unique website in the marketplace do not correlate with the corrected total very well. Their elimination is warranted on the basis that reducing the scale to only relevant items would make for a better, more parsimonious scale. It turns out that removing the item may further be motivated by anticipated increase in the reliability coefficient reported in the output (0.884).

If HTML newsletters, online payments, Internet as a tool, and customer communication with the agency were removed, the reliability of the scale would increase to 0.963.

To examine the impact of removing both items, each item was removed one at a time. This approach is necessary because the impact of removing one item changes the relationship of the other item with the changing total.

Developing a research question

Three major types of questions are causal/comparative, relationship, and impact. Use a literature review as a funnel to narrow a broad area of research into a manageable contribution, which leads to "The purpose of this study is..." Classic or landmark studies would be considered good content for the introductory, broad part of the literature review. Use incremental blocks of studies as a transition to more focused topics in your area of research. The final question should be something you are genuinely interested in finding the answer to so you have the motivation to complete the report as time progresses.

The question should have a question mark at the end. The question should ask if there is a cause/effect, what impacts what, or if there a relationship (what two or more things are related). Should have a construct or construct level at the start that is narrowed down to something very specific. Could "self concept" really mean "body image"? Does "achievement" mean achievement in math, reading, overall, or something else? Include operational definitions for parts of the question to define "achievement" if you are looking for relationship between achievement and yearly income. Perhaps achievement has several six factors to consider observing in the study.

Research problem statement template

Template:

This study will investigate the effects of (treatment) on (population). The investigation will be conducted (describe the setting). The present study, within the context of (the setting), will be concerned with the following research questions:

1. Does the procedure of (treatment) have (performance change) difference on (the experimental population) as compared to (the control population)?
2. Does the method of (treatment) - (restate method of treatment) - affect (population)?

Hypothesis 1: (Relate back to research question number 1). Hypothesis 2: (Relate back to research question number 2).

Example:

This study will investigate the effects of Digital Propensity on the preferences of insurance agency employees when information is presented with either text or graphics as the primary presentation format. The investigation will be conducted in a "real work" setting and will use a computer-based task. The present study, within the context of a work environment, will be concerned with the following research questions:

1. Does the presentation of graphics with text support more closely match the preferences of "Digital Natives" than the same content presented as text with graphical support?
2. Does the presentation of text with graphical support more closely match the preferences of "Digital Immigrants" than the same content presented as graphics with text support?

Sample Chapter organization of study

Chapter 1 of this study introduced the problem statement and described the specific problem addressed in the study as well as design components.

Chapter 2 presents a review of literature and relevant research associated with the problem addressed in this study.

Chapter 3 presents the methodology and procedures used for data collection and analysis.

Chapter 4 contains an analysis of the data and presentation of the results.

Chapter 5 offers a summary and discussion of the researcher's findings, implications for practice, and recommendations for future research.

Validity issues in research

Internal validity - how valid were the procedures that I followed in terms of the research method to obtain the information I need External validity - how generalizable is the information learned from a study

Internal validity

• History effect - did something recently happen to one of the sample groups that would make it differentiate from the other groups. For example, if you ask small children about their fears of ghosts, some of the children may report less fear if they are surveyed near Halloween.
• Maturation - when the study doesn't control for natural growth; in other words, the natural maturation of the group is the actual cause of improvement in the study instead of the treatment
• Testing - what are the effects of a pretest on a posttest; familiarity with the instrument impacts the results on the second test
• Instrumentation - is an issue when instrumentation changes during a study; for example, calibration of individual opinions in a observation study could have differences in results just because a one rater might think an event is a 5 on a 1-10 scale, and another rater might think the same event is a 7. The result grades on essay papers depending on the mood of the instructor. If the publisher comes out with a new version of a test and the previous test isn't available. Then you need to see how closely the two tests are correlated.
• Statistical regression - also known as regression to the mean; for example, getting a perfect score on the SAT only leaves the possibility of going down in score. The more extreme the results, the more likely each is to move towards the mean because of the error that put them in the extreme in the first place like guessing on questions that make the difference between a perfect score and a lower score.
• Differential selection - people that you pick are not representative of the population in the first place;
• Experimental mortality - differentially loose respondents; for example, you might not even know who you lost or picked in the first place. Loosing the extremes in both cases of a paired study could be the actual cause of change rather than the treatment
• Selection maturation interaction - as part of the selection, the sample actually has unknown differences that make the differences in the results instead of the treatment. For example, if teaching children basketball, while the children may start with similar free-throw ability, one child may improve faster than another simply because of superior hand-eye coordination, rather than the basketball instruction.

External validity

• Interactive or reactive effects of testing - example: giving a pretest to the people in the population sample may impact their performance during observation, so the pretest has an interaction effect that limits the generalizability of the results to the general population because the whole population did not take the pretest.
• Interaction and selection biases - the treatment only works for the selection, so can't be generalized; may be a result of the environment the selected group was performing in
• Reactive effects of experimental arrangements
• Multiple treatment interference - when you have more than one treatment, how do you know which one made the difference? Something like sequencing presentation of treatments may make a difference.

Bibliography

Campbell, D. T. & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Chicago, R. McNally.

Predicting the performance of interpreting instruction based on Digital Propensity Index score in text and graphic formats

Practitioners have proposed that Digital Natives prefer graphics while Digital Immigrants prefer text. While Instructional Design has been extensively studied and researched, the impact of the graphical emphasis in instructional designs as it relates to digital propensity has not been widely explored. Specifically, this study examined the performance of students when presented with text-only and graphic-only instructional formats. The purpose of this study was to test the relationship between Digital Propensity Index scores of individuals and their performance when interpreting online instruction. A sample of students from the population of a large metropolitan university received the Digital Propensity Index questionnaire, which is a measure of an individual's time spent interacting with digital media. Each student was randomly assigned varying formats of a computer-based instructional unit via a public survey. The instructional unit consisted of the DPI questionnaire and six tasks related to the Central Florida commuter rail system.

Participants were asked to answer the DPI questionnaire on a website by clicking on a link in an emailed invitation. Following the DPI questionnaire, participants were randomly assigned to one of two groups. Group One saw three instructional tasks shown in text and shuffled in random order. Each task was displayed on its own webpage. By submitting an answer to the task, the group progressed through the website to the next task. Group Two saw graphic tasks first, again, shuffled in random order. After the first three tasks, the groups swapped instructional formats to view the opposing group's initial questions. Participants were timed on how many seconds they spent reviewing each task. Each task had an assessment question to evaluate the learning outcomes of the instructional unit. Finally, the DPI score of the participant was matched with the time spent viewing each presentation format.

The findings indicate that DPI score had a statistically significant prediction of time spent navigating each type of instruction. Though the link between DPI score and time spent navigating instruction was statistically significant, the actual measurable time difference between navigating text and graphic formats was only a fraction of a second for each increment in DPI score. Limitations and potential future research related to the study are discussed as well.

Instructional Treatment Plan

As a class assignment, I developed an instructional treatment plan for the following objective: Given a virtual working environment, avoid communication problems between virtual team members when working on a project.

We chose Nelson's Collaborative Problem Solving (CPS) model because team skills are a higher order thinking activity. Nelson says CPS is not appropriate for memorized or procedural tasks and a CRT item would be "waste precious instructional time" and that "CPS is most appropriate when there is not a single answer to a question or best way of doing something" (Reigeluth, 1999, p. 247).

CPS was also chosen because the collaborative element in a virtual environment forces the students to experience the very topic they're studying and writing about. Each of the process activities in CPS is a small assessment of the enabling objectives since the group can't move on to complete the assignment until they have normed, agreed on the problem, defined roles, etc. I thought the true assessment of the enabling objectives would be performed as part of the debriefing, reflection, and discussion at the end of the unit.

Many texts have the condition, behavior, and criteria/degree elements of objectives (Dick, Carey, & Carey, 2005, pp. 123-139; Kubiszyn & Borich, 2003, p. 82; Simonson, Smaldino, Albright, & Zvacek 2006, p. 129; Stolovitch & Keeps, 1999, p. 356) and that Hirumi added audience (2005, ¶ 6). We tried to follow the samples in Dick, et al. (2005, p. 139), for the condition/degree of assessing intellectual skills by specifying things like "at least five peer-reviewed journals" in the sample assessment rubric. The other assessment rubric items were modeled from Dr. Hirumi's performance criteria for this assignment and peer evaluation for the end of the course.

I believe I can ask the question, "could I observe the learner doing this?" (Kubiszyn et al., 2003, p. 80; Dick et al., 1999, p. 127) to each of the objectives we listed in the treatment plan as part of the measurable behavior component of the objectives. Granted, the observation would not have an associated percentage completion or accuracy, but that goes back to the complexity of multiple solution theory in CPS. Dick et al, p. 139 has several samples without a CRT item as the assessment criterion for intellectual skill objectives.

Ways to improve the treatment plan are to include some of the degree elements from the assessment rubric earlier in the objectives, add more description and content to the instructional events, create an activity for forming and norming teams and an activity for role selection, and investigate taking some fuzziness out of the objectives.

References

Dick, W., Carey, L., & Carey, J. O. (2005). The systematic design of instruction (6th ed.). Boston, MA: Pearson Allyn and Bacon.

Hirumi, A. (2005, August 1). Unit 4: Performance objectives supplement. Retrieved September 9, 2005, from http://webct.ucf.edu/eme6613c/Unit04/u04info.html

Kubiszyn, T., & Borich, G. (2003). Educational testing and Measurement: Classroom application and practice (7th ed.). Hoboken, NJ: Wiley / Jossey-Bass Education.

Reigeluth, C. (1999). Instructional-design theories and models: 2nd ed. Mahwah, NJ: Lawrence Erlbaum Associates.

Simonson, M., Smaldino, S., Albright, M., & Zvacek, S. (2006). Teaching and learning at a distance: Foundations of distance education (3rd ed.). Upper Saddle River, NJ: Prentice Hall, Inc.

Stolovitch, H. D., & Keeps, E. J. (eds.). (1999). Handbook of human performance technology: Improving individual and organizational performance worldwide (2nd ed.). San Francisco, CA: Jossey-Bass / Pfeiffer