CHAPTER THREE

The researcher used an inductive process, seeking out details and discovering patterns as they emerged throughout the electronic portfolio process. Technical background and attitudes toward technology, as reported in the computer literacy questionnaire, were compared to student perceptions of the portfolio process. The process of designing the portfolio was investigated from the perspective of the course professor and the research designer to determine strengths and weaknesses in organization and presentation. Technological processes within the real-life context of a college computer lab were examined for problematic hardware and software issues. The electronic portfolio process was described and analyzed on an individual basis with no clear set of anticipated outcomes.

This study was exploratory in nature, attempting to promote further understanding of an innovative technology. The technical background of the individuals participating in the project was diverse, requiring non-standardized methods of investigation (Patton, 1980, p. 88). Strengths and weaknesses of the process and technical problems encountered by the participants varied according to the individual's background and interest in technology. Lincoln and Guba (1985) discussed the use of qualitative methods as being adaptable when dealing with multiple realities. Participants in this study represented multiple realities in terms of attitude toward and expertise in technology.

Detailed information was obtained through the participant’s descriptions to further understand and improve the process of creating electronic portfolios. The researcher interacted closely with some of the students involved in the study. The researcher observed and actively assisted personally in portfolio preparation within the lab setting with the first group of participants. The study was guided by the following questions:

1. What effect does incorporating technology have on the development of a portfolio for teacher candidates?

2. To what extent does the electronic portfolio process encourage self-assessment and reflection?

3. In what ways does the electronic portfolio provide evidence of student learning and achievement in line with course objectives?

4. What are the problems encountered in putting together the portfolio electronically?

5. What do students perceive as the strengths and/or weaknesses of creating a portfolio electronically?

6. What are the course professor's perceptions concerning the effectiveness of the electronic portfolio as a tool for assessment?

Previous to this study, the paper portfolio binder had been established as the primary assessment for the reading methodology course in this university teacher preparation program. Students were expected to collect paper documents, photographs, graphic examples, and audio-visual artifacts as evidence of meeting course objectives. Providing evidence of reflective practice was considered a significant component of the portfolio process. The electronic portfolio provided a computer framework for documenting the meeting of course criteria and providing evidence of self-reflection and self-assessment. A template was designed to accommodate the use of digital text, graphics, audio, and video files in order to provide multimedia evidence of the achievement of course objectives. Students wrote cover sheet reflections explaining, describing, and defending the artifacts they chose as evidence of their achievement.

The electronic portfolio project was constantly evolving during the course of this study due to revisions in state credential standards and adjustments made because of technical problems. The initial template was presented to the first group of students in an interactive web design based on hypertext markup language (HTML). The course objectives were linked directly to state certification standards on the World Wide Web (WWW). After determining that some students considered the web design complicated and confusing, the researcher created a second alternative template using an interactive multimedia authoring program, Hyperstudio. The software featured tools for inserting text boxes, graphics, audio, and video, as well as provide interactive functions which allowed for linking within the program, to other software programs, and to the internet. The second semester course objectives were modified to incorporate revised state certification standards and the implementation of Reading Instruction Competency Assessment (RICA).

In addition, the course professor and the computer lab technician were asked to provide observational data, opinions, and insights into the portfolio process through interviews, phone conversations, and e-mail correspondence. A field journal was kept in order to provide an historical record of the researcher's involvement in the process of developing templates for the multimedia computer portfolio. The researcher provided a narrative of significant developments, changes, and problems that occurred during the study. E-mail correspondence between the course professor and the researcher was analyzed to provide additional historical data. The researcher provided additional technical help and advice within the computer lab setting, as well as through e-mail correspondence with some of the participants.

Open-ended interviews with volunteer student participants were conducted after the completion of the course portfolio by the researcher (see Appendix B). A flexible question guide was followed and responses were audio-taped and transcribed. Interviews, which lasted approximately 30 to 45 minutes, were conducted after the student completed the reading methods course. The one on one interview was scheduled at the participant’s convenience and was held on campus in an available school of education classroom. The interviewer took notes during the course of each interview and asked for responses to be clarified or expanded upon when it was considered useful to the study. Additional responses or clarification were accomplished through e-mail correspondence. The researcher initially explained the purpose of the interview and requested permission to tape the interview. The researcher attempted to remain open to issues of personal interest to the participant and ask questions to further develop emerging topics and patterns. Participants were encouraged to expand upon the interview questions and talk about their experiences openly and freely. All interview text was imported into the qualitative software package, Ethnograph (Seidel, 1998), for analysis.

Electronic portfolios were examined after the participants had completed the course and had received their course evaluations from the professor. Portfolios of all participating students were collected on a zip disk and transferred to a CD-ROM disk. Student generated reflective cover sheet texts within the electronic portfolio framework were imported into the qualitative software for analysis along with interview and e-mail text. The cover sheet texts were examined primarily for evidence of self-reflection and self-assessment. Texts from the portfolios were examined for reference to personal perceptions and attitudes toward the electronic portfolio project. The researcher made no attempt to evaluate the quality of the electronic portfolio artifacts as assessment in reference to the achievement of course objectives.

An open-ended e-mail interview with the course professor was conducted after the portfolio project was completed (Appendix D). The course professor was asked about her expectations for the electronic portfolio project and her reasons for using technology in teacher preparation. She was asked to describe design changes and technical adjustments made during the course of the project. She described how she assessed the portfolio products and what she perceived student attitudes to be throughout the portfolio process. Adjustments in software design, as well as curricular changes in the portfolio format, were documented throughout the study. Data on the initial design phase of creating a template for the electronic portfolio organized around course objectives was obtained through e-mail correspondence and records of informal conversations with the course professor. A research journal was used to record field procedure notes. Throughout the study, the researcher kept a thorough narrative of the technical processes, the problems that occurred with software and hardware, the changes made to improve the templates and electronic resources, and the course revisions made for subsequent terms. In addition, a semi-structured interview with the computer lab technician was taped and transcribed.

An informed consent form was provided for everyone participating in the study (Appendix E). Participants were granted the right to withdraw from the study at any time. Interview participants received a CD-ROM of their electronic portfolio for helping provide their perspective on the portfolio experience. Data obtained throughout this study was collected and saved in digital format in order to be used as evidence in a multimedia computer presentation.

The data analysis software package, Ethnograph, v. 5.0 (Seidel, 1998), was used to manage narrative documents and transcripts, allowing for storing, browsing, indexing, and coding of all text. Ethnograph allowed the researcher to explore documents and search for patterns and themes that emerged from the text. As the data was explored, text annotations were coded and an index system was established. Data for this study included research questions, emerging ideas, categories of people, interview text, field notes, e-mail responses, open-ended questions, portfolio analysis text, and other documents related to the electronic portfolio project. Search tools within the software were used to link, explore, and ask questions in order to determine relationships and establish hierarchies within the data.

Transcriptions of the interviews from all participating students provided the text for the initial coding and indexing of data. Other text imported into the software program included student generated cover sheet text from within the portfolios, as well as text from the researcher’s e-mail and research journal. Text from interviews and e-mail correspondence with the course professor and the computer lab technician provided additional data. Most of the data was imported from Microsoft Word documents, although some of the cover sheet text was taken directly from the text of the Hyperstudio portfolios. The Ethnograph editor’s reformat function was used to convert data files into the 40 character, hanging indent format required by the software for analysis. Additional preparation of the data was made by editing the text to include identifiers which were followed by a colon (:) and contextual comments which followed the plus (+) sign.

Imported text from a student interview is presented in the following example (Figure 1). The student’s name, as well as the key terms from the interview questions, are contextual comments following the plus (+) sign. "CP1:" refers to the researcher asking the first question and the "Q1:" refers to the student’s answer. "CP1a:" refers to a follow-up question to clarify the student response.

Figure 1. Example of interview text from Ethnograph.

+Lisa – Interview – Fall Group

+Process - 1

CP1: Could you briefly describe the

process you used to put your

electronic portfolio together? What

computer software/hardware did you

use, how did you go about it?

Q1: I used the Hyperstudio program and I

used the portfolio that had been

pre-done, the template for it, but I

took it apart and put it back

together the way I wanted it to connect.

I set up how I wanted the computer

part of it first. Then I went in and

plugged in what I wanted to add for

the objectives and artifacts and

things.

CP1a: What kind of changes did you make

to the template?

Q1a: There were links between stacks

that didn't make a whole lot of sense

to me, so I changed the links. The

biggest thing that I did--there was

one assignment that I wanted to use

pieces of in different stacks. I

had to take it all out and make it

into a new stack, so I ended up linking.

There are some pages that look the same but

they're actually two different pages

that I ended up double creating so it

would link the way I wanted it to.

In addition to interview text, the cover sheet text was imported for analysis. The electronic portfolio framework was designed to give students an opportunity to include artifacts as evidence of achieving course objectives. For each objective, students prepared a cover sheet providing support for their choice of artifacts as evidence of their competence in meeting established criteria. Students reflected upon each course objective and assessed their proficiency in each area. Once the cover sheet reflections were imported into Ethnograph, an additional parent code for teaching was added to the family tree structure to accommodate expanded discussion of literacy, learning, curriculum, and instruction.

Determination of the coding hierarchy remained flexible throughout the study. The level one parent code words were based on the following themes:

1. The process of creating a portfolio electronically: student’s personal approach and attitude toward the assignment, the amount of time spent on the project, methods of collecting artifacts, design decisions made, and advice for improving the process.
2. The electronic portfolio product: references to demonstrating achievement of course objectives, the qualities of the product, assignments, artifacts, and comparisons to paper portfolio projects.
3. The use of the electronic portfolio for assessment: evidence of self-reflection, demonstration of competencies and proficiencies, personal ownership of the product, and evidence of successful performance of course criteria.
4. The use of technology: the development of technical skills, the need for support, hardware and software problems encountered, and the future use of technology in teaching.
5. The purpose of the electronic portfolio as a framework for discussion of teaching: course content, methodology, curriculum, literacy, content standards, strategies, research, resources, lesson plans, and classroom management.

Table 1. Key Terms for Interview Questions in Relationship to Original Parent Code Words

Table 2. Primary Coding System

Table 3. Words Used in the Coding of Student Attitudes

Figure 2. Example of coded segment of text file.

+Barbara - Interview - Spring Group 1

+process - 1 4

CP1: Could you briefly describe the 6

process you used to put your 7

electronic portfolio together? (What 8

computer software and hardware did you 9

use?) 10

#-ORGANIZE $-ARTIFACTS %-COMPETENCE

Q1: The first step was organizing 12 -#-$-%

material I wanted to use, the 13 | | |

artifacts. I looked at competencies 14 |-$ |

$-TECHNOLOGY

required. The second step was to 15 |-$-%

%-COLLECT *-ARTIFACTS

apply the technology. Once all the 16 |-$-%-*

artifacts were gathered and collected, 17 | -% |

$-HYPER $-FIELDWORK

I began to enter field experiences in 18 |-$ |

%-TEMPLATES

Hyperstudio. The second template was 19 |-$-%-*

$-CUT/PASTE

easier to use. I cut and pasted in 20 |-$-%

the material. 21 -#-$

+evidence - 2 23

CP2: What types of evidence did you 25

choose to place in your portfolio? How 26

did you select your evidence? 27

!-FIELDWORK #-ARTIFACTS

Q2: My fieldwork experience, artifacts 29 !-#

included lesson plans created for 30 |

$-JOURNAL

actual courses, critiques of journal 31 |-$

!-REFLECTION

articles, reflection papers, I 32 ! |-$

!-FOUNDATION $-OTHERCLASS

implemented my foundations teaching 33 ! |-$

!-KIDSBOOK

management plan, my children's book. 34 !-#-$

+methods - 3 36

CP3: What methods did you use to get 38

your evidence into the computer? 39

(typed, scanned, cut and paste?) 40

!-SCANNED !-TYPE !-CUT/PASTE

Q3: Scanning, typing, cut and paste, 42 !

!-VIDEO !-WEBSITE

video, web, audio sampling. 43 !

+purpose - 4 45

CP4: What did you see as the purpose of 47

your electronic portfolio? 48

#-REFLECT $-PURPOSE

Q4: I saw it as primarily a 50 -#-$

%-HELPATT

reflectionary tool. It was helpful to 51 -# |-%

know you were gaining what you should 52 | |

get out of the course. It fulfilled 53 -# |-%

%-OBJECTIVES

various purposes, but I used it more 54 | |-%

*-JOB

for reflection on goals. It could 55 | |-%-*

also be a tool for interviews. 56 -#-$ -*

Figure 3. Code counts for references to technical processes.

Combined searches for code words provided specific information in reference to files and lines each imported transcript. An example of a combined search for the code word, "experience," is provided in Figure 4.

Figure 4. Combined search results for experience code word.

Combined Search Results Search Code EXPERIENCE

====================================== Lines 96 to 98 of File PENNY ======================================

A lot of the people in the class were not already computer literate and so she wanted them to improve their knowledge.

====================================== Lines 282 to 283 of File PENNY ======================================

except my computer knowledge was expanded.

====================================== Lines 70 to 71 of File MAGGIE ======================================

Q4: One of the purposes was to get me used to a new computer program,

====================================== Lines 72 to 74 of File MAGGIE ======================================

because I know that's one of the big things that we need to start getting used to now is all the technology.

====================================== Lines 361 to 362 of File MAGGIE ======================================

Probably just more basic knowledge on how to use it.

====================================== Lines 367 to 370 of File MAGGIE ======================================

Q14e: Oh yes. I think they're really neat and I think that kids can learn a lot from them to. You can put so much into it and personalize it so much.

Figure 5. Example of summary output of advice code word.

Frequency data for the code word, " assessment," generates a report as it appears in Figure 6. The count for each file is stated along with the percent across all files (top number) and the percent within the file (bottom number).

Figure 6. Example of frequency count for assessment code word.

File:LISA

---------- ----- ---- ---------- ----- ----

ASSESS 15 0.06

1.00

File:PAULA

---------- ----- ---- ---------- ----- ----

ASSESS 20 0.08

1.00

File:BARBARA

---------- ----- ---- ---------- ----- ----

ASSESS 20 0.08

1.00

File:MAGGIE

---------- ----- ---- ---------- ----- ----

ASSESS 14 0.05

1.00

Figure 7. Example segment output of aesthetic code word.

With Kids-Parent = AESTHETIC

SEARCH CODE: AESTHETIC

$-AESTHETIC

files. Other than that, the arranging 171 -# -$

#-CONSISTENT

of it. I wanted it to look 172 -# $

%-THELOOK

consistent and I spent a lot of time 173 | $ -%

on the visual, the esthetic. 174 -# -$ -%

Triangulation of multiple data sources involved comparing the perspectives of people from different points of view, particularly the student participants, the course professor, the computer lab technician, and the researcher (Patton, 1980, pp. 330-331). Participants in the study were asked to review findings and give feedback during the course of the study in an effort to validate the accuracy of the information and determine whether it matches reality (Merriam, 1988). Member checks were conducted throughout the study through telephone, e-mail, and postal correspondence in order to clarify any conflicting statements or confusing data.

An audit trail was established and maintained to facilitate an external audit to establish levels of dependability and confirmability of this study (Lincoln and Guba, 1985). This research conformed to the audit trail classifications made by Halpern (1983) in an attempt to provide reliability and enhance dependability (Cited in Lincoln and Guba, 1985, pp. 382-392). Raw data was collected and sufficient evidence was provided concerning data reduction and analysis. Data reconstruction and synthesis established hierarchies of concepts and categories. Process notes were maintained; intentions and dispositions were recorded; and valid instruments were developed. The audit trail included audio-tapes of interviews, transcriptions of interviews, hard copies of all documents coded by the qualitative software package, including e-mail correspondence and text from the researchers diary. Data imported into the software package, Ethnograph, resulted in the generation of the following documents: the research codebook, individual code counts, and code searches using segment, summary, and frequency outputs.

An assumption in this study was that all participants answered interview questions about their perceptions and actions truthfully, freely, and frankly. It was assumed that the interviews, observations, and collection of electronic portfolios provided a complete and useful database of information upon which interpretations and conclusions could be drawn.

Limitations of the study included the lack of generalizability of findings beyond the specific research setting. The transferability of the research findings to another situation cannot be validated, although providing sufficient information might make findings applicable to a new situation depending on the degree of similarity (Lincoln and Guba, 1985). The reliability of this study was limited because it would be impossible to replicate in another context.

An additional limitation of the study related to the objectivity of the researcher. Patton recommended "empathic neutrality" (1980, p. 55). He stated that empathy "is a stance toward the people one encounters, while neutrality is a stance toward the findings" (1980, p. 58). He stated that the neutral researcher needs to be non-judgmental while trying to report what is found in a balanced way. This study was limited because participation in this study was completely optional and voluntary. Data obtained from the students who volunteered to be interviewed and willingly agreed to have their portfolios analyzed may not be representative of an entire class of teacher candidates.