Evaluation as a Learning Experience at Howard University
By RODNEY D. GREEN and ARVILLA PAYNE-JACKSON, Ph.D.
with SUSAN BONTHRON
Dr. Rodney D. Green has been a faculty member at Howard University for twenty-eight years in the Department of Economics, and founded and leads the Center for Urban Progress, within which the Center for the Advancement of Service Learning and the William F. Goodling Even Start Family Literacy Program are housed. Dr. Arvilla Payne-Jackson has been a faculty member at Howard University for twenty-nine years in the Department of Sociology and Anthropology and has intensified her long-standing implementation of service-learning pedagogy over the last five years. Together they have found a way to help students learn ethnographic methods while serving, observing and evaluating a family literacy program in the District of Columbia. Much of the following text was excerpted from a paper they co-authored.
During the 1960s, Rodney Green became an activist engaged in social change, so it’s not surprising that, in his academic life, he felt the need for “a hands-on educational approach to social change.” He founded the Center for Urban Progress (CUP) within Howard University as a research center studying urban issues, “but we soon decided that we should try to get funding in collaboration with partnering groups in the community in order to effect real change.” The university had been “land-banking” (buying up local properties) for years, “so the University was sitting on forty-five dilapidated houses. Some of our partners wanted to revitalize the area and neighborhood, so we began working with Community Development Corporations (CDCs) such as the Peoples Involvement Corporation, and Manna, Inc., through the Georgia Avenue Community Renaissance Initiative.
“Of course there is always tension between pure research —the ‘ivory tower’—and active, hands-on learning in the community,” said Dr. Green. “We wanted to do these projects, but discovered that doing direct service by hiring staff people to run a program meant that we ran the risk of being marginalized within the University, which after all focuses primarily on academic programs. So integration was key to success.”
The Center for the Advancement of Service-Learning
“We figured out that student and faculty involvement in the community was important, but so was connecting with core academics. So service-learning became the pedagogical centerpiece of our community initiatives, since it structures service so that it feeds into the learning process in academic courses.” Adds Dr. Green, “there are three divisions in CUP: Research & Evaluation, Community Initiatives and Demonstration Projects, and Curricular Initiatives. The Center for the Advancement of Service Learning (CASL), the heart of the Curricular Initiatives Division, helps bridge the other two divisions.” Established in 2000 with a grant from the Corporation for National and Community Service, and led by Dr. Janet Griffin-Graves, CASL works toward integrating service-learning into existing courses and curricula throughout the University and provides extra-curricular and co-curricular service-learning opportunities for all Howard University students. Dr. Green concluded by noting, “CUP is also promoting academic programs in Community Development. There is now a Community Development minor which embraces service-learning pedagogy, and we are working toward several advanced degree and certificate programs in this field. The more embedded we are in the academic curriculum of the University, the more service-learning is understood to be a valid pedagogy at the University.”
The Even Start Family Literacy Program
The Howard University Center for Urban Progress hosts one of the four locally administered, federally funded Even Start Family Literacy programs in DC. Even Start aims to help break the cycle of poverty and illiteracy by providing multicultural literacy experiences for economically disadvantaged parents and their children in grades K-3. Students and faculty at Howard University have been actively involved in tutoring and mentoring for the program since its inception in 1997.
"The Even Start classroom has moved to the top floor. When I enter I see Dr. Lorry, Rosa, and Carla. Richard, a new tutor, is also present. They are all talking with Hermilita and Maria, new Hispanic mothers, each with one child joining the program…I like this room better than the old room. There are more windows. The walls are decorated with alphabet letters and brightly colored pictures of objects."
Jacqualine Reid, Ethnographic Methods student
Evaluation as Service-Learning
Not long after the program began, anthropology professor Dr. Arvilla Payne-Jackson approached Dr. Green about the possibility of engaging her students in the Ethnographic Methods class to conduct qualitative program evaluation through participating in, observing, and reflecting on the Even Start program. Since then, her students have provided useful feedback on the program that larger-scale national evaluations could not offer, and in the process were able to learn about and practice data collection, observation, notation, and interviewing — all important components of ethnographic research. These efforts have been enhanced by the receipt of two minigrants from CASL.
In their paper, “Evaluation as a Form of Experiential Learning: The Howard University Even Start Program,” Drs. Payne-Jackson and Green describe the merits of this partnership between a community-based program and the University students: “Harnessing the energy and creativity of undergraduate students to support community-based projects is one of the great merits of course-based service-learning initiatives. Combined with the reciprocal impact on the students themselves, service-learning experiences are among the most productive elements of University-Community partnerships.”
More than Broad Evaluations Are Needed
Stressing the need for locally generated, qualitative evaluation, Payne-Jackson and Green note that “evaluation is often considered a purely professional activity, conducted by credentialed, experienced professionals to assess the impacts and outcomes of social programs….” The Payne-Jackson & Green paper goes on to explain how the federally mandated evaluation of Even Start programs requires national, state and local evaluations, which must meet performance measures in the four core areas of adult education, parenting education, early childhood education, and parent and child together time.
State evaluations often review the same data collected for the national evaluation and use them to compare the effectiveness of sub grantee programs in each state. “Such evaluations, which usually include large amounts of data gathered from many local versions of a program that are then aggregated to yield overall outcomes, do not always provide insights into the ways a particular program can be enhanced, since the audience for the evaluation tends to be national policy makers. These national evaluations often provide information used by federal program officers to argue for continued funding for a particular program. As such, they are essential for the overall federal budgetary process.
From the local perspective, however, such evaluations often seem to provide little return. In fact, their “one-size-fits-all” limited scope has led to spurious negative findings at the national level which have been used to justify reductions in federal funding for a very useful program.”
Student Preparation and Action: The Service-Learning Framework
The Ethnographic Methods class is a one-semester course conducted once per year that focuses on teaching and learning qualitative research methods (including participant observation) to study social issues. Service-learning is the natural pedagogy to meet the course’s objectives. To prepare students for the service-learning evaluation project, the Director of the Even Start Program and the Executive Director of CUP attend the ethnography class to give students an overview of the program, its mission, goals, program format, and program literature. Students fill in volunteer forms and sign up for specific times to attend the program and assist the teachers with either the adults or the children or help with other needs of the program such as recruiting.
A literature review on literacy and second language learning familiarizes students with local and national statistics as well as challenges and successes that programs encounter. Classroom instruction in preparation for the fieldwork includes training in note-taking, observation, mapping, conducting interviews and data analysis. The qualitative evaluation itself consists of mapping the surrounding community, participant observation in the program, interviewing participants and leadership, data analysis and writing of the final report. Mapping focuses on discovering a community’s capacities, assets, and resources, including the physical appearance of houses and businesses, upkeep of the streets and alleys, the type of litter, types of cars, interaction of people on the street, and so on. In addition to careful in-class preparation in methodology, students read the reports of previous cohorts of students in this class as well before going into the community setting. Comparisons are made to determine what changes have occurred in the neighborhoods and their possible implications for the program.
"There were child[ren]’s pictures covering every inch of the wall. A wall called the ‘Word Wall’ was sectioned off for words of the day… Children’s books were waiting anxiously in the shelves to be read by the children. I noticed that the mentors had helped the kids plant lima bean seeds in a bag and place them on the windows. I then thought that not only was this a place to learn how to read, but it was a place to have fun and relax from the struggles of their everyday life...."
Leonard Mason, Ethnographic Methods student
Students in the Ethnography class are required to attend one or more of the Even Start sessions each week. Their first task is to provide full descriptions of the building and the rooms used for the classes and socialization. This includes the arrangement of tables, and chairs, decorations and learning materials, interaction among teachers and participants, and between parents and children. They also provide general demographics of participants including age, gender, occupation, and educational level. Students describe lesson plans, materials used, and participant responses. They keep weekly journals to assist them in evaluating their observations and the patterns of behavior and interaction that happen during their time working with the program. Students develop separate interview protocols for staff, parents and children. The interviews are designed to gain insight into what was working well, what challenges staff and participants face, and what each would like changed to make the program more effective.
The final report includes analysis of data from the asset mapping—assessing the vitality of the community, its assets and areas of change. Thematic analysis is conducted on the other data and yields information on areas of strength and challenges of the program, bringing issues to the table such as management style questions, materials used and group dynamics. The report is submitted to the Executive Director of CUP, who, in turn, shares it with program staff. These findings influence personnel changes and developments that the formal traditional evaluation is not designed to address. Thus this process engages the students directly in the development of a program.
"Mr. Casto Romero is well liked by both the participants and other teachers… I observed several times how he brought clarity to the reading or writing of a student by explaining grammar or translating what had been written into Spanish. The students appreciate his dedication to the program…. I was impressed by the support in the room while students read their paragraphs aloud or showed their artwork."
A.J. Critchfield, Ethnographic Methods student
Student and Agency Reflections
Reflecting on their experience allows students to bridge the gap between academe and their communities. They gain practical experience and learn how to apply what they learn in the classroom in the real world. Several students have spent more than the required one session a week in the program and continue their commitment to Even Start or another literacy program after the course is completed. All the students have concurred in the importance of being able to make a difference and to be part of a community working for change. Reflections made in an interview with the Even Start Family Literacy program director, Ms. Michelle Coghill, reveal the benefits of the experiential learning evaluation model from the perspective of the agency. According to Ms. Coghill, among the benefits are “that it provides valuable insights into the complexities of serving low-income, low-literate, and low-skilled individuals. By focusing more on the dynamic process of behavior and lifestyle change, the service-learning model demonstrates how traditional evaluation methods have failed to capture the true impact of programs like Even Start.”
Dr. Payne-Jackson noted that a typical difficulty in partnerships between universities and community organizations is the transience of the student population versus the ongoing character of a social program. Although Howard University students participate in the Even Start program long enough to develop a relationship with the adults and children, nevertheless having too many students coming and going can be disruptive. Originally, the entire Ethnographic Methods class participated in the Even Start program and conducted qualitative evaluations. As result of consultations between the professor and the agency director, the Ethnographic Methods class now offers several service-learning choices; only two students will be carrying out qualitative evaluation of the Even Start program this semester to insure an optimal fit.
The unique qualities and benefits of this type of service-learning evaluation model include the willingness of participants to open up to and share their backgrounds and experiences with students. Participants feel that they have information to contribute to the student’s understanding of their lives, families, communities and challenges.
Through their engagement with participants, students come to understand the many mutable factors of participants’ lives. They learn to appreciate the small steps the participants make toward achieving literacy and self-sufficiency, such as successfully following through with a referral service like counseling or obtaining food from a food bank, calling when they will miss a class, volunteering to hand out flyers at a neighborhood school, or spending more time with their children.
Results from Student Evaluations
The reports submitted by students provide a narrative that program directors can use to measure program success. The qualitative data help program directors make informed decisions about programmatic changes and draw attention to areas that can go unnoticed in traditional evaluation formats. For example, the 2000 Report noted that African American and Hispanic families tended to eat dinner separately, which brought the issue of cross-cultural relationships more into focus. The students’ reports helped staff identify challenges and possible solutions. Recruitment was a major concern for the program staff. New strategies were developed in order to attract African American families, such as reconciling the schedule of parents to the time the program was offered and extending the hours of the program in order to allow more families to participate. Misperception of the program as being overly intensive was identified as another possible problem that had a direct impact on the recruitment efforts.
Challenges identified as inherent in the structure of the program included: space, more diverse staff, and in-house assessment. Space was identified as a major problem for the program in earlier years. The classroom was open and the parent and children’s groups were run concurrently. This tended to be distracting or, in the words of one person, “chaotic.” It was also noted that more space would allow the staff to display the work of parents, which would help to motivate them. Several recommended finding a new location for the program altogether as the physical environment (especially the lack of windows in the room) at Bruce Monroe made it feel “like a prison.” This finding helped the director prevail upon the local school administration for improved space, helping solve this problem. Because of the large number of Hispanic families, additional bi-lingual staff members were recommended; as a result, a bilingual Literacy AmeriCorps member was recruited to help meet this challenge. Another recommendation was to incorporate assessment into the structure of the program in order to provide feedback to staff on a regular basis. This too was accomplished.
Recommendations for enhancing the program included expanding the offerings to include GED preparation, information on social programs and parenting skills. The program recently moved to Southeast DC, the most distressed part of the city, according to Dr. Green. There are more men in the program now, and the program emphasis has evolved from bilingual literacy to include securing GEDs as well. Dr. Green noted that “we work to help improve the overall self-sufficiency level of the participants. For instance, we have a separate workforce development program to which Even Start participants can be referred to help them find work or get improvement in their current jobs. We’ve had a fair amount of success with this holistic approach; some Even Start participants have gained promotions, and several unemployed participants got jobs.”
A powerful benefit of this approach to evaluation is that it helps program participants see the value of community and university partnerships. Participants have the opportunity to change their perceptions about the “ivory tower” and students become a part of the communities where they attend school and see the relevance of their education for community empowerment. Students’ experience in the evaluation of the Even Start program provides them with a deeper appreciation for the importance of establishing relationships with the people with whom they work; this experience in turn builds their commitment to the families and the project, often leading to decisions to invest more of their time and energy in making the project a success. At the same time, the host agency, Even Start, receives useful qualitative evaluations of its program and improves its ability to meet the holistic needs of its clients as a result. The suggestions stemming from the service-learning evaluation cannot be derived from the federally mandated three-tier evaluation process. Accordingly, a service-learning-based evaluation may well turn out to be a more useful tool for local programs insofar as the audience for the evaluation is, in fact, the local agency, not federal policy makers.
MORE from the Journal! Essays l Articles l Reflections l Reviews l Literacy Corner l Events
© copyright 1995-2017, Community Works Institute (CWI)
All rights reserved. CWI a non-profit educational organization
CONTENT USE POLICY No material contained within this web site may be reproduced in print, by electronic or other means, without permission. All materials contained in this web site remain the sole and exclusive property of CWI, or the author if designated by arrangement.