|
|
|
Collective
Portfolio Assessment
Assessment rests on two legs: student perceptions of their experience
and actual student work of graduating seniors. This is why the
department’s assessment plan also includes samples of student
work in a "Collective Portfolio." This portfolio is
currently being constructed. In the near future this site will
present an analysis of the quality of student work of graduating
seniors.
The Collective Portfolio is composed of a sample of papers from
majors in all of the core, required upper division courses in
the major. Each year, the Assessment Committee solicits samples
of A, B, and C work from faculty who teach these courses. The
selected papers are then put into the department’s "Collective
Portfolio." The papers reviewed are from the academic year
1998-1999. These papers represent the first set of papers collected
in the portfolio. The Sociology Department’s Committee
on Assessment decided to initially look at the papers in the
Department’s Collective Portfolio by focusing on only two
of the department’s eleven learning goals. The Committee
selected only two goals to start with, since it has no track
record of evaluating papers in a collective portfolio. The thought
was to start slowly to see how the assessment of actual student
papers might work and what could be learned from doing this
exercise. In future years, the department will select other
learning goals for analysis in the Collective Portfolio. The
two selected goals represent a disciplinary goal (#5) and a
general education, university goal (#11). As the Committee read
the papers, the members focused on the patterns that were present
in the students’ work.
Learning Goals Assessed in the Portfolio
The two learning goals selected for this initial assessment
–
(#5) The sociology major at CSU Sacramento will be expected
to study, review, and reflect on basic concepts in sociology
and their fundamental theoretical interrelations. Under this
learning goal, the student should be able to define, give examples,
and demonstrate the relevance of concepts such as, culture,
socialization, stratification, social structure, institutions,
differentiations by race/ethnicity, gender, age, and social
class, etc.
(#11) The sociology major at CSU Sacramento will be expected
to have the writing skills necessary to communicate effectively
with persons whom they encounter in their work, civil obligation,
and personal life. Under this goal, the student should be able
to a) write a clear and concise report of the findings from
sociological data analysis; and b) write a clear and concise
sociological analysis and account of a social event, topic,
issue, or problem.
Assessment Categories or Rubrics
Before reading the papers, the Committee decided to use a simple
set of categories to start with. Over time, these categories
will probably be expanded and become more focused. The assessment
categories used for the initial assessment are –
a. What do sociology majors do that is outstanding?
b. What do sociology majors do that is satisfactory?
c. What do sociology majors do that needs improvement?
d. What might the department do to help students improve?
Assessment Results
a. Outstanding Work
For the learning goal on writing a research report, a few students
did outstanding work. These students were able to elaborate
their ideas and present a more substantial account both of the
concepts they were using as well as the literature review. For
example, in the literature review, students were able to go
beyond merely describing a published piece of research as it
relates to their topic, but were also able to criticize the
research they were reviewing. In other words, they went beyond
description to critical thinking about the literature This is
an outstanding accomplishment for an undergraduate. Few of our
students are able to do this, in part because it requires a
good deal of knowledge about the topic and previous research
done on the topic. These papers also show the ability to move
between levels of analysis in sociology–between the micro
and macro, between the self and social structure, and so forth.
The outstanding papers were also organized very well. The paragraphs
were logically ordered, there were transitions between paragraphs,
each paragraph had a thesis sentence, and paragraphs covered
only one topic. Less complex sentences tended to be used, which
made the writing more clear and concise. Literature reviews,
library research, and citations were done well. They were complete,
up-to-date, and related to the topic and thesis. These papers
reflect precision in the use of sociological concepts too. The
concepts were stated explicitly and defined reasonably well,
and these papers provided clear empirical referents for the
concepts used. In other words, the authors provide an empirical
account of what the concept means. The outstanding work represents
the ability to move from the descriptive level to the analytical
level.
b. Satisfactory Work
For the leaning goal on writing a research report, most students
performed satisfactorily. They were able to communicate their
ideas and write a report based on findings from sociological
data analysis. Most cited current literature, most used proper
citation methods, most were able to explain the independent
and dependent and control variables in their research paper
(the research design), and most presented clear conclusions.
Even so, satisfactory work often contains errors in writing,
but they tend to be minor and don’t interfere with communication
of the topic or ideas. In addition, the satisfactory papers
were organized in terms of defining the research question, the
literature review, research design, data analysis, conclusions,
and shortcomings of the study. In brief, they were able to follow
the acceptable organization for writing a research paper. This
is what we would expect given their course work in research
methods.
For the learning goal on the application and understanding of
basic concepts, most students were able to adequately describe
the empirical world of their observations, but were less able
to clearly link those observations to a clearly presented and
concisely defined sociological concept. In addition, in some
of the papers that were satisfactory, the student tried to cover
too many concepts and consequently, failed to provide either
a clear conceptualization of the concept or link the concept
to the empirical world of their observations. When there are
too many, some of them are covered superficially. The satisfactory
work is represented by good, sometimes excellent work at the
descriptive level, detailing observations in the natural laboratory
of society. This work is weaker at the analytical or conceptual
level. The student authors don’t move easily from the descriptive
to the analytical level.
However, satisfactory work often does show that the student
has learned to think sociologically, to use the sociological
imagination. These papers do show that the student has taken
empirical observations and reflected on them through the eyes
of sociological analysis, even though the examples and the writing
don’t reveal this as well as they could. An example of
this is a satisfactory paper which involved observing an African
American church ceremony. Although the scripture and the sermon
were about the "brotherhood of Christ," the student
was observing the inequalities in the church from the point
of view of the power structure within the church. This student
was able to give some examples that illustrated the concept
of the power structure, and the student did discover that she
was being a sociologist, seeing the event, at least in part,
as a sociologist would. But there could have been more empirical
examples and the concept of power structure could have been
more adequately conceptualized.
c. Work that Needs Improvement
For the learning goal on writing, some sociology majors do not
write as well as they should or could. They often do not have
good transitions between paragraphs, cover more than one topic
in a paragraph, have some grammatical problems (wrong verb tense).
Some sentences can be too complex and tangled, so that the idea
conveyed is not clear. Citations are also not consistently done
in the research report. Sometimes the thesis of the paper is
not stated clearly. Conclusions are not complete. Sometimes
there are factually incorrect statements. Subheadings in the
paper tend not to be present, which leads to loose organization
of the ideas. One of the characteristics of student work that
needs improvement is that it tends to be superficial. There
is not a deep account of the topic or the research problem or
the literature review. Sometimes the problem is not stated too
clearly or is too oversimplified. The work tends to skim the
surface, although the authors do have basic ability at following
the design for writing a research report.
For the goal of understanding and applying sociological concepts,
some of the sociology majors do not have a clear idea or understanding
of the concept they are writing about. Hence, their writing
tends to lack precision. Their use of concepts is too broad,
not precise. The concepts tend to be "spoken about"
not carefully defined or conceptualized. They tend to be confused
about the meaning, or implicitly think they know the meaning
without making the meaning explicit, or keep hidden in their
writing what they do know about the meaning of the concept.
As a consequence, their work is often at the descriptive level,
pointing out what they observed, saw, heard, and so forth. Even
at the descriptive level their accounts tend to be too brief
and superficial.
d. What can be done to improve the student’s work?
One way sociology faculty may be able to improve the writing
of students who do not write well is to try to have them write
less complicated, shorter sentences. Assignments that are longer,
need to require students to use subheadings within the paper
that might help them to organize their ideas more clearly and
coherently.
Even the satisfactory work of students could be improved on
this point.
One way to improve students work with sociological concepts
is to provide them with opportunities ( several assignments)
that require them to provide clear empirical referents and examples
for the concepts. If students can point to the empirical world
and use characteristics of that world as a way to understand
abstract concepts, they would be in a better position to both
understand the concept more clearly and to write about it more
concisely. Another way to possibly improve students’ ability
to use and apply sociological concepts is to provide assignments
that are specifically structured. The assignments may need to
be shorter so that more of them can be required. An example
of such an assignment may require the student to take one concept,
go to the literature to look up various conceptualizations of
the concept, choose one definition and then show explicitly
how the empirical observations the student is making can be
seen through the concept. If students had several of these briefer
assignments, they might improve their ability to move from observation
to analysis.
Still another way to help student improve their work writing
and applying sociological concepts is to provide them with opportunities
for writing and then revising and rewriting. This would have
the benefit of allowing students to see faculty or graduate
assistant feedback and then learn from that feedback in the
revision. This would provide students with practice and with
the possibility of receiving help. If students had more opportunities
for doing this, they would also learn how difficult it is to
do this well. They would also have the chance to see themselves
improving as they do the assignments. Providing more opportunity
for our students to write and revise should help them move from
description to analysis with more clarity.
The department may need to see if there would be resources to
hire some of its excellent graduate students or one of the part-time
faculty to help faculty who teach the core courses–especially
sociology 150, 125, 193 and 194–with the writing and rewriting
options. Faculty in the research methods courses already have
most students write and rewrite parts of their research projects
over the course of two semesters. This is possible in those
courses because the department attempts to limit enrollment
to no more than 25 in each section.
Summary Reflections on the Collective Portfolio
The process of assessment is still under review as much as the
student learning the department is seeking to assess. How we
arrive at our judgments, the criteria we use to assess the collective
portfolio, and the committee’s suggestions for improving
how we teach and how students might learn better are still open
for discussion and negotiation. Still, the process itself, however
imperfect, has moved us an important step forward.
The biggest problem our students face in reaching the two learning
goals assessed in the collective portfolio must be dealt with
on many levels and will require patience and reason. It is clear,
nonetheless, that problems with writing and discipline specific
analytical reasoning will require resources. These include,
but are not limited to, faculty time necessary to integrate
and effectively evaluate written analytical projects, especially
in the core courses with larger enrollments.
Regardless of resources available for implementing pedagogical
and curriculum changes, a major issue that will have to be considered
is how we address these changes as a department and not as individual
faculty members. This collective effort will have to be sensitive
to the intellectual integrity of individual faculty and also
to the fact that there are many different ways to effectively
teach the discipline of sociology. The worst thing we could
do is to force one approach over the other.
Finally, this is the first set of papers in the first department
collective portfolio. Over time the department will be able
to see more clearly trends in patterns in student work. The
results of this initial analysis may, in part, reflect the sample
of papers from 1998-1999 and not recurring patterns over a period
of time.
|
|
|
