|
Collective Portfolio
Assessment
Assessment rests
on two legs: student perceptions of their experience and actual
student work of graduating seniors. This is why the department’s
assessment plan also includes samples of student work in a "Collective
Portfolio." This portfolio is currently being constructed.
In the near future this site will present an analysis of the quality
of student work of graduating seniors.
The Collective Portfolio is composed of a sample of papers from
majors in all of the core, required upper division courses in the
major. Each year, the Assessment Committee solicits samples of A,
B, and C work from faculty who teach these courses. The selected
papers are then put into the department’s "Collective
Portfolio." The papers reviewed are from the academic year
1998-1999. These papers represent the first set of papers collected
in the portfolio. The Sociology Department’s Committee on Assessment
decided to initially look at the papers in the Department’s
Collective Portfolio by focusing on only two of the department’s
eleven learning goals. The Committee selected only two goals to
start with, since it has no track record of evaluating papers in
a collective portfolio. The thought was to start slowly to see how
the assessment of actual student papers might work and what could
be learned from doing this exercise. In future years, the department
will select other learning goals for analysis in the Collective
Portfolio. The two selected goals represent a disciplinary goal
(#5) and a general education, university goal (#11). As the Committee
read the papers, the members focused on the patterns that were present
in the students’ work.
Learning Goals Assessed in the Portfolio
The two learning goals selected for this initial assessment –
(#5) The sociology major at CSU Sacramento will be expected to study,
review, and reflect on basic concepts in sociology and their fundamental
theoretical interrelations. Under this learning goal, the student
should be able to define, give examples, and demonstrate the relevance
of concepts such as, culture, socialization, stratification, social
structure, institutions, differentiations by race/ethnicity, gender,
age, and social class, etc.
(#11) The sociology major at CSU Sacramento will be expected to
have the writing skills necessary to communicate effectively with
persons whom they encounter in their work, civil obligation, and
personal life. Under this goal, the student should be able to a)
write a clear and concise report of the findings from sociological
data analysis; and b) write a clear and concise sociological analysis
and account of a social event, topic, issue, or problem.
Assessment Categories or Rubrics
Before reading the papers, the Committee decided to use a simple
set of categories to start with. Over time, these categories will
probably be expanded and become more focused. The assessment categories
used for the initial assessment are –
a. What do sociology majors do that is outstanding?
b. What do sociology majors do that is satisfactory?
c. What do sociology majors do that needs improvement?
d. What might the department do to help students improve?
Assessment Results
a. Outstanding Work
For the learning goal on writing a research report, a few students
did outstanding work. These students were able to elaborate their
ideas and present a more substantial account both of the concepts
they were using as well as the literature review. For example, in
the literature review, students were able to go beyond merely describing
a published piece of research as it relates to their topic, but
were also able to criticize the research they were reviewing. In
other words, they went beyond description to critical thinking about
the literature This is an outstanding accomplishment for an undergraduate.
Few of our students are able to do this, in part because it requires
a good deal of knowledge about the topic and previous research done
on the topic. These papers also show the ability to move between
levels of analysis in sociology–between the micro and macro,
between the self and social structure, and so forth.
The outstanding papers were also organized very well. The paragraphs
were logically ordered, there were transitions between paragraphs,
each paragraph had a thesis sentence, and paragraphs covered only
one topic. Less complex sentences tended to be used, which made
the writing more clear and concise. Literature reviews, library
research, and citations were done well. They were complete, up-to-date,
and related to the topic and thesis. These papers reflect precision
in the use of sociological concepts too. The concepts were stated
explicitly and defined reasonably well, and these papers provided
clear empirical referents for the concepts used. In other words,
the authors provide an empirical account of what the concept means.
The outstanding work represents the ability to move from the descriptive
level to the analytical level.
b. Satisfactory Work
For the leaning goal on writing a research report, most students
performed satisfactorily. They were able to communicate their ideas
and write a report based on findings from sociological data analysis.
Most cited current literature, most used proper citation methods,
most were able to explain the independent and dependent and control
variables in their research paper (the research design), and most
presented clear conclusions. Even so, satisfactory work often contains
errors in writing, but they tend to be minor and don’t interfere
with communication of the topic or ideas. In addition, the satisfactory
papers were organized in terms of defining the research question,
the literature review, research design, data analysis, conclusions,
and shortcomings of the study. In brief, they were able to follow
the acceptable organization for writing a research paper. This is
what we would expect given their course work in research methods.
For the learning goal on the application and understanding of basic
concepts, most students were able to adequately describe the empirical
world of their observations, but were less able to clearly link
those observations to a clearly presented and concisely defined
sociological concept. In addition, in some of the papers that were
satisfactory, the student tried to cover too many concepts and consequently,
failed to provide either a clear conceptualization of the concept
or link the concept to the empirical world of their observations.
When there are too many, some of them are covered superficially.
The satisfactory work is represented by good, sometimes excellent
work at the descriptive level, detailing observations in the natural
laboratory of society. This work is weaker at the analytical or
conceptual level. The student authors don’t move easily from
the descriptive to the analytical level.
However, satisfactory work often does show that the student has
learned to think sociologically, to use the sociological imagination.
These papers do show that the student has taken empirical observations
and reflected on them through the eyes of sociological analysis,
even though the examples and the writing don’t reveal this
as well as they could. An example of this is a satisfactory paper
which involved observing an African American church ceremony. Although
the scripture and the sermon were about the "brotherhood of
Christ," the student was observing the inequalities in the
church from the point of view of the power structure within the
church. This student was able to give some examples that illustrated
the concept of the power structure, and the student did discover
that she was being a sociologist, seeing the event, at least in
part, as a sociologist would. But there could have been more empirical
examples and the concept of power structure could have been more
adequately conceptualized.
c. Work that Needs Improvement
For the learning goal on writing, some sociology majors do not write
as well as they should or could. They often do not have good transitions
between paragraphs, cover more than one topic in a paragraph, have
some grammatical problems (wrong verb tense). Some sentences can
be too complex and tangled, so that the idea conveyed is not clear.
Citations are also not consistently done in the research report.
Sometimes the thesis of the paper is not stated clearly. Conclusions
are not complete. Sometimes there are factually incorrect statements.
Subheadings in the paper tend not to be present, which leads to
loose organization of the ideas. One of the characteristics of student
work that needs improvement is that it tends to be superficial.
There is not a deep account of the topic or the research problem
or the literature review. Sometimes the problem is not stated too
clearly or is too oversimplified. The work tends to skim the surface,
although the authors do have basic ability at following the design
for writing a research report.
For the goal of understanding and applying sociological concepts,
some of the sociology majors do not have a clear idea or understanding
of the concept they are writing about. Hence, their writing tends
to lack precision. Their use of concepts is too broad, not precise.
The concepts tend to be "spoken about" not carefully defined
or conceptualized. They tend to be confused about the meaning, or
implicitly think they know the meaning without making the meaning
explicit, or keep hidden in their writing what they do know about
the meaning of the concept. As a consequence, their work is often
at the descriptive level, pointing out what they observed, saw,
heard, and so forth. Even at the descriptive level their accounts
tend to be too brief and superficial.
d. What can be done to improve the student’s work?
One way sociology faculty may be able to improve the writing of
students who do not write well is to try to have them write less
complicated, shorter sentences. Assignments that are longer, need
to require students to use subheadings within the paper that might
help them to organize their ideas more clearly and coherently.
Even the satisfactory work of students could be improved on this
point.
One way to improve students work with sociological concepts is to
provide them with opportunities ( several assignments) that require
them to provide clear empirical referents and examples for the concepts.
If students can point to the empirical world and use characteristics
of that world as a way to understand abstract concepts, they would
be in a better position to both understand the concept more clearly
and to write about it more concisely. Another way to possibly improve
students’ ability to use and apply sociological concepts is
to provide assignments that are specifically structured. The assignments
may need to be shorter so that more of them can be required. An
example of such an assignment may require the student to take one
concept, go to the literature to look up various conceptualizations
of the concept, choose one definition and then show explicitly how
the empirical observations the student is making can be seen through
the concept. If students had several of these briefer assignments,
they might improve their ability to move from observation to analysis.
Still another way to help student improve their work writing and
applying sociological concepts is to provide them with opportunities
for writing and then revising and rewriting. This would have the
benefit of allowing students to see faculty or graduate assistant
feedback and then learn from that feedback in the revision. This
would provide students with practice and with the possibility of
receiving help. If students had more opportunities for doing this,
they would also learn how difficult it is to do this well. They
would also have the chance to see themselves improving as they do
the assignments. Providing more opportunity for our students to
write and revise should help them move from description to analysis
with more clarity.
The department may need to see if there would be resources to hire
some of its excellent graduate students or one of the part-time
faculty to help faculty who teach the core courses–especially
sociology 150, 125, 193 and 194–with the writing and rewriting
options. Faculty in the research methods courses already have most
students write and rewrite parts of their research projects over
the course of two semesters. This is possible in those courses because
the department attempts to limit enrollment to no more than 25 in
each section.
Summary Reflections on the Collective Portfolio
The process of assessment is still under review as much as the student
learning the department is seeking to assess. How we arrive at our
judgments, the criteria we use to assess the collective portfolio,
and the committee’s suggestions for improving how we teach
and how students might learn better are still open for discussion
and negotiation. Still, the process itself, however imperfect, has
moved us an important step forward.
The biggest problem our students face in reaching the two learning
goals assessed in the collective portfolio must be dealt with on
many levels and will require patience and reason. It is clear, nonetheless,
that problems with writing and discipline specific analytical reasoning
will require resources. These include, but are not limited to, faculty
time necessary to integrate and effectively evaluate written analytical
projects, especially in the core courses with larger enrollments.
Regardless of resources available for implementing pedagogical and
curriculum changes, a major issue that will have to be considered
is how we address these changes as a department and not as individual
faculty members. This collective effort will have to be sensitive
to the intellectual integrity of individual faculty and also to
the fact that there are many different ways to effectively teach
the discipline of sociology. The worst thing we could do is to force
one approach over the other.
Finally, this is the first set of papers in the first department
collective portfolio. Over time the department will be able to see
more clearly trends in patterns in student work. The results of
this initial analysis may, in part, reflect the sample of papers
from 1998-1999 and not recurring patterns over a period of time.
|
