Report from
"Building the Reviewed Collection" Focus Group
at the DLESE Leadership Conference.
Bozeman, MT June 2000.
The charge to this focus group was to contribute to the early design of a procedure
to select resources for the "Reviewed Collection" of DLESE, identifying
the "best" resources within a larger "Open Collection."
The focus group began with a presentation by Kim Kastens. First , she reviewed
decisions made at the Portals to the Future Workshop, including the rationale
for having both reviewed and open collections, plus the seven selection criteria
(see http://www.dlese.org/documents/reports/panelreports/panel_3.html).
Then she presented a straw man plan for a review procedure which includes input
from the user-community via a web-mediated community-review recommendation engine,
from library staff, and from specialists. This plan is designed to be scalable
to handle the large number of resources projected for inclusion in the DLESE
collection, and supports the DLESE vision of a library built by and for the
community.
In the main activity of the Focus Group, participants filled out paper mock-ups
of a first design of two web-submitted forms that would be used by community-reviewers.
Under the proposed plan, only members who were registered with DLESE as educators
would be able to access the evaluation forms. However, within the group of people
in the DLESE membership database as "educators," community-reviewers
would step forward rather than being recruited by an editor.
First, focus group participants were first asked to recall a specific situation
in which they had looked at a web resource, but decided not to use it in their
teaching, and fill out a form appropriate for that situation. The information
from that form will go to the creator of the resource, and a digested version
will go to the Discovery System staff to catch situations when the resource
is misdirected rather than poor.
In the subsequent discussion, the following suggestions came forward concerning
the form for educators who looked at the resource but decided not to use it:
- Add additional radio buttons for other possible reasons
for not using the resource:
- "It doesn’t meet standards (pull down
menu to pick state or national standards)."
- "not appropriate for my class because __________________."
- "I didn’t think it would capture the
interest of my students."
- "I don’t have ready access to appropriate
technology, specifically _________."
- "I didn’t like the pedagogical approach.
I would have preferred __________."
- Capture more information on the "too difficult"
category. Add "What was too difficult: math? Navigation? Science concepts?
Other ________."
- Capture early input to "robustness/sustainability"
criterion by having a radio button for "I experienced a bug or technical
problem, which was ____________."
- Capture early input to the "scientific accuracy"
criterion by having a radio button for "I found a factual error, which
was ________________."
- Capture suggestions for the creator, using same wording
as on the other form: "In the space below, please provide additional
comments on the resource. Your suggestions will be sent anonymously to the
creator of the resource, for possible use in improving the resource."
The second form was a more extensive questionnaire for educators
who did use the resource under review for helping students learn. The information
from this form will go to the creator (all information anonymously), to the
editor/gatekeeper of the reviewed collection (after a threshold in quality and
quantity of reviews is passed), and to be posted with the resource (teaching
tips only). Again, focus group participants were asked to recall a specific
circumstance when they had used a web-based educational resource, and fill out
a paper mockup of a web form with that situation in mind.
The following suggestions were offered about the web-form for educators who
tested a resource with real learners:
- Important new questions: "Would you use this resource
again?" "Would you recommend this resource to a colleague?"
- Use a Likert scale (e.g. strongly agree, agree, neutral,
disagree, strongly disagree) instead of a numerical scale for the quantitative
metrics.
- Break down the three quantitative scores into subcategories,
i.e. instead of "Pedagogical Effectiveness" ask for scores on a
half dozen or so concrete, observable phenomena which can be combined into
a composite score for Pedagogical Effectiveness; instead of "Ease of
Use for Faculty and Students" put "Ease of Use for Learners"
separate from "Ease of Use for Teachers".
- Re-word "motivational/inspirational" question
to be more personal: "My learners were motivated to learn/work/understand
the Earth and/or environment through use of this resource".
- Capture additional input to the "well-documented"
criterion by asking "Were there parts of the resource that were poorly
documented? Please specify:________".
- Capture early input to the "robustness/sustainability"
criterion by asking "Did you find any bugs or experience any technical
difficulties using the resource? If so, please specify: _________".
- Capture early input to the scientific accuracy criterion
by asking: "Did you find any factual errors in the resource? If so, please
specify:_________".
- Capture additional information about school type: urban/suburban/rural.
- Consider a form for students to fill out, especially
for the "inspirational/motivational" criterion. Get around computer
access problem by having it be a printable form.
The participant-annotated paper mock-ups of the review forms have turned out
to be a valuable resource. We plan to revise the forms taking into account comments
from the Bozeman conference and repeat the focus group, possibly asynchronously
over the Collections list-server.
The following suggestions were offered as to what should happen next, after
the community-review results are in hand:
- Consider using an algorithm to weight the quantitative
data that varies with intended level of students (for example, the weighting
of "motivational/inspirational" relative to "scientific accuracy"
might be higher in an elementary school resource than in an undergraduate
resource).
- Consider using an algorithm to weight the quantitative
data that varies with type of resource.
- The present plan calls for a review for scientific accuracy
by a scientist selected by an editor/gatekeeper (as one of the last steps,
following the community review). The focus group favors having an education
review board to parallel the science review board, with a review by a selected
education specialist paralleling the review by the selected science specialist.
[In subsequent discussion, two distinguished members of the focus group agreed
to form the nucleus of this education review board.]
- The present plan for the "technical robustness"
criterion calls for a Quality Assurance (QA) testing program with some of
the features of the QA testing procedures used by commercial publishers of
educational software. A focus group member whose educational software product
has been through a commercial QA process emphasized the importance of setting
up a close communication between the QA people and the creator/developer.
Other issues raised in the focus group:
- A non-representative cross-section of the community will
step forward as reviewers (Discussion: all methods of selecting reviewers,
including journals and NSF, yield non-representative cross-section. Need to
establish peer-pressure sense of obligation that if you use DLESE resources
you owe the community and the creator a review.)
- One focus group participant with experience with K-12
teachers reviewing instructional materials doubts that the quality of the
reviews from teachers will be adequately rigorous.
The next day, at the town meeting, there was a discussion of diversity and
the digital divide. During this discussion, it occurred to me that the proposed
DLESE review process could help with this problem. At the end of the review
form for educators who have used the resource, we could add an additional optional
question, something along the lines of: "[optional] DLESE is especially
seeking resources that have worked well in challenging learning/teaching situations.
Please estimate what percentage of the learners with whom you used this resource
are: ( ) learning disabled ( ) physically disabled ( ) economically disadvantaged
( ) English as a second language ( ) minority underrepresented in science (
) urban setting, limited access to Nature." Then, whenever a resource had
scores above a certain threshold in a situation with more than a certain percentage
of students in any of the harder to teach categories, the review package would
get sent automatically to a member of the Editorial Board designated as the
diversity watchdog. Picking from the resources brought to his/her attention
in this way, and maybe communicating with the reviewers/testers for additional
input, the diversity watchdog would assemble web pages of resources "Recommended
for students with limited English proficiency", "Recommended for urban
students," "Recommended for place-bound students" and whatever
other categories seem useful.