DLESE Collections Committee

Draft Action Plan.

Written 2/28/00; slightly revised 3/3/00.



The Action Plan for the DLESE Collections Committee has two components:

  1. Advance the Collections Action Items articulated at the Coolfont Meeting, and
  2. Prepare proposals for the National Science, Mathematics, Engineering and Technology Education Digital Library (NSDL).


Advance the Coolfont Action Items.

1. Circulate the draft collection policy including criteria for acceptance to potential users for comments. Revise collections policy after community input.

Barbara DeFelice has indicated interest in pushing forward this activity. Mechanism not yet defined.

2. Define what "well-documented" means for each category of materials.

At Coolfont, we got as far as saying that "data should have metadata; assignments for students should have scoring rubrics." The GDL metadata group under the leadership of Kathryn Ginger is pushing hard and fast on this action item. We need to interact with them, particularly to assure that education-type documentation is included, in addition to data-oriented documentation. Some examples of "education-type" documentation could be: learning styles for which activity is particularly well-suited, level of math background required, answer key and scoring rubric for assessments. Chris DeLeonardo has agreed to be point-person for this activity.

3. Draft guidelines for what "pedagogical effectiveness" means.

This Coolfont action item was written when we envisioned that the burden of proof would be on the creator/provider of the resource to demonstrate that "student learning has occurred." If we adopt the Community-review mechanism outlined in the document "Thoughts about how to build a DLESE Collection, K. Kastens 2/14/00," then the nature of this action item changes. The guidelines become guidelines for the classroom tester/reviewer as well as for the creator. Such guidelines can be more pragmatic and empirical, less theoretical and less bound up in any particular ideology about what constitutes good pedagogy. Martin Ruzek has been thinking deeply about this action item.

4. Research recommendation engines and methods for review and evaluation.

Kim Kastens has been thinking about this, with some advice from Michelle Lamberson. The results of our thinking so far are in "Thoughts about how to build a DLESE Collection, K. Kastens 2/14/00," If the community-review concept in that document is acceptable to the DLESE Steering Committee, a recommendation-engine will play a major role in building the DLESE library by filtering the vast universe of existing relevant web resources. This action item will be advanced in the course of writing the "Collections Track" proposal to NSDL, see below.

5. Establish mechanisms for providing academic career credit to creator/suppliers of materials.

John Butler is leading this charge. His thoughts to date are summarized in his "Academic Recognition Task Force document" circulated on the DLESE Collections listserver. His plan for the coming year begins with gathering input from the geosciences community by means of a survey administered to the producers of the "best practices" resources on the Virtual Geoscience Professor web site. The survey will emphasize the producer's experiences with respect to the evaluation of their efforts by their peers and administration. Based on this survey, a preliminary set of guidelines will be developed by the Academic Recognition Task Force. The preliminary guidelines will be shared with the appropriate DLESE Committees, and discussed by a focus group (20-25 people) at the GSA meeting in Reno in November 2000. Implementation of strategies for enhancing academic recognition will begin in January 2000. Preliminary ideas include:

6. Develop scenarios for DL use, and coordinate them with collection development.

The use-case library development strategy that GDL has adopted means that scenarios are one of the main communication vehicles by which the DLESE rank-and-file can influence what gets built. The Collections Committee contributed its first scenario, the "Brad and Felice" scenario, back at Coolfont, following Doug Gordin's intuition that this would important and useful. Since Coolfont, we have sent in additional scenarios, most recently two scenarios dealing with academic career credit. Now that we understand how scenarios/use-cases feed into the GDL effort, we will remain alert for ideas that are surfacing in our discussions which aren't embodied in any of the existing use-cases, and write scenarios to embodied these new ideas. Kim Kastens will nudge this along, in her role as Committee Chair.

7. Develop specifications for prototype collections.

Dave Mogk will worry about this, in his role as liaison between Collections and GDL.


Prepare NSDL Proposals.

1. NSDL "Services" Track Proposal.

Martin Ruzek is spearheading the development of a proposal to the "Services" track of the NSDL, to support the development of JESSE, the Journal of Earth System Science Education. This proposal speaks to the bullet of the "services" track RFP which deals with "services to collection providers" in the form of "peer review mechanisms for quality assurance."

JESSE uses a journal-style peer review system, in which editors select reviewers based on their expertise. As a journal, JESSE will seek to inform its subscribers of what is new and excellent in the field of Earth System Education. The NSDL funding will be used to set up an infrastructure that will become self-sustaining and persist indefinitely.

2. NSDL "Collections" Track Proposal.

This proposal covers several activities pertaining to the NSDL RFP request for proposals that "aggregate and actively manage a subset of the digital library's content within a coherent theme or specialty," with the "theme or specialty" being Earth System Education.

The first activity under this proposal is to cast a wide net to find all the resources out there in the universe of Web content which are pertinent to Earth System Education. Much of this has been done by the Virtual Geosciences Professor and other collections of hotlinks. But there remains a perception that only the "low-hanging fruit" has been found, and that additional valuable resources can be found with more hunting. Chris DeLeonardo will spearhead this effort to aggregate a "wide and deep" collection of Earth-System Science-relevant resources.

The NSDL RFP requires "evidence that the proposed aggregation of resources will support the very best SMET education at all levels -- education that is inquiry-driven, active and engaging." Thus an important aspect of the Collections proposal must be to create a filtering system that will identify the best materials from the broad and deep collection. How this filtering system will work is explored in the document "Thoughts about how to build a DLESE Collection, K. Kastens 2/14/00." To summarize, we adopt the seven selection criteria established by the Coolfont Collections Committee, and implement several of them through a web-mediated "community review procedure," summarized in the following table:

Coolfont Selection Criteria. How to implement.
Well documented. Review by library staff.
Importance/Significance. More than N (threshold number to be chosen) educators from DLESE community tried this resource in their classroom.
Pedagogical Effectiveness.
Ease of use for students and faculty.
Inspirational or motivational for students.
On-line questionnaire filled out by educators who used resource in their classroom.
Accuracy, as evaluated by scientists. Invited review by a scientist, recruited by an editor.
Robustness/sustainability. Testing by a quality assurance professional.

Kim Kastens will take responsibility for implementing the filtering system.

The third activity under this proposal is to assess the scope and balance of the collection as it grows. If this were an ordinary book-library being built from scratch for a new community, there would a librarian watching the growth of the collection to make sure it was complete and well balanced. With a library based on contributed resources, there is a danger that the library will grow in a lopsided way. It could becoming rich in materials based on data from a well-organized and well funded government agency, while remaining poor in resources pertaining to another more diffuse and less organized discipline, or could grow rich in materials for undergraduates while remaining poor in materials for K-3. The product of this activity will be a report to the funding agencies and to the community calling attention to areas in which the library collections remains thin. Barbara DeFelice will take responsibility for this activity.

An unresolved question is whether the Collections Track proposal needs to include infrastructure to actually host and serve a large volume of resources. One school of thought says that DLESE is a distributed collection, and that each resource remains on the creator's server with only a collection of links and indices and service-providing software residing on DLESE-funded computers. Another school of thought says that the resources in the "inner-circle," the resources which have met all seven selection criteria, including being tested for robustness/sustainability, should be hosted at DLESE to ensure that they remain alive and available. Otherwise, the resource can gradually degrade or metamorphose into something other than the resource that was reviewed and given the DLESE stamp of approval. We need guidance on this point.

3. Coordination between the Proposals.

The "Services Track" and "Collections Track" proposals need to be well-integrated. Reviewers need to understand the relationship and division of labor between the two proposals. Some of the points that we need to emphasize include:

"Services" Track JESSE "Collections" Track filtering system
Focus on what is new and excellent in EESE. Focus on backlog of materials out there in the universe of Web content.
Infrastructure to persist indefinitely. Finite task. Does a job; then closes shop.
Traditional editor-mediated "peer-review". "Community review mechanism" mediated by web-based recommendation engine.

The review mechanisms used by the two proposals may be further compared, as follows:

Traditional "Peer-Review" Proposed "Community Review"
Reviewers are selected by their expertise by an editor. Reviewers step forward from the community.
Reviewers examine the material, or a description of the material, in their home or office. Reviewers test the material in their classroom with live students.
Typically two reviews. A larger number of reviews is possible: five? ten?
Volume through-put limited by editor-time. Volume throughput can be larger.
Editor filters out unfair or malicious reviews. Tests carried out in inappropriate context can be auto-discarded. Unfair or malicious reviews can be swamped out by numerical weight of good reviews.

Although the community-review mechanism has several appealing-sounding attributes, it is a venture into uncharted territory. The editor-mediated traditional peer-review mechanism is tried and true. It is probably in DLESE's best interests to have both review processes operating in parallel in the early stages of the development of the library. At the end of the Phase I of the NSDL funding, the best aspects of both review procedures can be merged into a common structure to go forward into the future.