This document summarizes the recommendations that emerged from the Workshop on the Quality of the DLESE Broad Collection, held at Lamont-Doherty Earth Observatory, June 30 to July 1, 2003. Workshop attendees were Kim Kastens (chair), Susan Buhr, Barbara DeFelice, Christopher DiLeonardo, Ben Domenico, Karon Kelly, Suzanne Larsen, Michael Mayhew, Rajul Pandya, John Saylor, Randy Sachter, Tamara Sumner, Neil Holzman (notetaker). Joining by phone were Holly Duvaul, Katy Ginger, Mary Marlino, and John Snow. David Mogk provided a written briefing paper. Full notes of the Workshop, with links to presentations and documents provided, will be available via the DLESE website.
These recommendations cover two topics: (1) Criteria and Priorities for developing the DLESE Broad Collection, and (2) Procedures to implement these priorities and recommendations.
In each recommendation, the text in italic constitutes wording of the recommendation agreed up by the workshop participants. In many cases, the recommendation will need further fleshing out. The non-italic text that follows the italicized recommendations contains points to be taken into account in fleshing out or implementing the recommendation.
Terms: The "DLESE Broad Collection," refers to a collection of resources that have met some threshold criteria of quality and relevance. This is contrasted with the "DLESE Reviewed Collection," which refers to a collection of resources that have been judged excellent against a stringent set of seven selection criteria. "Themed Collections" will be used to refer to collections of resources that have been accessioned as a group into DLESE, under the terms of the Interim Collection Accession Policy. This is contrasted with "The DLESE Community Collection," which refers to resources that have been contributed and cataloged as individual items.
"Accession threshold criterion" means that we think resources not meeting this criterion should be excluded from the DLESE Collections. "Favor" or "give high priority to" means that we think that limited resources (money, staff time) should be preferentially directed towards growing the collection in this direction, by gathering resources or accessioning collections that have these attributes.
In this context, DLESE needs to decide whether we are indeed a library about "Earth System Education" or in fact a library about "Earth System Science Education." "Earth System Science Education" connotes a library of resources useful for teaching or learning about the natural, physical and biological sciences of the Earth System. The term "Earth System Education" could be construed to mean a domain that extends substantially beyond natural/physical/biological science, including, for example, environmental policy, environmental law, environmental economics, environmental health sciences, and environmental engineering. We need to bite the bullet and articulate our content boundary for this first, fundamental criterion to be effective.
Tools available to help with the defining process include the AAAS Concept Maps and the Earth System Education vocabulary, which is currently under development.
While fleshing out the definition of "Earth System Education," it would also be useful to be more specific about what is meant by "relevant to." How far does DLESE want to go towards providing resources from cognate fields (e.g. chemistry, statistics) that undergird understanding of the Earth System? It would be helpful to people trying to apply this criterion (e.g. collections developers) to provide examples of resources that do and do not meet the criterion.
This criterion needs to be fleshed out operationally, perhaps with examples of what would and would not be allowed, or a checklist of what is expected. We recognize that it is impossible to guarantee that any piece of software is 100% free of bugs. A workable way to implement this criterion may be to require that it be "free from defects that inhibit the intended use of the resource."
The rationale for this recommendation is that knowledge of where a resource comes from--its author, sponsor or publisher--helps the user to judge how much credence to give the resource. In general, we consider that completely unattributed resources are among the more suspicious documents on the World Wide Web.
As a best practice, we further recommend that information be provided within the resource to allow users to contact the author or sponsoring organization with questions or suggestions.
"Scientific accuracy" was highly valued by the K-12 educators in Sumner et al's (2002) focus groups, and by the DLESE stakeholders surveyed by the Collection Committee Academic Career Recognition. It is highly valued by the Workshop participants, and by our sponsors at the National Science Foundation. This criterion is stated as "DLESE strives...." in recognition that we just can't guarantee that no errors of science will slip into the library, given the intricacy of science and the breadth of DLESE's content domain.
The workshop participants recognize that there may be important exceptions to the practice of including "only resources that reflect sound scientific principles and ... current scientific knowledge." Historical documents, which represent the scientific knowledge of a previous generation rather than "current scientific knowledge," can be important for learning about the history and epistemology of science. Resources expressing one point of view on a controversial topic might be considered by some to be outside of "sound scientific practices," and yet be valuable for teaching critical thinking skills such as evaluating viewpoints and weighing the effects of motives and biases.
Most workshop participants favor developing some distinctive way to identify resources that could have value if used in a specific pedagogical context (e.g. history of science, critical thinking), but which do not necessarily reflect current, mainstream scientific understanding. (See point (4) under "Procedures" below.)
To be effective and yet fair, this policy requires: (1) a readily-accessible mechanism for members of the DLESE Community to complain that they think a resource is inappropriate, and (2) a fair and transparent mechanism for the organization to make decisions about whether to decline or remove a resource that someone has complained about (see below under Procedures.)
The phrase "... to describe the resource as it sees fit" means that DLESE staff can change metadata submitted by a resource contributor. Examples might include removing self-congratulatory phrasing from the description metadata field, or noting that a point of view expressed in a resource is not widely held. This is compatible with the statement currently in the DLESE Terms of Use: "If the user does contribute comments, reviews, metadata, or other information, the user voluntarily grants DLESE or UCAR or its affiliates a nonexclusive, royalty-free, perpetual, irrevocable, and full right to use, reproduce, modify, adapt, publish, translate, create derivative works from, distribute, and display the contribution throughout the world in any media."
Workshop participants cited examples of resources that could potentially be used for learning or teaching about the Earth System but which were not intended for this purpose and did not include pedagogical content (e.g. a site providing fuel efficiencies of different cars). Although workshop participants would not necessarily reject such a site if it was contributed and cataloged by a community member, they thought that DLESE's limited time and money were better spent on resources for which the pedagogical application was more obvious.
Workshop participants wished to make a firm statement about educational quality of the resources in the Broad Collection. However, we could not come up with an all-encompassing test or measure that could be applied at the gateway to the Broad Collection, given the variety of resource types in the DLESE Collection, and the variety of pedagogical philosophies in the DLESE Community. We consider that clearly-articulated learning goals and assessment mechanisms are common attributes of good learning resources, no matter the resource type, and thus seeking out these attributes should foster high collection quality.
This recommendation implies that DLESE must maintain a capability for monitoring where gaps and thin spots exist in the collection.
DLESE should inform the community of potential resource creators and collection builders of where gaps and thin spots exist in the Collection, and should provide assistance (e.g. workshops, evaluation tools, best practices guidance) to help creators develop needed resources.
An historical example would be the shift from an undergraduate focus at the time of the Coolfont meeting (summer 1999), to a current focus in which K-12 is prominent. This refocusing began as a policy decision, and was implemented as a shift in the balance of what resources were collected by the funded gathering/cataloging groups.
An upcoming example would be georeferencing, the capability to search for resources by geospatial (location) specification. This recommendation would put a priority on gathering of resources or accessioning of collections with geospatial information, in the lead up to the release of georeferencing capabilities in DLESE.
Many of the participants in the focus groups of K-12 educators studied by Sumner et al (2002) were strongly opposed to the presence of advertising in DLESE resources, objecting that advertising distracts learners from the task at hand. Yet workshop members could think of counterexamples of resources that were otherwise good, but did include advertising. There was a lively discussion on the DLESE Collections listserver about the acceptability of resources with advertising for the DLESE Broad Collection, with strong opinions on both sides. Thus workshop participants felt that the presence of advertising could not be used as an accession threshold, but that resource gatherers should be encouraged to favor resources that do