This document summarizes the recommendations that emerged from the Workshop on the Quality of the DLESE Broad Collection, held at Lamont-Doherty Earth Observatory, June 30 to July 1, 2003. Workshop attendees were Kim Kastens (chair), Susan Buhr, Barbara DeFelice, Christopher DiLeonardo, Ben Domenico, Karon Kelly, Suzanne Larsen, Michael Mayhew, Rajul Pandya, John Saylor, Randy Sachter, Tamara Sumner, Neil Holzman (notetaker). Joining by phone were Holly Duvaul, Katy Ginger, Mary Marlino, and John Snow. David Mogk provided a written briefing paper. Full notes of the Workshop, with links to presentations and documents provided, will be available via the DLESE website.
These recommendations cover two topics: (1) Criteria and Priorities for developing the DLESE Broad Collection, and (2) Procedures to implement these priorities and recommendations.
In each recommendation, the text in italic constitutes wording of the recommendation agreed up by the workshop participants. In many cases, the recommendation will need further fleshing out. The non-italic text that follows the italicized recommendations contains points to be taken into account in fleshing out or implementing the recommendation.
Terms: The "DLESE Broad Collection," refers to a collection of resources that have met some threshold criteria of quality and relevance. This is contrasted with the "DLESE Reviewed Collection," which refers to a collection of resources that have been judged excellent against a stringent set of seven selection criteria. "Themed Collections" will be used to refer to collections of resources that have been accessioned as a group into DLESE, under the terms of the Interim Collection Accession Policy. This is contrasted with "The DLESE Community Collection," which refers to resources that have been contributed and cataloged as individual items.
"Accession threshold criterion" means that we think resources not meeting this criterion should be excluded from the DLESE Collections. "Favor" or "give high priority to" means that we think that limited resources (money, staff time) should be preferentially directed towards growing the collection in this direction, by gathering resources or accessioning collections that have these attributes.
In this context, DLESE needs to decide whether we are indeed a library about "Earth System Education" or in fact a library about "Earth System Science Education." "Earth System Science Education" connotes a library of resources useful for teaching or learning about the natural, physical and biological sciences of the Earth System. The term "Earth System Education" could be construed to mean a domain that extends substantially beyond natural/physical/biological science, including, for example, environmental policy, environmental law, environmental economics, environmental health sciences, and environmental engineering. We need to bite the bullet and articulate our content boundary for this first, fundamental criterion to be effective.
Tools available to help with the defining process include the AAAS Concept Maps and the Earth System Education vocabulary, which is currently under development.
While fleshing out the definition of "Earth System Education," it would also be useful to be more specific about what is meant by "relevant to." How far does DLESE want to go towards providing resources from cognate fields (e.g. chemistry, statistics) that undergird understanding of the Earth System? It would be helpful to people trying to apply this criterion (e.g. collections developers) to provide examples of resources that do and do not meet the criterion.
This criterion needs to be fleshed out operationally, perhaps with examples of what would and would not be allowed, or a checklist of what is expected. We recognize that it is impossible to guarantee that any piece of software is 100% free of bugs. A workable way to implement this criterion may be to require that it be "free from defects that inhibit the intended use of the resource."
The rationale for this recommendation is that knowledge of where a resource comes from--its author, sponsor or publisher--helps the user to judge how much credence to give the resource. In general, we consider that completely unattributed resources are among the more suspicious documents on the World Wide Web.
As a best practice, we further recommend that information be provided within the resource to allow users to contact the author or sponsoring organization with questions or suggestions.
"Scientific accuracy" was highly valued by the K-12 educators in Sumner et al's (2002) focus groups, and by the DLESE stakeholders surveyed by the Collection Committee Academic Career Recognition. It is highly valued by the Workshop participants, and by our sponsors at the National Science Foundation. This criterion is stated as "DLESE strives...." in recognition that we just can't guarantee that no errors of science will slip into the library, given the intricacy of science and the breadth of DLESE's content domain.
The workshop participants recognize that there may be important exceptions to the practice of including "only resources that reflect sound scientific principles and ... current scientific knowledge." Historical documents, which represent the scientific knowledge of a previous generation rather than "current scientific knowledge," can be important for learning about the history and epistemology of science. Resources expressing one point of view on a controversial topic might be considered by some to be outside of "sound scientific practices," and yet be valuable for teaching critical thinking skills such as evaluating viewpoints and weighing the effects of motives and biases.
Most workshop participants favor developing some distinctive way to identify resources that could have value if used in a specific pedagogical context (e.g. history of science, critical thinking), but which do not necessarily reflect current, mainstream scientific understanding. (See point (4) under "Procedures" below.)
To be effective and yet fair, this policy requires: (1) a readily-accessible mechanism for members of the DLESE Community to complain that they think a resource is inappropriate, and (2) a fair and transparent mechanism for the organization to make decisions about whether to decline or remove a resource that someone has complained about (see below under Procedures.)
The phrase "... to describe the resource as it sees fit" means that DLESE staff can change metadata submitted by a resource contributor. Examples might include removing self-congratulatory phrasing from the description metadata field, or noting that a point of view expressed in a resource is not widely held. This is compatible with the statement currently in the DLESE Terms of Use: "If the user does contribute comments, reviews, metadata, or other information, the user voluntarily grants DLESE or UCAR or its affiliates a nonexclusive, royalty-free, perpetual, irrevocable, and full right to use, reproduce, modify, adapt, publish, translate, create derivative works from, distribute, and display the contribution throughout the world in any media."
Workshop participants cited examples of resources that could potentially be used for learning or teaching about the Earth System but which were not intended for this purpose and did not include pedagogical content (e.g. a site providing fuel efficiencies of different cars). Although workshop participants would not necessarily reject such a site if it was contributed and cataloged by a community member, they thought that DLESE's limited time and money were better spent on resources for which the pedagogical application was more obvious.
Workshop participants wished to make a firm statement about educational quality of the resources in the Broad Collection. However, we could not come up with an all-encompassing test or measure that could be applied at the gateway to the Broad Collection, given the variety of resource types in the DLESE Collection, and the variety of pedagogical philosophies in the DLESE Community. We consider that clearly-articulated learning goals and assessment mechanisms are common attributes of good learning resources, no matter the resource type, and thus seeking out these attributes should foster high collection quality.
This recommendation implies that DLESE must maintain a capability for monitoring where gaps and thin spots exist in the collection.
DLESE should inform the community of potential resource creators and collection builders of where gaps and thin spots exist in the Collection, and should provide assistance (e.g. workshops, evaluation tools, best practices guidance) to help creators develop needed resources.
An historical example would be the shift from an undergraduate focus at the time of the Coolfont meeting (summer 1999), to a current focus in which K-12 is prominent. This refocusing began as a policy decision, and was implemented as a shift in the balance of what resources were collected by the funded gathering/cataloging groups.
An upcoming example would be georeferencing, the capability to search for resources by geospatial (location) specification. This recommendation would put a priority on gathering of resources or accessioning of collections with geospatial information, in the lead up to the release of georeferencing capabilities in DLESE.
Many of the participants in the focus groups of K-12 educators studied by Sumner et al (2002) were strongly opposed to the presence of advertising in DLESE resources, objecting that advertising distracts learners from the task at hand. Yet workshop members could think of counterexamples of resources that were otherwise good, but did include advertising. There was a lively discussion on the DLESE Collections listserver about the acceptability of resources with advertising for the DLESE Broad Collection, with strong opinions on both sides. Thus workshop participants felt that the presence of advertising could not be used as an accession threshold, but that resource gatherers should be encouraged to favor resources that do not include distracting advertising.
This recommendation is not intended to exclude for-fee genuine educational materials.
This recommendation is not intended to exclude educationally-valuable sites that are underwritten by corporate sponsorship (e.g. Geological Society of America).
"Free or low cost to educational users" was previously considered and rejected by the Steering Committee as an accession threshold criterion ("filter") for the Broad Collection. Workshop participants acknowledge that in some cases the best resource for a particular purpose may cost more than a nominal amount of money, and thus agree that cost should not be an accession threshold criterion. All else being equal, however, workshop participants favored a collection comprised mostly of resources that are free or low-cost to educational users.
Had the criteria above been in place from the beginning of DLESE, some problem resources would not have been gathered. (Based on Chris DiLeonardo's gathering experience.)
Had the criteria above been in place from the beginning of DLESE, some problem resources would have been identified at the cataloging or metadata QA step. (According to Holly Devaul, the person hands-on responsible for metadata QA.)
Approximately 100 resources currently in the DLESE Broad Collection (out of approximately 3600 total) might become candidates for deaccessioning under the criteria proposed above. (Estimate by Holly Devaul.)
Stating so many of the guidelines for admission to the Broad Collection in terms of "DLESE favors...." or "DLESE places a high priority on...." will work best if there continue to be funded gatherers for the foreseeable future (gathering individual resources and/or building collections) who are explicitly committed to work towards these guidelines. "The community" of volunteer resource contributors acting on its own through the open cataloger is not likely to create a library that happens to align with the attributes that "DLESE favors".
Recommended Procedures for Community Collection resources:
Funded gathering/cataloging teams would apply the new criteria as they gather and catalog resources, both the accession threshold criteria and the "favor/prioritize" criteria.
At the metadata QA step, resources would be scrutinized against the new criteria by the DPC staff, especially resources that did not come from a recognized gathering/cataloging group. Obvious cases (out of scope, no attribution) could be excluded from the library at this step.
Problematic resources would be referred to a Review Board, constituted by and overseen by the Collection Committee. Consideration of a resource by the Review Board would take place post-cataloging, pre-ingestion.
The Review Board would make a recommendation about whether or not to accession that item. The item then would or would not be accessioned.
Collection Committee should seek to learn heuristically from consideration of these specific problem cases to refine criteria and priorities, outreach message, etc., so as to decrease abundance of future problem cases.
Note that the "Community Collection" covered by this recommendation includes resources submitted to the DLESE Broad Collection, either via a funded gathering/ cataloging effort (e.g. current Foothill College/AGI team) or from the community via the open DLESE Cataloging Tool.
The Review Board described in this recommendation could also examine resources about which there has been a complaint from a member of DLESE community, as envisioned under Criteria Recommendation (5) above.
A minority opinion among the Workshop participants was that all resources submitted for the DLESE Broad Collection should be examined by a Review Board, in order to ensure quality. The workshop as a whole did not favor this option at this time because: (a) the problems seem concentrated in only a small fraction of resources, and it seemed unnecessary to spend additional time and energy on the many non-problematic resources, and (b) such a review process seemed too far removed from DLESE's founding vision as an open collection built by the community.
This recommendation is compatible with the existing Deaccession Policy.
No criticism of the prior gathering or cataloging decisions is implied by this recommendation; those choices were made at a different time under different ground rules.
Beyond stating the general principle that Themed Collections items should meet the same criteria as Community Collection items, the workshop participants felt that we don't yet have sufficient experience with accession of Themed Collections to recommend a detailed procedure to check for relevance and quality of resources. A working group was scheduled to meet one week after our workshop to review the documentation produced by the first set of collections seeking to be accessioned as themed collections. We asked participants in that working group to observe that process with an eye towards strengthening DLESE's oversight of the quality of resources arriving via themed collections.
The workshop did not consider criteria or procedures for annotation collections. We request that the upcoming metadata workshop consider this question as part of their discussion of metadata and annotations.
We recommend that DLESE explore developing some distinctive way to identify and provide access to resources that could have value if used in a specific pedagogical context (e.g. history of science, critical thinking), but which do not necessarily reflect current, mainstream scientific understanding.
Workshop participants differed as to what that distinctive identification should be. Options discussed include:
provide access to such resources via a route outside the "DLESE branded" collections, perhaps via topical webpages on the model of the the PBS site on Evolution, or through a "Developmental Collection" with less stringent criteria than the Broad Collection.
include such resources in the DLESE Broad Collection but have a distinctive icon or label appear in the return on the DLESE Discovery System, and
include such resources in the DLESE Broad Collection, but include specific wording in the first few lines of the description clarifying the pedagogical context in which the resource would be recommended for use.
With respect to the "Developmental Collection" concept, we note that at the time the DLESE Community Plan was developed, the Broad Collection was envisioned as serving as a developmental collection: "The library will identify these resources as relevant but unreviewed, applying the philosophy of caveat emptor. The DLitself can serve as a 'testbed' for reviewing and field-testing educational materials, and positive feedback via communications networks hosted by the DL could provide the basis for the documentation of educational effectiveness required for an item to move into the peer-reviewed section of the DL" (DLESE Community Plan, p. 31).
Workshop participants cautioned that if DLESE does develop some distinctive way to identify and provide access to resources which take a strong point of view on environmental controversies, that all advocacy positions, from Greenpeace to the World Coal Institute, should be treated the same.