Follow-up to a workshop at DH2015, Sydney, 29 June 2015
David Joseph Wrisley, American University of Beirut @DJWrisley
Katherine Hart Weimer, Rice University @kathy_weimer
Karl Grossner, Stanford University @kgeographer
The goals, methods, and products of Digital Humanities projects take many forms and may be assessed at various stages of their lifecycle: as planned work described in grant proposals, formally or informally at interim stages of progress, and at completion in independent or publisher-solicited reviews. Peer review, both formal and informal, has a very meaningful role in scholarship, however, the digital format challenges traditional assessment criteria. The intent of this document is to provide an opportunity for community input into an expansion of the existing DH Commons review criteria, in particular, for geographically inflected scholarly works.
The activities and products of DH projects can fall into one or more of several broad categories, including but not limited to: data collections; digital object collections (e.g. images of manuscripts, maps, and other artifacts); scholarly editions of texts; computational analyses of works, events, or activity and visualizations thereof (with or without interactivity); interpretive artistic works; and so on. Interactive products of DH projects may include interpretation and argumentation in extended prose elements, or be an accompaniment to a traditional print publication.
Clearly there is no set of detailed metrics that can apply to all DH projects and all circumstances of review, but DHCommons, an initiative of CenterNet, has developed a set of review guidelines as a “baseline for commentary” by reviewers, as well as submission guidelines for project teams.
We have taken those guidelines as a starting point, and encourage readers to become familiar with the base DH Commons document noted above. We have elaborated on some elements with respect to geographically inflected work, and ask you to evaluate their usefulness as criteria in assessing a few varied projects. Our goals are to explore how work in the GeoHumanities require us to expand upon, and perhaps revise if needed, what DHCommons has put forward. The original DH Commons text appears in plain text below in order to provide context. Note: GeoHumanities-specific elaborations appear in orange, underlined and in bold in this text.
This section asks reviewers to consider the overall field contribution of the project, paying special attention to gains made following the initial funding period (if applicable). We are particularly interested in how digital methods and modes of presentation offer new ways to address ongoing scholarly conversations or contribute to Digital Humanities methodology.
How does the project advance contemporary discussions within its particular subject area?
Does the project fully engage with current scholarship in the field?
Apart from work in the subject field, is literature relevant to geospatial methodologies and relevant applications cited? are methods or their application novel?
Do the digital methods employed offer unique insights into the project’s key questions?
How does mapping and/or geospatial analyses contribute to this work? Is it essential? Is it appropriate for the kind of project?
Which geo-humanistic aspect(s) of this project makes the greatest contribution (analysis, narrative, data, etc)?
Rather than narrowly about web or interface design, this section also asks reviewers to confront the project’s methodological and scholarly aims from a more disciplinary perspective. As much as possible, we want to move from thinking in terms of a form/content divide toward a full consideration of the digital project as purpose-built scholarship. In other words, the digital in digital humanities scholarship should make clear and critical interventions into important field topics, rather than simply being a mode of delivery.
Does the interface effectively communicate and facilitate the goals, purpose, and argument of the project?
How do the design and content elements of the project interact and integrate with one another?
Discuss usability of the interface(s) from the perspective of a reader/researcher; if possible, also discuss usability from the perspective of current user experience best practices?
Are chosen analytic methods (if any) suitable for addressing the research question?
Are the data used accurate, open/transparent and appropriate to the stated research questions?
Are the maps complete and coherent; do they convey the stated or computed results well?
Is the uncertainty inherent in all analysis conveyed in the map and/or written results?
Do they adhere to basic cartographic principles for legibility and clarity of communication, e.g. “separate meaningful characteristics and […] portray likenesses, differences, and interrelationships” well? 
Preservation can mean a number of different practices in a digital context. From long-term data storage to continued frontend browser optimization, the one thing that unifies preservation as a category is human commitment to upkeep and best practices. Preservation may be addressed in different ways depending on the short or long term goals of the project.
Have relevant best practices and standards been followed for markup and metadata? Has the project used responsive design?
Is documentation available about the project? Is information provided about who, why, when, and how different responsibilities were assigned?
How is the project hosted? Through a university server? A commercial host? A non-profit organization? Is there evidence of ongoing commitment to support of the project at the level of hosting? Is there similar evidence of ongoing support from project personnel?
Is there a preservation and maintenance plan for the interface, software, and associated databases (multiple copies, mirror sites, collaboration with data archives, etc.)? Is the project fully exportable/transferable? Are the data downloadable?
Is the data provenance described and discussed? Is the scope of the data articulated? Are the authors’ manipulations of the data documented in prose or code (and thus reproducible)? Are reused data appropriately cited?
Is the software being used proprietary, open-source, or editable by multiple programs? Are there clear plans for future accessibility? Will researchers have access to project material and/or metadata outside of a web-based interface? Is any rationale for keeping material/code private provided?
Will the project continue to be updated? If not, is the end of the project discussed? Is it complete or archivable now?