Keeping in the spirit of Sandy’s post on collaboration vs. “ownership,” I wanted to mention Fitzpatrick’s idea of peer review, share my hesitancy about her diagnosis of the problem and solution, and hopefully hear what everyone else thought about it.
In Planned Obsolescence, Fitzpatrick considers From Book Censorship to Academic Peer Review by Mario Biagioli (full text at http://innovation.ucdavis.edu/people/publications/Biagioli%202008%20Censorship_review.pdf) to describe how “peer review functions as a self-perpetuating disciplinary system, inculcating the objects of discipline into becoming its subjects” (Fitzpatrick 22). As Biagioli puts it, “subjects take turns at disciplining each other into disciplines” in academia (12). This concept makes sense across types of peer review; Biagioli focuses on the royal academies and the associated “republic of letters” as a way to conceptualize peer review beyond a singular project, and I am also thinking of contemporary practices that are designed to evaluate and recalibrate a power dynamic (like the time I realized that the department head in the back of a classroom was actually there to evaluate the instructor).
This entire process of peer review, but particularly familiar version that Fitzpatrick considers in her first and third chapters in detail, is wrapped up in notions of who counts as a peer. We have discussed the idea of collaboration throughout the semester, starting with the notion that DH projects often accommodate, even require, a variety of skills and contributions; Sandy’s post speaks to this point and flags the critical “decision point about whose contributions to include” in the first place as a good place to start for identifying a project’s collaborators and expanding our notion of a peer. All of this points to a more inclusive notion of the peer which, in turn, aligns with a field like DH that strives to be participatory and democratic in multiple senses of the words.
The peer review process that Fitzpatrick outlines in Chapter 3 seems like a good place to start putting this expanded idea of the peer into practice. She compares how digital commenting functions as one level of peer review for projects such as “Holy of Holies,” Iraq Study Group Report, her own article “CommentPress: New (Social) Structures for New (Networked) Texts,” Expressive Publishing, and a digital release of The Golden Notebook (112-117), describing a spectrum of options from an entirely open commenting feature where any reader could leave a comment to relatively closed off systems where only select readers could provide feedback. As I made my way through this chapter, the phrase “the wisdom of the crowd” (which we first encountered in the context of The Digital Humanities Manifesto 2.0 as described in “This Is Why We Fight” by Lisa Spiro) kept coming to mind. From my perspective, this notion underlies Fitzpatrick’s model for online peer review, which strives to be a social, open process while “managing the potential for chaos” (117). (Granted, this chaotic or more generally negative mob/mass/crowd was much more familiar to me from French history, Romantic literature, early urban sociology, and general concern about trolling, but I have come around to the idea that the crowd can be a force for good in so many DH contexts.)
However, Fitzpatrick also notes that the author of Expressive Processing experienced that “the preexistence of the community was an absolute necessity” (116) to make its comment structure useful. This experience logically translates to other projects: peer review that turns to the “wisdom of the crowd” can only be as helpful as its crowd. I see how the crowd might offer more variety of feedback and how a more expansive notion of peer review in general could magnify the voices of individuals who may not have gotten the chance to participate in the process otherwise, whether because they fall slightly outside of academic circles, have not yet acquired the prestige to “do peer review” for a publisher, or any other reason. But to become a member of that peer review community or crowd — one of the seven women with commenting privileges on The Golden Notebook, for example — in the first place, I see the same social and technical barriers to access that we have talked about in class. As a result, I am struggling to see how a more democratic comment structure in digital spaces changes the disciplinary power dynamic of peer review. In your reading, does Fitzpatrick’s proposed version of peer review (in certain contexts) adequately address this power dynamic?
I put together a blog post yesterday, saved it in draft, then must have deleted the content somehow. This is probably good because you starting the conversation about Fitzpatrick obviates the need for it.
So, first off, thanks for starting this convo.
You said, “As a result, I am struggling to see how a more democratic comment structure in digital spaces changes the disciplinary power dynamic of peer review.”
I have lots of questions about the model she’s proposing, but I think she’s addressing your concerns about the power dynamics in the following ways:
1. Openness. As you mention, the proposal is to go through the peer review process in the open. There is no black box around the process. Any authority economics (okay, I shouldn’t make up terms), that is, currency of gravitas and expertise, is available to be interrogated. Any cliques are operating in the open.
No longer are reviewers anonymous in the same way — i.e. rendering the author “blind” to them. No longer is it in the hands of the publisher/editor to determine who is a suitable authority in the work’s discipline. Which leads to….
2. “Review of reviewers.” I think she calls this “peer-to-peer review” — which is certainly a clever meme. I don’t think anyone could ever come up with a system that seeks to fully democratize, or crowd-source, the creation of scholarly works and preserve “quality”. We discussed expertise in our group in the second class. Expertise, experience and dedication are integral factors in creating important work. Her models might incorporate the “Wisdom of the Crowd” but I’m not sure anyone can really argue that that gets us where we want to go.
I guess the potential strength of reviewing the reviewers is that contributors are rated by peers both for the work they put up for review and the reviews they make of others’ work — and thus they gain power within the community by contributing well to the community. Again I suppose, ideally, the power dynamics, (always present and necessary), by virtual of being communally conditioned, should then always be directed towards improving the community itself.
That all said, I’m very suspicious of the capacity of online communities to approach this ideal. When trying to treat social problems through technological architecture, I think one always runs up against barriers. The big question is whether human behavior can be changed enough by design to change human “nature”. To be more concrete, Fitzpatrick proposes gating full access to community facilities behind reputation ratings — which leads to situations like you describe around “The Golden Notebook”. Somehow this is better than the noise of a non-gated process, yes, because of the aforementioned questions of expertise and quality. But, besides it being a process that’s publicly viewable, it slowly begins to resemble the status quo.
While I do think that openness, at least, is an improvement, I’m not particularly sanguine about the possibility that open publishing will be adopted any time soon in such a critical mass that it can have a huge impact on the things that Fitzpatrick is concerned with, namely, shifting values in the tenure process to something that more accurately reflects what is important in contemporary scholarship (sorry, I know I could say that better) and showing the public at large that scholarship is important to the world. As she writes in her intro:
“The crisis we face, after all, does not stop with the book, but rather extends to the valuation of the humanities within the university, and of institutions of higher education within the culture at large. We tend to dismiss the public disdain for our work and our institutions as a manifestation of the ingrained anti-intellectualism in U.S. culture, and perhaps understandably so, but until we take responsibility for our culture’s sense of our irrelevance, we cannot hope to convince it otherwise….” (PO, intro)
Once again, thanks for bringing this up.
I liked Fitzpatrick’s takedown of the existing peer review structure, but I am deeply suspicious of her proposed reputation-based community as a solution.
Reviewing the reviews seems more sensical to me than reviewing the reviewers. This is how models such as Amazon ratings, Wikipedia, Reddit comments, and Stack Overflow solutions work. Reviewing the reviewers can turn quickly into reviewing more than the quality of their output.
How open is it really? As existing reputations get bigger in a community they dominate more share of voice, making it harder for new voices to be heard unless amplified by someone with a big reputation. What indication do we have that people will amplify based on merit in an online space, rather than letting their own internal biases play out as is currently a problem in traditional methods?
Once a person has a big reputation, we have a familiar moral hazard. They are now vested in the system status quo. Change poses the risk of loss of reputation/power.
I also think she is too blasé in her suggestion of algorithms determining reputation score. As the fantastic book Weapons of Math Destruction taught me, opaque algorithms with social power and insufficient feedback mechanisms are fertile fields for injustice.
These are just a few of my misgivings about Fitzgerald’s proposed model. Overall, her solution seems a little half-baked for being in the relatively non-dynamic form of a book. I think this is great fodder for a series of blog posts and online review where more voices could jump in to flag and discuss these issues. This is actually a pretty good, if inadvertent, demonstration of the value of community pre-review before publishing.
Disclaimer: I am about 2/3 of the way through the book, finishing tonight.
*Correction to last paragraph: Fitzpatrick, not Fitzgerald
** Highly recommend Weapons of Math Destruction, by Cathy O’Neil. https://weaponsofmathdestructionbook.com/
Short answer: no.
First, as you stated, it’s not all that fleshed out.
Second, if social media has taught us anything, it’s that people can be sucked into a quagmire of negativity extremely quickly, whether they mean to or not.
Third, we have the nature of Academia. Academia is littered with fierce, angry rivalries between either individual scholars or proponents of opposing theories. Frequently, Academia looks like a particularly unruly fandom. Add that to the how people behave in social media, and It wouldn’t take much for this crowd sourced peer review to turn into a flame war (to use an old school term) or worse.
Now, we could make the argument that the easiest way to handle the rivalries and hatreds that are in Academia would be to have moderators to watch over this behavior. That job is the very definition of the word “thankless”.
What organization hosts this crowdsourced reviewing process? That organization will have a great deal of power over what gets prioritized for review.
I could continue asking logistical questions.
I am not saying peer review is perfect, far from it. We’ve all read those journal articles that were unholy messes which should not have made it through any kind of review process. I’m just not sure how this is an improvement.
I like your comparison to a fandom here! Fandoms can be generative, productive communities in which strong relationships are forged…. or they can be terrible morasses of backbiting and not-wholly-explicable conflict — sometimes in the same community.
I’m tempted to say that the size and the engagement level of the community is key. I don’t know if it’s true, but it’s an appealing idea. A journal is already a kind of community; it has a group of readers and writers who engage with it frequently and substantively, and others who are more peripheral. This is the strength of the scholarly journal as a format; it allows writers to reach the readers of that particular journal. However, journals vary widely in both their reach and the investment of their audience.
Fitzpatrick uses the example of open review in Science, which failed because there wasn’t really a built-in audience. She gives some reasons for that, but I think it’s also worth considering that Science is a very large journal covering a variety of topics. So, two problems: there’s no guarantee that any given article will be of particular interest to any given reader, but also, if something DID take off, people wouldn’t already be familiar enough with each other to know whether to take someone seriously or roll their eyes at an already well-known opponent. I suspect that things might go differently at a more specific journal. (To keep with the fandom theme — I wonder how that experiment would go at Transformative Works and Cultures!)
Which doesn’t REALLY solve the problem you point out where people have very strong commitments to their particular theories and then proceed to behave unprofessionally or abusively toward those who take a different perspective. But then again, it’s not really new, is it? I’m thinking of professional listservs (these are still really popular in librarianship, I don’t know about elsewhere), and this sort of thing definitely happens there. But #notalllistervs, I guess? Many of them are lovely. A few of them are bad (some notoriously so), usually because of one or two contributors who don’t know when to stop typing. Either a moderator steps in to stop the conversations that have gone wrong, or else other members of the listserv unsubscribe.
To the extent it works, this model is possible because a listserv usually consists of tens or perhaps hundreds of subscribers, not thousands or tens of thousands. The solutions Fitzpatrick proposes seem to have in mind a much larger scope. But I’m not sure that’s the kind of community that we ought to have in mind.
Re: smaller scope v. larger scope
If one of the points of DH is to bring people from various other disciplines together, keeping things at a small or fragmented scale defeats that purpose, doesn’t it? I mean, if that’s the direction we direction we go in, aren’t we admitting that fragmentary approaches work better?
Further, small forums could very easily be dominated by a small group of people, which could shut out scholarship from those they don’t like/don’t agree with/don’t value. This is just peer review 2.0.
Re: Power dynamics
Also, if it’s a forum or a listserv, the moderator has tremendous power. We’re trading in one kind of power structure for another, and I’m not sure it’s an improvement.
Yeah, I actually think the idea of building community, which Fitzpatrick emphasizes at several points, inherently puts some limits on the breadth of who can be included, because a community, inherently, isn’t everybody. I’m not sure she recognizes this in the book? But it’s part of what community is.
It’s certainly possible to build more inclusive communities, or communities that have a greater range of perspective, or communities with more flexible boundaries, but a community is never everybody.
On the other hand, the more small communities there are, the less the power of one specific moderator really matters (that is, it might matter a great deal within that community, but the moderator’s power is limited to one space among many. (Further problems could of course arise from that, but that’s true of every solution to every problem!)
In any case, it’s a much smaller change, yes, but I do actually think it’s an improvement because it comes with a greater degree of accountability.