Appropriateness of primary materials? Thoughts on peer review

It's been a while, but I am finally (sort of) getting back to addressing some feedback that my colleague and I got on an article we are working on with regard to MOOCs.  My colleague, Zaharias, thought it would be a great idea to sit down and make an (initial) typology of issues around the development of MOOCs. The abstract was accepted for a special issue of a journal, but our final version was not.  This I found a bit odd, but I've taken the peer reviewer's comments to heart, and I am thinking of ways of addressing them.  Some, in my mind, are valid.  I "live" in my head, so for me terms like cMOOC, xMOOC, FSLT12, CCK11, MobiMOOC, connectivism and so on are second nature.  I think to some audiences there might be some need to explain what this alphabet soup means.  Other comments that I also took to heart revolve around typos that just sneaked under our radars. After 8200 words, and multiple readings, who could blame us ;-) As a newly minted editor, though, for the CIEE journal I realize how annoying this might be for peer reviewers and editors :)

Anyway, there was some feedback, though, that I take issue with. In order to do this initial typology our literature review spanned over 100 sources ranging from academic, peer reviewed articles, to conference presentations, to news items from Inside Higher Ed and the Chronicle, as well as blog posts from respected leaders in the field, such as Stephen Downes, George Siemens, and people who've run their own MOOCs.  We opted to only use academic literature that was free on the web in an open access format.  I do have access to library research databases, however I find it very odd, philosophically, to study open education and to rely on so many closed access journals in order to do my research.  I could definitely see the peer reviewer's point that we should go after more (closed access) academic articles  to increase the pool of our data, however I take serious issue with the following advice that we got:
The collected materials in this manuscript are not high quality document owing that most of them are informal documents. The authors should collect academic high quality papers to survey and analyze.
Just because something is not in an academic, peer-reviewed, journal doesn't mean that it does not have some sort of quality, and it does not deserve attention.  There have been quite a few blog posts out there that were pre-cursors to academic articles that someone in an academic article down the road deemed to have sufficient quality to publish. Furthermore, if someone has published in peer reviewed articles, and regularly gives conference presentations and keynotes, don't their blog posts carry more weight than mine or someone who's a novice in the subject?  And, last but not least, what happens in professional publications such as the Chronicle and IHE never makes it to academic publishing (see for example the issues with FOE MOOC and #massiveteaching). These things need accounting and addressing, and if they never make it to publication, for one reason or another, they become invisible to educational research. This seems like a big failure to me.

Another thing that piqued my interest is the idea of laying down the groundwork.  I understand that there needs to be some level of definition of terms in academic research, and when I eventually do a dissertation I expect that I will be defining a lot of terms before I get to the heart of my own experiments.  That said, in an academic article, which usually has a limit on the amount of words you can put in it, how much space do you use for definition of terms, and how much do you devote to the heart of the matter? How reasonable is it to expect that, in the age of Google and in light of your providing of a full reference list, that you don't necessarily need to briefly describe terms like xMOOC, cMOOC, CCK, and PLENK? At the end of the day, I am not averse to defining such terms (in addition to any existing citations) but I do wonder how this is counted against the authors of papers when there are defined word limits for articles?

Your thoughts? Any other peculiarities of peer review you have a bone to pick with? :)

Image from: http://www.phdcomics.com/comics/archive/phd081508s.gif

Comments

That peer reviewer comment may mean that you neglected to cite the reviewer's work. And if the reviewer does not recognize the work of authorities in the field as being of academic high quality, then perhaps the reviewer's qualifications should be called into question.

The extent to which terms need to be defined, I think, depends on the expected audience. Amongst people who are engaged in research around MOOCs, it is reasonable to expect c- and x-MOOCs to be common knowledge. CCK and PLENK might need a brief description for those without enough background in the history. I wonder if citing and explaining things that are common knowledge doesn't diminish author credibility.

My two cents anyway.
LOL, Paul, that's a pretty cynical view (re: you didn't cite me, so it's junk) :-) I don't doubt that there are people like that out there though :)
As a non-academic, but someone who might find himself reading this work, I would need a fair bit of googling to keep up with this. I am almost certainly not the key demographic for this article, though. However, that brings up two thoughts:

1. Should this type of writing be more accessible to other professionals who are not perhaps completely involved in the topic, without a jargon dictionary?

2. Would this article be digital? Couldn't some excellent hypertextual footnoting help here? I mean… we have the technology…
Brian you bring up a valid point: accessibility. Most academic articles are written with a specific language type in mind (the academic language), so the audience tends to be mostly academics. That said, there are many professionals out there that would benefit a lot from updates in the research literature but don't have access to it. Access here I would define in two ways:


"Physical" access in that many articles are behind paywalls and are expensive to access (some are up to $30 per article! Insane!)


"cognitive" access (for lack of a better word) because readers have not necessarily been apprenticed into the lingo of the discipline.


Part of the problem, in my opinion, is the issue of tenure and promotion in higher education. A professor needs to publish in high impact journals (which are exclusionary, in many senses of the word) in order to have that cache, and get to keep their job. Of course, this has the effect of keeping people out. Pretty silly if you ask me. In order for the system to change, however, you need people on the inside willing to take a risk of not getting tenure by experimenting with open publishing (open in many different ways). Of course if one person does it he/she is a target. If many people do it, there is power there! :)

Popular posts from this blog

Academic Facepalm (evaluation edition)

Discussion forums in MOOCs are counter-productive...well, sort of...

Latour: Third Source of Uncertainty - Objects have agency too!