vol. 14, no. 1 (2019) Theory and Research (peer-reviewed) DOI: http://dx.doi.org/10.21083/partnership.v14i1.4764 CC BY-NC-ND 4.0 Hype or Real Threat: The Extent of Predatory Journals in Student Bibliographies H. Rainer Schira Acting University Librarian Brandon University schirar@brandonu.ca https://orcid.org/0000-0003-3915-8262 Chris Hurst Systems Librarian Brandon University hurst@brandonu.ca https://orcid.org/0000-0003-1759-5680 Abstract Predatory publishing has risen with the development of open access publishing. This study examines how many potential predatory journals were used by Brandon University students by analyzing their bibliographies. In total, 245 bibliographies including 2,359 citations were analyzed. Of the 1,485 citations to journals in these citations, five were found to cite journals on Beall’s List of Predatory Journals and Publishers. The probable sources of these journals in the students’ bibliographies were examined. Keywords predatory publishing; open access publishing; academic libraries, bibliographic analysis Introduction This study was an attempt to quantify the presence of predatory journals in university students’ bibliographies. Predatory open access journal publishing, often shortened to Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) predatory publishing, has risen with the development of open access publishing. The term was coined by Jeffrey Beall (2010), who subsequently created a list of journals and publishers that met his criteria for being predatory. In a comparative review of nine open access journal publishers, Beall (2010) used the term “predatory publisher” to describe publishers whose mission was to exploit the author-pays open access model for their own profit (p. 15). These publishers spam email lists with calls for papers and invitations to serve on nominal editorial boards. They provide little or no peer review, and they have no plan for digital preservation (Beall, 2010). Our concern here is this particular sense of predatory publishing: open access predatory publishing. These predators have two types of victims. The first kind is the naïve scholar with research that could be published in a non-predatory journal but who is inveigled into paying to publish in a predatory journal. The second victim is scholarly enterprise itself. Because predatory publishers will publish an article if someone pays the fee, articles that would not normally meet the test of peer review are available to anyone with a web browser. While articles in predatory journals may not appear in selective indexes like Web of Science or Scopus, they can certainly be found through Google, the search tool of choice for many university students. Therefore, it is worth investigating whether articles from predatory publishers have appeared in students’ bibliographies. There has been concern in the library and scholarly communications communities about the impact of predatory publishing. The lack of peer review in predatory publications makes the publication less reliable. Predatory publishers therefore erode the quality of scholarly communication and may lead to the use of misinformation, undermining the usefulness of published information in general. Students are beginning to learn about their chosen field of study. Using potentially unchecked and possibly inferior information published in predatory journals can harm their education. There have been studies of the presence of predatory journals in subject indexes and aggregated collections of open access resources (Nelson & Huffman, 2015; SomozaFernández, Rodríguez-Gairín, & Urbano, 2016), but no one has investigated if students are finding and using those articles in their assignments. This is a study of students’ bibliographies to see if citations to predatory journals are found in those bibliographies. We are also curious to see if the percentage of predatory journal articles in student bibliographies is higher than the percentage of predatory journals in subject indexes. Literature Review Scholarly publishing and peer review started in 17th century Europe when the Royal Society of London and the Académie Royal des Sciences of Paris were chartered by the English and French kings respectively. Experimental reports and findings were published by the in-house journals of both societies under the watchful eyes of the editors who acted as both the kings’ licensors and the judges of the research’s worth (Biagoli, 2002; Lee, Sugimoto, Zhang, & Cronin, 2013). Through the 18th and 19th 2 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) centuries, the modern procedures of peer review, where submissions to the society were sent for vetting by a select group of a society’s members, became codified (Speier, 2002). Scholarly publishing was largely the purview of academic societies until after the Second World War. Led by the entrepreneur Robert Maxwell, Pergamon Publishing epitomized the commercial publishers that profited from the work of scholars. As Maxwell himself observed, one journal could not be swapped for another, and so publishing journals was “a perpetual financing machine” (Buranyi, 2017, para 45). Libraries became a captive market for scholarly journal subscriptions during a time when library budgets were often stagnant or being cut. Between 1987 and 2003, the Consumer Price Index (CPI) increased 67.79% while the average serial unit cost for the Association of Research Libraries (ARL) increased 215.34%. The budget for journals— in particular Science, Technology, and Medical (STM) journals—crowded out the budget for other library expenses. This was dubbed the “Serials Crisis” (Greco, Wharton, Estelami, & Jones, 2006, p. 156). Although some not-for-profit academic publishers raised their subscription prices aggressively, the largest increases in subscription prices came from the commercial publishers, epitomized by Elsevier, which merged with Pergamon in 1991 (Buranyi, 2017). Resentment of scholarly publishers was further stoked by the publishers’ insistence that any scholar publishing in a scholarly journal sign over the copyright of the article to the publisher. These resentments came to a boiling point in the Academic Spring of 2012, when a call for a boycott of Elsevier’s journals by a mathematician led to an online petition and a general stand against all for-profit academic publishing. The “Academic Spring” has not maintained its momentum, but there is a lingering sentiment among some scholars that for-profit publishers are immoral capitalists exploiting academic labour (Brienza, 2012). In this broad sense, some scholars would consider all for-profit publishers to be predatory publishers. The most substantial reaction to the Serials Crisis was the Open Access Movement. The first freely available electronic journals began appearing in the mid-1990s. Walt Crawford (2002), a senior analyst at the Research Libraries Group and a prolific writer on open access, described 86 journals available as free refereed scholarly journals in 1995. In 2002, the Budapest Open Access Initiative would formalize the movement with the goal of creating “world-wide electronic distribution of the peer-reviewed journal literature and completely free and unrestricted access to it by all scientists, scholars, students and other curious minds” (Budapest Open Access Initiative, 2002, para 1). Many groups responded to the initiative, but none more than the Scholarly Publishing and Academic Resources Coalition (SPARC), a coalition of more than 200 research institutions that spearheaded a number of education and advocacy campaigns to promote open access in the United States and Europe (Savenije, 2004; Suber, 2006). These freely available, digital, online journals often published with less restrictive copyright and licensing terms were not the only means of realizing this goal, but they were one of the main tools. 3 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) In 2002, approximately 22% of the 775 Newly Created journals, which is a term from Ulrich’s Periodical Directory, were open access journals; in 2011, 35% of the 1,499 Newly Created journals were open access journals, although there was a falloff in 2012 and 2013 (Gu & Blackmore, 2016). Another study estimated that 19.4% of the approximately 1,346,000 scientific papers published in 2006 were available in open access form (Björk, Roos, & Lauri, 2009). Clearly, open access journals are a significant part of scholarly publishing. Some open access journals can publish by virtue of grants or support from a host institution. Other open access journals charge processing fees to the scholars for publishing the articles. These fees are not unique to open access publishing: many smaller non-open access journals have used page charges or similar fees to publish their journals. However, it soon became evident that some of the newer open access publishers were far more interested in collecting fees than they were in promoting scholarship. Beall (2010, 2012, 2013a, 2013b, 2017) is one of the most influential figures in the study of predatory publishers. He has written much on the subject, and many still use his list of potential predatory publishers as a guideline even though it was taken down in January 2017. Since he coined the term in 2010, the definition of predatory publishers and predatory journals has grown. Beall (2012b) expanded on this definition by stating that predatory publishers use deception to appear legitimate, entrapping researchers into submitting their work and then charging them to publish it. Some prey especially on junior faculty and graduate students, bombarding them with spam e-mail solicitations. Harvesting data from legitimate publishers’ websites, they send personalized spam, enticing researchers by praising their earlier works and inviting them to submit a new manuscript. Many of these bogus publishers falsely claim to enforce stringent peer review, but it appears they routinely publish article manuscripts upon receipt of the author fee. Some have added names to their editorial boards without first getting permission from the scientists they list, among other unethical practices. (p. 23) Beall (2017a) added, “many claim to be scholarly institutes, scholarly societies, or associations, when they're really just a sole proprietor running multiple journals from a dwelling. Some copy the titles of existing journals or create titles very close to those of respected journals” (p. 55). Tin et al. (2014) provide an excellent summary of the criticisms associated with predatory publishing, adding that predatory publishers appoint fake academics to editorial boards, mimicking the name or website style of established journals, combine two or more fields that are usually not combined, lift the aims and scopes of established journals, integrate words such as “global” in front of well known titles, and pretend to be indexed and abstracted in library databases services (p. 73). 4 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) To help with the identification of predatory journals and publishers, Beall created a list of potential predatory publishers and standalone journals in 2008. He took the list down in January 2017, but many found it so useful that it was quickly archived. No other such list has been created since, so many librarians and scholars still refer to it for guidance. Following the lead of Nelson and Huffman (2015) and Somoza-Fernández, RodríguezGairín, and Urbano (2016), we decided to use it as the reference point for predatory journals as well, making it possible for us to compare our data with theirs. This list has many proponents (Butler, 2013; Tin et al., 2014) but has also incited much criticism (Anderson, 2015; Berger & Cirasella, 2015; Crawford, 2014). As a response to his critics, Beall published a list of criteria for determining predatory open access publishers in 2012. Again, many embraced these criteria, while Olivarez, Bales, Sare, and vanDuinkerken (2018) have said that “they recommend discretion because the criteria used are so general that an evaluator might label a scholarly journal as ‘predatory’ when it is not” (p. 53) and that “even traditional model journals might classify as predatory” (p. 61) using these criteria. The authors set out to “assess the subjectivity of Beall’s Criteria for Determining Predatory Open-Access Publishers, the same criteria by which he determines his lists” (p. 55). They conclude that the list is very subjective and that many proven non-predatory journals could be considered predatory under these criteria. They acknowledge that there is debate about inclusion of content on the list and worry that a reader of the list might automatically assume that the list is 100% correct, rather than a beginning point for discussion. The list is certainly not perfect, but it can still serve as a basis to identify potential predatory journals. After Beall coined the description “predatory journals” much has been written about the existence of predatory journals, what makes a journal predatory, how to tell if a journal is predatory, and how to avoid publishing in them. Studies were also done to determine if any of these predatory journals have made their way into library databases. Nelson and Huffman (2015) examined “the extent to which these publishers and their journals, as identified by Beall, are present in select academic databases and in a directory commonly utilized in libraries” (p. 170). They looked at Beall’s predatory publishers list and determined how many of those publishers and their journals were present in a number of databases. They found two publishers (out of 3,758, or 0.05%) and six journals (out of 16,555, or 0.04%) in Gale Academic OneFile, six publishers (out of 3,801, or 0.16%) and 55 journals (out of 13,787, or 0.40%) in EbscoHost Academic Search Complete, 41 publishers (out of 5,695, or 0.72%) and 299 journals (out of 21,174, or 1.41%) in Proquest Central, and 123 publishers (out of 5,456, or 2.25%) and 812 journals (out of 9,709, or 8.36%) in the Directory of Open Access Journals (DOAJ). They concluded that they “have uncovered a significant presence of predatory content in library subscription databases, which may already have had a deleterious effect on student research” (p. 190). Somoza-Fernández et al. (2016) take a different approach. They only used Beall’s list of standalone journals (rather than including the list of publishers as well) to “verify to what extent the journals considered predatory have been selected for appearing in more than 100 bibliographic databases” (p. 732). The list of potential predatory stand 5 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) alone journals totaled 944. Inclusion rates for these journals in commercial databases range from 1.48% (14 journals out of approximately 9,450) in EbscoHost Academic Search Premier, to 0.95% (39 out of 18,495) in the Web of Science, to 0.25% (56 out of 22,409) in Scopus. These results indicate that “there is no significant presence of predatory journals in bibliographic databases” (p. 736). The authors also found that 143 journals on Beall’s list of standalone journals were present in DOAJ. The authors, however, do not provide the percentage of predatory journals in DOAJ. They also do not state how many journals were in DOAJ at the time of the study, nor exactly when the study was done. It is therefore difficult to compare their findings to the previous study (Nelson & Huffman, 2015), other than to say that 20 more journals were included in 2016 than in 2015. Neither of the studies includes information about Google Scholar and Google. Kommissarov and Murray (2016) report that “a large fraction of students report using popular search engines for finding information for projects involving library research” (p. 424). Their research shows that 39% of the students surveyed started their research using Google and 15% used Google Scholar (compared to 42% who started in the library catalogue, library website, or electronic databases). Presumably, a Google search has the potential to find predatory journals. However, no study yet exists to identify how likely it is for someone to find predatory journals that way. What this research sets out to do is to see if predatory articles appear at all in students’ bibliographies, no matter where they might have come from. Methodology Study population and sampling Brandon University is a primarily undergraduate university based in Brandon, Manitoba, with a satellite campus in Winnipeg, Manitoba. The university has five faculties or schools (Arts, Science, Health Studies, Education, and Music) with graduate programs in Music, Education, Psychiatric Nursing, Rural Development, and Environmental & Life Sciences. The university has roughly 2,288 full-time and 847 part-time undergraduate students, 224 full-time and 166 part-time graduate students, and 220 faculty members. Data collection In order to find out how many potentially predatory journals were used by Brandon University students in the Winter Semester (January–April) 2017, the authors of this study decided to analyze students’ bibliographies. Our first step was to get Brandon University Research Ethics Committee approval. We then contacted faculty at Brandon University through email and asked them to self-identify if they had an assignment due in their class that required the use of academic, scholarly, peer-reviewed research articles. We then gained their permission to come and speak to their students during class time. At that time, we described the project to students. We told them how we would collect their bibliographies from their professors and asked for permission to get copies of their research assignment bibliographies. Students were told that they were 6 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) under no obligation to participate, and that participation would, in no way, be linked to their grades for that class. One professor (in two of her classes), however, decided on her own to award an extra 2% to each participants’ final grade. This incentive was not under our control, and no students in other classes received incentives of any kind. In describing the project to the students, we sought to give the students full knowledge of the project in plain language so that they would be able to give fully informed consent, as outlined in Article 3.2(b) of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS 2) (Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, & Social Sciences and Humanities Research Council of Canada, 2014, p. 29). This description included our research focus of predatory journals and a definition of predatory journals. This may have introduced a bias into the data as the students may have consciously or unconsciously tried to avoid using predatory journals. We feel that withholding or not fully explaining our research focus would not allow the students to make a fully informed decision regarding participation in our project. Perhaps a different data collection design would allow research to proceeed ethically without encountering this bias. To make it as easy as possible for the professor to provide us with the bibliographies, we let them choose either to let us make photocopies of the bibliographies only or to forward an electronic copy of the entire student’s paper to us after it was handed in but before it was graded. After receiving an electronic copy, we deleted everything other than the bibliography. We received most of the bibliographies in electronic format. We did not track how well a student was graded on the paper or what GPA they had overall. In the cases where we received bibliographies from group assignments, we only retained those for which we had permission from every student involved. Data analysis Each bibliography was tagged to keep track of which student and class it was from. The authors of this study transcribed all information from the bibliographies to an Excel spreadsheet and tagged it by type (article, book, book chapter, news, website, government document, etc.). Although the general definition of a website is any site on the internet, in our tagging, it had a much more specialized meaning. An entry was tagged a “website” if it was a site on the internet, there was a URL in the bibliographic entry, and the URL did not lead to a scholarly journal or book. We only analyzed those that were tagged as articles. When a citation had an abbreviated title, we replaced it with the long format. When a citation was wrong (e.g., the student used the journal publisher information rather than the journal title), we corrected it. We also corrected all spelling mistakes to make it more likely to find a match for the article. A list of journal titles for matching the journal tiles in the bibliographies was created by linking together the lists from publishers, aggregators (like JSTOR), and services (like DOAJ). This list is referred to hereafter as the publishers list. This list was standardized by removing the leading articles. Each entry in the publishers list consisted of the journal title; the journal’s print ISSN; the journal’s electronic ISSN; the starting date of the journal at the publisher, aggregator, or service; the end date (if applicable, because 7 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) of the journal ending, the journal moving to another publisher, or the aggregator no longer having access to the journal after a date) of the journal at the publisher, aggregator, or service; the publisher, aggregator, or service; the collection of the publisher, aggregator, or service where the journal can be found; and a descriptor of the publisher, aggregator, or service. We used four descriptors: “large” for large publishers/aggregators, “small” for small publishers, “open access” for open access journals, and “free access” for journals that are freely available but have not declared themselves to be open access. Piwowar et al. (2017) have called this last category of articles bronze open access. There are no formal criteria by which a publisher or aggregator is considered either large or small. As such, there are intermediate publishers or aggregators who could arguably be classified with either descriptor. However, the data will demonstrate that the largest sources for journal articles are indeed large publishers and aggregators. For each entry in the students’ bibliographies, the journal titles were matched against the publishers list. If there was a unique match on the title and the year of publication of the article and the bibliography fell between the starting and end dates of the publisher, aggregator, or service, then the journal was considered to be matched. If the year of publication was outside the starting and end dates of the journal in the publishers list, the entry was put aside for further checking. If the entry was matched to more than one entry in the publishers list, then the year of publication was checked against the starting and end dates of the entries in the publishers list to determine the most likely match. The general rule was that articles with a recent publication year were more likely to be accessed from a publisher’s site and hence matched to the publisher’s entry while articles with an older publication year were more likely to be accessed from an aggregator’s site and hence matched to an aggregator’s entry. If the year of publication was outside the starting and end dates of all the matching journals in the publishers list, the entry was put aside for further checking. If the entry was not matched by title to the publishers list or the year of publication of the article was outside the start and end dates of any title match, the title from the bibliography was googled. If the result of the Google search indicated that there should have been a match (usually because the journal is published by a major publisher in the publishers list), the publishers list was searched by keyword or ISSN to see if the journal was present with a slightly varying title. If this was the case, either the title from the bibliography or the title from the publishers list was altered so there would be a uniform tile between the two lists and consequently a title match search. If the result of the Google search indicated that there was no match, then the publisher of the journal was recorded and a descriptor for the publisher was assigned based on available information, primarily the journal publisher’s website. Each of the publishers or journal titles where the year of publication of the article was after 1995 was checked against Beall’s List. (Articles published in or before 1995 predate the current conception of predatory publishing.) If either the publisher or the journal was found on Beall’s List, then the descriptor was changed to “predatory.” 8 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) Results We gathered 245 bibliographies with 2,359 citations from 19 courses. The breakdown of what kind of material was cited can be found in Table 1. Table 1 Kind and Number of Material Cited in Student Bibliographies Kind of Material Cited Number of Citations Average Publication Year Book 368 1999 Journal 1,485 2008 Website 420 2014 Other 86 1998 Total 2,359 2008 The other category includes citations of newspaper articles, theses, government documents, conference papers, television episodes, and personal communications. These were primarily citations of online sources but included some print. The website category refers to sites on the internet that are not scholarly and did not fit into the other category (e.g., tourism sites, organizational websites, blogs). The distribution of types of publishers for the journal citations can be found in Table 2. Table 2 Distribution by Publisher Type of Journal Citations Type of Publisher Count Percentage Free Access 105 7.07% Large 1,204 81.08% Open Access 93 6.26% Predatory 5 0.34% Small 78 5.25% Total 1,485 The Directory of Open Access Journals (DOAJ) is a service and not an aggregator in the formal sense of actually providing content the way an aggregator like EbscoHOST or ProQuest does. DOAJ does aggregate information about open access journals and is a distinct collection within the Alma/Primo discovery system, which allows our patrons to 9 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) search both our local collections and remote indexes and full text collections simultaneously. Therefore we have treated DOAJ as an aggregator. Because of Nelson and Huffman’s (2016) work, we carefully examined all the citations associated with the DOAJ data, but none of these citations were associated with the journals on Beall’s list. The HighWire Press (Free Journals) is a long standing source of free online full-text articles published with the assistance of HighWire Press. The site has never described itself as an open access site but essentially functions as one. It is a distinct collection within the Alma/Primo discovery system. PubMed Central Journals are part of a free archive of biomedical and life sciences journal literature at the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM). This is a distinct collection within the Alma/Primo discovery system. The Cell Press Free Archive is as collection of free issues made available by Cell Press. This is a distinct collection within the Alma/Primo discovery system. Table 3 Twenty Largest Publishers/Aggregators by Number of Citations Publisher Type of Publisher Count Percentage Elsevier Large 309 20.81% Taylor and Francis Large 197 13.27% John Wiley Large 172 11.58% Springer Publisher Large 134 9.02% JSTOR Large 105 7.07% Sage Publishing Large 102 6.87% DOAJ Open Access 90 6.06% HighWire Press (Free Journals) Free Access 83 5.59% Oxford University Press Large 38 2.56% Cambridge University Press Large 28 1.89% Nature Journals Online Large 23 1.55% Amercian Psychological Association Large 22 1.48% Project Muse Large 21 1.41% PubMed Central Free Access 16 1.08% Wolters Kluwer Large 8 0.54% Annual Reviews Large 7 0.47% NRC Research Press Large 7 0.47% 10 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) Publisher Type of Publisher Count Percentage American Medical Association Large 6 0.40% BioOne Large 6 0.40% Cell Press Free Archives Free Access 5 0.34% The five predatory journals found in the bibliographies came from five distinct students writing papers in three courses. Two of those courses were Biology courses (two students in a third-year course, and one in a fourth-year course), and one was a thirdyear Political Science course. The possible indexes where the five predatory journals could be found are outlined in Table 4. Table 4 Predatory Journals and Possible Discovery Origins PubMed Google Scholar International Journal of Biology Yes No Yes Pakistan Journal of Biological Sciences No Yes Yes International Journal of Business and Social Science Yes No Yes Open Journal of Political Science No Yes Yes International Journal of Agriculture and Forestry No No Yes Journal Primo (Free EJournals Collection) None of the five journals was found to be indexed in EbscoHOST Academic Search Premier, DOAJ, or Web of Science. All of the cited journals found on Beall’s list could be searched through Google Scholar. All journals were also found in Ulrichsweb, which is not an indexing service for articles, but the online equivalent of Ulrich’s Periodical Directory. As such, it contains detailed metadata on all journals. Brandon University uses Alma as an integrated library service and Primo as a discovery service. One of the prepackaged aggregated collections available within Alma is the Free E-Journals Collection, which contains over 20,000 freely available electronic journals. Two of the cited journals found on Beall’s list were also found in this collection; consequently, articles from these journals would have appeared in searches in Primo. 11 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) Two of the cited journals found on Beall’s list were found in the list of journals included in PubMed. Neither journal is indexed in Medline. Journals that are not indexed in Medline have an NlmId number that starts with “101,” which these two journals do. Both journals have their content deposited in PubMed Central. Discussion We analyzed 1,485 citations for journals and found that five of them were citations for journals found on Beall’s list. We have no data that indicates whether that is a high or low rate of occurence. To determine this, we would have to know two data points. The first is the percentage of articles from journals on Beall’s list in articles found on the web. The second is the percentage of articles from journals on Beall’s list in articles found in all students’ citations. Both data points cannot be feasibly obtained. On a more practical level, we would like to see this study reproduced elsewhere to verify if the 0.34% of citations for journals on Beall’s list is a typical percentage. If the 0.34% was a typical percentage, is this consistent with the widespread perception of predatory journals as an existential danger to the scholarly enterprise in general and open access publishing in particular? A different way of looking at this is that 2.4% (five out of 203) of free or open access journals cited in the student bibliographies were on Beall’s list. If a long term goal is to build trust in free and open access journals, then a finding that 2.4% of their citations are from potentially predatory journals may be disquieting. Are these journals actually predatory? The journals on Beall’s list cited in the students’ papers have an argument that since they were cited, they are fulfilling their role as dissemination vehicles for research. It is beyond the scope of this study to evaluate the quality of the papers cited or to judge whether they should have been published. Could these be examples of worthy scholarship published in unworthy journals because naïve scholars were enticed by the journals’ advertising? A larger question that needs to be answered is just what is a predatory journal? Beall’s list was one person’s attempt to list predatory journals on a binary basis: either a journal was predatory or it wasn’t predatory. A more nuanced classification scheme is needed. The attributes of predation should be described and journals should be graded on those attributes. Therefore journals can be slightly predatory, somewhat predatory, or very predatory. This isn’t to suggest that some journals aren’t obviously predatory and that most journals are obviously not predatory. It’s just to say that there is a spectrum of predatory behaviour for journals. Do these results indicate that undergraduates need to be instructed on predatory journals? Well known indexes like Web of Science did not index the journals on Beall’s list. However, two of the cited journals on Beall’s list were found in the university’s discovery service, and two different cited journals were found in PubMed. More importantly, all five cited journals on Beall’s list could be found on Google Scholar and consequently on Google. If Google is the first choice of many undergraduates, then they need to be taught to recognize predatory journals. 12 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) Limitations and Considerations for Future Studies Full disclosure of the purpose of the study to students could have influenced their behaviour. There was no other way to ethically collect this data within this research design. It is possible that students might have gone out of their way not to use predatory journals in their bibliographies. In general, we have no way of knowing if and how many predatory articles were found but rejected by students. We also have no sense of what students know about predatory publishing and if they feel the need to search out and reject such articles. We intend to do a follow-up study. We want to measure student intent in the research process. The form of these measurements is to be determined. We could add an interview component to the research to ask if the students considered avoiding articles from predatory journals in their research process. We encourage others to do similar studies to add to the data we have already collected. One of the professors offered a free 2% to be added to students’ final grade if they participated in the study. We had a high participation rate for that class, which might have influenced the overall results. Conclusion This study was an attempt to quantify the presence of predatory journals in university students’ bibliographies. This attempt was necessarily limited by the circumstances of the data collection: the data came from bibliographies gathered in one semester, reflecting the courses and the specific essay assignments. Gathering bibliographies from a different semester may have yielded different results. We welcome similar studies to add to these results. We examined 245 bibliographies totalling 2,359 citations. From the 1,485 journal articles citations, five could be matched to Beall’s list. Therefore, 0.34% of all journal articles, and 2.4% of open access aricles, were from potentially predatory publishers. So what could all this mean? If the presence of any articles in student bibliographies from predatory publishers is a concern, then this result is concerning. Google Scholar is likely to be a choice for student essay writers for the forseeable future, and all five predatory journals found in this study were indexed in Google Scholar. If 0.34% of all journal articles is roughly the percentage of journal articles in predatory journals compared to the number of articles overall in scholarly journals, then our results may reflect the general chance of choosing an article from a predatory journal from any randomly selected set of articles. Nelson and Huffman (2005) found that there was a 1.9% incidence of predatory journals in the aggregated databases and indexes that they examined. Somoza-Fernandez et al. (2016) found there was 0.23% incendence of predatory journals in the indexes they examined. Our results are close to these. This would be consistent with the chance hypothesis. 13 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) The problem with this hypothesis is that it discounts the intent of the student, who is looking for the best articles for the student’s paper. If students are taught about predatory journals as part of information literacy instruction, then students may choose not to use predatory journals. At least, they may be cognizant that predatory journals exist and should be avoided. For this to be the case they must have the knowledge to determine which journals are predatory and which are not. We did not include a mechanism in this study to investigate the intents and processes of students in their research. In further studies, we need to include a component where we ask students about their research processes. Our anecdotal evidence seems to suggest, however, that the average students does not know much about predatory journals. Like many studies, this one raises more questions than it answers. We welcome more quantitative research that examines the impact of predatory publishing. Disclosure Author Rainer Schira is a member of Partnership’s editorial board as a layout editor. In order to minimize any conflict of interest, this article was guest edited by Giovanna Badia of McGill University. This article went through the same double-blind peer review process as all other articles in this journal. References Anderson, R. (2015, May 11). Should we retire the term “predatory publishing”? [web log comment]. Beall, J. (2010). Predatory open-access scholarly publishers. The Charleston Advisor, 11(4), 10–17. Beall, J. (2012a). Predatory publishers are corrupting open access. Nature, 489(7415), 179. Beall, J. (2012b). Predatory publishing. The Scientist, 26(8), 22–23. Beall, J. (2013a). Medical publishing triage – chronicling predatory open access publishers. Annals of Medicine and Surgery, 2(2), 47–49. Beall, J. (2013b). Predatory publishing is just one of the consequences of gold open access. Learned Publishing, 26, 79–84. Beall, J. (2016). Beall’s list of predatory journals and publishers. 14 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) Beall, J. (2017a). Predatory journals, peer review, and education research. New Horizons in Adult Education & Human Resource Development, 29(1), 54–58. Beall, J. (2017b). What I learned from predatory publishers. Biochemia Medica, 27(2), 273–278. Berger, M., & Cirasella, J. (2015). Beyond Beall’s list: Better understanding of predatory publishers. College and Research Libraries News, 76, 132–135. Biagioli, M. (2002). From book censorship to academic peer review. Emergences: Journal for the Study of Media & Composite Cultures, 12(1), 11–45. Björk, B.-C., Roos, A, & Lauri, M. (2009). Scientific journal publishing: Yearly volume and open access availability. Information Research, 14(1). Brienza, C. (2012). Opening the wrong gate? The academic spring and scholarly publishing in the humanities and social sciences. Publishing Research Quarterly, 28(3), 159–171. Budapest Open Access Initiative. (2002, February 14). Read the Budapest open access initiative. Buranyi, S. (2017, June 27). Is the staggeringly profitable business of scientific publishing bad for science? The Guardian. Butler, D. (2013). Investigating journals: The dark side of publishing. Nature, 495(7442): 433–435. Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, & Social Sciences and Humanities Research Council of Canada. (2014). Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans [PDF file]. Crawford, W. (2002). Free electronic refereed journals: Getting past the arc of enthusiasm. Learned Publishing, 15(2), 117–123. Greco, A. N., Wharton, R. M., Estelami, H., & Jones, R. F. (2006). The state of scholarly journal publishing: 1981–2000. Journal of Scholarly Publishing, 37(3), 155–214. Gu, X., & Blackmore, K. L. (2016). Recent trends in academic journal growth. Scientometrics, 108(2), 693–716. Komissarov, S., & Murray, J. (2016). Factors that influence undergraduate informationseeking behavior and opportunities for student success. Journal of Academic Librarianship, 42(4), 423–429. Lee, C. J., Sugimoto, C. R., Zhang, G. & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17. 15 Partnership: The Canadian Journal of Library and Information Practice and Research, vol. 14, no. 1 (2019) Nelson, N. & Huffman, J. (2015). Predatory journals in library databases: How much should we worry? The Serials Librarian, 69(2), 169–192. Olivarez, J. D., Bales, S., Sare, L., & vanDuinkerken, W. (2018). Format aside: Applying Beall's criteria to assess the predatory nature of both OA and non-OA library and information science journals. College & Research Libraries, 79(1), 52–67. Piwowar, H., Priem, J., Larivière, V., Alperin, J. P., Matthias, L., Norlander, B., Farley, A., West, J., & Haustein, S. (2017). The state of OA: A large-scale analysis of the prevalence and impact of Open Access articles. PeerJ 6(e4375). Savenije, B. (2004). The SPARC initiative: A catalyst for change. LIBER Quarterly, 14(3–4), 412–439. Somoza-Fernández, M., Rodríguez-Gairín, J.-M., & Urbano, C. (2016). Presence of alleged predatory journals in bibliographic databases: Analysis of Beall's list. Profesional De La Información, 25(5), 730–737. Spier, R. (2002). The history of the peer-review process. Trends in Biotechnology, 20(8), 357–358. Suber, P. (2006). Open access in the United States. In N. Jacobs (Ed.), Open access: Key strategic, technical and economic aspects (pp. 149–160). Oxford, UK: Chandos Publishing. Tin, L., Ivana, B., Biljana, B., Bibić, L. I., Dragan, M., & Dušan, S. (2014). Predatory and fake scientific journals/publishers—A global outbreak with rising trend: A review. Geographica Pannonica, 18(3), 69–81. 16