School of Geographical sciences, University of Bristol
Contact email: lenka.hasova@bristol.ac.uk
Link to data (within this repository): Reproducibility survey (Responses).csv
This is an informal sheet describing the responses to R&R survey. The purpose of this is to
- Generate discussion within the Department
- Generate discussion within a separate research group
- Inform on the current state of R&R
- Identify the biggest blocker of R&R within the department
- Report with the results of the survey a. Report.pptx b. Reoprt.pdf c. README.md
- Data - all the responses from the survey in csv file a. Reproducibility survey (Responses).csv
- Python Notebook with code to create the figures a. reproducibility.ipynb
- Each Figure as .png
We have collected 70 responses within the school of Geographical sciences. Almost 36% of respondents identify as Human Geographers and 64% as Physical Geographers.
Almost 25% of respondents do not identify themselves with a particular research group or don't want to disclose this information. The rest of the respondents identify predominantly with Glaciology Centre (20%) or Research Initiative for the Dynamic Global Environment (23%).
Q1 - According to you, how important is the Reproducibility and Replicability of Geographical research to ...
In general, respondents feel that this topic is very important to their research as well as to our department and the future of geographical analysis.
However, there is a clear distinction between Human geographers, who feel that R&R is not as important or moderately important for their own research, and Physical geographers, who feel that R&R is much more important for their own research.
| Human Geographers |
|---|
 |
| Physical Geographers |
|---|
 |
Here we can clearly see the differences by each Research Group. While Quantitative Spatial Science and Research Initiative for the Dynamic Global Environment provided more stable answer, where all the domains are important for them, the Historical and Cultural Research Group is the most varied in its responses. For them, the R&R is more important for our department than for their own research.
| To Your Own research |
|---|
 |
| To Our Department |
|---|
 |
| To future of Geographical Analysis |
|---|
 |
Most of the respondents (91%) consider the R&R in their research to some, moderate or great extent, while only 8.5% of respondents is less inclined to R&R in their own research.
We can observe a clear distinction between the Human and Physical geographers. All of the physical geographers consider R&R in their research at least to some extent, while this cannot be said about Human geographers.
This correlates also with the responses grouped by a specific Research group. The respondents identifying with Political Economy and Historical and Cultural Research Group are those who consider R&R the least or not at all.
Although there is a big proportion of people who publish their codes, data or preprints, There are quite a few people (34% of all respondents) who does not include open-source practices in their research.
This proportion of respondents that don't include the open-source practices in their research is visibly greater in Human Geography. The general open source publishing practices are much more present within the Physical Geographers.
| Human Geographers |
|---|
 |
| Physical Geographers |
|---|
 |
In the category 'Other' can be found those respondents who haven't published anything yet, publish with a different body than stated (government,..), publish on request or those for whom publishing is not relevant
Q4 - Please tell us what restricts you from publishing your code/data/reproducible examples of your research?
The respondents identified, in general, lack of skills, tools or knowledge and not having enough time as the biggest blockers when it comes to publishing their research.
For Human Geographers, however, is also a major problem sensitive data or research. In comparison, Physical Geographers identified 'No time' as a stronger factor than any other.
| Human Geographers |
|---|
 |
| Physical Geographers |
|---|
 |
In the category 'Other' can be found those respondents for whom publishing is irrelevant or those who are restricted by copyright or temporal scale of the research.
22 respondents commented on this issue. From those, roughly 72% is expressing the importance of R&R and would be happy to attend any courses, seminar or department-wide discussions aiming to increase our knowledge of this topic. Nevertheless, Roughly 20% of respondents that provided comments feel that R&R is not relevant to their research, there is no need for it or it is simply not suitable for some research designs.
Example of some of the comments:
Training on how to do this and what the different options are is needed
R&R is essential, but seems low on the list of many researchers' and institutions' priorities. There are still many analyses that lack a standardised, scientifically-verified best practice procedure, meaning that it can be difficult to even compare results from different field seasons.\n\nImproving the conditions of the laboratories (e.g. for the instruments to work more reliably) would help to flatten the baseline of analyses, making it easier to trust in technical replicates. It would also make it more logistically feasible to run multiple analyses and hence to verify conclusions through independent methods. Similarly, buying some communal field equipment (e.g. peristaltic pumps) would make it easier to increase sample size in the field, making it easier to trust in-field replicates.
Never thought about it in these terms before now and don't deem it of significance to what I research or teach. I can see how it might be relevant to some.
I would be very interested in attending R&R workshops. I don't have much experience of how things are done at Bristol, but I have benefited in the past from established data management infrastructure/staff. This has massively helped in collating and publishing data and the organisation of samples.
Would be good to have a training course. So far everything has been very learning by doing and trial and error with R&R.
I think it is great you are collecting responses on this! A seminar on R&R or perhaps a mini conference (say a half day with lots of short pop up talks and opportunity for breakout groups) on some good practices of R&R across the department would be helpful and inspiring i.e. how are we using github or data archives, who is good to talk to if I am interested in publishing my data? I don't think we place a lot of emphasis on it in the department currently so raising awareness would be a helpful first step.
I have to say it's the first I've come across the terminology. It seems important, even if greased towards more scientific modes of research. I think human geographers could have a fruitful discussion about what it means for them.
Kedron, P., Frazier, A.E., Trgovac, A.B., Nelson, T. and Fotheringham, A.S. (2021), Reproducibility and Replicability in Geographical Analysis. Geogr. Anal., 53: 135-147. Availiable here
Michael F. Goodchild, A. Stewart Fotheringham, Peter Kedron & Wenwen Li (2020), Forum on Reproducibility and Replicability in Geography, Annals of the American Association of Geographers, DOI: 10.1080/24694452.2020.1806030, Availiable here
Gelman, A. and Eric Loken. “The garden of forking paths : Why multiple comparisons can be a problem , even when there is no “ fishing expedition ” or “ p-hacking ” and the research hypothesis was posited ahead of time ∗.” (2019). Availiable here
Huntington-Klein, Nick and Arenas, Andreu and Beam, Emily and Bertoni, Marco and Bloem, Jeffrey and Burli, Pralhad H and Chen, Naibin and Grieco, Paul L.E. and Ekpe, Godwin and Pugatch, Todd and Saavedra, Martin Hugo and Stopnitzky, Yaniv, The Influence of Hidden Researcher Decisions in Applied Microeconomics. IZA Discussion Paper No. 13233, Available at SSRN