1 00:00:05,239 --> 00:00:07,259 [music] 2 00:00:09,825 --> 00:00:11,845 [music] 3 00:00:15,360 --> 00:00:19,039 Uh welcome back to the 4 00:00:19,039 --> 00:00:22,080 research software engineering track and 5 00:00:22,080 --> 00:00:24,560 uh thanks to everybody who's stayed. Uh 6 00:00:24,560 --> 00:00:26,880 we have one more talk before lunch. Uh 7 00:00:26,880 --> 00:00:29,679 that's uh Stephanie Chong talking about 8 00:00:29,679 --> 00:00:33,240 sighting software. 9 00:00:33,405 --> 00:00:35,425 [applause] 10 00:00:38,800 --> 00:00:41,600 Thank you everyone for being here. So 11 00:00:41,600 --> 00:00:43,120 before I start, I just want to do a 12 00:00:43,120 --> 00:00:45,120 quick bit of housekeeping. There's quite 13 00:00:45,120 --> 00:00:47,280 a few references in this talk. I had a 14 00:00:47,280 --> 00:00:48,480 bit of trouble getting them on the 15 00:00:48,480 --> 00:00:50,320 slide, so I've wedged them in as best as 16 00:00:50,320 --> 00:00:52,559 possible. There are a few slides with 17 00:00:52,559 --> 00:00:54,399 just references. When the video is 18 00:00:54,399 --> 00:00:56,000 available, you can press pause to read 19 00:00:56,000 --> 00:00:58,399 those. Alternatively, the full reference 20 00:00:58,399 --> 00:01:00,320 list will be in the Python AU academic 21 00:01:00,320 --> 00:01:01,520 proceedings which will be available 22 00:01:01,520 --> 00:01:03,920 after this conference. And for any 23 00:01:03,920 --> 00:01:05,840 fellow pedants in the room who recognize 24 00:01:05,840 --> 00:01:08,240 I'm using APA 7, but notice that I have 25 00:01:08,240 --> 00:01:09,840 omitted hanging indenteds. That's for 26 00:01:09,840 --> 00:01:11,840 reasons of space. I've also used a few 27 00:01:11,840 --> 00:01:15,040 footnotes. So, please do forgive me. All 28 00:01:15,040 --> 00:01:17,759 right, into the talk. 29 00:01:17,759 --> 00:01:20,240 So in publications, researchers site 30 00:01:20,240 --> 00:01:22,560 previous work that informed, was used or 31 00:01:22,560 --> 00:01:25,119 was relied on. Depending on their field, 32 00:01:25,119 --> 00:01:27,119 they can site all manner of things. They 33 00:01:27,119 --> 00:01:29,360 might site papers, theorems, data sets, 34 00:01:29,360 --> 00:01:32,240 pottery, carvings, coins, film, heaps of 35 00:01:32,240 --> 00:01:35,439 different stuff. Software is widely used 36 00:01:35,439 --> 00:01:37,840 and produced in research. In fact, a lot 37 00:01:37,840 --> 00:01:39,840 of modern software would be simply 38 00:01:39,840 --> 00:01:42,079 impossible without research, without 39 00:01:42,079 --> 00:01:44,159 software. Sorry. A lot of modern 40 00:01:44,159 --> 00:01:46,320 research would be simply impossible 41 00:01:46,320 --> 00:01:48,320 without software. 42 00:01:48,320 --> 00:01:50,320 Despite this, software is often not 43 00:01:50,320 --> 00:01:53,280 cited at all or when it is cited is 44 00:01:53,280 --> 00:01:57,000 often cited improperly. 45 00:01:57,119 --> 00:01:59,600 Powers and Bullard analyzed 90 articles 46 00:01:59,600 --> 00:02:01,759 from the field of biology. 47 00:02:01,759 --> 00:02:04,000 59 of those articles mentioned software 48 00:02:04,000 --> 00:02:06,719 comprising a total of 286 software 49 00:02:06,719 --> 00:02:10,720 mentions. Yet 56% of those mentions had 50 00:02:10,720 --> 00:02:13,200 no corresponding entry in the pub in the 51 00:02:13,200 --> 00:02:16,560 reference list. Of the 286 software 52 00:02:16,560 --> 00:02:19,440 mentions, 31% merely mentioned the name 53 00:02:19,440 --> 00:02:22,560 of the software in the text. In 5% of 54 00:02:22,560 --> 00:02:24,959 cases, there was simply a URL in the 55 00:02:24,959 --> 00:02:27,840 text or the footnote. And in 1% of 56 00:02:27,840 --> 00:02:29,520 cases, they merely said they use 57 00:02:29,520 --> 00:02:31,520 software without even mentioning the 58 00:02:31,520 --> 00:02:33,120 name. 59 00:02:33,120 --> 00:02:35,120 Now note, I did adapt this table from 60 00:02:35,120 --> 00:02:37,120 the paper, but it uses the original data 61 00:02:37,120 --> 00:02:40,400 from the paper. 62 00:02:40,400 --> 00:02:42,480 So why aren't researchers necessarily 63 00:02:42,480 --> 00:02:45,040 citing software? My guess is there's 64 00:02:45,040 --> 00:02:46,720 quite a few reasons, but I thought I'd 65 00:02:46,720 --> 00:02:50,319 outline three on this slide. I suspect a 66 00:02:50,319 --> 00:02:52,160 lot of researchers are simply unaware 67 00:02:52,160 --> 00:02:54,080 that they can and should be citing 68 00:02:54,080 --> 00:02:55,760 software. 69 00:02:55,760 --> 00:02:57,200 There also appears to be a lack of 70 00:02:57,200 --> 00:02:59,200 awareness of best practices for citing 71 00:02:59,200 --> 00:03:01,360 software amongst authors, reviewers, and 72 00:03:01,360 --> 00:03:03,280 editors. 73 00:03:03,280 --> 00:03:05,280 Additionally, citation information for 74 00:03:05,280 --> 00:03:07,440 software packages is not always clearly 75 00:03:07,440 --> 00:03:09,599 provided. 76 00:03:09,599 --> 00:03:11,440 So, what exactly are you meant to 77 00:03:11,440 --> 00:03:13,360 actually site? 78 00:03:13,360 --> 00:03:17,760 Software itself can be directly cited. 79 00:03:17,760 --> 00:03:19,440 Additionally, sometimes there are also 80 00:03:19,440 --> 00:03:21,360 other artifacts, for example, a journal 81 00:03:21,360 --> 00:03:23,360 paper or a conference paper that 82 00:03:23,360 --> 00:03:26,000 describes a software. 83 00:03:26,000 --> 00:03:28,159 In these instances, it is increasingly 84 00:03:28,159 --> 00:03:30,319 considered best practice to site both 85 00:03:30,319 --> 00:03:33,360 the software itself and the paper. Don't 86 00:03:33,360 --> 00:03:36,239 just site one or the other. Site both. 87 00:03:36,239 --> 00:03:38,000 And then make sure you include the full 88 00:03:38,000 --> 00:03:40,720 citations in the reference list. Now the 89 00:03:40,720 --> 00:03:42,879 reason this is important is if it's not 90 00:03:42,879 --> 00:03:45,280 in the reference list it is unlikely to 91 00:03:45,280 --> 00:03:47,360 be discovered by abstracting indexing 92 00:03:47,360 --> 00:03:49,760 services reducing the discoverability of 93 00:03:49,760 --> 00:03:52,480 this software and also um preventing 94 00:03:52,480 --> 00:03:53,920 people from getting attribution and 95 00:03:53,920 --> 00:03:56,239 credit for their work 96 00:03:56,239 --> 00:03:58,239 and most of this talk will be on citing 97 00:03:58,239 --> 00:04:00,959 the software itself. 98 00:04:00,959 --> 00:04:02,959 Okay. So this is an example and I've 99 00:04:02,959 --> 00:04:05,040 picked the software package pandas and 100 00:04:05,040 --> 00:04:06,720 that's because pandas in its field is 101 00:04:06,720 --> 00:04:08,720 extremely wellknown and extremely well 102 00:04:08,720 --> 00:04:10,879 used. Now these screenshots are from 103 00:04:10,879 --> 00:04:13,599 Google Scholar. Google Scholar is not a 104 00:04:13,599 --> 00:04:15,840 perfect abstract and index and indexer 105 00:04:15,840 --> 00:04:17,919 particularly a software but honestly 106 00:04:17,919 --> 00:04:19,600 none of the abstracting and indexing 107 00:04:19,600 --> 00:04:21,519 services are perfect particularly a 108 00:04:21,519 --> 00:04:23,440 software but I'm using Google Scholar 109 00:04:23,440 --> 00:04:25,040 for this talk simply because you don't 110 00:04:25,040 --> 00:04:29,520 need an account to view um its pages. So 111 00:04:29,520 --> 00:04:31,199 you can see on the right of the screen 112 00:04:31,199 --> 00:04:33,280 that the sci paper that describes 113 00:04:33,280 --> 00:04:36,320 pandas, Google Scholar has resolved over 114 00:04:36,320 --> 00:04:39,840 17,000 citations. 115 00:04:39,840 --> 00:04:41,440 And you can see on the left hand side of 116 00:04:41,440 --> 00:04:43,360 the screen that Google Scholar has 117 00:04:43,360 --> 00:04:47,919 resolved 1,01 citations of um several 118 00:04:47,919 --> 00:04:51,040 versions of the software itself. Now, 119 00:04:51,040 --> 00:04:53,199 the purpose of this screenshot is to 120 00:04:53,199 --> 00:04:55,520 illustrate that it's not futile to put 121 00:04:55,520 --> 00:04:57,759 effort into both citing software and 122 00:04:57,759 --> 00:04:59,840 making your own software easier to site 123 00:04:59,840 --> 00:05:02,320 cuz software can get cited and does get 124 00:05:02,320 --> 00:05:04,000 cited. 125 00:05:04,000 --> 00:05:05,440 These are some references that didn't 126 00:05:05,440 --> 00:05:07,039 fit on their slides. You can look at 127 00:05:07,039 --> 00:05:09,120 those in the video. 128 00:05:09,120 --> 00:05:12,000 So, why should we be citing software? 129 00:05:12,000 --> 00:05:13,600 There are several benefits of citing 130 00:05:13,600 --> 00:05:16,160 software which include facilitating the 131 00:05:16,160 --> 00:05:18,240 replication, reproducibility and 132 00:05:18,240 --> 00:05:20,800 transparency of research. It can 133 00:05:20,800 --> 00:05:22,960 encourage the reuse and extension rather 134 00:05:22,960 --> 00:05:25,680 than reinvention of software. 135 00:05:25,680 --> 00:05:27,600 It can inform researchers about software 136 00:05:27,600 --> 00:05:30,080 that is relevant to their field. Can 137 00:05:30,080 --> 00:05:31,520 give attribution and credit for 138 00:05:31,520 --> 00:05:34,160 developers of the software. Can assist 139 00:05:34,160 --> 00:05:35,840 with determining research impact which 140 00:05:35,840 --> 00:05:37,360 can be beneficial for professional 141 00:05:37,360 --> 00:05:40,000 recognition. provides insight into when 142 00:05:40,000 --> 00:05:41,360 and how software is being used for 143 00:05:41,360 --> 00:05:44,400 research and can promote software as a 144 00:05:44,400 --> 00:05:46,639 first class research output in its own 145 00:05:46,639 --> 00:05:48,560 right. 146 00:05:48,560 --> 00:05:50,240 So in this talk I'm going to use the 147 00:05:50,240 --> 00:05:52,000 example of the software package scores a 148 00:05:52,000 --> 00:05:54,160 few times and that's simply because I'm 149 00:05:54,160 --> 00:05:56,240 one of the contributors to scores and I 150 00:05:56,240 --> 00:05:57,840 know about some of the decisions we made 151 00:05:57,840 --> 00:06:00,880 and some of the lessons we learned. So I 152 00:06:00,880 --> 00:06:02,000 want to just give a little bit of 153 00:06:02,000 --> 00:06:03,520 information about the package just so 154 00:06:03,520 --> 00:06:05,360 you can understand its size and scope 155 00:06:05,360 --> 00:06:08,000 when looking at it citations. Scores is 156 00:06:08,000 --> 00:06:09,680 an open-source Python package that 157 00:06:09,680 --> 00:06:11,520 contains metrics for the verification, 158 00:06:11,520 --> 00:06:13,600 evaluation, optimization of forecasts 159 00:06:13,600 --> 00:06:16,319 and models. It's used by researchers and 160 00:06:16,319 --> 00:06:18,160 scientific organizations in multiple 161 00:06:18,160 --> 00:06:21,120 countries. It has over 30 contributors. 162 00:06:21,120 --> 00:06:23,919 It was first released in 2023. 163 00:06:23,919 --> 00:06:25,840 As of August this year, it' been 164 00:06:25,840 --> 00:06:28,240 downloaded more than 125,000 times from 165 00:06:28,240 --> 00:06:30,479 Pippi, which puts in the top 10% by 166 00:06:30,479 --> 00:06:32,800 number of downloads globally, and over 167 00:06:32,800 --> 00:06:35,520 13,000 times from Condor. 168 00:06:35,520 --> 00:06:38,000 In July 2024, we published a paper in 169 00:06:38,000 --> 00:06:39,759 the journal of open-source software and 170 00:06:39,759 --> 00:06:40,960 you can see at the top of the screen 171 00:06:40,960 --> 00:06:42,720 that Google Scholars resolved 26 172 00:06:42,720 --> 00:06:44,720 citations for that. And at the same 173 00:06:44,720 --> 00:06:46,720 time, we started depositing copies of 174 00:06:46,720 --> 00:06:48,800 scores on Zenotto. And you can see 175 00:06:48,800 --> 00:06:51,199 Google Scholars resolved two citations 176 00:06:51,199 --> 00:06:54,160 for our software on Zenoto. 177 00:06:54,160 --> 00:06:57,039 So scores is no longer completely 178 00:06:57,039 --> 00:06:59,360 unheard of, but we're not yet as famous 179 00:06:59,360 --> 00:07:02,240 as something like Pandas. 180 00:07:02,240 --> 00:07:04,080 Now I'm putting this slide up here just 181 00:07:04,080 --> 00:07:06,080 in case anyone's thinking what is Zenodo 182 00:07:06,080 --> 00:07:07,520 and what's that got to do if anything. 183 00:07:07,520 --> 00:07:10,479 Anyway, this screenshot shows the top of 184 00:07:10,479 --> 00:07:12,240 the landing page for the scores record 185 00:07:12,240 --> 00:07:15,440 on Zeno. Zenodto is really cool. It's an 186 00:07:15,440 --> 00:07:17,520 open-source open access repository 187 00:07:17,520 --> 00:07:20,720 developed by CERN. The score source code 188 00:07:20,720 --> 00:07:23,520 is hosted on GitHub. But when we release 189 00:07:23,520 --> 00:07:25,919 a version, we also deposit a copy of the 190 00:07:25,919 --> 00:07:27,520 source code for that version onto 191 00:07:27,520 --> 00:07:30,720 Zenodo. Cenodto then preserves that copy 192 00:07:30,720 --> 00:07:32,319 of the version of the source code so 193 00:07:32,319 --> 00:07:34,160 that it will be available for the long 194 00:07:34,160 --> 00:07:37,759 term. Cenodto also issues a DOI which is 195 00:07:37,759 --> 00:07:40,000 a globally unique persistent identifier 196 00:07:40,000 --> 00:07:42,240 for each version and when we ask people 197 00:07:42,240 --> 00:07:44,319 to site our scores software what we 198 00:07:44,319 --> 00:07:46,080 actually ask them to site is a Zenotto 199 00:07:46,080 --> 00:07:48,880 record for that version. There are also 200 00:07:48,880 --> 00:07:51,039 other similar archival repositories for 201 00:07:51,039 --> 00:07:53,360 example figshare. 202 00:07:53,360 --> 00:07:55,360 So going back to the benefits of citing 203 00:07:55,360 --> 00:07:57,199 software 204 00:07:57,199 --> 00:07:59,120 and bringing that as using scores as an 205 00:07:59,120 --> 00:08:01,360 example. 206 00:08:01,360 --> 00:08:03,039 So the implementation of the same 207 00:08:03,039 --> 00:08:05,120 verification metric can vary between 208 00:08:05,120 --> 00:08:07,759 software packages. So when researchers 209 00:08:07,759 --> 00:08:09,759 site that they view scores, this aids 210 00:08:09,759 --> 00:08:12,319 with reproducibility and transparency as 211 00:08:12,319 --> 00:08:14,319 it makes clear which implementation of 212 00:08:14,319 --> 00:08:16,960 the metric was used. It can also help 213 00:08:16,960 --> 00:08:18,720 with attribution and credit. For 214 00:08:18,720 --> 00:08:20,400 example, you can see the awful list is 215 00:08:20,400 --> 00:08:22,479 longer on the zenoto entry than it is on 216 00:08:22,479 --> 00:08:24,960 the papers. So when people site both the 217 00:08:24,960 --> 00:08:27,280 paper and the software itself, it means 218 00:08:27,280 --> 00:08:29,120 the developers who came on board after 219 00:08:29,120 --> 00:08:31,120 the paper was published also get credit 220 00:08:31,120 --> 00:08:33,039 for their work. 221 00:08:33,039 --> 00:08:34,640 It can also assist with determining 222 00:08:34,640 --> 00:08:36,560 research impact. When people site 223 00:08:36,560 --> 00:08:38,560 scores, we can see what they've done 224 00:08:38,560 --> 00:08:40,399 with it, where they're working, how 225 00:08:40,399 --> 00:08:41,919 they're using it, and that can help 226 00:08:41,919 --> 00:08:43,839 establish that scores has research 227 00:08:43,839 --> 00:08:46,240 impact. It's also been really 228 00:08:46,240 --> 00:08:48,000 interesting how scores is being used in 229 00:08:48,000 --> 00:08:50,000 the wild. For example, we already knew 230 00:08:50,000 --> 00:08:51,279 that some people using us in the 231 00:08:51,279 --> 00:08:53,839 geocciences, but it was only because of 232 00:08:53,839 --> 00:08:55,680 citations that we discovered that some 233 00:08:55,680 --> 00:08:57,360 epidemologists have actually started 234 00:08:57,360 --> 00:08:59,760 using us as well. 235 00:08:59,760 --> 00:09:01,200 Here's some references that didn't fit 236 00:09:01,200 --> 00:09:03,360 on this slide. 237 00:09:03,360 --> 00:09:06,080 So, how do you actually site software? 238 00:09:06,080 --> 00:09:07,760 This should be really easy, right? You 239 00:09:07,760 --> 00:09:09,040 should be able to click a few buttons 240 00:09:09,040 --> 00:09:11,040 and get a perfectly formatted citation 241 00:09:11,040 --> 00:09:13,040 out of a reference manager or AI every 242 00:09:13,040 --> 00:09:16,320 time without fail. Correct. 243 00:09:16,320 --> 00:09:20,560 Maybe, maybe not. So even when metadata 244 00:09:20,560 --> 00:09:22,560 is complete and accurate, reference 245 00:09:22,560 --> 00:09:24,560 managers, for example, Zodoro, Endnote, 246 00:09:24,560 --> 00:09:27,360 Mendlay, they may or may not correctly 247 00:09:27,360 --> 00:09:29,680 format software citations. They can 248 00:09:29,680 --> 00:09:31,839 still be really useful. I use Zottoro to 249 00:09:31,839 --> 00:09:34,160 help me do the references of this talk, 250 00:09:34,160 --> 00:09:36,000 but they're not perfect, and so you need 251 00:09:36,000 --> 00:09:37,760 to check the outputs and adjust them as 252 00:09:37,760 --> 00:09:39,440 required. 253 00:09:39,440 --> 00:09:41,360 If you use AI to generate your 254 00:09:41,360 --> 00:09:43,279 citations, you need to check its 255 00:09:43,279 --> 00:09:46,000 outputs. Humans are still responsible 256 00:09:46,000 --> 00:09:47,680 for what they submit, including 257 00:09:47,680 --> 00:09:49,200 references. 258 00:09:49,200 --> 00:09:51,519 For example, earlier in this year, um 259 00:09:51,519 --> 00:09:54,320 the archive, as in ARXIV, the major 260 00:09:54,320 --> 00:09:56,959 print pre-print server, clarified that 261 00:09:56,959 --> 00:09:59,120 hallucinated references will count as 262 00:09:59,120 --> 00:10:01,680 incontrovertible evidence that authors 263 00:10:01,680 --> 00:10:04,160 have not checked their LLM outputs and 264 00:10:04,160 --> 00:10:08,360 will result in a one-year ban. 265 00:10:08,399 --> 00:10:10,800 So what are best practices in how to 266 00:10:10,800 --> 00:10:13,279 site software? 267 00:10:13,279 --> 00:10:15,279 The force 11 software citation 268 00:10:15,279 --> 00:10:17,200 implementation working group created a 269 00:10:17,200 --> 00:10:20,240 software citation guide cats atl 2021. 270 00:10:20,240 --> 00:10:21,600 Now if you search for this on the 271 00:10:21,600 --> 00:10:23,040 internet, there's a very good chance 272 00:10:23,040 --> 00:10:24,480 you're going to find a different but 273 00:10:24,480 --> 00:10:27,680 related force 11 document from 2016. So 274 00:10:27,680 --> 00:10:29,120 if you do want to find this document, 275 00:10:29,120 --> 00:10:30,880 make sure you're looking for the 2021 276 00:10:30,880 --> 00:10:32,800 one. 277 00:10:32,800 --> 00:10:34,640 The citation guide was compiled with 278 00:10:34,640 --> 00:10:36,560 input from representatives from 279 00:10:36,560 --> 00:10:39,519 academia, major publishers, for example, 280 00:10:39,519 --> 00:10:41,519 Spring of Nature, WY, Taylor and 281 00:10:41,519 --> 00:10:43,440 Francis, from professional 282 00:10:43,440 --> 00:10:44,959 organizations, societies and 283 00:10:44,959 --> 00:10:46,959 associations, for example, the ILE E and 284 00:10:46,959 --> 00:10:49,600 the American Geoysical Union 285 00:10:49,600 --> 00:10:51,440 from research institutions for example 286 00:10:51,440 --> 00:10:53,279 ENCAR and the German climate computing 287 00:10:53,279 --> 00:10:56,160 center and also from crossre and data 288 00:10:56,160 --> 00:10:58,000 site. 289 00:10:58,000 --> 00:10:59,600 Multiple publishers have now 290 00:10:59,600 --> 00:11:01,200 incorporated this guidance into their 291 00:11:01,200 --> 00:11:04,320 software citation requirements. 292 00:11:04,320 --> 00:11:06,399 So their software citation guide 293 00:11:06,399 --> 00:11:08,160 suggests including the following 294 00:11:08,160 --> 00:11:11,120 following elements in your citations. 295 00:11:11,120 --> 00:11:13,040 The creator, so the authors or the name 296 00:11:13,040 --> 00:11:15,040 of the team that created the software. 297 00:11:15,040 --> 00:11:17,680 The title, so the name of the software. 298 00:11:17,680 --> 00:11:19,760 The publication venue. Now this trips 299 00:11:19,760 --> 00:11:22,079 people up a little bit. Preferably for 300 00:11:22,079 --> 00:11:23,680 the publisher, you should list an 301 00:11:23,680 --> 00:11:25,440 archive or repository that issues 302 00:11:25,440 --> 00:11:27,360 persistent identifiers. for example, 303 00:11:27,360 --> 00:11:30,160 Zenodo or FigShare software heritage, 304 00:11:30,160 --> 00:11:32,000 but otherwise a publisher can be the 305 00:11:32,000 --> 00:11:33,680 company or the institution who created 306 00:11:33,680 --> 00:11:35,440 the software or otherwise where the 307 00:11:35,440 --> 00:11:37,360 software is hosted, for example, GitHub 308 00:11:37,360 --> 00:11:39,360 or GitLab. 309 00:11:39,360 --> 00:11:40,720 You should include the date of the 310 00:11:40,720 --> 00:11:43,920 release or version and an identifier and 311 00:11:43,920 --> 00:11:45,839 preferably this should be a persistent 312 00:11:45,839 --> 00:11:48,560 identifier, for example, a DOI. But when 313 00:11:48,560 --> 00:11:50,720 the DOI or other persistent identifier 314 00:11:50,720 --> 00:11:52,959 is not available, you can use a URL for 315 00:11:52,959 --> 00:11:54,399 where the software exists. For example, 316 00:11:54,399 --> 00:11:57,360 a URL to the GitHub repo. Now, the next 317 00:11:57,360 --> 00:11:59,519 two elements may or may not be required. 318 00:11:59,519 --> 00:12:01,120 It depends on the software citation 319 00:12:01,120 --> 00:12:03,839 system you're using. When required, you 320 00:12:03,839 --> 00:12:05,200 should list the version of the software 321 00:12:05,200 --> 00:12:06,639 being referenced. And if there's no 322 00:12:06,639 --> 00:12:08,399 version information, provide the date of 323 00:12:08,399 --> 00:12:11,200 access. And a small number of citation 324 00:12:11,200 --> 00:12:14,160 types, for example, API APA require a 325 00:12:14,160 --> 00:12:15,600 description. For example, computer 326 00:12:15,600 --> 00:12:17,760 software in square brackets. That's 327 00:12:17,760 --> 00:12:19,040 really not required in many of them, 328 00:12:19,040 --> 00:12:21,120 just some. 329 00:12:21,120 --> 00:12:22,639 So, I'm not going to talk to everything 330 00:12:22,639 --> 00:12:24,639 on this slide, but I just want to put a 331 00:12:24,639 --> 00:12:26,639 slide up with some software citations 332 00:12:26,639 --> 00:12:28,639 just so you can see them. For the 333 00:12:28,639 --> 00:12:31,360 purposes of this slide, I've used APA 7 334 00:12:31,360 --> 00:12:33,200 simply because it includes all the 335 00:12:33,200 --> 00:12:34,720 elements from the 4th 11 reference 336 00:12:34,720 --> 00:12:37,200 guide. If you're writing a journal 337 00:12:37,200 --> 00:12:38,720 article, you should follow the 338 00:12:38,720 --> 00:12:40,639 guidelines provided by the journal, but 339 00:12:40,639 --> 00:12:43,600 they do tend to be along these lines. So 340 00:12:43,600 --> 00:12:45,440 on the screen I've included two examples 341 00:12:45,440 --> 00:12:47,680 of open- source software. One closed, 342 00:12:47,680 --> 00:12:49,440 two with named authors and one with a 343 00:12:49,440 --> 00:12:51,760 group author and two with a persistent 344 00:12:51,760 --> 00:12:54,720 identifier and one with a URL. So I'm 345 00:12:54,720 --> 00:12:56,480 just going to now in a moment talk a 346 00:12:56,480 --> 00:12:57,920 little bit more about versions and 347 00:12:57,920 --> 00:12:59,839 persistent identifiers because I've seen 348 00:12:59,839 --> 00:13:01,760 them trip up people quite a lot and then 349 00:13:01,760 --> 00:13:03,360 I'll move on to the next section of the 350 00:13:03,360 --> 00:13:06,560 talk. Versions. 351 00:13:06,560 --> 00:13:08,480 Virgin information can be important for 352 00:13:08,480 --> 00:13:11,600 in for reproducibility and transparency. 353 00:13:11,600 --> 00:13:13,200 software differs and sometimes 354 00:13:13,200 --> 00:13:15,839 significantly between versions and bugs 355 00:13:15,839 --> 00:13:19,200 can be corrected between versions. 356 00:13:19,200 --> 00:13:21,200 But for some reason, some citation 357 00:13:21,200 --> 00:13:23,760 styles and some journals do not allow 358 00:13:23,760 --> 00:13:25,839 you to include version information in 359 00:13:25,839 --> 00:13:28,160 the reference list entry. I honestly 360 00:13:28,160 --> 00:13:30,079 find this completely baffling. If you 361 00:13:30,079 --> 00:13:32,079 reference a book, you're always able to 362 00:13:32,079 --> 00:13:34,639 say if it's a specific edition, but for 363 00:13:34,639 --> 00:13:36,160 some reason, you're not always able to 364 00:13:36,160 --> 00:13:38,240 for software. 365 00:13:38,240 --> 00:13:40,079 Additionally, even when the citation 366 00:13:40,079 --> 00:13:42,639 style does allow or even requires the 367 00:13:42,639 --> 00:13:44,720 version information to be included, I 368 00:13:44,720 --> 00:13:45,920 have seen the version information 369 00:13:45,920 --> 00:13:48,240 nonetheless submitted. For example, I 370 00:13:48,240 --> 00:13:49,839 know an author who submitted to a 371 00:13:49,839 --> 00:13:51,839 journal that had excellent software 372 00:13:51,839 --> 00:13:53,760 citation guidelines, totally in line 373 00:13:53,760 --> 00:13:56,480 with the force 11 suggestions, and they 374 00:13:56,480 --> 00:13:58,399 had a really thorough, really diligent 375 00:13:58,399 --> 00:14:00,240 copy editor helping them format all 376 00:14:00,240 --> 00:14:02,959 their um references. that copy editor 377 00:14:02,959 --> 00:14:04,720 erroneously removed all the virgin 378 00:14:04,720 --> 00:14:06,639 information from the software citations 379 00:14:06,639 --> 00:14:08,560 despite their software citation guide 380 00:14:08,560 --> 00:14:10,800 saying it was a required element and 381 00:14:10,800 --> 00:14:12,720 that was in spite of the copy editor 382 00:14:12,720 --> 00:14:14,959 being really diligent. So the author saw 383 00:14:14,959 --> 00:14:16,399 it on time and they managed to get it 384 00:14:16,399 --> 00:14:18,160 back in before it went to print. But it 385 00:14:18,160 --> 00:14:20,000 is something to be aware of and it does 386 00:14:20,000 --> 00:14:22,079 suggest to me that the information about 387 00:14:22,079 --> 00:14:24,079 software citation is not necessarily 388 00:14:24,079 --> 00:14:25,760 adequately passed on to all the people 389 00:14:25,760 --> 00:14:28,320 in the publishing ecosystem. 390 00:14:28,320 --> 00:14:30,240 So, if you're prevented from including 391 00:14:30,240 --> 00:14:31,760 version information directly in the 392 00:14:31,760 --> 00:14:33,519 reference list, but you still want to 393 00:14:33,519 --> 00:14:34,959 let the readers know which version 394 00:14:34,959 --> 00:14:37,600 you're citing, options include you can 395 00:14:37,600 --> 00:14:39,120 mention that version information in the 396 00:14:39,120 --> 00:14:41,519 body of the text or when you've got a 397 00:14:41,519 --> 00:14:43,440 persistent identifier for a specific 398 00:14:43,440 --> 00:14:45,120 version. It is a roundabout way for 399 00:14:45,120 --> 00:14:46,399 readers to figure out which version 400 00:14:46,399 --> 00:14:48,880 you're referring to. Now, I'm just going 401 00:14:48,880 --> 00:14:50,560 to talk a little bit about persistent 402 00:14:50,560 --> 00:14:52,399 identifiers because it comes up over and 403 00:14:52,399 --> 00:14:54,160 over again in this talk because, as far 404 00:14:54,160 --> 00:14:55,600 as I can tell, this comes up over and 405 00:14:55,600 --> 00:14:57,839 over again in real life. 406 00:14:57,839 --> 00:15:02,000 URLs can and do break. I assume everyone 407 00:15:02,000 --> 00:15:03,920 in this audience has encountered at 408 00:15:03,920 --> 00:15:07,120 least one broken URL. 409 00:15:07,120 --> 00:15:09,440 In contrast, persistent identifiers are 410 00:15:09,440 --> 00:15:11,760 intended to be longlasting references to 411 00:15:11,760 --> 00:15:13,360 resources. 412 00:15:13,360 --> 00:15:14,720 They are frequently managed by 413 00:15:14,720 --> 00:15:16,959 institutions with the express aim of 414 00:15:16,959 --> 00:15:20,240 ensuring the links continue to resolve. 415 00:15:20,240 --> 00:15:21,839 So there's a number of different type of 416 00:15:21,839 --> 00:15:23,680 persistent identifiers, but probably the 417 00:15:23,680 --> 00:15:25,440 one you'd most commonly encounter and 418 00:15:25,440 --> 00:15:26,720 the one that's probably most relevant 419 00:15:26,720 --> 00:15:29,600 for software citation is a DOI. So a 420 00:15:29,600 --> 00:15:31,920 digital object identifier that have 421 00:15:31,920 --> 00:15:36,240 globally unique alpha numeric strings. 422 00:15:36,240 --> 00:15:38,720 Wherever possible, use a persistent 423 00:15:38,720 --> 00:15:41,440 identifier rather than URL. But when a 424 00:15:41,440 --> 00:15:43,120 persistent identifier is not available, 425 00:15:43,120 --> 00:15:46,399 then it is fine to use a URL. 426 00:15:46,399 --> 00:15:48,240 Okay. So you've decided that it's worth 427 00:15:48,240 --> 00:15:50,079 citing software. you know why you should 428 00:15:50,079 --> 00:15:52,320 do it, but when you go to do it, how do 429 00:15:52,320 --> 00:15:53,759 you find what it is you should be 430 00:15:53,759 --> 00:15:55,519 citing? 431 00:15:55,519 --> 00:15:57,839 Some software packages are great. They 432 00:15:57,839 --> 00:16:00,560 provide clear, easy to find instructions 433 00:16:00,560 --> 00:16:01,759 saying, "Here's how to site my 434 00:16:01,759 --> 00:16:02,959 software." And they give you all the 435 00:16:02,959 --> 00:16:05,519 information you need. So, if a package 436 00:16:05,519 --> 00:16:07,199 does that though, that information tends 437 00:16:07,199 --> 00:16:08,639 to be in one or more of the following 438 00:16:08,639 --> 00:16:10,800 locations. The readme, it could be a 439 00:16:10,800 --> 00:16:13,279 textbased description or a badge. In 440 00:16:13,279 --> 00:16:15,279 docu, the documentation website, there 441 00:16:15,279 --> 00:16:17,759 may be a citation guide or similar. 442 00:16:17,759 --> 00:16:19,680 There's often citation files. So there's 443 00:16:19,680 --> 00:16:21,360 a few different types of citation files, 444 00:16:21,360 --> 00:16:23,839 but one example is citation.cf, which is 445 00:16:23,839 --> 00:16:25,440 usually in the root directory or for R 446 00:16:25,440 --> 00:16:29,040 packages, citation in int subdirectory. 447 00:16:29,040 --> 00:16:31,040 Some packages have embedded mechanisms 448 00:16:31,040 --> 00:16:33,440 to generate the citation. For example, R 449 00:16:33,440 --> 00:16:35,040 has a citation function that you can 450 00:16:35,040 --> 00:16:37,839 call to retrieve citation information. 451 00:16:37,839 --> 00:16:39,600 So node and fig share allow you to 452 00:16:39,600 --> 00:16:42,000 export citation metadata. 453 00:16:42,000 --> 00:16:43,839 And for GitHub, for repositories that 454 00:16:43,839 --> 00:16:46,160 have a citation.cf CFF file, it will 455 00:16:46,160 --> 00:16:48,160 display a site this repository link on 456 00:16:48,160 --> 00:16:49,759 the right hand sidebar of the landing 457 00:16:49,759 --> 00:16:53,440 page. However, note if that GitHub repo 458 00:16:53,440 --> 00:16:56,399 in its citation.cf file asks you to site 459 00:16:56,399 --> 00:16:58,320 more than one artifact, for example, the 460 00:16:58,320 --> 00:17:00,639 software itself and the paper that site 461 00:17:00,639 --> 00:17:02,639 this repository link will only show one 462 00:17:02,639 --> 00:17:04,959 of those. So even if that appears on the 463 00:17:04,959 --> 00:17:06,400 right hand sidebar, it is actually best 464 00:17:06,400 --> 00:17:09,919 to check the citation file directly. 465 00:17:09,919 --> 00:17:11,360 So what do you do if you want to site 466 00:17:11,360 --> 00:17:13,120 software and you can't find any of those 467 00:17:13,120 --> 00:17:15,120 instructions? The first thing I want to 468 00:17:15,120 --> 00:17:17,280 say is frankly it's far more frustrating 469 00:17:17,280 --> 00:17:19,199 and difficult and when I've had to do it 470 00:17:19,199 --> 00:17:21,199 myself, I have found myself thinking the 471 00:17:21,199 --> 00:17:23,760 odd unkind thought. 472 00:17:23,760 --> 00:17:25,919 Nonetheless, software can still be cited 473 00:17:25,919 --> 00:17:27,839 even when no citation instructions are 474 00:17:27,839 --> 00:17:30,400 provided. Just follow the referencing 475 00:17:30,400 --> 00:17:32,320 guidelines of the citation style you're 476 00:17:32,320 --> 00:17:34,160 using. List the creators, list the 477 00:17:34,160 --> 00:17:36,880 title, the publisher, and so on. What do 478 00:17:36,880 --> 00:17:38,240 you do if you can't find all that 479 00:17:38,240 --> 00:17:40,799 information? Firstly, don't panic. 480 00:17:40,799 --> 00:17:42,799 Gather as much as you can and do your 481 00:17:42,799 --> 00:17:44,240 best. 482 00:17:44,240 --> 00:17:46,559 Some citation styles also have specific 483 00:17:46,559 --> 00:17:47,840 guidance for handling missing 484 00:17:47,840 --> 00:17:49,360 information. For example, if the dates 485 00:17:49,360 --> 00:17:51,600 unknown, you might write ND. And the 486 00:17:51,600 --> 00:17:53,520 Force 11 software citation guide I 487 00:17:53,520 --> 00:17:54,960 mentioned earlier also has further 488 00:17:54,960 --> 00:17:56,240 guidance for handling missing 489 00:17:56,240 --> 00:17:58,000 information. 490 00:17:58,000 --> 00:17:59,840 And just a friendly reminder, wherever 491 00:17:59,840 --> 00:18:02,000 possible, use persistent identifiers 492 00:18:02,000 --> 00:18:04,640 rather than URLs. 493 00:18:04,640 --> 00:18:06,880 So, on to the best of the best of this 494 00:18:06,880 --> 00:18:08,960 talk, how to make your own software 495 00:18:08,960 --> 00:18:12,080 easier for other people to site. 496 00:18:12,080 --> 00:18:13,919 I think this sort of boils down to five 497 00:18:13,919 --> 00:18:17,200 key steps. Firstly, choose a license for 498 00:18:17,200 --> 00:18:19,200 your software. 499 00:18:19,200 --> 00:18:21,840 Next, version and release your software 500 00:18:21,840 --> 00:18:24,080 even if you only release a single 501 00:18:24,080 --> 00:18:26,240 version because this gives people a 502 00:18:26,240 --> 00:18:28,400 specific and dated version or versions 503 00:18:28,400 --> 00:18:31,039 that they can site. 504 00:18:31,039 --> 00:18:33,120 And if you're permitted to deposit your 505 00:18:33,120 --> 00:18:34,880 software in a preservation repository 506 00:18:34,880 --> 00:18:37,120 that issues persistent identifiers, for 507 00:18:37,120 --> 00:18:39,200 example, cenodo or fig share and some 508 00:18:39,200 --> 00:18:41,600 institutional repositories. 509 00:18:41,600 --> 00:18:43,280 So these three things, and we'll come to 510 00:18:43,280 --> 00:18:45,120 the next two steps in a moment, are good 511 00:18:45,120 --> 00:18:46,240 to do if you want to make your own 512 00:18:46,240 --> 00:18:48,080 software easier to site. But there's 513 00:18:48,080 --> 00:18:49,679 also a whole lot of other benefits for 514 00:18:49,679 --> 00:18:51,919 doing this. For example, doing these 515 00:18:51,919 --> 00:18:54,559 things goes, pardon the pun, a fair way 516 00:18:54,559 --> 00:18:56,480 towards meeting the fair principles 517 00:18:56,480 --> 00:18:58,240 where fair stands for findable, 518 00:18:58,240 --> 00:19:01,280 accessible, interoperable, and reusable. 519 00:19:01,280 --> 00:19:03,280 and fair principles increasingly being 520 00:19:03,280 --> 00:19:06,720 adopted across the research landscape. 521 00:19:06,720 --> 00:19:08,559 Also, increasingly journals are 522 00:19:08,559 --> 00:19:10,720 requesting or requiring that software is 523 00:19:10,720 --> 00:19:12,559 to be opened and published with a 524 00:19:12,559 --> 00:19:14,640 persistent identifier if it forms part 525 00:19:14,640 --> 00:19:16,320 of your research. So, if you're in a 526 00:19:16,320 --> 00:19:18,160 position to do these three steps, it's 527 00:19:18,160 --> 00:19:21,280 just a really good thing to do. 528 00:19:21,280 --> 00:19:24,000 Next, if you can consider publishing a 529 00:19:24,000 --> 00:19:25,440 journal paper that describes your 530 00:19:25,440 --> 00:19:27,039 software. 531 00:19:27,039 --> 00:19:29,760 So there's lots of reasons um to write a 532 00:19:29,760 --> 00:19:31,679 paper about your software, but in terms 533 00:19:31,679 --> 00:19:33,520 of specifically at making your software 534 00:19:33,520 --> 00:19:35,760 easier to site, reasons include 535 00:19:35,760 --> 00:19:38,240 abstracting and indexing systems are 536 00:19:38,240 --> 00:19:40,080 currently much better at recognizing 537 00:19:40,080 --> 00:19:43,520 paper citations than software citations. 538 00:19:43,520 --> 00:19:45,360 Additionally, researchers may be more 539 00:19:45,360 --> 00:19:47,600 familiar with referencing journal papers 540 00:19:47,600 --> 00:19:49,039 rather than referencing the software 541 00:19:49,039 --> 00:19:51,840 itself. So, I know an actual case where 542 00:19:51,840 --> 00:19:53,440 a researcher knew they were supposed to 543 00:19:53,440 --> 00:19:54,880 site both the software and the journal 544 00:19:54,880 --> 00:19:57,520 paper but chose to only site the paper. 545 00:19:57,520 --> 00:19:59,440 And I suspect that this happens quite a 546 00:19:59,440 --> 00:20:01,120 lot. So, if you have both the paper and 547 00:20:01,120 --> 00:20:02,640 the software itself, it kind of hedges 548 00:20:02,640 --> 00:20:05,360 your bets. Also, if you're not in a 549 00:20:05,360 --> 00:20:06,960 position to open source your software 550 00:20:06,960 --> 00:20:09,120 and it has to remain closed, you still 551 00:20:09,120 --> 00:20:10,640 may be able to write a journal paper 552 00:20:10,640 --> 00:20:12,720 about your closed software. And there 553 00:20:12,720 --> 00:20:14,240 are definitely papers out there about 554 00:20:14,240 --> 00:20:16,240 closed software that can and do get 555 00:20:16,240 --> 00:20:18,160 cited. So, it is an alternative way to 556 00:20:18,160 --> 00:20:19,840 get cited if you can't open your 557 00:20:19,840 --> 00:20:21,919 software. 558 00:20:21,919 --> 00:20:24,160 And lastly, but most importantly, 559 00:20:24,160 --> 00:20:26,880 provide clear and easy to find citation 560 00:20:26,880 --> 00:20:28,960 instructions. Don't be one of the 561 00:20:28,960 --> 00:20:30,799 packages that makes people curse under 562 00:20:30,799 --> 00:20:32,080 their breath when they go to try and 563 00:20:32,080 --> 00:20:34,000 site you. Be one of the packages where 564 00:20:34,000 --> 00:20:35,440 they just breathe a sigh of relief and 565 00:20:35,440 --> 00:20:38,080 go, "Oh, this one's going to be easy." 566 00:20:38,080 --> 00:20:40,480 So, include a citation metadata file. 567 00:20:40,480 --> 00:20:42,640 For example, citation.cf in your root 568 00:20:42,640 --> 00:20:44,559 directory or citation of R packages in 569 00:20:44,559 --> 00:20:46,320 the in subdirectory. There are other 570 00:20:46,320 --> 00:20:48,960 ones as well such as code matter. As 571 00:20:48,960 --> 00:20:50,640 well as that provide your recommended 572 00:20:50,640 --> 00:20:52,799 citation in your readme and your 573 00:20:52,799 --> 00:20:55,360 documentation or website. And if you 574 00:20:55,360 --> 00:20:57,440 have a journal article request both the 575 00:20:57,440 --> 00:20:59,200 software itself and the journal paper 576 00:20:59,200 --> 00:21:01,200 should be cited. Now you might think 577 00:21:01,200 --> 00:21:02,720 that's a little bit overkill. Why would 578 00:21:02,720 --> 00:21:04,640 you put your citation suggestions in 579 00:21:04,640 --> 00:21:06,720 more than one place? Well, the citation 580 00:21:06,720 --> 00:21:08,640 files are designed to be both human and 581 00:21:08,640 --> 00:21:11,120 machine readable. your documentation is 582 00:21:11,120 --> 00:21:12,880 not necessarily going to be machine 583 00:21:12,880 --> 00:21:14,640 readable and different people look for 584 00:21:14,640 --> 00:21:16,559 information in different places. So, 585 00:21:16,559 --> 00:21:18,240 it's definitely not overkill and it's 586 00:21:18,240 --> 00:21:20,000 definitely not a problem to put your 587 00:21:20,000 --> 00:21:21,600 instructions in more than one place and 588 00:21:21,600 --> 00:21:23,200 they're far more likely to be found that 589 00:21:23,200 --> 00:21:24,880 way. 590 00:21:24,880 --> 00:21:27,760 So, if we go back to the package scores 591 00:21:27,760 --> 00:21:29,520 and some of the lessons we learned about 592 00:21:29,520 --> 00:21:32,000 sighting, originally we didn't provide 593 00:21:32,000 --> 00:21:33,600 any citation instructions for our 594 00:21:33,600 --> 00:21:35,679 package at all. So, some people start to 595 00:21:35,679 --> 00:21:37,440 use our package and in papers would just 596 00:21:37,440 --> 00:21:38,799 say something like, "Oh, yeah, we use 597 00:21:38,799 --> 00:21:41,679 scores. Here's a link to the GitHub. 598 00:21:41,679 --> 00:21:43,280 Once we published the Journal of Open 599 00:21:43,280 --> 00:21:46,559 Source software paper in 2024, we asked 600 00:21:46,559 --> 00:21:48,240 people to please site that. And those 601 00:21:48,240 --> 00:21:49,919 citations did start rolling in, which 602 00:21:49,919 --> 00:21:52,159 you can see with the 26 citations at the 603 00:21:52,159 --> 00:21:55,679 top. But it took us another 16 months in 604 00:21:55,679 --> 00:21:57,679 February this year to actually ask them 605 00:21:57,679 --> 00:22:00,240 to site the software itself. Until we 606 00:22:00,240 --> 00:22:02,240 did that, nobody cited the software 607 00:22:02,240 --> 00:22:04,960 itself. And those citations are just 608 00:22:04,960 --> 00:22:06,720 starting to appear which you can see 609 00:22:06,720 --> 00:22:08,480 with the two citations on the software 610 00:22:08,480 --> 00:22:10,559 itself on the bottom of the screen. So 611 00:22:10,559 --> 00:22:12,480 what are the lessons we learned? Once 612 00:22:12,480 --> 00:22:14,720 you're ready to do so, you can provide 613 00:22:14,720 --> 00:22:16,320 citation instructions to the software 614 00:22:16,320 --> 00:22:18,480 itself. There is no need to wait for a 615 00:22:18,480 --> 00:22:20,799 published paper. And then if you do 616 00:22:20,799 --> 00:22:23,360 publish a paper, ask people to site both 617 00:22:23,360 --> 00:22:26,240 your paper and the software itself. 618 00:22:26,240 --> 00:22:28,240 Here's a slide with some further reading 619 00:22:28,240 --> 00:22:30,000 stuff I didn't specifically refer to but 620 00:22:30,000 --> 00:22:31,360 are really useful if you want some more 621 00:22:31,360 --> 00:22:33,200 info. And here are some references that 622 00:22:33,200 --> 00:22:34,720 didn't fit on their slides. And there's 623 00:22:34,720 --> 00:22:36,320 not much time, but if there's time, I'm 624 00:22:36,320 --> 00:22:37,840 open to questions. But if I need to 625 00:22:37,840 --> 00:22:41,105 stop, that's fine, too. Thank you. 626 00:22:41,105 --> 00:22:43,125 [applause] 627 00:22:51,840 --> 00:22:53,360 So, I'm not sure. Are we seeing if 628 00:22:53,360 --> 00:22:54,799 there's questions or we finishing? 629 00:22:54,799 --> 00:22:56,400 Yes, we do have time for questions. 630 00:22:56,400 --> 00:22:58,960 Thank you very much. So, we've got about 631 00:22:58,960 --> 00:23:00,640 five minutes for questions. And as 632 00:23:00,640 --> 00:23:04,760 usual, please wait for the microphone. 633 00:23:08,480 --> 00:23:10,400 Hi, thank you for the talk. Just 634 00:23:10,400 --> 00:23:14,080 wondering um what do you do with 635 00:23:14,080 --> 00:23:15,760 um a software package that has 636 00:23:15,760 --> 00:23:17,919 dependency like imagine you have a 637 00:23:17,919 --> 00:23:20,799 package let's call it fu that internally 638 00:23:20,799 --> 00:23:24,240 using heavily numpy would you include 639 00:23:24,240 --> 00:23:26,320 sort of all the entire hierarchy into 640 00:23:26,320 --> 00:23:27,919 your citation or 641 00:23:27,919 --> 00:23:30,559 that's a really good question um the 642 00:23:30,559 --> 00:23:32,480 conventions around software citation are 643 00:23:32,480 --> 00:23:34,240 not as well developed yet as they are 644 00:23:34,240 --> 00:23:35,919 say in other areas of research and I 645 00:23:35,919 --> 00:23:37,360 think I'm hoping as that evolves over 646 00:23:37,360 --> 00:23:38,799 time these answers will become more 647 00:23:38,799 --> 00:23:40,880 clear-cut cut. Um I've gone back to this 648 00:23:40,880 --> 00:23:43,600 slide. So the second dot point, the 649 00:23:43,600 --> 00:23:45,440 software citation checklist for authors 650 00:23:45,440 --> 00:23:48,960 by Chu Hong from 2019. Um that has a 651 00:23:48,960 --> 00:23:50,720 really good section on it on sort of 652 00:23:50,720 --> 00:23:52,159 working out what software you should 653 00:23:52,159 --> 00:23:53,600 maybe try and site and sort of some of 654 00:23:53,600 --> 00:23:55,760 the rationale. Um at the moment the 655 00:23:55,760 --> 00:23:57,760 convention seems to be not to site the 656 00:23:57,760 --> 00:23:59,360 dependencies and just site the main 657 00:23:59,360 --> 00:24:01,919 package. But I guess if you think some 658 00:24:01,919 --> 00:24:03,520 key dependencies are really crucial, 659 00:24:03,520 --> 00:24:05,440 it's a good idea and it is a little bit 660 00:24:05,440 --> 00:24:07,600 of a tough one. Um but yeah the second 661 00:24:07,600 --> 00:24:09,600 the reference and the second dot points 662 00:24:09,600 --> 00:24:11,760 probably a good set of yeah guidelines 663 00:24:11,760 --> 00:24:15,360 you could use to sort of inform that. 664 00:24:15,360 --> 00:24:16,559 Other questions? 665 00:24:16,559 --> 00:24:17,840 Yes. 666 00:24:17,840 --> 00:24:19,520 Hi this might be answered in the same 667 00:24:19,520 --> 00:24:22,640 kind of checklist and area. Is there any 668 00:24:22,640 --> 00:24:25,279 benefit to citing the particular version 669 00:24:25,279 --> 00:24:27,279 of the software um to the authors 670 00:24:27,279 --> 00:24:31,279 themselves or is it um okay to site I 671 00:24:31,279 --> 00:24:33,840 saw that the Zenoto website had the um 672 00:24:33,840 --> 00:24:35,600 it recommended doing the general version 673 00:24:35,600 --> 00:24:39,039 rather than um the specific versions or 674 00:24:39,039 --> 00:24:40,799 that's a really good question. So one 675 00:24:40,799 --> 00:24:42,559 advantage if you site the specific 676 00:24:42,559 --> 00:24:44,240 version is that reproducibility and 677 00:24:44,240 --> 00:24:46,400 transparency. For example, if there is a 678 00:24:46,400 --> 00:24:49,679 change between version 1 whatever and 679 00:24:49,679 --> 00:24:51,279 version two, that might be significant 680 00:24:51,279 --> 00:24:53,919 to the research. Um, if you site the 681 00:24:53,919 --> 00:24:55,600 specific version used, hopefully it will 682 00:24:55,600 --> 00:24:57,360 include the contributors to that 683 00:24:57,360 --> 00:24:59,520 specific version. Um, in terms of the 684 00:24:59,520 --> 00:25:01,360 site all versions, the not does provide 685 00:25:01,360 --> 00:25:03,360 that, which is really interesting. The 686 00:25:03,360 --> 00:25:05,200 problem is is that the abstracting 687 00:25:05,200 --> 00:25:07,520 indexing systems still don't quite know 688 00:25:07,520 --> 00:25:11,120 how to handle that DOI. So, you can use 689 00:25:11,120 --> 00:25:13,120 it, but maybe it's more foolproof to use 690 00:25:13,120 --> 00:25:14,720 the exact version one. But again, it's 691 00:25:14,720 --> 00:25:16,320 one of those things of the ecosystem 692 00:25:16,320 --> 00:25:17,919 around this is still evolving and maybe 693 00:25:17,919 --> 00:25:19,279 that answer will be different down the 694 00:25:19,279 --> 00:25:22,000 road. Um, does that answer the question? 695 00:25:22,000 --> 00:25:24,400 Thanks. 696 00:25:24,400 --> 00:25:26,720 Um, my really poorly thought out 697 00:25:26,720 --> 00:25:29,919 question is about the incentives for 698 00:25:29,919 --> 00:25:33,520 software engineers to do the work. Uh, 699 00:25:33,520 --> 00:25:35,039 and it's a small amount of work we're 700 00:25:35,039 --> 00:25:35,440 talking about. 701 00:25:35,440 --> 00:25:35,919 Yeah. 702 00:25:35,919 --> 00:25:37,600 Um, 703 00:25:37,600 --> 00:25:41,840 if like I'm I am a software engineer who 704 00:25:41,840 --> 00:25:44,400 works on research stuff. Um, I don't 705 00:25:44,400 --> 00:25:46,400 really care about papers though. I care 706 00:25:46,400 --> 00:25:48,480 about the experimental results. 707 00:25:48,480 --> 00:25:50,159 Um, 708 00:25:50,159 --> 00:25:53,760 it really feels like so much of this 709 00:25:53,760 --> 00:25:58,080 could be automated by a system that 710 00:25:58,080 --> 00:26:01,679 points to a Pippi release rather than 711 00:26:01,679 --> 00:26:04,320 having to go and make DDIs elsewhere and 712 00:26:04,320 --> 00:26:06,880 add extra files and stuff like that. So 713 00:26:06,880 --> 00:26:08,960 I don't know like it my question might 714 00:26:08,960 --> 00:26:10,880 be what are the incentives for software 715 00:26:10,880 --> 00:26:12,400 engineers who don't care about papers 716 00:26:12,400 --> 00:26:14,480 and my other question might be why 717 00:26:14,480 --> 00:26:16,880 aren't we automating this based on all 718 00:26:16,880 --> 00:26:18,240 of the infrastructure that's already out 719 00:26:18,240 --> 00:26:18,880 there 720 00:26:18,880 --> 00:26:20,159 right I'm going to answer that in two 721 00:26:20,159 --> 00:26:21,279 parts and I'm going to start with the 722 00:26:21,279 --> 00:26:22,559 automation bit and then come back to the 723 00:26:22,559 --> 00:26:24,960 incentives so if you're doing software 724 00:26:24,960 --> 00:26:26,159 particularly if it's open you need a 725 00:26:26,159 --> 00:26:27,919 license anyway basically so that's sort 726 00:26:27,919 --> 00:26:30,559 of a step you should be doing versioning 727 00:26:30,559 --> 00:26:32,080 and releasing your software is probably 728 00:26:32,080 --> 00:26:33,760 just a good idea anyway so that's 729 00:26:33,760 --> 00:26:35,039 probably also something that you should 730 00:26:35,039 --> 00:26:36,400 be 731 00:26:36,400 --> 00:26:37,840 Depending where you work, you may or may 732 00:26:37,840 --> 00:26:38,960 not be able to open your software and 733 00:26:38,960 --> 00:26:40,000 you may or may not be able to put it 734 00:26:40,000 --> 00:26:41,360 somewhere like Zenoto. So, first of all, 735 00:26:41,360 --> 00:26:43,279 that could be a barrier. If you are able 736 00:26:43,279 --> 00:26:45,840 to um there are platforms I said, but 737 00:26:45,840 --> 00:26:48,400 Zenoto actually has an easy integration 738 00:26:48,400 --> 00:26:50,320 function with GitHub where you set it up 739 00:26:50,320 --> 00:26:52,240 once and then every time you do a tagged 740 00:26:52,240 --> 00:26:53,919 release, it just autoflows to Zenoto. 741 00:26:53,919 --> 00:26:55,360 So, we set that up for scores and it 742 00:26:55,360 --> 00:26:57,520 means that we went from, you know, 20 743 00:26:57,520 --> 00:26:58,880 minutes work the first time or something 744 00:26:58,880 --> 00:27:00,720 to like boom, instantly works the next 745 00:27:00,720 --> 00:27:02,000 time. So, there are ways you can 746 00:27:02,000 --> 00:27:03,360 automate it to make it easier for 747 00:27:03,360 --> 00:27:04,559 yourself. I'm less familiar with the 748 00:27:04,559 --> 00:27:05,679 other platforms or whether or not they 749 00:27:05,679 --> 00:27:07,520 got the automation, but it does at least 750 00:27:07,520 --> 00:27:09,440 exist between GitHub and Zenoto. So, you 751 00:27:09,440 --> 00:27:11,440 absolutely can automate some of it. Now, 752 00:27:11,440 --> 00:27:13,600 in terms of the benefits to researchers 753 00:27:13,600 --> 00:27:15,279 who don't care about papers, which is a 754 00:27:15,279 --> 00:27:17,919 really valid thing um sometimes you need 755 00:27:17,919 --> 00:27:19,440 to convince your bosses or your 756 00:27:19,440 --> 00:27:21,279 organization or your funders or people 757 00:27:21,279 --> 00:27:22,799 grants that you actually what you're 758 00:27:22,799 --> 00:27:25,600 doing has meaningful impact in the 759 00:27:25,600 --> 00:27:28,240 research sector. So citation metrics, 760 00:27:28,240 --> 00:27:30,880 even if it doesn't individually benefit 761 00:27:30,880 --> 00:27:33,120 the software developers, may actually 762 00:27:33,120 --> 00:27:35,520 really help make sense to your managers 763 00:27:35,520 --> 00:27:37,279 or to your funding organizers, 764 00:27:37,279 --> 00:27:38,480 especially if they come from that 765 00:27:38,480 --> 00:27:40,320 scientific background where citations 766 00:27:40,320 --> 00:27:42,880 are seen as currency. Also, people 767 00:27:42,880 --> 00:27:44,400 citing the software does mean you can 768 00:27:44,400 --> 00:27:46,240 say, well, we know that it's used in 769 00:27:46,240 --> 00:27:48,400 this organization here and that agency 770 00:27:48,400 --> 00:27:49,760 over there and by this company here, and 771 00:27:49,760 --> 00:27:51,520 we know people using it for A, B, C, and 772 00:27:51,520 --> 00:27:53,360 D. So, it does help to actually just not 773 00:27:53,360 --> 00:27:55,120 go, oh, no, we tell we promise you our 774 00:27:55,120 --> 00:27:56,640 software is useful to people. you can 775 00:27:56,640 --> 00:27:58,640 use it as kind of definitive concrete 776 00:27:58,640 --> 00:28:02,000 proof and real examples. Um, so perhaps 777 00:28:02,000 --> 00:28:04,080 that can be a motivating factor and I 778 00:28:04,080 --> 00:28:06,240 guess a more nebulous one might just be 779 00:28:06,240 --> 00:28:08,240 to contribute to the broader landscape 780 00:28:08,240 --> 00:28:10,960 of research. Um, does that answer your 781 00:28:10,960 --> 00:28:12,320 question? 782 00:28:12,320 --> 00:28:14,240 And that's all the time that we have for 783 00:28:14,240 --> 00:28:17,039 questions. Uh, and uh, here's the 784 00:28:17,039 --> 00:28:18,159 traditional gift. Oh, 785 00:28:18,159 --> 00:28:21,109 thank you. [applause]