1
00:00:13,040 --> 00:00:21,120
Hello. I'm humbled to be presenting from 
the Lenape, Shawnee, Wyandot Miami, Ottawa,  

2
00:00:21,120 --> 00:00:29,840
Potawatomi and to acknowledge the thousands of 
Native American who call Northeast Ohio home. 

3
00:00:30,880 --> 00:00:34,720
My name is Ee and I'm the Director of 
Infrastructure from the Python Software  

4
00:00:34,720 --> 00:00:43,280
Foundation, or PSF. The PSF is a U.S. 501(c)3 
nonprofit that serves the Python community through  

5
00:00:43,280 --> 00:00:49,840
our mission to promote, protect and advance the 
Python Programming Language and the growth of a  

6
00:00:49,840 --> 00:00:59,200
diverse community of Python developers. We defend 
the Trademark, Copyright and Intellectual Property  

7
00:00:59,200 --> 00:01:05,360
for the language, awarding grants to support 
Python focus groups and events around the world,  

8
00:01:06,000 --> 00:01:14,800
providing fiscal sponsorship to extent tax exempt 
status and producing and underwriting PyCon US. 

9
00:01:20,160 --> 00:01:24,480
In my role, at the PSF, I'm responsible 
for many of the online services that  

10
00:01:24,480 --> 00:01:31,120
Pythonistas use around the world. I don't 
necessarily build or develop all of them,  

11
00:01:31,120 --> 00:01:36,480
but work to support volunteers to provide the 
community services on the Python.org domain. 

12
00:01:37,200 --> 00:01:49,280
Today, I want to talk about PyPI, to some, the 
cheese shop. Yes, it still redirects. But humbly,  

13
00:01:49,280 --> 00:01:54,880
my preferred description is PyPI, where Python 
developers go to cheat on their homework. 

14
00:01:56,560 --> 00:02:03,920
Alas, PyPI is the Python Package 
Index. It's the de facto place to find,  

15
00:02:03,920 --> 00:02:12,400
install and publish Python packages online.
Richard's initial invitation prompted stories  

16
00:02:12,400 --> 00:02:17,600
from the PyPI trenches of where there 
are many. I've been a caretaker for  

17
00:02:17,600 --> 00:02:23,360
this service for around eight years, primarily 
focused on the bits and bobs behind the scenes  

18
00:02:23,360 --> 00:02:28,800
that are needed to keep the service online.
In that time, I've contributed to two major  

19
00:02:28,800 --> 00:02:40,400
infrastructure overhauls, codebase rewrite 
and deprecations. I've also been there as the  

20
00:02:40,400 --> 00:02:45,120
person carrying the pager on the front line for 
incidents and package availability of the service.  

21
00:02:46,080 --> 00:02:49,840
So, yes, there are many stories 
from that time, in those trenches. 

22
00:02:51,440 --> 00:03:00,240
However, in considering the opportunity to 
delivery of a Keynote, security breaches,  

23
00:03:03,520 --> 00:03:11,040
branded software cycle and even our own 
mistakes simply seemed like it would have been  

24
00:03:11,040 --> 00:03:25,280
uninspiring to talk about the successes PyPI 
has seen and what challenges PyPI has seen. 

25
00:03:28,160 --> 00:03:32,240
So, with that, here's an outline 
for the rest of this talk.  

26
00:03:32,240 --> 00:03:37,040
I'll share an overview of PyPI's history 
from my perspective, highlighting some of  

27
00:03:37,040 --> 00:03:41,600
the critical changes that made it what it is 
today and quantifying the growth it has seen  

28
00:03:43,120 --> 00:03:50,880
and the current era and some of the challenges it 
faces today and the future of PyPI based on that. 

29
00:03:54,240 --> 00:03:59,440
Fundamentally, the core goal of PyPI hasn't 
changed since it was described and implemented  

30
00:03:59,440 --> 00:04:07,120
in the early 2000s, PyPI came into existence as a 
solution to the problem of "where do I find Python  

31
00:04:07,120 --> 00:04:19,120
software?" While predecessor existed, they didn't 
offer the standardization necessary. So given  

32
00:04:19,120 --> 00:04:37,360
that, Richard Jones submitted PEP 301, package 
and metadata. It came on November 1, 2002, eight  

33
00:04:37,360 --> 00:04:43,920
days before the PEP was accepted. About 12 hours 
after the commit to version control I could find.  

34
00:04:45,520 --> 00:04:52,720
Thus, PyPI was born. Here's the earliest caption 
of the page, by archive.org. At this point,  

35
00:04:52,720 --> 00:05:00,560
PyPI is fulfilling PEP 301. Maintainers of 
software can create new releases. Users who  

36
00:05:00,560 --> 00:05:05,440
want new Python software can find and install 
it from here if they're using the latest Python. 

37
00:05:06,880 --> 00:05:11,440
By the end of 2003, just under 
300 projects exist on the index.  

38
00:05:14,080 --> 00:05:19,760
This slide reads as intended. This is the point 
in PyPI history that I always like to pause at  

39
00:05:19,760 --> 00:05:27,360
and enjoy. Imagine, it's 2004, right now, PyPI's a 
pure, hypermedia dream compared to what it's like  

40
00:05:27,360 --> 00:05:34,160
to run a web service. Maintainers are responsible 
for hosting their downloadable artifacts  

41
00:05:34,160 --> 00:05:48,000
and PyPI is just a centralized index that allows 
others to programmatically search. Theoretically,  

42
00:05:48,000 --> 00:05:55,120
given the advancements of internet infrastructure, 
PyPI, as it existed, could have been served from  

43
00:05:55,120 --> 00:06:03,120
the co-located server it was hosted on. 
As a point of interest, that very server,  

44
00:06:03,840 --> 00:06:07,360
located in Netherlands, was 
decommissioned early last month. 

45
00:06:10,160 --> 00:06:17,200
So that brings us to the first fundamental 
shift for PyPI since its inception. Even  

46
00:06:17,760 --> 00:06:21,840
given steady growth of the index, as far 
as submissions and usage by the community,  

47
00:06:22,400 --> 00:06:27,600
self hosting files was something that not every 
maintainer had interest or even means to do.  

48
00:06:28,640 --> 00:06:36,800
As such, in March 2005, the ability to upload 
files to PyPI itself was added. And the inflection  

49
00:06:36,800 --> 00:06:41,040
in this curve, shown total project counts 
submitted to the index, makes clear that this  

50
00:06:41,040 --> 00:06:46,960
was the right thing to do. People wanted to share 
their Python software and hosting was clearly a  

51
00:06:46,960 --> 00:06:54,000
barrier that was keeping them from being able to 
do so. Adding upload hosting was a good thing.  

52
00:06:55,840 --> 00:07:01,120
But it came with caveats. And those caveats 
include the infrastructure necessary to store  

53
00:07:01,120 --> 00:07:06,000
these files, humans to ensure good backups 
and recover from disaster, if necessary,  

54
00:07:06,720 --> 00:07:10,560
all of which were available, thanks 
to the volunteers working on PyPI  

55
00:07:11,120 --> 00:07:18,400
and the hosting provided by XS4ALL at the time.
I don't mean to gloss over the efforts of the  

56
00:07:18,400 --> 00:07:29,440
many volunteers and contributors, but from 2005 
to around 2013, not much really changed about  

57
00:07:29,440 --> 00:07:35,600
PyPI or its backing infrastructure. But what 
did change is that got a new coat of paint.  

58
00:07:39,360 --> 00:07:46,320
Oh, and, the popularity and adoption of Python and 
thus the Python Package Index saw extreme growth.  

59
00:07:48,800 --> 00:07:58,320
PyPI had 36,000 projects, 185,000 releases 
and about 62 gigabytes of stored files. 

60
00:08:00,880 --> 00:08:06,080
The specific data and number on those graphs 
are interesting. But for the sake of this talk,  

61
00:08:06,080 --> 00:08:12,240
the most important thing to keep in mind 
is the pattern of growth. There was a time  

62
00:08:12,240 --> 00:08:18,080
and will be, again when cognizance of 
the exponential growth won't be quite as  

63
00:08:25,040 --> 00:08:28,000
[Indiscernible] led to painful 
results for the community. 

64
00:08:29,920 --> 00:08:37,040
By 2010, PyPI was suffering outages frequently 
enough that it was being commissioned and  

65
00:08:37,040 --> 00:08:48,960
maintained by volunteers. These outages directly 
impacted all users of PyPI, making installations  

66
00:08:48,960 --> 00:08:53,360
and downloads impossible. As well as stopping 
maintainers from being able to publish their  

67
00:08:53,360 --> 00:09:00,640
own releases. This post by Jacob Kaplan Moss 
was a common response on Twitter to let those  

68
00:09:00,640 --> 00:09:08,560
programmers keep moving. Mirroring did carry 
the community for many years, but ultimately,  

69
00:09:08,560 --> 00:09:18,400
what PyPI at this point a core piece of internet 
infrastructure needed was, well, infrastructure. 

70
00:09:20,960 --> 00:09:26,960
Hosting improvements in 2012 to migrate from 
those co-located to the cloud at the Oregon State  

71
00:09:26,960 --> 00:09:36,880
University Open Source Lab provided scalability to 
PyPI, after a decade of being online. This effort,  

72
00:09:36,880 --> 00:09:42,800
led by Noah Kantrowitz, did provide substantially 
more resources, taking off some of the pressure.  

73
00:09:44,080 --> 00:09:49,280
Unfortunately, the demands of the other 
Python services being hosted at OSUOSL  

74
00:09:49,280 --> 00:09:56,320
and the codebase itself meant that for 
parts of the backend, the scaling up  

75
00:09:56,320 --> 00:09:59,600
needed to meet the demand wasn't 
feasible in this environment. 

76
00:10:02,000 --> 00:10:11,840
And, in January 2013, the Python Wiki Host was 
attacked leading to a security breach. This led to  

77
00:10:11,840 --> 00:10:17,200
a commitment, by the Python Software Foundation, 
to fill a part-time systems administrator role  

78
00:10:17,200 --> 00:10:25,040
for the PSF infrastructure. Myself, and Benjamin 
W. Smith, applied for this role. Committing 130  

79
00:10:25,040 --> 00:10:30,400
hours of paid work per quarter and sharing a 24 
hour on-call rotation to respond to incidents.  

80
00:10:32,480 --> 00:10:38,400
We were hired for this paid role. It is ultimately 
what latched me into the Python infrastructure.  

81
00:10:40,080 --> 00:10:45,280
I began to put work into ensuring that the PSF 
infrastructure paged as little as possible.  

82
00:10:48,640 --> 00:10:56,080
Much of it was focused on PyPI. That didn't 
change that the hosting weren't keeping up. 

83
00:10:58,800 --> 00:11:04,400
Far and away, the largest infrastructure 
improvement PyPI has ever seen is the  

84
00:11:04,400 --> 00:11:11,120
implementation of Fastly as a CDN or Content 
Delivery Network. The commitment Fastly has made,  

85
00:11:13,280 --> 00:11:20,240
that they, themselves, benefit from, is 
outstanding. Donald Stuff flipped the switch  

86
00:11:20,240 --> 00:11:28,480
to move PyPI behind this CDN. Unfortunately, 
statistics for PyPI's usage before this point  

87
00:11:28,480 --> 00:11:35,120
are likely lost to history, but the 
statistics from Fastly, that we do have,  

88
00:11:35,120 --> 00:11:42,160
we can see just how dire the infrastructure 
situation was and how badly a CDN was needed. 

89
00:11:42,160 --> 00:11:49,040
In its first full day serving at the edge 
for PyPI, Fastly served 5.6 million requests.  

90
00:11:49,600 --> 00:11:53,680
And almost 800 gigabytes of data 
accessibility transfer on behalf of PyPI.  

91
00:11:54,480 --> 00:11:58,320
With over two-thirds of that traffic 
being diverted from our backends.  

92
00:11:59,600 --> 00:12:05,120
In its first full month, we start to get a better 
idea for the scale that PyPI was already at in  

93
00:12:05,120 --> 00:12:12,000
2013. If you have a calculator, that's around 
85 requests per second, which isn't unheard  

94
00:12:12,000 --> 00:12:18,800
of for a single nowadays, but given the PyPI 
implementation, it was very impressive it was  

95
00:12:18,800 --> 00:12:24,400
staying online at all before the CDN. More 
stark is that data accessibility transfer.  

96
00:12:25,040 --> 00:12:29,200
If you've paid a hosting bill, I'm guessing 
you might be getting nervous right now. 

97
00:12:31,600 --> 00:12:37,360
The success of PyPI's CDN rollout with Fastly 
and the reliability provided for installs to  

98
00:12:37,360 --> 00:12:42,240
continue even when PyPI, itself, was 
down, allowed for the deprecation.  

99
00:12:49,680 --> 00:12:53,840
This also addressed the outstanding security 
concerns of the mirroring infrastructure. 

100
00:12:55,840 --> 00:13:02,400
In late 2013, the PSF and PyPI were some of the 
first using Rackspace's Open Source Support.  

101
00:13:04,240 --> 00:13:08,880
This, combined with me committed and 
paid time working on the infrastructure,  

102
00:13:08,880 --> 00:13:13,440
led to a complete rebuild on 
to Rackspace's public cloud.  

103
00:13:14,720 --> 00:13:22,560
There was a failover, capable 
backend as PyPI continued to grow. 

104
00:13:24,400 --> 00:13:29,040
While outages naturally continued for 
PyPI itself, they became less frequent.  

105
00:13:32,400 --> 00:13:38,160
Outages for the hosting solutions chosen by 
maintainers, who opted to host their own files,  

106
00:13:38,160 --> 00:13:46,080
also led to confusion and frustration. 
Supporting direct uploads to PyPI,  

107
00:13:46,080 --> 00:13:53,040
for it to host, was just an additional feature. 
In the 13 years that PyPI had been online, many  

108
00:13:53,040 --> 00:13:59,520
externally hosted files had either gone missing or 
were hosted on flaky or insecure infrastructure. 

109
00:14:00,880 --> 00:14:04,240
With the reliable CDN, and 
less outage prone backends,  

110
00:14:04,960 --> 00:14:10,960
10 years after PyPI started to 
support uploads, PEP 470 was proposed.  

111
00:14:14,880 --> 00:14:20,720
Its acceptance effectively completed the 
core feature set we know from PyPI today. 

112
00:14:22,160 --> 00:14:27,680
So, overall, with commitments to the 
infrastructure by volunteer contributors,  

113
00:14:27,680 --> 00:14:36,800
community based support, companies, via In 
Kind sponsorship, 2012 to 2014 provides a  

114
00:14:36,800 --> 00:14:48,400
foundational shift and grow, it did. With careful 
tweaks and optimizations, like interning caching,  

115
00:14:49,360 --> 00:14:59,360
migration of file storage to Amazon S3, removing 
noncritical features, PyPI saw a continued growth  

116
00:14:59,360 --> 00:15:04,880
trend from mid 2013 on to April of 2018 
with minimal for us to keep the lights on.  

117
00:15:06,400 --> 00:15:13,520
A 4,000% growth in daily request volume and 5, 
000 growth in data accessibility transferred  

118
00:15:13,520 --> 00:15:21,440
during that time. So, this brings us to the end of 
PyPI's first 15 and a half years. Collectively, we  

119
00:15:21,440 --> 00:15:28,480
can call everything up to this point, the history 
of PyPI. Even with the bumps along the way,  

120
00:15:28,480 --> 00:15:42,000
it's hard to call it anything but successful.
PyPI hosts 140,000 projects. 975,000 releases  

121
00:15:42,000 --> 00:15:49,600
with 1.3 million release files. Just over a 
terabyte of data accessibility. I think it's  

122
00:15:49,600 --> 00:15:55,280
worthy of a moment to reflect just how far PyPI 
came on a shoestring budget and a skeleton crew.  

123
00:15:56,720 --> 00:16:07,840
For over a decade and a half, PyPI was able to 
grow exponentially. We owe immense gratitude.  

124
00:16:11,680 --> 00:16:16,960
By this time, the codebase and infrastructure 
behind PyPI were starting to show their age. 

125
00:16:18,320 --> 00:16:23,040
It was becoming harder and harder to onboard 
new contributors into a codebase that was so  

126
00:16:23,040 --> 00:16:32,880
architected with valid assumptions from 2002. None 
of the robust frameworks or standards that we take  

127
00:16:32,880 --> 00:16:39,360
today that we take for granted today existed. PyPI 
used what was available and logical at the time.  

128
00:16:40,640 --> 00:16:44,160
Even for those who contributed 
to PyPI on a regular basis,  

129
00:16:44,720 --> 00:16:50,960
it made building out new features cumbersome and 
a lack of thorough tests made deployments risky.  

130
00:16:53,040 --> 00:16:59,680
This entire era of PyPI's history was mainly about 
keeping the lights on rather than developing new  

131
00:16:59,680 --> 00:17:05,360
features to support the Python community.
For a fairly comprehensive summary for the  

132
00:17:05,360 --> 00:17:13,040
state of PyPI, I recommend checking out Donald 
Stufft's post, Powering the Python Package Index. 

133
00:17:15,680 --> 00:17:21,040
So, that brings us to the current era of PyPI 
and a new set of challenges facing the service.  

134
00:17:22,640 --> 00:17:30,160
By now, most of us are familiar with this 
[Indiscernible] of PyPI. The transition into  

135
00:17:30,160 --> 00:17:34,960
the current era of PyPI may not have registered 
as the massive undertaking it has been. 

136
00:17:36,480 --> 00:17:42,880
This era began rather quietly with Donald Stufft 
joining Hewlett Packard Enterprise in 2015.  

137
00:17:43,680 --> 00:17:48,720
During Donald's 18 months there, Hewlett 
Packard allowed him a full time commitment  

138
00:17:48,720 --> 00:17:54,720
to Python Packaging. In addition to a plethora of 
work done throughout the Python Packaging Space,  

139
00:17:56,080 --> 00:17:59,840
he did a complete rewrite of PyPI codebase.  

140
00:18:03,600 --> 00:18:10,720
Today, that effort is what powers PyPI and its 
name is warehouse. Donald set out with ambitious,  

141
00:18:10,720 --> 00:18:23,120
but clear goals. Some of the most notable are 
comprehensive test coverage, extreme cachability,  

142
00:18:23,120 --> 00:18:30,720
a modern web framework, security mindedness, 
creating a maintainable and extensive codebase.  

143
00:18:31,680 --> 00:18:38,880
For anyone who's been a part of a full write 
of an existing codebase, with ambitious goals,  

144
00:18:38,880 --> 00:18:46,560
it isn't trivial or fast, especially with 
a committed team of approximately one.  

145
00:18:47,440 --> 00:18:57,440
So, ultimately, Warehouse could do most of the 
things, but didn't have the features necessary. 

146
00:19:01,360 --> 00:19:07,360
At around this time, having identified finishing 
Warehouse would require features for development,  

147
00:19:07,360 --> 00:19:14,320
design and project management, Donald, along with 
others, co-founded the Python Software Foundation  

148
00:19:14,320 --> 00:19:20,080
Packaging Working Group. This working group was 
formed to support large efforts in improving,  

149
00:19:20,800 --> 00:19:27,920
the packaging system through fundraising and 
grant oversight. By forming this work group,  

150
00:19:27,920 --> 00:19:32,720
the PSF was able to provide crucial, 
administrative support necessary to  

151
00:19:32,720 --> 00:19:47,520
apply for and delivery on substantial grants and 
awards for work in the Python Packaging Program. 

152
00:19:48,400 --> 00:19:56,720
In their Foundational Technology, a $170,000 
award was funded to finish parity for Warehouse,  

153
00:20:00,160 --> 00:20:01,760
decommission the original service.  

154
00:20:02,880 --> 00:20:08,800
It was accomplished under budget with a team 
of three developers and a project manager.  

155
00:20:10,240 --> 00:20:14,480
In addition to the deliverables of the 
award, there were incredible byproducts  

156
00:20:17,200 --> 00:20:25,040
and extensive improvements to the documents. In 
large part, the success of this work is due to  

157
00:20:25,040 --> 00:20:31,920
the influence of Smer, our project manager.
So, from my perspective, the true start of  

158
00:20:31,920 --> 00:20:38,320
the current era of PyPI began on April 
16, Warehouse went live as "the" PyPI. 

159
00:20:40,720 --> 00:20:44,400
The results of this current era 
have consistently met the goals.  

160
00:20:46,800 --> 00:20:53,280
Make it appealing via grant and award 
applications by the Packaging Working Group.  

161
00:20:57,040 --> 00:21:03,200
The PSF Packaging Working Group has 
received funding for multiple projects.  

162
00:21:04,880 --> 00:21:13,840
In 2018, an $80, 000 contract was proposed to 
the Open Technology Fund's core infrastructure  

163
00:21:13,840 --> 00:21:23,680
program. It could be completed in 2019 and brought 
some of the first, major features. It delivered  

164
00:21:23,680 --> 00:21:34,400
Multifactor Authentication, via TOTP and WebAuthn 
and audit logging for projects and user accounts. 

165
00:21:35,440 --> 00:21:41,120
To help make PyPI more readily available to the 
global Python community, this work also delivered  

166
00:21:41,120 --> 00:21:51,680
as accessibility audit and accessibility fixes, 
it enabled localization features. So now, among  

167
00:21:51,680 --> 00:22:00,320
the 13 languages PyPI is available in, you can 
browse in German, Spanish, and even Australian. 

168
00:22:02,320 --> 00:22:06,400
Underway right now, thanks to 
$100,000 from Facebook Research,  

169
00:22:07,840 --> 00:22:13,680
is the update framework for PEP 458. 
This gift also funded development  

170
00:22:13,680 --> 00:22:18,800
of infrastructure of automated 
analysis and prototype malware. 

171
00:22:20,800 --> 00:22:26,880
Worthy of note, this track record enabled 
the Packaging Working Group to simultaneously  

172
00:22:26,880 --> 00:22:34,720
and successfully apply for both the Mozilla Open 
Source Support Awards and the Chan Zuckerberg  

173
00:22:34,720 --> 00:22:41,920
Initiative's Essential Open Source Software 
for Science, funded $407,000 worth of work  

174
00:22:43,280 --> 00:22:47,520
on pypi in 2020.
There reason I note this, it  

175
00:22:47,520 --> 00:23:00,000
is the first time that the PSF Packaging Working 
Group secured funding. This is a major milestone  

176
00:23:00,000 --> 00:23:06,640
in fulfilling the Packaging Working Group's goals.
Now, all of these improvements are incredible,  

177
00:23:07,440 --> 00:23:12,640
and as a community, we are benefiting from 
there. Some of the biggest wins, since PyPI  

178
00:23:12,640 --> 00:23:18,320
was relaunched, has been in the maintenance of 
PyPI itself. A new administrator, bringing our  

179
00:23:18,320 --> 00:23:30,720
total up to three. A team of five moderators that 
can help triage, a new team of issue triagers.  

180
00:23:33,760 --> 00:23:38,800
A new team of translation project maintainers 
and 100 contributors to translating PyPI,  

181
00:23:40,240 --> 00:23:43,920
nearly 80 unique contributors to 
the PyPI codebase since its release. 

182
00:23:45,120 --> 00:23:52,320
Once again, I'm glad to be able to point you 
to Dustin Ingram, who does an excellent job  

183
00:23:52,320 --> 00:23:59,360
summarizing the current state of PyPI in this 
era. So, all in all, I think celebration of the  

184
00:23:59,360 --> 00:24:04,640
results we've seen is warranting. As a codebase, 
the effort put in has rewarded us with something  

185
00:24:04,640 --> 00:24:11,840
extensive and maintainable that truly is PR's 
welcome. Not to say this is a happily ever  

186
00:24:11,840 --> 00:24:17,840
after story yet. We still have some fundamental 
challenges that we face in the current era.  

187
00:24:18,480 --> 00:24:23,200
Many of these are open questions and frankly, 
we don't have solid answers for how to overcome  

188
00:24:23,200 --> 00:24:29,920
them at this time. The first is user support. 
While our team of moderators has been crucial  

189
00:24:29,920 --> 00:24:38,560
in helping us state up to date with requests, 
there still remains a large backlog of requests,  

190
00:24:38,560 --> 00:24:46,800
which are either security sensitive or require 
access to resolve. There is only one person  

191
00:24:46,800 --> 00:24:52,880
who's paid to support PyPI as it continues to 
experience a growing userbase, not to mention,  

192
00:24:52,880 --> 00:24:59,840
it's only a slice of that paid person's time. 
So many of these backlogs are blocked on them. 

193
00:25:01,200 --> 00:25:07,280
The next is project management. PyPI has seen its 
greatest strides forward under reliable project  

194
00:25:07,280 --> 00:25:12,800
manager. All of our success, we've fulfilled 
grants and award said, incorporated some amount  

195
00:25:12,800 --> 00:25:20,880
of project management. In 2020, we had to take 
a step back from applying for and taking work,  

196
00:25:21,520 --> 00:25:32,160
the project management was taken a toll on other 
initiatives. We have one project and substantial  

197
00:25:32,160 --> 00:25:36,320
funding from Google that's waiting until we're 
in a place to responsibly start that work. 

198
00:25:38,320 --> 00:25:46,640
You might have noticed the graph, 
indeed, that pattern has only continued.  

199
00:25:47,280 --> 00:25:59,840
As of the end of June, PyPI hosts 312,000 
projects. 224 releases with 2.27 million files.  

200
00:26:05,040 --> 00:26:08,800
The growth and the size of data accessibility 
storage has far outpaced most of our other  

201
00:26:08,800 --> 00:26:16,080
metrics. While we've identified some of the 
growth and working with publishers to address  

202
00:26:16,080 --> 00:26:23,120
their specific use cases, large 
files, specifically binary wheels, it  

203
00:26:24,560 --> 00:26:33,440
is taking up moderator time and it makes 
turning the knob further unattainable. 

204
00:26:37,040 --> 00:26:41,360
Mostly, these large distributions are reaching to 
the point they're having [Indiscernible] affects,  

205
00:26:41,920 --> 00:26:50,960
most notably in data accessibility transfer. 
In Quarter 2, 2021, PyPI served 62 Petabyte,  

206
00:26:53,440 --> 00:26:59,120
even with 99% and Fastly providing that 
data accessibility transfer for free,  

207
00:26:59,840 --> 00:27:12,080
the 1% of our traffic is substantial that 
it needed a quick migration from Amazon S3  

208
00:27:12,080 --> 00:27:24,000
to Google Cloud storage in order to be better 
stewards of the donated resources that make  

209
00:27:24,000 --> 00:27:29,360
operating PyPI possible. Data accessibility 
transfer, alone, was quickly consuming the  

210
00:27:29,360 --> 00:27:33,840
generous credits we had with Amazon that 
allowed us to run PyPI backends there. 

211
00:27:34,480 --> 00:27:40,800
So, I'd like to take another opportunity to extend 
a heartfelt thank you to Fastly, for everyone who  

212
00:27:40,800 --> 00:27:48,400
works on, maintains, supports and uses PyPI. While 
our growth has been a challenge, only 1% of that  

213
00:27:48,400 --> 00:27:54,640
actually growth is felt at the PyPI backends. 
This has enabled us to be where we are today,  

214
00:27:54,640 --> 00:28:02,080
with the limited resources that we have. What 
PyPI does would not be possible without Fastly. 

215
00:28:02,080 --> 00:28:05,760
So that's more or less what PyPI has 
been up to over the past few years.  

216
00:28:07,600 --> 00:28:12,880
And frankly, at this point, it's safe 
to say that the codebase, behind PyPI,  

217
00:28:12,880 --> 00:28:19,360
has reached a point of sustainability and that is 
the greatest success of this current era of PyPI.  

218
00:28:20,240 --> 00:28:24,800
But the sustainability of the software is 
independent of the sustainability of operating  

219
00:28:24,800 --> 00:28:30,880
the service. This challenge is the one that I 
believe must be addressed in the next era of PyPI. 

220
00:28:32,960 --> 00:28:36,800
So, I know it's taken some time 
for us to get to this point,  

221
00:28:36,800 --> 00:28:42,800
but it's the crux of what I want to motivate 
here. To activate the entire Python community  

222
00:28:43,520 --> 00:28:48,640
to think about how we can collectively enable 
the operational sustainability of PyPI into  

223
00:28:48,640 --> 00:28:55,120
the future. When I use this term, I'm focusing 
primarily on what it takes to keep PyPI online  

224
00:28:55,120 --> 00:29:00,640
and available for the community. Realistically, we 
can break this down into three major components,  

225
00:29:01,520 --> 00:29:09,353
the human labor necessary, respond to incidents 
and support users, the financials to cover month  

226
00:29:09,353 --> 00:29:17,920
to month expenditures and the organizational 
structure to manage those concerns. 

227
00:29:20,160 --> 00:29:25,920
The Python Software Foundation has served as 
the organizational backstop for PyPI since its  

228
00:29:25,920 --> 00:29:33,600
inception. Initially, this didn't have much. 
But it has taken on more and more of a role.  

229
00:29:36,560 --> 00:29:42,160
This not only includes the work the Packaging 
Working Group has done, but hiring me.  

230
00:29:43,200 --> 00:29:49,040
Now again, I have to make clear that my full 
time job is not just PyPI. It's a relatively  

231
00:29:49,040 --> 00:29:54,880
small fraction of what I do on a day to day 
basis. All other labor that goes into PyPI  

232
00:29:54,880 --> 00:30:00,480
is community and volunteer driven. Some is 
done by employees of companies, but those  

233
00:30:00,480 --> 00:30:07,280
engagements tend to be driven by scratching an 
itch or occasionally, there are people paid full  

234
00:30:07,280 --> 00:30:15,280
time to contribute to open source. But the 
current state is we only have me and only  

235
00:30:15,280 --> 00:30:20,720
part of my time paid to operate the service.
I don't think we can call that sustainable.  

236
00:30:21,520 --> 00:30:25,440
If you know how often I ride a bicycle, 
you might call it downright risky. 

237
00:30:27,040 --> 00:30:31,600
Now, excluding my salary and overhead 
internal of the PSF    that we'll  

238
00:30:31,600 --> 00:30:44,800
discuss shortly the monthly cost that 
the PSF pays is $0. Well, kind of...as  

239
00:30:44,800 --> 00:30:50,720
it is, we're grateful to have these companies who 
provide In Kind sponsorship of PyPI by letting  

240
00:30:50,720 --> 00:31:02,400
us use their services. They are off the shelf 
offerings or formalized sponsorship agreements  

241
00:31:02,400 --> 00:31:09,600
with the PSF. Managing sponsorships creates some 
overhead costs within the PSF that we absorb.  

242
00:31:12,560 --> 00:31:18,160
But in the interest of sustainability, we have to 
keep track of the value of these donated services.  

243
00:31:18,160 --> 00:31:21,360
So, let's look at August of 201, just last month.  

244
00:31:23,040 --> 00:31:30,880
Here's what it would cost if the PSF paid 
out of pocket, $8,000 in servers and $5, 000  

245
00:31:33,920 --> 00:31:42,960
to run our analytics pipeline, $4,200 to store 
upload files and the data accessibility transfer  

246
00:31:45,360 --> 00:31:54,400
and $1,000 in tooling from monitoring, error 
logging, telemetry. In the last month, $18,  

247
00:31:54,400 --> 00:32:03,680
000 for PyPI. That is excluding what our 
CDN costs would be if we paid Fastly retail. 

248
00:32:06,800 --> 00:32:16,080
And if we remember back earlier growth graphs, the 
costs is following a similarly alarming pattern. 

249
00:32:18,880 --> 00:32:24,320
So, I think the hard truth is that we 
must admit that the current status of PyPI  

250
00:32:24,320 --> 00:32:30,880
is not even remotely financially sustainable. If 
there came a time that one of our In Kind sponsors  

251
00:32:30,880 --> 00:32:39,840
abruptly ceased providing services, PyPI is 
designed, from the ground up, to extract itself  

252
00:32:39,840 --> 00:32:49,840
from operating on any brand of dependency. 
While we work to implement a new integration  

253
00:32:49,840 --> 00:32:55,440
and migrate to a new provider, the PSF would be 
on the hook to pay the bill for those services  

254
00:32:55,440 --> 00:33:03,040
in the interim. Now that is not a problem. 
The PSF has worked diligently to build a  

255
00:33:03,040 --> 00:33:09,200
reserve of funds over the last decade to 
ensure that critical parts can survive.  

256
00:33:11,600 --> 00:33:17,600
So, considering the retail costs to operate PyPI, 
all but one of those services would be tolerable  

257
00:33:17,600 --> 00:33:24,400
for the PSF to cover for a few months at 
most. If they had to pay PSF CDN bill,  

258
00:33:24,960 --> 00:33:33,120
almost regardless of the provider, it would be 
more than 50% of our reserve in a single month. 

259
00:33:35,840 --> 00:33:41,600
The impact that 2020 had on the PSF's 
reserves is obvious. For the first time,  

260
00:33:41,600 --> 00:33:46,800
a major block of those funds was necessary 
to continue the operations of the nonprofit.  

261
00:33:47,520 --> 00:33:51,840
Primarily, as a result of the lost 
revenue from PyCon 2020's cancellation.  

262
00:33:52,640 --> 00:34:01,920
In June, assets were around $4 million. The 
very risk to our financial sustainability  

263
00:34:01,920 --> 00:34:09,280
directly risks our organizational sustainability.
So, in assessing the operational sustainability  

264
00:34:09,280 --> 00:34:16,880
of PyPI, it's hard to make a case we have such 
sustainability. In the aftermath of PyCon 2020's  

265
00:34:16,880 --> 00:34:25,760
cancellation, they are committed to ensuring 
the nonprofit. We believe those concerns can  

266
00:34:25,760 --> 00:34:33,840
be accomplished, hand in hand. While we haven't 
made any firm plans, there are things already  

267
00:34:33,840 --> 00:34:39,120
in motion with the intent to build more towards 
more sustainable PyPI and more sustainable PSF.  

268
00:34:40,720 --> 00:34:48,880
The PSF revamped and relaunched our sponsorship 
program related to PyPI. As a result,  

269
00:34:48,880 --> 00:34:54,400
a portion of sponsorship funds are earmarked 
for PyPI. This is a departure from the past,  

270
00:34:54,400 --> 00:34:57,600
where all sponsorship funds were 
part of the PSF general fund. 

271
00:35:01,920 --> 00:35:09,520
Also on the sponsorship front, Bloomberg, in 
addition to their direct sponsorship of the PSF,  

272
00:35:09,520 --> 00:35:22,480
committed to $250,000 in funding for Python 
Packaging Manager role. Shamika Mohanan started  

273
00:35:22,480 --> 00:35:30,000
the PSF as a packaging project manager. Shamika 
will be working on efforts similar to what the  

274
00:35:30,000 --> 00:35:39,200
Packaging Working Group has undertaken 
and [Indiscernible] with the ecosystem.  

275
00:35:42,080 --> 00:35:47,760
But a critical component of Shamika's work will be 
in helping the PSF build a more sustainable PyPI. 

276
00:35:48,960 --> 00:35:55,520
Now, this is the place where we face the most 
uncertainty in looking forward for PyPI. I hope,  

277
00:35:55,520 --> 00:36:01,040
by this point, we can agree that while PyPI has 
made enormous strides in becoming sustainable,  

278
00:36:03,120 --> 00:36:06,560
operating the service, as-is, 
simply can't be sustained.  

279
00:36:08,080 --> 00:36:14,720
So what are our options and what will we 
explore? Perhaps the status quo is sufficient.  

280
00:36:15,360 --> 00:36:18,560
We've seen success in securing 
funding for big lift efforts,  

281
00:36:19,920 --> 00:36:26,240
retaining In Kind sponsorship and squeaking by 
with the volunteer and paid labor that's available  

282
00:36:26,240 --> 00:36:32,000
and honestly, no matter what, we will probably 
continue to do these things, but historically,  

283
00:36:32,000 --> 00:36:37,920
that has led to what we see over the last 
18 months, when large scale funded efforts,  

284
00:36:38,720 --> 00:36:45,440
the people who are paid to do that work go on 
to do other paid work. The effort to reconstruct  

285
00:36:45,440 --> 00:36:50,080
teams, when the next round of funding is 
available, is duplicated time and again. 

286
00:36:51,360 --> 00:37:00,000
We hope that having a packaging project manager 
can alleviate some of the duplication, but we need  

287
00:37:00,000 --> 00:37:08,320
to hire or rehire contractors. We're very lucky 
to be a high profile enough service that retaining  

288
00:37:08,320 --> 00:37:20,160
In Kind sponsorships isn't necessary hard.
Lastly, the results of relying on a skeleton  

289
00:37:20,160 --> 00:37:25,760
crew are simply not sufficient to keep up 
with the growth that PyPI is seeing. So,  

290
00:37:25,760 --> 00:37:30,160
unfortunately, the status quo just 
isn't working. It leaves a stagnation  

291
00:37:30,160 --> 00:37:34,960
and even getting behind on improvements to 
PyPI, because what limited resources we have  

292
00:37:36,720 --> 00:37:39,440
focus on maintaining the service 
and keeping the lights on.  

293
00:37:40,720 --> 00:37:45,440
So maybe it's a case where small iterations 
and changes could make up the difference?  

294
00:37:46,400 --> 00:37:51,360
More talks like these, features baked into PyPI 
that make clear what the true cost of running  

295
00:37:51,360 --> 00:37:57,120
the service is and ramping up donation efforts 
and seeking sponsorships to support the service.  

296
00:37:57,760 --> 00:38:02,960
And here, we've already started on some 
of those things and know    and we know  

297
00:38:02,960 --> 00:38:07,280
that they'll remain important in supporting the 
Python community, at large, and maintain PyPI. 

298
00:38:09,440 --> 00:38:17,040
The barrier I keep running up against is that all 
of the efforts we've undertaken and all of the  

299
00:38:17,040 --> 00:38:23,760
iterative improvements are not directly tied 
to the growth of PyPI. Of course, as Python,  

300
00:38:23,760 --> 00:38:29,920
and thus PyPI has grown in popularity, our ability 
to fund projects, attract new contributors and  

301
00:38:31,680 --> 00:38:37,680
have sponsorships have grown. They're just 
not growing in the same manner. These graphs  

302
00:38:37,680 --> 00:38:43,360
should illustrate that. The overall trend for PSF 
sponsorship, compared to the trend for usage on  

303
00:38:43,360 --> 00:38:48,880
PyPI is illustrated just by data accessibility 
transfer, indicate we have one that is linear,  

304
00:38:48,880 --> 00:38:54,880
at best. And one that is, well, exponential. 
There's a case to be made for establishing  

305
00:38:54,880 --> 00:39:01,120
some mechanism by which the funding available 
to PyPI correlates to its popularity and use. 

306
00:39:03,920 --> 00:39:08,400
And that's what motivates us to begin considering 
the ways that we can responsibly implement  

307
00:39:08,400 --> 00:39:17,920
features on PyPI that will provide real value to 
corporations. By creating a new revenue stream  

308
00:39:17,920 --> 00:39:24,960
for the PSF, we could become less resilient 
to fund our mission. Any revenue generated by  

309
00:39:24,960 --> 00:39:33,120
PyPI must initially go towards addressing some 
of the challenges we have. An entire paid team  

310
00:39:33,120 --> 00:39:39,680
working to operate PyPI and support all users 
of the service. Over time, if things go well,  

311
00:39:39,680 --> 00:39:44,880
it could reach the point where PyPI is 
sustaining itself. That's our long shot goal. 

312
00:39:46,640 --> 00:39:52,000
And preliminary discussions exploring this topic, 
there's clear consensus on some requirements.  

313
00:39:53,440 --> 00:39:58,400
None of these features should impact a 
reasonable use cases of PyPI. However,  

314
00:39:58,400 --> 00:40:03,920
being realistic, there are almost certainly some 
use cases that are extremely niche or far enough  

315
00:40:03,920 --> 00:40:14,800
beyond the limits of the service that may require 
intervention. All paid features will have to be  

316
00:40:14,800 --> 00:40:19,920
available to community organizations at no cost. 
We don't see the point in developing a feature  

317
00:40:19,920 --> 00:40:23,840
for PyPI that doesn't simultaneously 
support the community that we serve. 

318
00:40:24,880 --> 00:40:29,600
And lastly, all of these features should be 
developed on actual needs in the community.  

319
00:40:31,280 --> 00:40:39,680
Again, none of this is set in stone. Shamika 
and myself will be working on outreach needed  

320
00:40:39,680 --> 00:40:44,000
to fulfill the requirements and understand 
what a roadmap for this initiative looks like.  

321
00:40:45,840 --> 00:40:52,640
There are significant risks to getting a paid PyPI 
at all. It must comply with our nonprofit status.  

322
00:40:53,360 --> 00:41:02,800
It may alienate volunteers. Our sponsors, both 
In Kind and direct, may no longer support us.  

323
00:41:03,440 --> 00:41:07,600
And frankly, there they simply may not be 
the demand we anticipate for such a feature.  

324
00:41:08,800 --> 00:41:13,920
So, that's the big reveal. Although for 
those who pay close attention to PSF,  

325
00:41:13,920 --> 00:41:21,760
the idea was leaked in our annual impact report.
Now, I want to leave you with some ways that  

326
00:41:21,760 --> 00:41:26,640
you can get involved with all of this. I'll have 
URLs on the next slide where you can go for each.  

327
00:41:27,600 --> 00:41:33,760
The first is to donate. The PSF is and will 
remain a nonprofit. Every donation we receive  

328
00:41:33,760 --> 00:41:38,160
goes to continuing to fulfill our mission, 
including operating PyPI for our community.  

329
00:41:39,280 --> 00:41:47,360
Sponsor, encouraging your company to sponsor 
the PSF can amplify impact with more continuity.  

330
00:41:50,960 --> 00:41:55,760
Contribute. PyPI is and will continue 
to run as an open source project  

331
00:41:55,760 --> 00:42:00,800
and open source projects need much more than 
software development. We need help and input,  

332
00:42:04,320 --> 00:42:10,720
not just from backend development. If you're 
not familiar with the Python discussion forum,  

333
00:42:10,720 --> 00:42:16,000
but are interested in packaging in 
PyPI, take time to find us there,  

334
00:42:16,000 --> 00:42:22,160
participate in discussions and be on the lookout 
for announcements as we hit research phases. 

335
00:42:24,560 --> 00:42:33,840
And with that, I want to thank the organizers of 
PyConline Australia, the volunteers. And finally,  

336
00:42:33,840 --> 00:42:41,520
you, the attendees for taking the time to be here.
>> And, we're back. We're back. Excellent. Look,  

337
00:42:42,160 --> 00:42:48,640
thanks, Ee. That was fantastic. Look, I hope it 
gives some folks some ideas of where we've been,  

338
00:42:48,640 --> 00:42:53,440
how they might contribute and 
some of the very exciting future  

339
00:42:53,440 --> 00:43:01,840
of the Python Package Index in the future.
And, Ee is joining us from look, I just want  

340
00:43:01,840 --> 00:43:09,280
to say that as a child of the 1970s, having 
a conference Keynote call in from a campsite  

341
00:43:09,280 --> 00:43:18,480
to chat with our about 700 attendees is frankly 
mind blowing. So, welcome, Ee, to the conference. 

342
00:43:18,480 --> 00:43:22,160
EE DURBIN: I'm actually 
baffled that it worked at all. 

343
00:43:24,000 --> 00:43:26,320
[Laughter].
>> All right. Well, look, what are  

344
00:43:26,320 --> 00:43:30,800
you up to there, Ee? Just out of curiosity?
EE DURBIN: I'm going to ride my bike  

345
00:43:30,800 --> 00:43:39,840
in an excruciating long distance 
tomorrow. I'm camping before. 

346
00:43:39,840 --> 00:43:49,200
>> Fantastic. I'm an avid follower of your 
bicycle antics. So, have a heck of a fun ride. 

347
00:43:49,200 --> 00:43:52,480
EE DURBIN: Thank you.
>> We have a bunch of  

348
00:43:52,480 --> 00:43:55,600
really good questions that have come 
in from people watching the Keynote. 

349
00:43:57,920 --> 00:44:06,240
Look, the highest ranked question is, are there 
any security concerns when downloading a random  

350
00:44:06,240 --> 00:44:10,560
package from the Python Package 
Index? How careful should I be? 

351
00:44:10,560 --> 00:44:20,160
EE DURBIN: Yes. Simply choosing a random package 
to download comes with a random amount of risk.  

352
00:44:21,360 --> 00:44:26,320
Because PyPI is designed to be a 
public and freely available service  

353
00:44:26,320 --> 00:44:32,560
and tends to remain that way into the future, 
anyone is able to upload anything they like there.  

354
00:44:33,600 --> 00:44:36,160
And because of the   
[Laughter]. 

355
00:44:36,160 --> 00:44:41,680
Constraints of packages and Python and the tooling 
that exists, that means that executable code,  

356
00:44:41,680 --> 00:44:48,640
whether it's at install time or runtime, is always 
a risk, just in the same way that any bug or,  

357
00:44:48,640 --> 00:44:54,000
you know, system crashing feature could be, as 
well. So, the amount of risk that is reasonable,  

358
00:44:55,680 --> 00:45:03,840
when you're installing something that you know 
or that that has good what's the word for that?  

359
00:45:04,960 --> 00:45:13,600
Whatever. Has a good lineage. It is fairly 
low. Choosing a random package off of the index  

360
00:45:14,480 --> 00:45:20,160
is kind of risky because it is 
just as    it's just as free as  

361
00:45:20,160 --> 00:45:24,720
the internet in that way.
>> Yep. Very solid advice.  

362
00:45:26,480 --> 00:45:36,560
Our next question, I've always wondered this. 
Why did Python not have hierarchical namespaces?  

363
00:45:40,560 --> 00:45:43,920
I take responsibility for that.
[Laughter]. 

364
00:45:45,440 --> 00:45:48,800
What do you say, Ee?
EE DURBIN: [Audio cutting in and out]  

365
00:45:51,520 --> 00:45:54,400
and there was no need, sort 
of assumed for such a thing.  

366
00:45:55,360 --> 00:46:01,760
Thus, it was implemented and delivered immense 
value for a long time without namespaces.  

367
00:46:03,600 --> 00:46:09,040
Namespaces are a good idea, we should have 
more of them, is a very important string  

368
00:46:09,040 --> 00:46:14,080
that most of us are somewhat familiar with, 
from    from the [Indiscernible] Python. But,  

369
00:46:16,160 --> 00:46:22,320
why doesn't it have them now is effectively the 
question of the cost and time to implement them  

370
00:46:22,320 --> 00:46:29,520
and the return on that. And so, effectively, it's 
something we've recognized as having an importance  

371
00:46:29,520 --> 00:46:35,200
in the future of PyPI, not only because the amount 
of space on the index is slowly being exhausted,  

372
00:46:36,560 --> 00:46:42,080
as far as just the number of strings go. Or the 
number of usable strings go. But also, because  

373
00:46:42,080 --> 00:46:47,840
it creates credence for projects. So, if a 
corporation is able to have a namespace that  

374
00:46:47,840 --> 00:46:54,240
matches their corporate [no audio].
>>  

375
00:46:54,240 --> 00:47:00,080
Oh, no.
EE DURBIN: [No audio].  

376
00:47:00,080 --> 00:47:03,280
Similarly, community projects, like Django,  

377
00:47:04,400 --> 00:47:10,160
should absolutely have the ability to publish 
under their own namespace so that's the sort  

378
00:47:10,160 --> 00:47:14,800
of thing that we're really seeing as a very 
high potential for paid features on PyPI,  

379
00:47:15,840 --> 00:47:22,240
is to allow corporations to own their namespaces 
for a cost and allow community projects to own  

380
00:47:22,240 --> 00:47:24,840
their namespaces at no cost.
>>  

381
00:47:29,280 --> 00:47:34,480
We have okay. An amazing Keynote. That's 
more of a comment than a question,  

382
00:47:34,480 --> 00:47:38,720
but it does come with a question 
after it. Amazing Keynote. Thank you.  

383
00:47:40,160 --> 00:47:48,880
Just to kind of reiterate, what support does 
PyPI need from the community in 2021 and beyond? 

384
00:47:48,880 --> 00:47:54,000
EE DURBIN: Yeah. Certainly. I think there 
are a couple crucial things people can do  

385
00:47:54,000 --> 00:47:58,640
immediately, tomorrow and the next week and then 
some things that are definitely important over  

386
00:47:58,640 --> 00:48:05,920
the coming weeks, months, years. Immediately is 
talk to your company, if you work at a company  

387
00:48:05,920 --> 00:48:12,640
that uses Python, and thus uses PyPI almost 
assuredly, talk to them about sponsoring the PSF.  

388
00:48:13,600 --> 00:48:17,440
It's the by far, the most direct 
way that you can have impact today. 

389
00:48:19,200 --> 00:48:22,800
In the coming days and weeks, it 
could be contributing to the codebase,  

390
00:48:23,920 --> 00:48:29,120
helping us to resolve things, taking time 
to get familiar with it or deliver a feature  

391
00:48:29,120 --> 00:48:37,520
or bug fix that is valuable to the project.
In the near term, there's a pretty big research  

392
00:48:37,520 --> 00:48:44,800
phase around how to responsibly and appropriately 
approach paid features on PyPI so engaging that  

393
00:48:44,800 --> 00:48:50,240
is a really great way to get involved. 
And then long term, start paying for  

394
00:48:50,240 --> 00:48:58,560
those features if they make sense for you.
>> Fantastic. We are just pushing over 10:00,  

395
00:48:58,560 --> 00:49:02,320
but I have one more question, which 
is a little bit dear to my heart  

396
00:49:02,320 --> 00:49:07,360
and I wonder if you might have a quick answer to 
it. Is there any data accessibility on how much of  

397
00:49:07,360 --> 00:49:15,840
the traffic is caused by CI jobs downloading 
the same package hundreds of times a day? 

398
00:49:15,840 --> 00:49:20,960
EE DURBIN: I don't think we have direct 
data accessibility for that. But, there  

399
00:49:20,960 --> 00:49:26,320
are features in pypi that were added some time 
ago, that will help us answer those questions  

400
00:49:26,880 --> 00:49:29,840
and so if you're interested in 
answering a question yourself,  

401
00:49:31,120 --> 00:49:36,560
you can go look at the documentation for the 
BigQuery data set that Google helps us publish  

402
00:49:37,520 --> 00:49:45,520
for PyPI download statistics. And one of the 
little bits of metadata is pypi's best guess  

403
00:49:45,520 --> 00:50:03,280
as to whether or not that is instantiated.
>> There's a related question, too,  

404
00:50:03,280 --> 00:50:07,040
which I'm going to slightly paraphrase 
because it's [Indiscernible] specific.  

405
00:50:11,200 --> 00:50:17,760
Would it be worth talking to    or have we 
had conversations with vendors like GitHub,  

406
00:50:17,760 --> 00:50:22,080
to see whether they could have their caches  

407
00:50:22,640 --> 00:50:29,920
sitting between them and the cheese shop?
EE DURBIN: Yeah. So, I'm almost positive that  

408
00:50:29,920 --> 00:50:44,160
there are corporations out there that instantiate 
mirrors or caches in between. I do I'm fairly  

409
00:50:44,160 --> 00:50:50,720
certain that continuous integration is a big part 
of our bandwidth consumption, as is mirroring.  

410
00:50:52,480 --> 00:50:59,600
And as is, just, again, the popularity and usage 
of Python. It's not thank you for those links in  

411
00:50:59,600 --> 00:51:05,360
the chat, by the way. So, it's not something, 
again, we can be positive about the answer to,  

412
00:51:06,240 --> 00:51:11,120
but it is absolutely. If you run your 
own continuous integration jobs and  

413
00:51:11,120 --> 00:51:18,160
have control over cache, that contributes. If 
you're able to start caching your, you know,  

414
00:51:18,160 --> 00:51:25,280
pips wheel cache between builds, that can be 
a big advantage for you. And also, you know,  

415
00:51:25,280 --> 00:51:33,200
contribute to lowering that bandwidth overall.
>> I think what we might do is call time there  

416
00:51:33,200 --> 00:51:40,640
on this live chat and switch over to the text chat 
where we can continue having a more asynchronous  

417
00:51:40,640 --> 00:51:45,920
conversation and people can go off and grab a 
coffee, we will have a little bit of a break  

418
00:51:45,920 --> 00:51:53,280
before we cut into our next set of talks.
The next session is due to start at 10:30,  

419
00:51:54,080 --> 00:52:03,520
Australian Eastern Time. I've lost 
my window. Where are we? Here we go.  

420
00:52:04,640 --> 00:52:09,680
Too many windows open. We'll take 
a break, now. Thank you, again, Ee,  

421
00:52:09,680 --> 00:52:16,080
for that wonderful, wonderful Keynote. 
And we'll catch you in the text chat. 

422
00:52:16,080 --> 00:52:28,000
Thanks, everyone.
EE DURBIN: Thank you.