1
00:00:06,320 --> 00:00:11,499
[Music]

2
00:00:17,279 --> 00:00:21,359
welcome back

3
00:00:18,560 --> 00:00:23,920
our next speaker will be brett downing

4
00:00:21,359 --> 00:00:27,119
he'll be speaking on the topic of g

5
00:00:23,920 --> 00:00:28,960
streamer and ros roz a tale of two

6
00:00:27,119 --> 00:00:30,800
messaging frameworks

7
00:00:28,960 --> 00:00:32,640
brett is a hardware hacker whose passion

8
00:00:30,800 --> 00:00:34,000
for robotics dragged him into software

9
00:00:32,640 --> 00:00:35,920
development

10
00:00:34,000 --> 00:00:39,760
there before we're going to see either

11
00:00:35,920 --> 00:00:39,760
software or robotics

12
00:00:40,640 --> 00:00:45,600
brett will allow time for questions at

13
00:00:42,480 --> 00:00:47,760
the end so please enter any on the tab

14
00:00:45,600 --> 00:00:49,600
in venulis

15
00:00:47,760 --> 00:00:50,879
take it away brett

16
00:00:49,600 --> 00:00:52,879
okay

17
00:00:50,879 --> 00:00:54,480
um

18
00:00:52,879 --> 00:00:57,039
so i've

19
00:00:54,480 --> 00:00:58,960
been hacking on a software project to

20
00:00:57,039 --> 00:01:00,960
essentially bridge

21
00:00:58,960 --> 00:01:03,199
g streamer and rolls

22
00:01:00,960 --> 00:01:03,199
um

23
00:01:06,159 --> 00:01:10,720
pipelines are really cool if you were at

24
00:01:07,840 --> 00:01:13,680
my last talk i discussed a bit about how

25
00:01:10,720 --> 00:01:15,759
pipelines can help you

26
00:01:13,680 --> 00:01:18,320
build reliable software that can be

27
00:01:15,759 --> 00:01:21,439
extended maintained all the good stuff

28
00:01:18,320 --> 00:01:24,799
especially as it applies to

29
00:01:21,439 --> 00:01:27,360
advanced robotics meeting hardware

30
00:01:24,799 --> 00:01:29,200
but pipelines as a general approach to

31
00:01:27,360 --> 00:01:31,119
software make a lot of sense in a lot of

32
00:01:29,200 --> 00:01:32,320
applications

33
00:01:31,119 --> 00:01:33,680
you can

34
00:01:32,320 --> 00:01:35,360
use them for large software projects and

35
00:01:33,680 --> 00:01:37,200
then all of the components all of the

36
00:01:35,360 --> 00:01:39,520
functionality becomes quite small quite

37
00:01:37,200 --> 00:01:41,439
maintainable

38
00:01:39,520 --> 00:01:43,040
you can write new applications using

39
00:01:41,439 --> 00:01:44,960
proven utilities and you can teach old

40
00:01:43,040 --> 00:01:47,680
applications new tricks by creating new

41
00:01:44,960 --> 00:01:47,680
plugins for them

42
00:01:48,240 --> 00:01:53,040
pipeline elements practically operate in

43
00:01:50,880 --> 00:01:54,799
isolation elements pass messages between

44
00:01:53,040 --> 00:01:57,520
each other but they aren't really aware

45
00:01:54,799 --> 00:01:59,439
of each other beyond that

46
00:01:57,520 --> 00:02:01,520
when you build applications with a

47
00:01:59,439 --> 00:02:03,840
pipeline architecture the components can

48
00:02:01,520 --> 00:02:05,119
become very simple and can become very

49
00:02:03,840 --> 00:02:06,719
reliable

50
00:02:05,119 --> 00:02:08,720
if an element does more than one thing

51
00:02:06,719 --> 00:02:10,879
it usually just gets broken to more

52
00:02:08,720 --> 00:02:13,040
elements

53
00:02:10,879 --> 00:02:15,360
elements tend to inherit a lot of code

54
00:02:13,040 --> 00:02:17,360
from some base package that often

55
00:02:15,360 --> 00:02:19,680
includes the inspection tooling and the

56
00:02:17,360 --> 00:02:23,480
build system not everyone likes that but

57
00:02:19,680 --> 00:02:23,480
it certainly helps maintainers

58
00:02:23,680 --> 00:02:27,920
a pipeline diagram is

59
00:02:25,760 --> 00:02:29,440
easy to read and understand

60
00:02:27,920 --> 00:02:31,760
especially understand what the pipeline

61
00:02:29,440 --> 00:02:33,519
as a whole is trying to do

62
00:02:31,760 --> 00:02:36,000
inspection tools that work on the single

63
00:02:33,519 --> 00:02:38,160
pipeline element for the base element

64
00:02:36,000 --> 00:02:39,680
work on all elements

65
00:02:38,160 --> 00:02:41,760
the elements themselves share a lot of

66
00:02:39,680 --> 00:02:43,680
code conventions making them easier to

67
00:02:41,760 --> 00:02:47,120
read easier to share

68
00:02:43,680 --> 00:02:47,120
easier to resurrect

69
00:02:47,440 --> 00:02:51,440
gstreamer is a pipeline framework

70
00:02:49,280 --> 00:02:53,360
designed for broadcast media it has very

71
00:02:51,440 --> 00:02:56,080
strict timing and every media converter

72
00:02:53,360 --> 00:02:56,080
you could think of

73
00:02:56,319 --> 00:03:00,080
gstreamer's timing is so strict that

74
00:02:58,159 --> 00:03:02,080
it's been picked up for gravity wave

75
00:03:00,080 --> 00:03:03,599
gravity wave observatories

76
00:03:02,080 --> 00:03:05,280
if you want to build new multimedia

77
00:03:03,599 --> 00:03:07,040
applications and you want all of the

78
00:03:05,280 --> 00:03:09,120
codecs and drivers

79
00:03:07,040 --> 00:03:10,959
and a proven pipeline architecture use

80
00:03:09,120 --> 00:03:13,599
geostreamer

81
00:03:10,959 --> 00:03:15,840
gstream has so many drivers for cameras

82
00:03:13,599 --> 00:03:19,360
that it's starting to show up as a core

83
00:03:15,840 --> 00:03:21,200
dependency for self-driving cars

84
00:03:19,360 --> 00:03:23,920
ros on the other hand is a pipeline

85
00:03:21,200 --> 00:03:27,280
framework built for robots it shows up

86
00:03:23,920 --> 00:03:29,760
in consumer robotics vacuum cleaners

87
00:03:27,280 --> 00:03:32,159
it moves stuff around warehouses and it

88
00:03:29,760 --> 00:03:35,360
forms the core of a lot of driver assist

89
00:03:32,159 --> 00:03:36,480
technologies certainly in research

90
00:03:35,360 --> 00:03:37,920
um

91
00:03:36,480 --> 00:03:40,319
between the two

92
00:03:37,920 --> 00:03:41,760
they have basically the same topology um

93
00:03:40,319 --> 00:03:44,120
so there's a bit of

94
00:03:41,760 --> 00:03:48,239
a jargon that they use different names

95
00:03:44,120 --> 00:03:50,159
for um i will be using the terms pretty

96
00:03:48,239 --> 00:03:52,159
much interchangeably and hoping that i'm

97
00:03:50,159 --> 00:03:53,760
clear

98
00:03:52,159 --> 00:03:55,920
uh

99
00:03:53,760 --> 00:03:59,000
ask questions if it's not clear i'm open

100
00:03:55,920 --> 00:03:59,000
to heckling

101
00:04:00,159 --> 00:04:04,400
so

102
00:04:00,879 --> 00:04:07,360
roz started round about 2007 with

103
00:04:04,400 --> 00:04:09,519
the aim to end a decades-long academic

104
00:04:07,360 --> 00:04:11,599
abandonware cycle

105
00:04:09,519 --> 00:04:13,360
every new student would throw out most

106
00:04:11,599 --> 00:04:14,879
of the work of the predecessor and spend

107
00:04:13,360 --> 00:04:16,799
all of their time fiddling with power

108
00:04:14,879 --> 00:04:18,639
systems and simple mechanics

109
00:04:16,799 --> 00:04:20,479
roz was developed to encourage

110
00:04:18,639 --> 00:04:21,759
roboticists to think

111
00:04:20,479 --> 00:04:24,160
uh to think about their robots as

112
00:04:21,759 --> 00:04:26,000
modular constructions to break them down

113
00:04:24,160 --> 00:04:28,560
into a few fairly consistent interfaces

114
00:04:26,000 --> 00:04:31,759
under a shared build tool

115
00:04:28,560 --> 00:04:34,000
researchers competing on a mapping or

116
00:04:31,759 --> 00:04:35,759
grasping problem could at least share

117
00:04:34,000 --> 00:04:38,160
the code that makes the robot drive

118
00:04:35,759 --> 00:04:38,160
around

119
00:04:38,240 --> 00:04:43,759
roz doesn't trust the coding practices

120
00:04:40,960 --> 00:04:45,520
of novice roboticists so every node is a

121
00:04:43,759 --> 00:04:47,199
wholly separate program

122
00:04:45,520 --> 00:04:49,440
this prevents a lot of conflicts and

123
00:04:47,199 --> 00:04:51,199
side effects

124
00:04:49,440 --> 00:04:53,280
ros nodes inherit a lot of functionality

125
00:04:51,199 --> 00:04:55,040
from ros so they almost always build

126
00:04:53,280 --> 00:04:57,840
using the default build system and the

127
00:04:55,040 --> 00:05:01,199
default dependency manager

128
00:04:57,840 --> 00:05:03,039
gstreamer was built to make it easy to

129
00:05:01,199 --> 00:05:04,560
build capable and extensible multimedia

130
00:05:03,039 --> 00:05:06,400
multimedia systems

131
00:05:04,560 --> 00:05:07,919
gstreamer wanted to aggregate new and

132
00:05:06,400 --> 00:05:10,000
unheard of

133
00:05:07,919 --> 00:05:12,320
multimedia processing techniques under

134
00:05:10,000 --> 00:05:14,160
one stable and versatile framework that

135
00:05:12,320 --> 00:05:16,800
could be easily included into other

136
00:05:14,160 --> 00:05:16,800
applications

137
00:05:17,199 --> 00:05:20,560
gstreamer provides mesh messaging

138
00:05:18,960 --> 00:05:22,320
framework and handles the scheduling

139
00:05:20,560 --> 00:05:24,880
problems associated with complex

140
00:05:22,320 --> 00:05:26,720
concurrent media streams gstreamer aims

141
00:05:24,880 --> 00:05:29,039
for everything to be easily included

142
00:05:26,720 --> 00:05:30,639
into even simple programs so it keeps

143
00:05:29,039 --> 00:05:33,199
all of its computing the pipeline

144
00:05:30,639 --> 00:05:35,759
elements under one process id making it

145
00:05:33,199 --> 00:05:35,759
one program

146
00:05:37,360 --> 00:05:41,919
gstreamer elements need to inherit a lot

147
00:05:39,680 --> 00:05:43,919
of code from gstreamer so almost all

148
00:05:41,919 --> 00:05:46,560
elements are built using the default

149
00:05:43,919 --> 00:05:46,560
build system

150
00:05:48,400 --> 00:05:53,199
fundamentally robot perception

151
00:05:51,520 --> 00:05:54,720
and robot response data most of the

152
00:05:53,199 --> 00:05:55,840
things that go through a robotics

153
00:05:54,720 --> 00:05:57,919
pipeline

154
00:05:55,840 --> 00:06:00,800
is the same kind of information as a

155
00:05:57,919 --> 00:06:03,039
streaming multimedia platform

156
00:06:00,800 --> 00:06:05,360
g streamer and roz chose the same

157
00:06:03,039 --> 00:06:08,000
topology as an answer to basically the

158
00:06:05,360 --> 00:06:09,680
same technical problem

159
00:06:08,000 --> 00:06:11,520
and then for very good reasons

160
00:06:09,680 --> 00:06:14,800
implemented that topology with nearly

161
00:06:11,520 --> 00:06:16,319
opposite design decisions

162
00:06:14,800 --> 00:06:18,240
gstreamer is

163
00:06:16,319 --> 00:06:20,560
difficult to use for distributed

164
00:06:18,240 --> 00:06:23,280
computing problems because it does not

165
00:06:20,560 --> 00:06:25,680
provide network transports sensibly

166
00:06:23,280 --> 00:06:29,280
passing that responsibility onto its

167
00:06:25,680 --> 00:06:31,120
elements and there's so many of those

168
00:06:29,280 --> 00:06:32,400
ros 2 is so flexible about message

169
00:06:31,120 --> 00:06:34,560
passing

170
00:06:32,400 --> 00:06:36,240
as a as a framework

171
00:06:34,560 --> 00:06:38,960
that you almost never need to care which

172
00:06:36,240 --> 00:06:41,039
computer your node is sitting on

173
00:06:38,960 --> 00:06:42,800
ros 2 gains flexibility at the cost of

174
00:06:41,039 --> 00:06:45,120
efficiency so while it can do image

175
00:06:42,800 --> 00:06:46,800
processing the overheads associated mean

176
00:06:45,120 --> 00:06:48,639
you don't want to use it for all of your

177
00:06:46,800 --> 00:06:50,000
streaming media

178
00:06:48,639 --> 00:06:51,599
rolls 2

179
00:06:50,000 --> 00:06:54,080
mitigates most of those overheads that

180
00:06:51,599 --> 00:06:55,919
were found in ros 1 but it still can't

181
00:06:54,080 --> 00:06:58,800
beat the strict tempo rendering that

182
00:06:55,919 --> 00:06:58,800
g-stream offers

183
00:07:00,160 --> 00:07:04,400
there's a couple of robot ros packages

184
00:07:02,160 --> 00:07:05,759
that use g-stream internally

185
00:07:04,400 --> 00:07:07,199
if you build robots that see you're

186
00:07:05,759 --> 00:07:09,199
probably going to look for at least one

187
00:07:07,199 --> 00:07:11,360
camera driver the chances are if you

188
00:07:09,199 --> 00:07:14,000
can't find a linux driver you'll find

189
00:07:11,360 --> 00:07:16,560
driver in g-streamer

190
00:07:14,000 --> 00:07:18,160
gs cam sticks a g streamer pipeline in a

191
00:07:16,560 --> 00:07:21,039
ros node and whatever comes out of the

192
00:07:18,160 --> 00:07:22,639
pipeline is put into an image message

193
00:07:21,039 --> 00:07:24,880
and then transported on the ros message

194
00:07:22,639 --> 00:07:24,880
bus

195
00:07:25,120 --> 00:07:29,120
if you build a robot that can hear or

196
00:07:26,960 --> 00:07:30,880
make noise raws is certainly not the

197
00:07:29,120 --> 00:07:33,360
best framework for capturing or

198
00:07:30,880 --> 00:07:36,160
processing real-time audio

199
00:07:33,360 --> 00:07:37,440
audio common uses g-streamer to handle

200
00:07:36,160 --> 00:07:39,520
raw audio

201
00:07:37,440 --> 00:07:42,479
raw and then uses roles to move

202
00:07:39,520 --> 00:07:44,560
compressed payloads between machines

203
00:07:42,479 --> 00:07:46,240
i was disappointed to discover that it

204
00:07:44,560 --> 00:07:48,639
discards all of the metadata and then

205
00:07:46,240 --> 00:07:51,120
supports exactly one mp3 bit rate

206
00:07:48,639 --> 00:07:54,319
transporting only binary payloads

207
00:07:51,120 --> 00:07:57,039
but this was a very early package um

208
00:07:54,319 --> 00:07:58,720
early 2010s i think

209
00:07:57,039 --> 00:08:00,479
that was only really intended for

210
00:07:58,720 --> 00:08:03,199
telepresence to

211
00:08:00,479 --> 00:08:03,199
one machine

212
00:08:04,800 --> 00:08:09,280
ros one without the package called

213
00:08:07,280 --> 00:08:10,240
plug-in lib moved everything over a

214
00:08:09,280 --> 00:08:11,680
socket

215
00:08:10,240 --> 00:08:12,400
through a through a buffer through the

216
00:08:11,680 --> 00:08:14,400
kernel

217
00:08:12,400 --> 00:08:17,759
and not very efficiently at that so

218
00:08:14,400 --> 00:08:19,360
copying raw images took forever

219
00:08:17,759 --> 00:08:22,960
the compression options were limited and

220
00:08:19,360 --> 00:08:22,960
i never did work out how to tune them

221
00:08:23,599 --> 00:08:28,000
plug-in lib was the first ros package to

222
00:08:25,680 --> 00:08:29,919
break the one node one process rules and

223
00:08:28,000 --> 00:08:31,680
allow messages to be passed between

224
00:08:29,919 --> 00:08:34,479
nodes by pointer

225
00:08:31,680 --> 00:08:36,479
without copying or serializing data

226
00:08:34,479 --> 00:08:38,880
nodes under plug-in lib share a couple

227
00:08:36,479 --> 00:08:42,000
of threads in a single process just like

228
00:08:38,880 --> 00:08:44,480
elements in a gstreamer pipeline

229
00:08:42,000 --> 00:08:46,720
ros 2 expanded plugin lib to cover all

230
00:08:44,480 --> 00:08:49,040
of the nodes making it easy to stuff all

231
00:08:46,720 --> 00:08:51,360
of your nodes into a single process not

232
00:08:49,040 --> 00:08:53,200
just the ones that inherited plug-in lib

233
00:08:51,360 --> 00:08:55,920
plug-in lib is still around it just has

234
00:08:53,200 --> 00:08:58,399
different use cases now

235
00:08:55,920 --> 00:09:01,279
the defining features of ros 2 over ros

236
00:08:58,399 --> 00:09:02,959
1 was pluggable transport systems

237
00:09:01,279 --> 00:09:05,600
one such transport option allows

238
00:09:02,959 --> 00:09:07,600
disparate processors on the same host to

239
00:09:05,600 --> 00:09:09,839
allocate messages

240
00:09:07,600 --> 00:09:10,880
to allocate message memory inside a

241
00:09:09,839 --> 00:09:13,600
daemon

242
00:09:10,880 --> 00:09:15,279
and then pass the message by pointer to

243
00:09:13,600 --> 00:09:16,080
downstream elements and more than one of

244
00:09:15,279 --> 00:09:18,480
them

245
00:09:16,080 --> 00:09:22,240
so you get zero copy messaging from one

246
00:09:18,480 --> 00:09:24,480
node to all downstream elements

247
00:09:22,240 --> 00:09:26,880
there's a theme here ros one's messaging

248
00:09:24,480 --> 00:09:28,880
efficiency was painful for more than a

249
00:09:26,880 --> 00:09:31,440
decade rost2

250
00:09:28,880 --> 00:09:33,600
fixed it in every way anyone could think

251
00:09:31,440 --> 00:09:33,600
of

252
00:09:33,839 --> 00:09:37,519
most of the time if you want to do some

253
00:09:36,080 --> 00:09:39,360
image processing

254
00:09:37,519 --> 00:09:40,560
you collect the image

255
00:09:39,360 --> 00:09:42,480
from somewhere

256
00:09:40,560 --> 00:09:44,240
turn it into an opencv matrix and then

257
00:09:42,480 --> 00:09:46,240
do your preliminary white balance and

258
00:09:44,240 --> 00:09:48,320
contrast adjustments in the privacy of

259
00:09:46,240 --> 00:09:51,440
your process before implementing your

260
00:09:48,320 --> 00:09:54,240
algorithm on that data

261
00:09:51,440 --> 00:09:56,800
opencv is great but does a node a ros

262
00:09:54,240 --> 00:09:59,360
node need that additional complexity

263
00:09:56,800 --> 00:10:01,279
should a roboticist implement color

264
00:09:59,360 --> 00:10:02,480
adjustments in every algorithm they

265
00:10:01,279 --> 00:10:04,240
write

266
00:10:02,480 --> 00:10:07,360
would other nodes benefit from the same

267
00:10:04,240 --> 00:10:09,680
preliminary number crunching

268
00:10:07,360 --> 00:10:12,000
in ros one that was very easy to answer

269
00:10:09,680 --> 00:10:14,160
transporting images was so nightmarishly

270
00:10:12,000 --> 00:10:16,560
expensive that you couldn't afford the

271
00:10:14,160 --> 00:10:18,560
cpu time involved to share anything but

272
00:10:16,560 --> 00:10:21,360
the final result it was cheaper to just

273
00:10:18,560 --> 00:10:23,839
reprocess data

274
00:10:21,360 --> 00:10:25,760
in ros2 it might be fine but then you're

275
00:10:23,839 --> 00:10:28,000
just burning developer time replicating

276
00:10:25,760 --> 00:10:32,800
a bunch of g-stream elements

277
00:10:28,000 --> 00:10:34,800
so the images on screen now are the the

278
00:10:32,800 --> 00:10:36,959
nodes and elements on screen now are

279
00:10:34,800 --> 00:10:40,880
practically equivalent in every way

280
00:10:36,959 --> 00:10:40,880
you could literally substitute them

281
00:10:43,920 --> 00:10:47,519
so

282
00:10:45,120 --> 00:10:49,839
if you want to use g streamer's ability

283
00:10:47,519 --> 00:10:52,880
to efficiently manipulate video streams

284
00:10:49,839 --> 00:10:55,839
and audio streams and multimedia in a

285
00:10:52,880 --> 00:10:58,560
robotics perception pipeline

286
00:10:55,839 --> 00:11:00,560
why not just bridge them

287
00:10:58,560 --> 00:11:02,240
so i ended up putting a ros node inside

288
00:11:00,560 --> 00:11:03,839
a g streamer element

289
00:11:02,240 --> 00:11:04,800
for

290
00:11:03,839 --> 00:11:06,800
g streaming this is kind of

291
00:11:04,800 --> 00:11:08,800
run-of-the-mill it's just yet another

292
00:11:06,800 --> 00:11:09,519
network transport for roz developers

293
00:11:08,800 --> 00:11:11,200
though

294
00:11:09,519 --> 00:11:13,440
this was a saw point that desperately

295
00:11:11,200 --> 00:11:15,839
needed some systematic action

296
00:11:13,440 --> 00:11:15,839
and

297
00:11:16,240 --> 00:11:18,800
i mean i knew it would get used but this

298
00:11:17,680 --> 00:11:21,200
is the first time i've had an open

299
00:11:18,800 --> 00:11:23,440
source project gain traction

300
00:11:21,200 --> 00:11:25,760
by putting whole g streamer pipelines

301
00:11:23,440 --> 00:11:29,600
inside a rolls node you could launch

302
00:11:25,760 --> 00:11:31,760
whole perception chains from a single

303
00:11:29,600 --> 00:11:35,120
bring up command in ros

304
00:11:31,760 --> 00:11:35,120
i'll demonstrate some of that later

305
00:11:35,760 --> 00:11:41,360
as every good software project begins i

306
00:11:38,000 --> 00:11:43,920
started by underestimating the problem

307
00:11:41,360 --> 00:11:45,519
roz and gstreamer are both pipelines but

308
00:11:43,920 --> 00:11:48,320
they're about as different as they can

309
00:11:45,519 --> 00:11:50,880
be while still being pipelines

310
00:11:48,320 --> 00:11:52,880
ros 2 relaxed the definition of a node a

311
00:11:50,880 --> 00:11:54,959
little more than ros1 so a node is

312
00:11:52,880 --> 00:11:57,200
really just anything that can announce

313
00:11:54,959 --> 00:11:59,040
that it's a ros node

314
00:11:57,200 --> 00:12:01,600
g stream elements on the other hand

315
00:11:59,040 --> 00:12:03,680
can't exist in isolation so the only

316
00:12:01,600 --> 00:12:05,600
practical way of bridging the two was

317
00:12:03,680 --> 00:12:08,560
for a g streamer element to announce

318
00:12:05,600 --> 00:12:08,560
that it's a ros node

319
00:12:09,360 --> 00:12:13,839
robotics tends to have

320
00:12:12,000 --> 00:12:15,600
many low bandwidth streams and a very

321
00:12:13,839 --> 00:12:17,839
relaxed view on time

322
00:12:15,600 --> 00:12:20,000
reactions should be as soon as possible

323
00:12:17,839 --> 00:12:21,680
but a robust control system can deal

324
00:12:20,000 --> 00:12:23,120
with modest latency in a bit of timing

325
00:12:21,680 --> 00:12:25,040
jitter

326
00:12:23,120 --> 00:12:26,560
if your video stutters though

327
00:12:25,040 --> 00:12:28,240
you get mad

328
00:12:26,560 --> 00:12:30,160
and

329
00:12:28,240 --> 00:12:34,320
on the other hand a seven second delay

330
00:12:30,160 --> 00:12:36,560
on live broadcast media is traditional

331
00:12:34,320 --> 00:12:38,480
uh special relativity says that a time

332
00:12:36,560 --> 00:12:40,639
without a place is meaningless

333
00:12:38,480 --> 00:12:42,959
in gstream other play in gstreamer the

334
00:12:40,639 --> 00:12:44,800
place is one hardware device typically a

335
00:12:42,959 --> 00:12:46,959
sound card and all of the clocks and

336
00:12:44,800 --> 00:12:48,560
timing is propagated from there

337
00:12:46,959 --> 00:12:50,320
gstreamer provides powerful

338
00:12:48,560 --> 00:12:52,959
synchronization algorithms to make sure

339
00:12:50,320 --> 00:12:55,360
audio from say your bluetooth headset

340
00:12:52,959 --> 00:12:58,399
matches video on maybe your second

341
00:12:55,360 --> 00:13:01,120
monitor they're not sharing clocks but

342
00:12:58,399 --> 00:13:02,959
they're in sync

343
00:13:01,120 --> 00:13:04,880
ros timestamps on the other hand always

344
00:13:02,959 --> 00:13:07,120
come with a frame id so for multiple

345
00:13:04,880 --> 00:13:09,279
robots with multiple clocks you can use

346
00:13:07,120 --> 00:13:11,760
the frame id to talk about which robot

347
00:13:09,279 --> 00:13:13,200
you derived the clock from for a time

348
00:13:11,760 --> 00:13:16,480
measurement

349
00:13:13,200 --> 00:13:19,360
roz can't guess how your dozen clocks

350
00:13:16,480 --> 00:13:20,880
are going to drift or interact

351
00:13:19,360 --> 00:13:23,440
and it can't guess how you're going to

352
00:13:20,880 --> 00:13:25,200
prioritize your time sources so it

353
00:13:23,440 --> 00:13:27,440
doesn't

354
00:13:25,200 --> 00:13:28,959
i'm reasonably certain these two stances

355
00:13:27,440 --> 00:13:31,440
on the nature of time can't be

356
00:13:28,959 --> 00:13:33,360
reconciled within either g stream or

357
00:13:31,440 --> 00:13:35,519
raws but there's a couple of tools that

358
00:13:33,360 --> 00:13:38,320
we can use to you know make it a bit

359
00:13:35,519 --> 00:13:38,320
easier to manage

360
00:13:39,120 --> 00:13:42,399
when you set up a

361
00:13:40,320 --> 00:13:44,639
piece of a pipeline for broadcast media

362
00:13:42,399 --> 00:13:45,839
you really don't want to start it all at

363
00:13:44,639 --> 00:13:47,519
once

364
00:13:45,839 --> 00:13:49,040
and you certainly don't want to touch it

365
00:13:47,519 --> 00:13:51,040
until you're done

366
00:13:49,040 --> 00:13:53,279
if part of the stream fails

367
00:13:51,040 --> 00:13:55,440
you usually want the stream to just

368
00:13:53,279 --> 00:13:56,959
all drop dead so that you can switch to

369
00:13:55,440 --> 00:13:58,560
an alternate stream

370
00:13:56,959 --> 00:14:00,399
when you set up a robot

371
00:13:58,560 --> 00:14:02,399
the control pipelines you use to walk to

372
00:14:00,399 --> 00:14:03,920
the kitchen might not be the same as the

373
00:14:02,399 --> 00:14:05,199
control pipelines you use to open the

374
00:14:03,920 --> 00:14:07,680
fridge

375
00:14:05,199 --> 00:14:09,760
if part of the software dies on a robot

376
00:14:07,680 --> 00:14:11,519
you want to turn on the remote control

377
00:14:09,760 --> 00:14:13,120
and have the robot discover the remote

378
00:14:11,519 --> 00:14:14,800
control

379
00:14:13,120 --> 00:14:16,000
so that you can drive it back to the lab

380
00:14:14,800 --> 00:14:18,480
instead of having to go and get the

381
00:14:16,000 --> 00:14:20,880
forklift

382
00:14:18,480 --> 00:14:23,279
gstreamer keeps metadata with the

383
00:14:20,880 --> 00:14:24,880
pipeline and it has a complicated format

384
00:14:23,279 --> 00:14:27,279
negotiation step that goes upstream and

385
00:14:24,880 --> 00:14:29,440
downstream and puts together the most

386
00:14:27,279 --> 00:14:32,240
efficient pipeline to do a given

387
00:14:29,440 --> 00:14:33,199
a given streaming task very well

388
00:14:32,240 --> 00:14:35,600
ros

389
00:14:33,199 --> 00:14:38,000
packs complete metadata with each image

390
00:14:35,600 --> 00:14:39,760
which gives it a bit more overhead but

391
00:14:38,000 --> 00:14:43,839
it means that node developers don't need

392
00:14:39,760 --> 00:14:46,320
to remember complicated things like

393
00:14:43,839 --> 00:14:48,160
video formats

394
00:14:46,320 --> 00:14:50,560
they can just process each image like it

395
00:14:48,160 --> 00:14:52,880
was new by promoting stateless software

396
00:14:50,560 --> 00:14:55,920
design you make it very easy for novice

397
00:14:52,880 --> 00:14:57,680
developers to write testable code

398
00:14:55,920 --> 00:15:00,240
this also means that you can play back a

399
00:14:57,680 --> 00:15:02,079
bag full of archive perception data into

400
00:15:00,240 --> 00:15:03,760
a new control system

401
00:15:02,079 --> 00:15:05,920
and the new control system will behave

402
00:15:03,760 --> 00:15:08,959
like it was on the actual robot this is

403
00:15:05,920 --> 00:15:10,880
great if you've just plowed 100 acres of

404
00:15:08,959 --> 00:15:12,240
farmland but you hit a rock along the

405
00:15:10,880 --> 00:15:14,160
way and tripped up the old controller

406
00:15:12,240 --> 00:15:16,079
you want to see if the new controller is

407
00:15:14,160 --> 00:15:17,680
tripped up by the same thing

408
00:15:16,079 --> 00:15:20,399
you can just snip it

409
00:15:17,680 --> 00:15:22,399
the the problem part of the

410
00:15:20,399 --> 00:15:23,680
the perception data feed it into the new

411
00:15:22,399 --> 00:15:26,399
control system

412
00:15:23,680 --> 00:15:28,000
and see how it behaves there's no

413
00:15:26,399 --> 00:15:30,160
setup thing at the beginning that you

414
00:15:28,000 --> 00:15:32,880
have to go and collect you can just

415
00:15:30,160 --> 00:15:32,880
start anywhere

416
00:15:32,959 --> 00:15:37,440
this difference though means that if a

417
00:15:34,959 --> 00:15:40,160
ros node is a source of data for a g

418
00:15:37,440 --> 00:15:42,480
stream of a pipeline the pipeline can't

419
00:15:40,160 --> 00:15:45,759
start its negotiation phase until the

420
00:15:42,480 --> 00:15:45,759
first ros message arrives

421
00:15:47,759 --> 00:15:50,959
so

422
00:15:49,040 --> 00:15:54,480
i plugged two pipelines together

423
00:15:50,959 --> 00:15:56,800
and i didn't realize how

424
00:15:54,480 --> 00:15:58,880
uh how much was hanging on this how much

425
00:15:56,800 --> 00:16:00,320
software needed this

426
00:15:58,880 --> 00:16:03,360
i've never felt like i was thrown

427
00:16:00,320 --> 00:16:03,360
further into the deep end

428
00:16:05,519 --> 00:16:08,560
i had a lot of people doing very strange

429
00:16:07,440 --> 00:16:09,920
things with

430
00:16:08,560 --> 00:16:11,600
perception data

431
00:16:09,920 --> 00:16:13,360
robotics

432
00:16:11,600 --> 00:16:15,839
i started getting issues showing up on

433
00:16:13,360 --> 00:16:19,759
an issue tracker

434
00:16:15,839 --> 00:16:19,759
i started having bugs and bug reports

435
00:16:20,000 --> 00:16:24,480
some of them would you know

436
00:16:21,600 --> 00:16:27,440
were the same bug over and over so

437
00:16:24,480 --> 00:16:30,959
i had to set up regression tests and and

438
00:16:27,440 --> 00:16:32,880
build tests uh ros is a

439
00:16:30,959 --> 00:16:35,680
ros has two stable editions at any given

440
00:16:32,880 --> 00:16:37,199
time both are supported by one api but

441
00:16:35,680 --> 00:16:38,880
there's a rolling addition ahead of time

442
00:16:37,199 --> 00:16:40,959
so you can kind of

443
00:16:38,880 --> 00:16:42,480
hitch your build tools to the rolling

444
00:16:40,959 --> 00:16:44,320
the rolling release and then get sort of

445
00:16:42,480 --> 00:16:46,560
advanced notice for when an api changes

446
00:16:44,320 --> 00:16:46,560
on you

447
00:16:47,600 --> 00:16:51,160
licensing clarity

448
00:16:52,320 --> 00:16:56,560
i had a company show up asking for

449
00:16:55,120 --> 00:16:57,759
a bit more clarity on the license i was

450
00:16:56,560 --> 00:16:59,519
offering it under

451
00:16:57,759 --> 00:17:02,399
they wanted to use it for

452
00:16:59,519 --> 00:17:02,399
as far as i can tell

453
00:17:02,720 --> 00:17:07,839
using robot perception data on safe

454
00:17:05,439 --> 00:17:09,600
uh air transport

455
00:17:07,839 --> 00:17:10,720
uh drone traffic control that sort of

456
00:17:09,600 --> 00:17:12,799
thing

457
00:17:10,720 --> 00:17:14,240
um

458
00:17:12,799 --> 00:17:17,760
autonomous robots and certainly

459
00:17:14,240 --> 00:17:20,559
self-driving cars is not a thing um

460
00:17:17,760 --> 00:17:22,000
no one currently has a fully autonomous

461
00:17:20,559 --> 00:17:24,480
anything

462
00:17:22,000 --> 00:17:26,959
but semi-autonomous certainly so driver

463
00:17:24,480 --> 00:17:29,200
operate driver interaction operator

464
00:17:26,959 --> 00:17:30,400
interaction during a semi-autonomous

465
00:17:29,200 --> 00:17:32,720
procedure

466
00:17:30,400 --> 00:17:35,520
everyone wants that and gstream is a

467
00:17:32,720 --> 00:17:37,280
great tool to put together a

468
00:17:35,520 --> 00:17:38,640
means of moving perception data from one

469
00:17:37,280 --> 00:17:41,280
machine to another

470
00:17:38,640 --> 00:17:42,320
really really efficiently

471
00:17:41,280 --> 00:17:43,600
uh

472
00:17:42,320 --> 00:17:45,039
had people doing strange things with

473
00:17:43,600 --> 00:17:47,280
cameras underwater

474
00:17:45,039 --> 00:17:49,440
they wanted to do stereo imaging but

475
00:17:47,280 --> 00:17:51,360
their

476
00:17:49,440 --> 00:17:52,480
their pressure envelope meant that they

477
00:17:51,360 --> 00:17:55,360
couldn't

478
00:17:52,480 --> 00:17:56,880
actually connect the two cameras as a

479
00:17:55,360 --> 00:17:58,480
stereo rig they had to treat them as

480
00:17:56,880 --> 00:18:00,640
independent cameras and then stitch them

481
00:17:58,480 --> 00:18:03,120
after which included things like time

482
00:18:00,640 --> 00:18:03,120
stitching

483
00:18:04,880 --> 00:18:09,280
and of course when you make something

484
00:18:06,640 --> 00:18:11,120
that feels like infrastructure

485
00:18:09,280 --> 00:18:14,320
in something that sort of

486
00:18:11,120 --> 00:18:16,320
bridges two worlds of self-driving cars

487
00:18:14,320 --> 00:18:17,760
and machine learning all these memes of

488
00:18:16,320 --> 00:18:20,240
course it's going to become homework at

489
00:18:17,760 --> 00:18:20,240
some point

490
00:18:21,039 --> 00:18:24,160
feature creep was

491
00:18:22,640 --> 00:18:26,320
a thing

492
00:18:24,160 --> 00:18:27,600
fortunately people in the roz community

493
00:18:26,320 --> 00:18:29,840
seem to have

494
00:18:27,600 --> 00:18:31,600
very realistic expectations and

495
00:18:29,840 --> 00:18:33,120
understanding of how much work something

496
00:18:31,600 --> 00:18:35,760
takes so

497
00:18:33,120 --> 00:18:37,440
i had some discussions with people about

498
00:18:35,760 --> 00:18:39,840
the weirdest camera formats i've ever

499
00:18:37,440 --> 00:18:39,840
heard of

500
00:18:40,400 --> 00:18:44,559
now i kind of started building this

501
00:18:42,080 --> 00:18:45,600
because i wanted robots to interact with

502
00:18:44,559 --> 00:18:47,360
people

503
00:18:45,600 --> 00:18:50,000
like a webrtc

504
00:18:47,360 --> 00:18:52,000
thing i actually wanted to use webrtc

505
00:18:50,000 --> 00:18:54,000
uh because i've got a lot of friends who

506
00:18:52,000 --> 00:18:55,200
do their work in workshop environments

507
00:18:54,000 --> 00:18:56,880
and they move around their workshop

508
00:18:55,200 --> 00:18:58,320
environment you can't just tie them to a

509
00:18:56,880 --> 00:19:01,120
chair and have a chat with them you need

510
00:18:58,320 --> 00:19:03,679
to follow them around

511
00:19:01,120 --> 00:19:06,799
this was especially for

512
00:19:03,679 --> 00:19:08,799
um immunocompromised friends in

513
00:19:06,799 --> 00:19:10,080
their workshop where they got to play

514
00:19:08,799 --> 00:19:11,360
with their tools but if you wanted to

515
00:19:10,080 --> 00:19:12,960
hang out at their workshop you had to do

516
00:19:11,360 --> 00:19:15,679
it without

517
00:19:12,960 --> 00:19:17,200
bringing yourself there

518
00:19:15,679 --> 00:19:18,960
um

519
00:19:17,200 --> 00:19:20,880
turns out people don't like fiddling

520
00:19:18,960 --> 00:19:24,000
with audio and ros

521
00:19:20,880 --> 00:19:27,120
not sure why

522
00:19:24,000 --> 00:19:29,520
uh gstreamer developers can treat this

523
00:19:27,120 --> 00:19:32,240
plug-in this collection of elements like

524
00:19:29,520 --> 00:19:35,360
any other network transport

525
00:19:32,240 --> 00:19:37,679
these elements aren't started by roz

526
00:19:35,360 --> 00:19:39,120
they're actually started by a pipeline

527
00:19:37,679 --> 00:19:41,919
so

528
00:19:39,120 --> 00:19:44,240
i had to wrap

529
00:19:41,919 --> 00:19:46,640
i'd wrap the

530
00:19:44,240 --> 00:19:48,880
i'll get to it

531
00:19:46,640 --> 00:19:51,120
jstreamer has rigorous definitions for

532
00:19:48,880 --> 00:19:54,080
binary encodings of images and audio

533
00:19:51,120 --> 00:19:56,640
which is a very good thing because the

534
00:19:54,080 --> 00:19:58,400
the endianness of an audio stream almost

535
00:19:56,640 --> 00:20:00,720
never matches the endingness of the

536
00:19:58,400 --> 00:20:03,600
system that it's carried by

537
00:20:00,720 --> 00:20:06,159
there's a lot of legacy code

538
00:20:03,600 --> 00:20:08,480
roz uses opencv for most of its image

539
00:20:06,159 --> 00:20:09,760
processing and opencv can deal with

540
00:20:08,480 --> 00:20:11,679
almost any

541
00:20:09,760 --> 00:20:13,679
binary packaging that gstreamer can

542
00:20:11,679 --> 00:20:15,760
throw at it likewise g streaming can

543
00:20:13,679 --> 00:20:17,200
describe and handle any encoding that

544
00:20:15,760 --> 00:20:18,960
roz can describe

545
00:20:17,200 --> 00:20:22,000
i didn't have to touch transcoding at

546
00:20:18,960 --> 00:20:23,280
all in the end the images image formats

547
00:20:22,000 --> 00:20:25,039
that there was a one-to-one

548
00:20:23,280 --> 00:20:26,960
correspondence with at least one format

549
00:20:25,039 --> 00:20:29,440
on both sides

550
00:20:26,960 --> 00:20:30,640
so the images the the the element and

551
00:20:29,440 --> 00:20:32,960
the node they just

552
00:20:30,640 --> 00:20:35,840
pass on a block of data without touching

553
00:20:32,960 --> 00:20:35,840
it

554
00:20:36,559 --> 00:20:39,520
roz

555
00:20:37,679 --> 00:20:41,600
a ros image topic can

556
00:20:39,520 --> 00:20:43,760
carry any kind of image but it's up to

557
00:20:41,600 --> 00:20:46,799
the node to read the message metadata

558
00:20:43,760 --> 00:20:48,240
when the image actually arrives

559
00:20:46,799 --> 00:20:49,679
jstreamer configures all of its elements

560
00:20:48,240 --> 00:20:52,080
ahead of time and that needs to know the

561
00:20:49,679 --> 00:20:55,200
image format and dimensions of the image

562
00:20:52,080 --> 00:20:57,760
before going from pause to playback so

563
00:20:55,200 --> 00:21:00,159
i had to add a tool to

564
00:20:57,760 --> 00:21:00,159
add a

565
00:21:00,400 --> 00:21:06,400
parameter so you can pass the ros node a

566
00:21:03,520 --> 00:21:08,080
format string to hand to g streamer so

567
00:21:06,400 --> 00:21:10,320
that g streamer can go straight from

568
00:21:08,080 --> 00:21:12,080
pause to playback it's a hack and i want

569
00:21:10,320 --> 00:21:13,840
to talk to a g streamer dev about

570
00:21:12,080 --> 00:21:14,799
renegotiating

571
00:21:13,840 --> 00:21:18,400
uh

572
00:21:14,799 --> 00:21:20,240
the pipeline format during runtime

573
00:21:18,400 --> 00:21:22,000
so if you know how to do that let me

574
00:21:20,240 --> 00:21:24,720
know

575
00:21:22,000 --> 00:21:27,120
uh if you leave the format string empty

576
00:21:24,720 --> 00:21:28,480
the ros source will

577
00:21:27,120 --> 00:21:32,799
block the pipeline from going to

578
00:21:28,480 --> 00:21:32,799
playback until the first message arrives

579
00:21:33,679 --> 00:21:37,840
gstreamer's sense of time is

580
00:21:35,840 --> 00:21:38,960
very strict very well defined it's

581
00:21:37,840 --> 00:21:41,840
really good

582
00:21:38,960 --> 00:21:43,760
uh genestreamer times timestamps

583
00:21:41,840 --> 00:21:44,720
buffers through its messaging framework

584
00:21:43,760 --> 00:21:47,919
as

585
00:21:44,720 --> 00:21:50,000
nanoseconds since playback start

586
00:21:47,919 --> 00:21:52,400
roz treats time as a relative thing and

587
00:21:50,000 --> 00:21:54,559
then frowns on the use of datums

588
00:21:52,400 --> 00:21:56,960
so the bridge nodes sample the various

589
00:21:54,559 --> 00:21:58,320
clocks at startup and then apply offsets

590
00:21:56,960 --> 00:22:01,760
so the times represent something

591
00:21:58,320 --> 00:22:01,760
sensible in both pipelines

592
00:22:02,960 --> 00:22:08,320
i ended up wrapping the gstreamer launch

593
00:22:05,520 --> 00:22:10,559
syntax with a python node that is a ros

594
00:22:08,320 --> 00:22:10,559
node

595
00:22:10,640 --> 00:22:15,120
ros node implemented in python

596
00:22:12,799 --> 00:22:16,880
a little like gscam does

597
00:22:15,120 --> 00:22:18,720
but the pipeline string can be passed in

598
00:22:16,880 --> 00:22:21,600
from configuration

599
00:22:18,720 --> 00:22:22,880
in ros so that you can package

600
00:22:21,600 --> 00:22:25,200
uh

601
00:22:22,880 --> 00:22:26,559
essentially a version controlled ros

602
00:22:25,200 --> 00:22:28,720
package that says

603
00:22:26,559 --> 00:22:30,559
this robot needs these packages and

604
00:22:28,720 --> 00:22:32,159
needs to start these nodes with this

605
00:22:30,559 --> 00:22:34,720
configuration all as one version

606
00:22:32,159 --> 00:22:38,520
controllable configuration file a

607
00:22:34,720 --> 00:22:38,520
configuration package

608
00:22:40,400 --> 00:22:43,679
now i wanted to use

609
00:22:45,679 --> 00:22:49,679
i wanted to use the raspberry pi camera

610
00:22:47,760 --> 00:22:52,480
over the internet

611
00:22:49,679 --> 00:22:54,799
so both ros and gstreamer allow

612
00:22:52,480 --> 00:22:56,559
parameters or properties the same thing

613
00:22:54,799 --> 00:22:58,799
to be altered on the fly allowing you to

614
00:22:56,559 --> 00:23:00,320
tune things as you go

615
00:22:58,799 --> 00:23:02,640
robots often

616
00:23:00,320 --> 00:23:04,799
often end up on the end of a very long

617
00:23:02,640 --> 00:23:05,600
internet connection a very long unstable

618
00:23:04,799 --> 00:23:06,799
one

619
00:23:05,600 --> 00:23:09,280
and the easiest way of limiting

620
00:23:06,799 --> 00:23:11,679
bandwidth is to reduce the bit rate on

621
00:23:09,280 --> 00:23:12,640
the camera

622
00:23:11,679 --> 00:23:14,559
so

623
00:23:12,640 --> 00:23:16,640
the easiest way of tuning the camera

624
00:23:14,559 --> 00:23:19,200
bitrate on the raspberry pi the h264

625
00:23:16,640 --> 00:23:21,760
compression in its gpu

626
00:23:19,200 --> 00:23:23,280
a bit of background the pi cam

627
00:23:21,760 --> 00:23:24,480
attaches to the pi in a low level

628
00:23:23,280 --> 00:23:26,400
interface

629
00:23:24,480 --> 00:23:29,039
that data goes straight to the gpu where

630
00:23:26,400 --> 00:23:31,840
it does hardware h.264 compression

631
00:23:29,039 --> 00:23:35,200
and then passes that onto the cpu

632
00:23:31,840 --> 00:23:37,280
through a bit of a memory bottleneck

633
00:23:35,200 --> 00:23:40,320
which then can be picked up by

634
00:23:37,280 --> 00:23:41,840
applications such as the pycam software

635
00:23:40,320 --> 00:23:43,679
which has been wrapped by a gstreamer

636
00:23:41,840 --> 00:23:45,520
node and is available in the public

637
00:23:43,679 --> 00:23:47,440
archives

638
00:23:45,520 --> 00:23:48,480
so if you want to

639
00:23:47,440 --> 00:23:51,039
adjust

640
00:23:48,480 --> 00:23:53,440
the bitrate of the h.264 encoding you

641
00:23:51,039 --> 00:23:57,039
need to change the g stream of property

642
00:23:53,440 --> 00:23:58,799
now that isolation property of

643
00:23:57,039 --> 00:24:01,120
of pipeline frameworks

644
00:23:58,799 --> 00:24:03,919
means that the pi cam is not guaranteed

645
00:24:01,120 --> 00:24:05,919
to exist in any given pipeline so if i

646
00:24:03,919 --> 00:24:08,799
wanted to capture

647
00:24:05,919 --> 00:24:11,120
the h.264 bit rate

648
00:24:08,799 --> 00:24:13,120
and expose it to ros i needed to capture

649
00:24:11,120 --> 00:24:15,360
all of the properties of

650
00:24:13,120 --> 00:24:18,640
all of the elements in every pipeline

651
00:24:15,360 --> 00:24:22,320
and expose them to roz the bit rate had

652
00:24:18,640 --> 00:24:23,760
to be captured by a general solution

653
00:24:22,320 --> 00:24:25,120
uh

654
00:24:23,760 --> 00:24:28,000
demo time

655
00:24:25,120 --> 00:24:28,000
so behind me

656
00:24:28,080 --> 00:24:32,960
last session i had a robert claw

657
00:24:30,880 --> 00:24:38,559
and i didn't point out in that session

658
00:24:32,960 --> 00:24:38,559
the second camera was this robot here

659
00:24:39,039 --> 00:24:42,480
uh

660
00:24:40,000 --> 00:24:44,080
so on the screen now is the

661
00:24:42,480 --> 00:24:46,480
all of the nodes that are currently

662
00:24:44,080 --> 00:24:47,279
running on that robot

663
00:24:46,480 --> 00:24:49,039
so

664
00:24:47,279 --> 00:24:50,400
i can go ahead and

665
00:24:49,039 --> 00:24:52,320
launch

666
00:24:50,400 --> 00:24:56,640
from a package

667
00:24:52,320 --> 00:24:58,240
some node with some configuration

668
00:24:56,640 --> 00:25:00,159
and that goes and loads up a gstreamer

669
00:24:58,240 --> 00:25:02,559
pipeline it goes and loads up a joystick

670
00:25:00,159 --> 00:25:03,440
driver

671
00:25:02,559 --> 00:25:04,880
which

672
00:25:03,440 --> 00:25:09,440
then show up

673
00:25:04,880 --> 00:25:09,440
and stitch together a pipeline using ros

674
00:25:10,080 --> 00:25:12,840
the joystick

675
00:25:12,080 --> 00:25:16,159
and

676
00:25:12,840 --> 00:25:17,120
the the teleoperation tools are exactly

677
00:25:16,159 --> 00:25:19,120
the same

678
00:25:17,120 --> 00:25:22,640
as i used to drive the claw but this

679
00:25:19,120 --> 00:25:24,640
time it's targeting a wheeled robot

680
00:25:22,640 --> 00:25:26,720
the mixer node

681
00:25:24,640 --> 00:25:28,270
handles the the vector from the joystick

682
00:25:26,720 --> 00:25:29,600
and then passes that on to

683
00:25:28,270 --> 00:25:32,000
[Music]

684
00:25:29,600 --> 00:25:34,720
the actual

685
00:25:32,000 --> 00:25:34,720
the motor system

686
00:25:34,960 --> 00:25:39,279
at the same time i've got the gstreamer

687
00:25:36,720 --> 00:25:40,720
pipeline

688
00:25:39,279 --> 00:25:42,720
which has a bunch of properties which

689
00:25:40,720 --> 00:25:44,720
can't be mapped

690
00:25:42,720 --> 00:25:45,919
ignoring gst objects

691
00:25:44,720 --> 00:25:49,200
ignoring

692
00:25:45,919 --> 00:25:49,200
structures that whatever

693
00:25:49,440 --> 00:25:53,120
so these are these are jstream of

694
00:25:50,880 --> 00:25:55,360
properties that are exposed by elements

695
00:25:53,120 --> 00:25:58,960
that can't be mapped in a type safe

696
00:25:55,360 --> 00:25:58,960
manner to a ros property

697
00:25:59,120 --> 00:26:03,679
it's picked up a joystick

698
00:26:02,080 --> 00:26:06,720
and it started

699
00:26:03,679 --> 00:26:06,720
the udp stream

700
00:26:16,159 --> 00:26:19,840
which gives me access to

701
00:26:18,320 --> 00:26:20,640
the image topic

702
00:26:19,840 --> 00:26:22,720
sorry

703
00:26:20,640 --> 00:26:24,799
access to the gstreamer

704
00:26:22,720 --> 00:26:25,679
compressed video feed

705
00:26:24,799 --> 00:26:27,360
as

706
00:26:25,679 --> 00:26:29,919
a ros

707
00:26:27,360 --> 00:26:32,720
uh image message

708
00:26:29,919 --> 00:26:35,360
so this here is a video

709
00:26:32,720 --> 00:26:36,840
in ros as a ros

710
00:26:35,360 --> 00:26:41,840
as ros

711
00:26:36,840 --> 00:26:41,840
message and then if i turn the motors on

712
00:26:42,720 --> 00:26:46,400
uh not that

713
00:26:45,279 --> 00:26:48,880
so

714
00:26:46,400 --> 00:26:50,720
this one is just

715
00:26:48,880 --> 00:26:52,400
a raspberry pi

716
00:26:50,720 --> 00:26:53,760
pi camera

717
00:26:52,400 --> 00:26:54,799
a

718
00:26:53,760 --> 00:26:56,559
servo

719
00:26:54,799 --> 00:26:58,480
server control breakout a couple of

720
00:26:56,559 --> 00:27:00,559
batteries couple of motor controllers

721
00:26:58,480 --> 00:27:02,640
and that's it the the whole thing can be

722
00:27:00,559 --> 00:27:06,000
put together for

723
00:27:02,640 --> 00:27:09,679
only 50 bucks

724
00:27:06,000 --> 00:27:11,919
i'm gonna put that down uh

725
00:27:09,679 --> 00:27:14,559
those ones um

726
00:27:11,919 --> 00:27:16,000
those ones debuted at

727
00:27:14,559 --> 00:27:17,360
those robots are actually the same as

728
00:27:16,000 --> 00:27:21,640
the

729
00:27:17,360 --> 00:27:21,640
the one that i'm about to drive

730
00:27:22,080 --> 00:27:26,240
uh

731
00:27:23,520 --> 00:27:27,760
these robots are again a pi zero

732
00:27:26,240 --> 00:27:29,440
with pi cam

733
00:27:27,760 --> 00:27:30,880
with motor controllers

734
00:27:29,440 --> 00:27:33,279
and that little mount there is for

735
00:27:30,880 --> 00:27:35,600
adjusting pole these debuted at

736
00:27:33,279 --> 00:27:37,120
the canberra python user group and we

737
00:27:35,600 --> 00:27:40,320
had a great time

738
00:27:37,120 --> 00:27:42,000
jousting with them in the park

739
00:27:40,320 --> 00:27:43,919
there's two of them

740
00:27:42,000 --> 00:27:46,559
and they are just identical they're

741
00:27:43,919 --> 00:27:47,840
exactly the same as the little robot

742
00:27:46,559 --> 00:27:50,799
it's just they're using skateboard

743
00:27:47,840 --> 00:27:50,799
wheels and larger batteries

744
00:27:51,360 --> 00:27:56,240
so if there's any questions i'm happy to

745
00:27:53,600 --> 00:27:56,240
take them now

746
00:27:56,480 --> 00:28:00,799
that's kind of all the

747
00:27:58,159 --> 00:28:02,240
demo that i needed to get through

748
00:28:00,799 --> 00:28:06,000
but i have a bunch of other stuff that i

749
00:28:02,240 --> 00:28:09,279
can show while i'm answering questions

750
00:28:06,000 --> 00:28:09,279
um particularly

751
00:28:09,679 --> 00:28:13,559
i want to show the

752
00:28:21,039 --> 00:28:24,559
so by using

753
00:28:22,240 --> 00:28:25,840
raws by using a pipeline framework

754
00:28:24,559 --> 00:28:27,360
i can using

755
00:28:25,840 --> 00:28:28,960
pipeline introspection tools to just

756
00:28:27,360 --> 00:28:31,840
discover everything

757
00:28:28,960 --> 00:28:31,840
on the network

758
00:28:32,399 --> 00:28:37,200
so here's a list of

759
00:28:35,120 --> 00:28:40,000
all the nodes on my

760
00:28:37,200 --> 00:28:41,360
local network so the raspberry pi and

761
00:28:40,000 --> 00:28:42,559
the laptop stuff is all discovered

762
00:28:41,360 --> 00:28:46,080
automatically i don't have to worry

763
00:28:42,559 --> 00:28:46,080
about where my nodes sit

764
00:28:46,240 --> 00:28:49,240
um

765
00:28:58,720 --> 00:29:01,520
the other one is

766
00:29:03,120 --> 00:29:08,240
i can use g streamer inspection tools

767
00:29:05,679 --> 00:29:10,720
if i tell it where i've built the

768
00:29:08,240 --> 00:29:13,120
gstreamer plugin

769
00:29:10,720 --> 00:29:15,200
i can inspect any of the

770
00:29:13,120 --> 00:29:17,600
elements that i've put together

771
00:29:15,200 --> 00:29:20,080
and gstreamer just provides

772
00:29:17,600 --> 00:29:22,399
automatic documentation features this is

773
00:29:20,080 --> 00:29:23,360
amazing for

774
00:29:22,399 --> 00:29:24,559
uh

775
00:29:23,360 --> 00:29:26,240
software

776
00:29:24,559 --> 00:29:27,039
conservancy

777
00:29:26,240 --> 00:29:28,880
um

778
00:29:27,039 --> 00:29:30,240
especially when you've got academics

779
00:29:28,880 --> 00:29:32,480
going through and

780
00:29:30,240 --> 00:29:33,360
um

781
00:29:32,480 --> 00:29:36,000
you know

782
00:29:33,360 --> 00:29:37,360
building building up robots where

783
00:29:36,000 --> 00:29:38,880
it's their first time touching

784
00:29:37,360 --> 00:29:42,159
electronics touching hardware touching

785
00:29:38,880 --> 00:29:43,520
software the ability for people to

786
00:29:42,159 --> 00:29:45,440
just write some code and have it

787
00:29:43,520 --> 00:29:47,120
automatically get documented and then

788
00:29:45,440 --> 00:29:49,039
you know read their own docs

789
00:29:47,120 --> 00:29:50,960
to figure out what's what might be wrong

790
00:29:49,039 --> 00:29:55,840
what might be misconfigured this is

791
00:29:50,960 --> 00:29:55,840
extremely valuable to novice developers

792
00:29:58,720 --> 00:30:05,919
ros gst bridge is the plugin name

793
00:30:02,159 --> 00:30:05,919
it's just a shared object

794
00:30:07,039 --> 00:30:10,799
the image sync is what i'm inspecting

795
00:30:08,559 --> 00:30:10,799
now

796
00:30:11,520 --> 00:30:16,159
uh all of this

797
00:30:13,520 --> 00:30:19,360
is inherited from

798
00:30:16,159 --> 00:30:19,360
the jstreamer framework

799
00:30:19,440 --> 00:30:25,200
and then right down at the bottom i have

800
00:30:21,679 --> 00:30:25,200
just what's novel about

801
00:30:26,000 --> 00:30:29,679
using raws

802
00:30:28,000 --> 00:30:31,840
in jstreamer

803
00:30:29,679 --> 00:30:34,240
so i ended up having to write

804
00:30:31,840 --> 00:30:35,840
not an enormous amount of code

805
00:30:34,240 --> 00:30:39,640
in order to make something that was

806
00:30:35,840 --> 00:30:39,640
really quite capable

807
00:30:51,039 --> 00:30:53,360
uh

808
00:30:53,760 --> 00:30:59,840
the other one is because i've loaded

809
00:30:56,640 --> 00:31:01,039
ros nodes into gstreamer elements i can

810
00:30:59,840 --> 00:31:04,880
put together

811
00:31:01,039 --> 00:31:04,880
you know additional snippets of pipeline

812
00:31:06,000 --> 00:31:10,000
using just

813
00:31:08,000 --> 00:31:11,440
gstreamer launch

814
00:31:10,000 --> 00:31:13,039
so

815
00:31:11,440 --> 00:31:14,720
something that's really common in

816
00:31:13,039 --> 00:31:15,679
robotics is you have high quality

817
00:31:14,720 --> 00:31:17,600
cameras

818
00:31:15,679 --> 00:31:20,159
in a high quality lab

819
00:31:17,600 --> 00:31:21,120
on a high quality network

820
00:31:20,159 --> 00:31:23,760
and

821
00:31:21,120 --> 00:31:25,600
your robot never encounters mud until

822
00:31:23,760 --> 00:31:27,279
it's too late

823
00:31:25,600 --> 00:31:29,039
so something you often want to do is

824
00:31:27,279 --> 00:31:30,799
either like do a bit of pre-processing

825
00:31:29,039 --> 00:31:33,799
or

826
00:31:30,799 --> 00:31:33,799
damage

827
00:31:34,159 --> 00:31:40,399
your video feed before passing it on to

828
00:31:37,360 --> 00:31:40,399
the control system

829
00:31:40,640 --> 00:31:46,840
so if i run that

830
00:31:42,640 --> 00:31:46,840
and pray to the demo gods

831
00:31:49,360 --> 00:31:52,320
then i should have

832
00:31:54,159 --> 00:32:00,200
some amount of debris

833
00:31:56,559 --> 00:32:00,200
on my camera feed

834
00:32:00,320 --> 00:32:07,279
that's provided by gstreamer

835
00:32:03,120 --> 00:32:11,640
so then on the node graph

836
00:32:07,279 --> 00:32:11,640
i've got all the joystick stuff

837
00:32:14,320 --> 00:32:16,799
joystick stuff all the way to motor

838
00:32:15,679 --> 00:32:18,720
control

839
00:32:16,799 --> 00:32:21,120
i've got the

840
00:32:18,720 --> 00:32:22,640
the ros node that is inside the

841
00:32:21,120 --> 00:32:24,640
gstreamer pipeline

842
00:32:22,640 --> 00:32:26,640
producing an image message

843
00:32:24,640 --> 00:32:28,880
hands it on to another ros node inside

844
00:32:26,640 --> 00:32:31,600
another g-stream pipeline

845
00:32:28,880 --> 00:32:34,240
which does pre-processing

846
00:32:31,600 --> 00:32:36,480
that comes out here

847
00:32:34,240 --> 00:32:37,600
which then gets picked up by

848
00:32:36,480 --> 00:32:40,640
the

849
00:32:37,600 --> 00:32:40,640
iqt gui

850
00:32:40,840 --> 00:32:47,039
interface um now i said i covered

851
00:32:44,640 --> 00:32:50,320
uh source and sync for

852
00:32:47,039 --> 00:32:50,320
images audio and text

853
00:32:55,440 --> 00:33:00,720
this install setup dot bash is a raw's

854
00:32:58,960 --> 00:33:03,200
invocation to set up some environment

855
00:33:00,720 --> 00:33:03,200
variables

856
00:33:03,360 --> 00:33:06,159
uh

857
00:33:04,640 --> 00:33:07,919
it is a little bit tedious and there are

858
00:33:06,159 --> 00:33:12,320
some people who have put together

859
00:33:07,919 --> 00:33:12,320
some extremely compelling bash aliases

860
00:33:13,440 --> 00:33:17,120
um

861
00:33:14,720 --> 00:33:19,120
so this

862
00:33:17,120 --> 00:33:20,559
so here i'm using g streamer launch

863
00:33:19,120 --> 00:33:22,080
syntax to

864
00:33:20,559 --> 00:33:24,080
invoke

865
00:33:22,080 --> 00:33:26,159
ros nodes

866
00:33:24,080 --> 00:33:28,159
uh

867
00:33:26,159 --> 00:33:31,120
ros image source

868
00:33:28,159 --> 00:33:33,840
puts it in some namespace

869
00:33:31,120 --> 00:33:35,200
uh i've got some

870
00:33:33,840 --> 00:33:39,039
rostopic

871
00:33:35,200 --> 00:33:40,960
and then i give the node some name

872
00:33:39,039 --> 00:33:42,640
the video convert is

873
00:33:40,960 --> 00:33:44,799
because i'm not taking chances of the

874
00:33:42,640 --> 00:33:46,480
live demo

875
00:33:44,799 --> 00:33:49,039
rolls text source

876
00:33:46,480 --> 00:33:51,679
to put a text overlay over the top

877
00:33:49,039 --> 00:33:53,120
so that one

878
00:33:51,679 --> 00:33:54,880
might show up

879
00:33:53,120 --> 00:33:56,320
i've got high dead sinks on because

880
00:33:54,880 --> 00:33:59,640
there's a lot of

881
00:33:56,320 --> 00:33:59,640
background clutter

882
00:34:06,080 --> 00:34:10,720
now all of those nodes again show up

883
00:34:08,560 --> 00:34:13,960
their topics when i ask for them there's

884
00:34:10,720 --> 00:34:13,960
text overlay

885
00:34:20,399 --> 00:34:24,639
let's minimize that

886
00:34:23,440 --> 00:34:28,639
so image

887
00:34:24,639 --> 00:34:32,639
processed image text this is the same

888
00:34:28,639 --> 00:34:32,639
image on a different ros topic

889
00:34:33,359 --> 00:34:36,639
and then i can publish to it

890
00:34:44,639 --> 00:34:49,839
text overlay it's a strongly typed

891
00:34:47,200 --> 00:34:49,839
message

892
00:34:50,159 --> 00:34:52,639
where am i

893
00:34:53,679 --> 00:34:56,079
that's a

894
00:35:01,599 --> 00:35:04,960
using the bash escaping rules

895
00:35:09,040 --> 00:35:14,960
and i've got text overlay on a video

896
00:35:13,280 --> 00:35:18,240
i've pulled

897
00:35:14,960 --> 00:35:20,960
what i've done i've taken

898
00:35:18,240 --> 00:35:22,960
a ros command line tool

899
00:35:20,960 --> 00:35:25,599
to produce

900
00:35:22,960 --> 00:35:27,200
messages onto what was that

901
00:35:25,599 --> 00:35:31,119
if i keep that going

902
00:35:27,200 --> 00:35:34,240
it'll show up in rqt

903
00:35:31,119 --> 00:35:36,560
so i've got the command line client

904
00:35:34,240 --> 00:35:39,680
submitting messages text messages to a

905
00:35:36,560 --> 00:35:42,240
text overlay topic which gets picked up

906
00:35:39,680 --> 00:35:43,760
by a ros node inside a gstream or

907
00:35:42,240 --> 00:35:47,359
pipeline

908
00:35:43,760 --> 00:35:49,119
which gets composited

909
00:35:47,359 --> 00:35:51,920
onto

910
00:35:49,119 --> 00:35:51,920
wow that's unhappy

911
00:35:54,079 --> 00:35:59,920
composited into my video feed and then

912
00:35:56,960 --> 00:36:02,880
emitted onto

913
00:35:59,920 --> 00:36:02,880
another ros topic

914
00:36:03,520 --> 00:36:06,560
again this is kind of unremarkable for

915
00:36:05,280 --> 00:36:07,839
jstreamer

916
00:36:06,560 --> 00:36:09,839
but

917
00:36:07,839 --> 00:36:13,440
this kind of flexibility

918
00:36:09,839 --> 00:36:16,400
with this many image manipulation tools

919
00:36:13,440 --> 00:36:18,640
is just something that rolls hasn't had

920
00:36:16,400 --> 00:36:21,119
yet

921
00:36:18,640 --> 00:36:23,599
and i've had a hell of a time

922
00:36:21,119 --> 00:36:26,720
a wonderful time

923
00:36:23,599 --> 00:36:28,320
talking to community members about

924
00:36:26,720 --> 00:36:31,440
you know

925
00:36:28,320 --> 00:36:33,200
what they're trying to achieve with

926
00:36:31,440 --> 00:36:34,400
this bridge package what they're trying

927
00:36:33,200 --> 00:36:37,359
to achieve with

928
00:36:34,400 --> 00:36:37,359
ross perception data

929
00:36:38,480 --> 00:36:41,480
um

930
00:36:45,760 --> 00:36:49,359
and then

931
00:36:47,599 --> 00:36:51,160
not just you know what community members

932
00:36:49,359 --> 00:36:54,160
are trying to do with perception data

933
00:36:51,160 --> 00:36:56,560
but uh

934
00:36:54,160 --> 00:36:58,400
you know what they think about

935
00:36:56,560 --> 00:37:00,000
uh my code what they think about uh

936
00:36:58,400 --> 00:37:02,240
testing practices

937
00:37:00,000 --> 00:37:03,839
like having an open source project just

938
00:37:02,240 --> 00:37:05,920
take off on you when you're not quite

939
00:37:03,839 --> 00:37:07,119
ready for it

940
00:37:05,920 --> 00:37:08,880
is

941
00:37:07,119 --> 00:37:11,599
as an experience but the ros community

942
00:37:08,880 --> 00:37:14,480
have been really quite

943
00:37:11,599 --> 00:37:14,480
gracious about it

944
00:37:14,560 --> 00:37:17,520
and of course when you do that much

945
00:37:15,920 --> 00:37:21,640
compositing you get a whole lot of

946
00:37:17,520 --> 00:37:21,640
latency and you can't control the robot

947
00:37:32,880 --> 00:37:36,320
hello

948
00:37:34,079 --> 00:37:37,839
hi brett i just came in to help you with

949
00:37:36,320 --> 00:37:40,000
the questions

950
00:37:37,839 --> 00:37:42,240
great thank you we have a couple of

951
00:37:40,000 --> 00:37:44,320
questions appeared

952
00:37:42,240 --> 00:37:46,480
first of all uh which approach do you

953
00:37:44,320 --> 00:37:48,839
prefer uh bridge the two different

954
00:37:46,480 --> 00:37:52,240
framework uh ros and g

955
00:37:48,839 --> 00:37:55,119
streamer or to provide a distributed

956
00:37:52,240 --> 00:37:58,560
framework that transparently subsumes

957
00:37:55,119 --> 00:37:58,560
ros and g streamer

958
00:38:00,000 --> 00:38:05,119
i don't think you can can subsume both

959
00:38:03,119 --> 00:38:07,359
ros and g streamer

960
00:38:05,119 --> 00:38:07,359
um

961
00:38:07,920 --> 00:38:12,560
both g streamer and raz are moving

962
00:38:09,760 --> 00:38:14,960
targets roz certainly is a bit

963
00:38:12,560 --> 00:38:17,119
less api stable than g streamer

964
00:38:14,960 --> 00:38:18,480
um

965
00:38:17,119 --> 00:38:21,760
the

966
00:38:18,480 --> 00:38:23,280
the api and api the binary interface

967
00:38:21,760 --> 00:38:25,359
stability in raws is something they're

968
00:38:23,280 --> 00:38:27,200
emphasizing within a release

969
00:38:25,359 --> 00:38:30,480
uh api compatibility should be

970
00:38:27,200 --> 00:38:30,480
maintained between two releases

971
00:38:30,880 --> 00:38:35,520
but there's not really

972
00:38:33,920 --> 00:38:36,240
much benefit

973
00:38:35,520 --> 00:38:38,000
in

974
00:38:36,240 --> 00:38:40,400
trying to subsume

975
00:38:38,000 --> 00:38:42,400
two enormous squiggly frameworks

976
00:38:40,400 --> 00:38:44,240
uh you would end up creating a whole

977
00:38:42,400 --> 00:38:47,599
pile of conventions that a lot of people

978
00:38:44,240 --> 00:38:48,960
just don't agree with because

979
00:38:47,599 --> 00:38:50,880
the conventions of roles are so

980
00:38:48,960 --> 00:38:51,920
dissimilar from the convention of g

981
00:38:50,880 --> 00:38:53,520
streamer

982
00:38:51,920 --> 00:38:55,520
so

983
00:38:53,520 --> 00:38:57,599
what i've ended up doing is

984
00:38:55,520 --> 00:39:00,000
essentially merging the two

985
00:38:57,599 --> 00:39:03,280
on both sides

986
00:39:00,000 --> 00:39:05,359
so gstreamer has a launch syntax

987
00:39:03,280 --> 00:39:06,400
which you just saw

988
00:39:05,359 --> 00:39:08,400
you can launch a bunch of stuff on the

989
00:39:06,400 --> 00:39:09,520
command line you can describe your

990
00:39:08,400 --> 00:39:10,480
pipeline

991
00:39:09,520 --> 00:39:12,400
as

992
00:39:10,480 --> 00:39:16,880
a bunch of stuff separated by

993
00:39:12,400 --> 00:39:19,359
exclamation marks so by putting

994
00:39:16,880 --> 00:39:21,200
pipelines in configuration strings in

995
00:39:19,359 --> 00:39:24,000
rows you can start g streamer pipelines

996
00:39:21,200 --> 00:39:24,000
in ros nodes

997
00:39:24,400 --> 00:39:28,320
likewise by placing ros nodes inside g

998
00:39:26,560 --> 00:39:31,839
streamer elements you can start ros

999
00:39:28,320 --> 00:39:34,640
pipelines in gstreamer

1000
00:39:31,839 --> 00:39:38,400
so i don't think that the two

1001
00:39:34,640 --> 00:39:40,560
should be subsumed by another framework

1002
00:39:38,400 --> 00:39:43,560
because they they can work so nicely

1003
00:39:40,560 --> 00:39:43,560
together

1004
00:39:44,000 --> 00:39:48,320
okay

1005
00:39:46,160 --> 00:39:50,560
the next question is is there a way i

1006
00:39:48,320 --> 00:39:52,720
can play with this stuff without having

1007
00:39:50,560 --> 00:39:54,560
all the physical equipment

1008
00:39:52,720 --> 00:39:57,440
absolutely

1009
00:39:54,560 --> 00:39:59,760
so there's simulation tools

1010
00:39:57,440 --> 00:39:59,760
um

1011
00:40:00,000 --> 00:40:04,640
so in my last talk uh i believe it was

1012
00:40:02,240 --> 00:40:08,480
recorded and released um

1013
00:40:04,640 --> 00:40:10,960
the claw behind me i have a full

1014
00:40:08,480 --> 00:40:12,960
i described it in ros sorry i described

1015
00:40:10,960 --> 00:40:15,920
the claw in openscad

1016
00:40:12,960 --> 00:40:19,040
printed it exported kinematics

1017
00:40:15,920 --> 00:40:21,520
descriptions from openscad into

1018
00:40:19,040 --> 00:40:23,520
rvs into some kinematics tools on the

1019
00:40:21,520 --> 00:40:26,400
command line and then ran a simulation

1020
00:40:23,520 --> 00:40:27,839
of the claw responding to joystick input

1021
00:40:26,400 --> 00:40:31,440
as though it was a real thing and in

1022
00:40:27,839 --> 00:40:33,680
that talk i had the physical model

1023
00:40:31,440 --> 00:40:33,680
um

1024
00:40:34,480 --> 00:40:39,520
the physical model

1025
00:40:36,560 --> 00:40:41,119
uh

1026
00:40:39,520 --> 00:40:44,240
doing exactly the same motions as the

1027
00:40:41,119 --> 00:40:46,079
simulated model at the same time so

1028
00:40:44,240 --> 00:40:48,319
you don't need

1029
00:40:46,079 --> 00:40:50,800
any real hardware at all

1030
00:40:48,319 --> 00:40:52,720
uh just a laptop running

1031
00:40:50,800 --> 00:40:54,000
usually just ubuntu linux

1032
00:40:52,720 --> 00:40:56,720
um

1033
00:40:54,000 --> 00:40:59,680
and then there's gazebo uh ignition

1034
00:40:56,720 --> 00:41:01,760
robotics has the gazebo

1035
00:40:59,680 --> 00:41:05,280
simulation environment which gives you

1036
00:41:01,760 --> 00:41:07,680
physics and then you can add plugins

1037
00:41:05,280 --> 00:41:10,000
the the gazebo simulation environment

1038
00:41:07,680 --> 00:41:12,640
has plug-ins which support g-streamer

1039
00:41:10,000 --> 00:41:14,319
pipelines so you can actually

1040
00:41:12,640 --> 00:41:16,640
i haven't played with that yet i'm

1041
00:41:14,319 --> 00:41:19,119
looking forward to that

1042
00:41:16,640 --> 00:41:20,560
so you can put a camera into your

1043
00:41:19,119 --> 00:41:22,160
simulation environment and then attach

1044
00:41:20,560 --> 00:41:23,839
it to a robot

1045
00:41:22,160 --> 00:41:26,880
and then have your control system just

1046
00:41:23,839 --> 00:41:29,119
everything is in simulation

1047
00:41:26,880 --> 00:41:29,119
great

1048
00:41:30,720 --> 00:41:33,200
so

1049
00:41:31,680 --> 00:41:35,280
what would you like to do now would you

1050
00:41:33,200 --> 00:41:38,079
like to continue the demo would you like

1051
00:41:35,280 --> 00:41:39,280
to finish or there are no more questions

1052
00:41:38,079 --> 00:41:41,920
currently

1053
00:41:39,280 --> 00:41:44,400
cool um

1054
00:41:41,920 --> 00:41:45,680
so there's the github to the

1055
00:41:44,400 --> 00:41:49,520
um

1056
00:41:45,680 --> 00:41:52,480
to the bridge that i've been building

1057
00:41:49,520 --> 00:41:54,079
the other thing i wanted to

1058
00:41:52,480 --> 00:41:57,200
maybe look into

1059
00:41:54,079 --> 00:42:00,240
maybe talk about was the

1060
00:41:57,200 --> 00:42:00,240
the launch syntax

1061
00:42:00,720 --> 00:42:03,839
with robots you usually have hardware

1062
00:42:02,480 --> 00:42:05,920
that moves a little bit slower than

1063
00:42:03,839 --> 00:42:07,680
software so you need to version control

1064
00:42:05,920 --> 00:42:09,280
like people like to have

1065
00:42:07,680 --> 00:42:11,200
robots

1066
00:42:09,280 --> 00:42:12,800
their software and their

1067
00:42:11,200 --> 00:42:14,319
control systems version controlled

1068
00:42:12,800 --> 00:42:15,200
alongside

1069
00:42:14,319 --> 00:42:18,319
um

1070
00:42:15,200 --> 00:42:19,200
alongside hardware it revisions

1071
00:42:18,319 --> 00:42:20,800
uh

1072
00:42:19,200 --> 00:42:23,119
so in this case i have

1073
00:42:20,800 --> 00:42:23,119
uh

1074
00:42:23,359 --> 00:42:27,520
launch descriptions um

1075
00:42:25,599 --> 00:42:30,079
this is the launch the ros launch syntax

1076
00:42:27,520 --> 00:42:32,000
in python it is just a python program

1077
00:42:30,079 --> 00:42:34,160
it's a little bit verbose and there's

1078
00:42:32,000 --> 00:42:35,119
actually some motion to

1079
00:42:34,160 --> 00:42:36,000
putting something together that's a

1080
00:42:35,119 --> 00:42:37,280
little

1081
00:42:36,000 --> 00:42:38,960
nicer

1082
00:42:37,280 --> 00:42:42,000
but you can see in this one

1083
00:42:38,960 --> 00:42:45,839
on the robot at least i've got

1084
00:42:42,000 --> 00:42:45,839
this is the motor driver node

1085
00:42:47,200 --> 00:42:52,000
and then i've got the pipeline

1086
00:42:49,599 --> 00:42:54,079
with the same package i've got the mixer

1087
00:42:52,000 --> 00:42:56,400
which handles joystick input

1088
00:42:54,079 --> 00:42:56,400
and then

1089
00:42:57,359 --> 00:43:02,880
uh handling the the hardware device to

1090
00:42:59,680 --> 00:43:02,880
synthesize the server commands

1091
00:43:03,280 --> 00:43:05,520
um

1092
00:43:08,480 --> 00:43:12,800
on the robot i load the raspberry pi

1093
00:43:10,240 --> 00:43:14,800
camera

1094
00:43:12,800 --> 00:43:16,560
oh yeah that was the other one uh

1095
00:43:14,800 --> 00:43:19,359
bitrate is

1096
00:43:16,560 --> 00:43:22,480
a jstreamer property so

1097
00:43:19,359 --> 00:43:22,480
where's the parameter table

1098
00:43:22,560 --> 00:43:26,319
the bitrate is a gstreamer property

1099
00:43:24,000 --> 00:43:28,480
which means that i can

1100
00:43:26,319 --> 00:43:30,960
let's

1101
00:43:28,480 --> 00:43:30,960
do this

1102
00:43:35,280 --> 00:43:39,119
so on the robot i've got the raspberry

1103
00:43:36,720 --> 00:43:41,440
cam udp

1104
00:43:39,119 --> 00:43:43,040
node this thing handles

1105
00:43:41,440 --> 00:43:45,200
the

1106
00:43:43,040 --> 00:43:46,880
raspberry pi camera source

1107
00:43:45,200 --> 00:43:50,800
the h.264

1108
00:43:46,880 --> 00:43:50,800
parsing and payloading

1109
00:43:51,520 --> 00:43:54,400
and these are all the properties for all

1110
00:43:52,960 --> 00:43:56,400
of those elements

1111
00:43:54,400 --> 00:44:00,079
which means

1112
00:43:56,400 --> 00:44:00,079
if i can find the bit rate

1113
00:44:00,240 --> 00:44:03,440
so this this node is running on my

1114
00:44:02,079 --> 00:44:05,599
laptop the raspberry pi camera is

1115
00:44:03,440 --> 00:44:06,839
attached to physical hardware

1116
00:44:05,599 --> 00:44:11,040
on the

1117
00:44:06,839 --> 00:44:11,040
robot i should be able to

1118
00:44:19,119 --> 00:44:23,160
if i select the node

1119
00:44:31,040 --> 00:44:36,079
camera node

1120
00:44:32,720 --> 00:44:38,720
a bit rate

1121
00:44:36,079 --> 00:44:38,720
i've lost it again

1122
00:44:46,160 --> 00:44:50,560
and i get that

1123
00:44:48,240 --> 00:44:53,839
it'll interrogate the robot and using

1124
00:44:50,560 --> 00:44:56,960
the ros pub sub um

1125
00:44:53,839 --> 00:44:58,480
metaphors on its bus it'll come out with

1126
00:44:56,960 --> 00:44:59,440
some integer value

1127
00:44:58,480 --> 00:45:02,800
now

1128
00:44:59,440 --> 00:45:05,839
the same parameter

1129
00:45:02,800 --> 00:45:05,839
i should be able to set

1130
00:45:11,040 --> 00:45:17,319
with a number

1131
00:45:13,760 --> 00:45:17,319
that is lower

1132
00:45:18,560 --> 00:45:23,920
and then on the device

1133
00:45:21,119 --> 00:45:25,839
it starts to reduce the quality of the

1134
00:45:23,920 --> 00:45:28,319
feed if i can drag that down

1135
00:45:25,839 --> 00:45:28,319
really low

1136
00:45:31,440 --> 00:45:34,640
you can see it gets real grainy very

1137
00:45:33,200 --> 00:45:38,240
quickly

1138
00:45:34,640 --> 00:45:38,240
i think i can drag that a little lower

1139
00:45:38,640 --> 00:45:45,040
until it's

1140
00:45:41,119 --> 00:45:45,040
you know scarcely visible so

1141
00:45:45,119 --> 00:45:48,319
this was actually inspired by

1142
00:45:47,040 --> 00:45:50,480
a

1143
00:45:48,319 --> 00:45:52,160
hackaday author that developed solarboy

1144
00:45:50,480 --> 00:45:53,359
which was a

1145
00:45:52,160 --> 00:45:57,119
cross-country

1146
00:45:53,359 --> 00:46:01,040
uh rover which was on the internet uh

1147
00:45:57,119 --> 00:46:03,839
this property mapping was intended as

1148
00:46:01,040 --> 00:46:04,720
an option for him to

1149
00:46:03,839 --> 00:46:06,720
uh

1150
00:46:04,720 --> 00:46:09,920
manage his bandwidth more sort of

1151
00:46:06,720 --> 00:46:09,920
automatically on the device

1152
00:46:11,200 --> 00:46:16,880
okay bret we just have a last question i

1153
00:46:13,839 --> 00:46:21,200
think before we wrap up

1154
00:46:16,880 --> 00:46:22,800
what runtime are you using on the esp32

1155
00:46:21,200 --> 00:46:24,319
oh

1156
00:46:22,800 --> 00:46:26,839
that was also my previous talk that was

1157
00:46:24,319 --> 00:46:29,680
the whole talk um the runtime on the

1158
00:46:26,839 --> 00:46:32,319
esp32 is called micro gross

1159
00:46:29,680 --> 00:46:34,319
and then it announces itself on

1160
00:46:32,319 --> 00:46:35,839
the the local network as a whole rolls

1161
00:46:34,319 --> 00:46:38,400
node uh

1162
00:46:35,839 --> 00:46:40,000
really cool very recent uh really

1163
00:46:38,400 --> 00:46:41,760
excited about that one go take a look at

1164
00:46:40,000 --> 00:46:43,359
the previous talk

1165
00:46:41,760 --> 00:46:45,680
excellent thank you

1166
00:46:43,359 --> 00:46:47,280
well thank you for the wonderful talk

1167
00:46:45,680 --> 00:46:48,960
and

1168
00:46:47,280 --> 00:46:51,040
that's it for now we have a 10 minute

1169
00:46:48,960 --> 00:46:55,640
break before the next session

1170
00:46:51,040 --> 00:46:55,640
great thank you thank you brett