1 00:00:00,480 --> 00:00:03,480 foreign 2 00:00:08,220 --> 00:00:12,360 hello everybody and welcome back to the 3 00:00:10,500 --> 00:00:13,980 our connected Universe specialist track 4 00:00:12,360 --> 00:00:16,039 this afternoon is everyone full of 5 00:00:13,980 --> 00:00:19,380 energy after afternoon tea 6 00:00:16,039 --> 00:00:20,939 it's better than I expected well done uh 7 00:00:19,380 --> 00:00:22,260 we're gonna get straight on into talks 8 00:00:20,939 --> 00:00:24,180 this afternoon I'm really excited to 9 00:00:22,260 --> 00:00:26,100 introduce Andy jelly to the stage uh 10 00:00:24,180 --> 00:00:28,560 who's going to be talking to us about 11 00:00:26,100 --> 00:00:30,720 gstreamer which is really exciting so 12 00:00:28,560 --> 00:00:32,940 without too much uh delay I'm going to 13 00:00:30,720 --> 00:00:35,520 hand over to our our 14 00:00:32,940 --> 00:00:37,739 hacker hobbyist in Chief in the track 15 00:00:35,520 --> 00:00:39,480 Andy cheers 16 00:00:37,739 --> 00:00:40,320 good afternoon everyone it's good to be 17 00:00:39,480 --> 00:00:42,180 here 18 00:00:40,320 --> 00:00:44,820 um so as Jack said I'm talking about 19 00:00:42,180 --> 00:00:48,300 python G streamer in particular focus on 20 00:00:44,820 --> 00:00:49,980 webrtc and uh rtsp 21 00:00:48,300 --> 00:00:52,320 um it's fairly dense like uh 22 00:00:49,980 --> 00:00:53,640 presentations so there's a slide deck if 23 00:00:52,320 --> 00:00:56,280 you've got a mobile device and I wanted 24 00:00:53,640 --> 00:00:57,600 to check that out and uh if I don't get 25 00:00:56,280 --> 00:00:58,620 through everything it's um we've got 26 00:00:57,600 --> 00:01:00,780 lots of questions please hit me up 27 00:00:58,620 --> 00:01:02,899 anytime until Tuesday or just catch up 28 00:01:00,780 --> 00:01:05,159 on Discord 29 00:01:02,899 --> 00:01:06,900 I'd like to begin today by acknowledging 30 00:01:05,159 --> 00:01:08,220 the Karana people the traditional 31 00:01:06,900 --> 00:01:09,060 customers the land on which We Gather 32 00:01:08,220 --> 00:01:10,740 today 33 00:01:09,060 --> 00:01:12,360 pay my respects to their orders past and 34 00:01:10,740 --> 00:01:13,680 present I extend that respect to 35 00:01:12,360 --> 00:01:16,400 Aboriginal and Torres Strait Islanders 36 00:01:13,680 --> 00:01:19,020 at peoples here today 37 00:01:16,400 --> 00:01:21,659 so uh what I hope to cover is uh 38 00:01:19,020 --> 00:01:22,979 obviously a g streamer 39 00:01:21,659 --> 00:01:26,400 um provide a bit of an overview about 40 00:01:22,979 --> 00:01:27,659 rtsp and webrtc bit of a refresher uh 41 00:01:26,400 --> 00:01:29,400 just quickly provide some Alternatives 42 00:01:27,659 --> 00:01:32,880 because there's many ways to skin this 43 00:01:29,400 --> 00:01:34,259 cat or solve this problem and then do a 44 00:01:32,880 --> 00:01:36,420 deep dive 45 00:01:34,259 --> 00:01:38,220 um and because uh it's hard to cover in 46 00:01:36,420 --> 00:01:40,079 a short time I'll there'll be links to 47 00:01:38,220 --> 00:01:42,540 source code nothing so you can check in 48 00:01:40,079 --> 00:01:45,119 your at your leisure 49 00:01:42,540 --> 00:01:48,020 so when when does it make sense to use 50 00:01:45,119 --> 00:01:51,060 uh g-string Miranda for a party and uh 51 00:01:48,020 --> 00:01:53,640 rtsb uh there's lots of applications 52 00:01:51,060 --> 00:01:55,259 these days uh teleconferencing um going 53 00:01:53,640 --> 00:01:57,119 out from drones looking at surveying 54 00:01:55,259 --> 00:01:59,159 surveying the world 55 00:01:57,119 --> 00:02:00,360 um trying to deal things with an 56 00:01:59,159 --> 00:02:02,280 emerging situation we've got to respond 57 00:02:00,360 --> 00:02:03,960 quickly we're low latency 58 00:02:02,280 --> 00:02:06,060 um media streaming makes a lot of sense 59 00:02:03,960 --> 00:02:07,259 and with the AI going to everything 60 00:02:06,060 --> 00:02:09,979 these days 61 00:02:07,259 --> 00:02:12,780 um like it's to attract investors 62 00:02:09,979 --> 00:02:14,640 and we're dealing with um multimodal 63 00:02:12,780 --> 00:02:17,099 large language models now I'm dealing 64 00:02:14,640 --> 00:02:19,200 with images and uh audio and so on is 65 00:02:17,099 --> 00:02:22,400 also a thing and uh and they'll be and 66 00:02:19,200 --> 00:02:22,400 there'll be robots everywhere you know 67 00:02:22,640 --> 00:02:27,060 driving down the streets in the form of 68 00:02:24,480 --> 00:02:28,800 self-driving cars and so the basic 69 00:02:27,060 --> 00:02:30,840 pipeline is going to be to so decodes 70 00:02:28,800 --> 00:02:32,819 the media you'll do some pre-processing 71 00:02:30,840 --> 00:02:33,959 on it then there'll be as analysis and 72 00:02:32,819 --> 00:02:35,760 that's something you might want to be 73 00:02:33,959 --> 00:02:37,200 doing in Python and then post process 74 00:02:35,760 --> 00:02:38,940 and re-encode again 75 00:02:37,200 --> 00:02:40,260 and if you're going for extra points 76 00:02:38,940 --> 00:02:42,180 it'd be really nice to do that entirely 77 00:02:40,260 --> 00:02:43,860 in the GPU which is possible with G 78 00:02:42,180 --> 00:02:46,379 streamer 79 00:02:43,860 --> 00:02:47,879 um so just as a bit of a distraction uh 80 00:02:46,379 --> 00:02:50,400 a personal 81 00:02:47,879 --> 00:02:51,959 um hobby horse when working across a lot 82 00:02:50,400 --> 00:02:53,640 of different domains 83 00:02:51,959 --> 00:02:55,440 um there's so many different Frameworks 84 00:02:53,640 --> 00:02:56,879 for all these things you need to uh glue 85 00:02:55,440 --> 00:02:59,360 together so you want to be able to take 86 00:02:56,879 --> 00:03:02,640 um information from a nursing sensors 87 00:02:59,360 --> 00:03:04,620 that might be combined with robotics in 88 00:03:02,640 --> 00:03:06,239 a much stream media off those robots to 89 00:03:04,620 --> 00:03:08,040 to your back end running through machine 90 00:03:06,239 --> 00:03:10,200 learning close the loop and send some 91 00:03:08,040 --> 00:03:11,760 some trolls things back again and all 92 00:03:10,200 --> 00:03:13,440 these it's just really much harder than 93 00:03:11,760 --> 00:03:14,519 it needs to be these days because I 94 00:03:13,440 --> 00:03:16,319 always have a Frameworks don't 95 00:03:14,519 --> 00:03:17,400 interoperate very well so I'm 96 00:03:16,319 --> 00:03:18,959 particularly interested in that solving 97 00:03:17,400 --> 00:03:20,760 that problem and I'm looking 98 00:03:18,959 --> 00:03:23,700 um that g streamer through through that 99 00:03:20,760 --> 00:03:26,519 lens and so um I would have a consistent 100 00:03:23,700 --> 00:03:28,260 set of apis for in this case for media 101 00:03:26,519 --> 00:03:30,420 streaming I want to be able to have 102 00:03:28,260 --> 00:03:32,879 those set up as a pipeline where I um 103 00:03:30,420 --> 00:03:34,680 create the components so I don't 104 00:03:32,879 --> 00:03:36,480 dynamically the gstrom elements I want 105 00:03:34,680 --> 00:03:38,040 to be able to inspect them and change 106 00:03:36,480 --> 00:03:40,019 their change their parents on the Fly 107 00:03:38,040 --> 00:03:42,000 and so I don't want this to be a black 108 00:03:40,019 --> 00:03:45,299 box and so that's why diving in Fairly 109 00:03:42,000 --> 00:03:47,580 deep on the programming aspect it 110 00:03:45,299 --> 00:03:49,739 um so to start a live demonstration 111 00:03:47,580 --> 00:03:51,180 um so what's what's going on here is the 112 00:03:49,739 --> 00:03:54,239 uh right in the middle there's an rtsp 113 00:03:51,180 --> 00:03:56,400 camera there's a white box and that's uh 114 00:03:54,239 --> 00:03:58,500 and the idea is take the the stream from 115 00:03:56,400 --> 00:04:00,360 that rtsp camera run it through a 116 00:03:58,500 --> 00:04:02,760 pipeline and then have uh have that 117 00:04:00,360 --> 00:04:04,260 presented as our web RC video so the web 118 00:04:02,760 --> 00:04:06,480 artistic part is all the blue parts we 119 00:04:04,260 --> 00:04:07,680 have to um set up our work bar two 120 00:04:06,480 --> 00:04:10,379 session through the signaling server up 121 00:04:07,680 --> 00:04:12,060 on the top left uh and so the browser 122 00:04:10,379 --> 00:04:14,040 has to interact with the uh the back end 123 00:04:12,060 --> 00:04:15,720 uh through the signaling server they 124 00:04:14,040 --> 00:04:17,340 negotiate and then that allows them to 125 00:04:15,720 --> 00:04:19,440 set up the uh the session so it's 126 00:04:17,340 --> 00:04:20,880 roughly what we're seeing and so um 127 00:04:19,440 --> 00:04:22,199 fingers crossed this works is the the 128 00:04:20,880 --> 00:04:24,240 Network's been up and down a bit for me 129 00:04:22,199 --> 00:04:26,280 today uh but hopefully we'll see as a 130 00:04:24,240 --> 00:04:27,660 web artistic view that's um being 131 00:04:26,280 --> 00:04:30,060 streamed to the back end and then 132 00:04:27,660 --> 00:04:32,100 appears on the browser so let's see how 133 00:04:30,060 --> 00:04:34,699 we go 134 00:04:32,100 --> 00:04:34,699 about that 135 00:04:35,580 --> 00:04:40,259 so um 136 00:04:37,020 --> 00:04:41,880 this is a run against a back end 137 00:04:40,259 --> 00:04:43,680 um up in the top left there's a number 138 00:04:41,880 --> 00:04:45,720 of devices which we can select so 139 00:04:43,680 --> 00:04:47,820 there's um our office cameras web RC 140 00:04:45,720 --> 00:04:49,740 camera I can also select the uh the 141 00:04:47,820 --> 00:04:52,199 camera on the on the laptop and also 142 00:04:49,740 --> 00:04:54,419 just a mock video and you can see in the 143 00:04:52,199 --> 00:04:56,940 back end we should be able to see the um 144 00:04:54,419 --> 00:04:59,580 the process is running so there's 145 00:04:56,940 --> 00:05:01,380 there's these devices I can look at yeah 146 00:04:59,580 --> 00:05:03,180 I like things to be 147 00:05:01,380 --> 00:05:04,740 white boxes rather than black boxes I 148 00:05:03,180 --> 00:05:06,419 can see inside what they're doing and 149 00:05:04,740 --> 00:05:08,340 then we can see that's the rtsp source 150 00:05:06,419 --> 00:05:09,600 so and so I'm just down there you can 151 00:05:08,340 --> 00:05:12,479 see it's um 152 00:05:09,600 --> 00:05:14,220 loading up four inches a second and uh 153 00:05:12,479 --> 00:05:18,979 hopefully what we'll see is uh of the 154 00:05:14,220 --> 00:05:18,979 pipeline starting up so it's yes sir 155 00:05:23,479 --> 00:05:27,120 okay 156 00:05:25,220 --> 00:05:28,199 so it's going to select the office 157 00:05:27,120 --> 00:05:30,720 camera 158 00:05:28,199 --> 00:05:32,460 and then try and start the uh the way 159 00:05:30,720 --> 00:05:33,900 back to see session 160 00:05:32,460 --> 00:05:35,520 and there we go that's that's if you're 161 00:05:33,900 --> 00:05:37,320 at our office in in the city in 162 00:05:35,520 --> 00:05:38,460 Melbourne it's uh it's like a somewhat 163 00:05:37,320 --> 00:05:40,800 sunny day 164 00:05:38,460 --> 00:05:43,740 and we can see in the back end there's 165 00:05:40,800 --> 00:05:46,199 now uh there's now a webrtc uh session 166 00:05:43,740 --> 00:05:48,240 pair there's the web there's the rtsp 167 00:05:46,199 --> 00:05:51,479 data source we're doing some processing 168 00:05:48,240 --> 00:05:54,120 Python and then writing that as webrtc 169 00:05:51,479 --> 00:05:56,940 so uh I can also um 170 00:05:54,120 --> 00:05:58,500 select a uh a different data source so 171 00:05:56,940 --> 00:06:01,520 you can go select the web camera on the 172 00:05:58,500 --> 00:06:01,520 uh on the laptop 173 00:06:02,820 --> 00:06:05,600 yes 174 00:06:06,300 --> 00:06:11,450 okay I'm going to fix it 175 00:06:09,060 --> 00:06:14,630 okay okay 176 00:06:11,450 --> 00:06:14,630 [Music] 177 00:06:17,880 --> 00:06:20,600 try that again 178 00:06:24,180 --> 00:06:27,900 here we go 179 00:06:25,740 --> 00:06:30,060 and you can see it's taking the web 180 00:06:27,900 --> 00:06:31,319 their web camera and streaming that 181 00:06:30,060 --> 00:06:33,240 through our um 182 00:06:31,319 --> 00:06:34,440 uh email model to do face detection and 183 00:06:33,240 --> 00:06:35,460 the Python's just putting a bounding box 184 00:06:34,440 --> 00:06:36,780 around it 185 00:06:35,460 --> 00:06:39,180 so let's uh 186 00:06:36,780 --> 00:06:41,900 that's sort of the AIM 187 00:06:39,180 --> 00:06:41,900 shut that down 188 00:06:45,419 --> 00:06:48,319 demo fail 189 00:06:48,660 --> 00:06:51,199 all right 190 00:06:51,720 --> 00:06:55,199 okay 191 00:06:53,400 --> 00:06:56,940 so just like to quickly cover some 192 00:06:55,199 --> 00:06:59,160 alternative approaches so everyone's 193 00:06:56,940 --> 00:07:01,259 probably pretty familiar with opencv and 194 00:06:59,160 --> 00:07:03,300 just in roughly one page of code you can 195 00:07:01,259 --> 00:07:05,220 see that we can in Orange do the G 196 00:07:03,300 --> 00:07:08,639 string apart where we create a G Street 197 00:07:05,220 --> 00:07:11,100 g-streamer pipeline as a string we then 198 00:07:08,639 --> 00:07:14,160 use a video capture to create a an 199 00:07:11,100 --> 00:07:15,479 opencv video in object we do the same 200 00:07:14,160 --> 00:07:16,500 thing for the output Pipeline and then 201 00:07:15,479 --> 00:07:18,600 we sit and loop where we just read 202 00:07:16,500 --> 00:07:20,819 images in Off Video in do a bit of 203 00:07:18,600 --> 00:07:21,840 processing and write out imagine a lot 204 00:07:20,819 --> 00:07:23,039 of people in the audience have done that 205 00:07:21,840 --> 00:07:24,960 already 206 00:07:23,039 --> 00:07:26,280 um which is which is nothing wrong with 207 00:07:24,960 --> 00:07:28,740 that approach it gives you all the 208 00:07:26,280 --> 00:07:29,940 access to opencv it's very simple 209 00:07:28,740 --> 00:07:31,620 um a little bit of downside is doing 210 00:07:29,940 --> 00:07:34,080 those I'm sure strings gets quite quite 211 00:07:31,620 --> 00:07:35,460 messy however the big problem is that um 212 00:07:34,080 --> 00:07:36,840 you've completely lost all access to 213 00:07:35,460 --> 00:07:38,880 dstreamer's capabilities because you 214 00:07:36,840 --> 00:07:40,979 just really got these uh video and video 215 00:07:38,880 --> 00:07:43,080 out objects and uh you can't access like 216 00:07:40,979 --> 00:07:46,020 meta buffers or anything else deep in 217 00:07:43,080 --> 00:07:47,039 g-streamer and also doesn't work for 218 00:07:46,020 --> 00:07:49,400 audio 219 00:07:47,039 --> 00:07:51,660 another approach is uh to use ffmpeg 220 00:07:49,400 --> 00:07:53,759 also not not too difficult you once 221 00:07:51,660 --> 00:07:56,280 again set up these uh these pipelines so 222 00:07:53,759 --> 00:07:57,780 input and output up the top process one 223 00:07:56,280 --> 00:07:59,160 process two and once again certain Loop 224 00:07:57,780 --> 00:08:01,919 just reading up reading bytes off 225 00:07:59,160 --> 00:08:03,599 process one uh doing some processing and 226 00:08:01,919 --> 00:08:04,259 running back out 227 00:08:03,599 --> 00:08:06,300 um 228 00:08:04,259 --> 00:08:07,919 also that works too 229 00:08:06,300 --> 00:08:10,199 um that's how it also handles audio 230 00:08:07,919 --> 00:08:12,660 ffmp's really powerful but it's just a 231 00:08:10,199 --> 00:08:13,979 wrapper around ffmpeg so you don't once 232 00:08:12,660 --> 00:08:15,960 again you can't leverage leverage tea 233 00:08:13,979 --> 00:08:18,840 streamer but still worth considering if 234 00:08:15,960 --> 00:08:20,520 that's um if that's what you need to do 235 00:08:18,840 --> 00:08:22,020 all right so let's let's get again 236 00:08:20,520 --> 00:08:23,819 Steve's Famous so just just for a 237 00:08:22,020 --> 00:08:26,220 reference I um included how to install 238 00:08:23,819 --> 00:08:27,419 gstreamer on Battlefield 22 it's not 239 00:08:26,220 --> 00:08:28,740 getting much easier these days a few 240 00:08:27,419 --> 00:08:31,080 years ago used to be quite a trial now 241 00:08:28,740 --> 00:08:32,219 it's just um you start install a few 242 00:08:31,080 --> 00:08:33,680 packages 243 00:08:32,219 --> 00:08:37,260 um set up your python environment 244 00:08:33,680 --> 00:08:38,520 install uh the python G object what that 245 00:08:37,260 --> 00:08:42,180 does that allows you to access all of 246 00:08:38,520 --> 00:08:43,919 the EG no mic libraries on on a Linux so 247 00:08:42,180 --> 00:08:47,100 if you get to gpk many things and in 248 00:08:43,919 --> 00:08:48,300 this case a g streamer and uh and if you 249 00:08:47,100 --> 00:08:50,160 can just run through a little sequence 250 00:08:48,300 --> 00:08:51,959 down the bottom of um importing the uh 251 00:08:50,160 --> 00:08:53,640 The Gnome inspector object and then 252 00:08:51,959 --> 00:08:55,560 doing a g streamer net you know that you 253 00:08:53,640 --> 00:08:57,360 know I'm just successfully installed and 254 00:08:55,560 --> 00:08:59,880 I'll put a link up the top to how to do 255 00:08:57,360 --> 00:09:01,680 that the online instructions and 256 00:08:59,880 --> 00:09:04,440 similarly for for Mac OS 257 00:09:01,680 --> 00:09:07,140 um you just use Brew to install GST 258 00:09:04,440 --> 00:09:07,680 python it's just everything's all in one 259 00:09:07,140 --> 00:09:09,899 um 260 00:09:07,680 --> 00:09:11,279 one big package and it's once again very 261 00:09:09,899 --> 00:09:12,839 easy test so it's that's kind of good 262 00:09:11,279 --> 00:09:14,040 news I don't I don't know how to do this 263 00:09:12,839 --> 00:09:15,839 on Windows 264 00:09:14,040 --> 00:09:19,320 not sure what's possible 265 00:09:15,839 --> 00:09:21,000 all right so some look at the rtsp now 266 00:09:19,320 --> 00:09:22,440 so it's just a quick profession if 267 00:09:21,000 --> 00:09:24,480 anyone's not familiar 268 00:09:22,440 --> 00:09:26,459 um what rtsp does is it but it says 269 00:09:24,480 --> 00:09:27,959 basically it gives the details and 270 00:09:26,459 --> 00:09:29,279 negotiates initiates 271 00:09:27,959 --> 00:09:32,940 um 272 00:09:29,279 --> 00:09:34,740 uh talking to an RTP server so RTP is 273 00:09:32,940 --> 00:09:36,959 the real-time protocol it just wraps a 274 00:09:34,740 --> 00:09:39,000 proper product like h.264 it's typically 275 00:09:36,959 --> 00:09:40,940 used for things like network cameras uh 276 00:09:39,000 --> 00:09:43,920 where you can um you know get an 277 00:09:40,940 --> 00:09:46,560 artistic video stream and what happens 278 00:09:43,920 --> 00:09:48,540 is you have uh you have your um your 279 00:09:46,560 --> 00:09:51,300 client which is um you know it could be 280 00:09:48,540 --> 00:09:52,800 um your your python code and you uh I 281 00:09:51,300 --> 00:09:54,779 would have the the orange boxes here 282 00:09:52,800 --> 00:09:57,000 representing the client side where the 283 00:09:54,779 --> 00:09:58,440 rtsb client will make some requests on 284 00:09:57,000 --> 00:10:01,500 the rtsp server which is your network 285 00:09:58,440 --> 00:10:03,180 camera it will describe some um what's 286 00:10:01,500 --> 00:10:05,220 your commands what's to send in terms of 287 00:10:03,180 --> 00:10:06,959 play and terminate extra setup options 288 00:10:05,220 --> 00:10:09,240 uh which is to you know say what sort of 289 00:10:06,959 --> 00:10:11,339 channels and resolutions and things and 290 00:10:09,240 --> 00:10:13,680 and the play commanders initiates that 291 00:10:11,339 --> 00:10:15,480 then gets the RTP protocol going from 292 00:10:13,680 --> 00:10:17,399 the RTP server aspect of the camera 293 00:10:15,480 --> 00:10:18,660 which sends you video back and then you 294 00:10:17,399 --> 00:10:20,339 can just terminate and tear it down so 295 00:10:18,660 --> 00:10:22,980 it's pretty it's a it's reasonably 296 00:10:20,339 --> 00:10:24,839 straightforward 297 00:10:22,980 --> 00:10:27,779 um how you can do that in gstream is you 298 00:10:24,839 --> 00:10:29,700 uh construct a a launch a string of 299 00:10:27,779 --> 00:10:32,519 pipeline ones so I'd say you want to 300 00:10:29,700 --> 00:10:35,040 access an RTP Source I want to um 301 00:10:32,519 --> 00:10:38,279 depacketize that to get the h264 video 302 00:10:35,040 --> 00:10:40,019 video code account uh they know you uh 303 00:10:38,279 --> 00:10:43,560 decode that and convert it into um 304 00:10:40,019 --> 00:10:45,899 effectively raw uh raw raw buffers that 305 00:10:43,560 --> 00:10:47,700 um G streamer can deal with and that the 306 00:10:45,899 --> 00:10:50,459 very this this string would display an 307 00:10:47,700 --> 00:10:51,959 RTP session on up on your um on your 308 00:10:50,459 --> 00:10:54,000 desktop at the bottom we just replace 309 00:10:51,959 --> 00:10:55,800 that video sync for the desktop with uh 310 00:10:54,000 --> 00:10:58,140 appsync which is how you access 311 00:10:55,800 --> 00:11:01,620 um G streamer plugins in in part within 312 00:10:58,140 --> 00:11:02,880 python so that probably that doing this 313 00:11:01,620 --> 00:11:04,320 from the command line is that they've 314 00:11:02,880 --> 00:11:05,700 got no way to access the the video 315 00:11:04,320 --> 00:11:07,440 stream from python so it's what we're 316 00:11:05,700 --> 00:11:10,079 about to get into 317 00:11:07,440 --> 00:11:11,880 uh so how this looks uh from a 318 00:11:10,079 --> 00:11:13,740 conceptual view is there's some data 319 00:11:11,880 --> 00:11:15,180 some data structures there's the the 320 00:11:13,740 --> 00:11:16,680 pipeline on the left and the pipeline 321 00:11:15,180 --> 00:11:19,140 consists of these the elements that 322 00:11:16,680 --> 00:11:22,320 we're constructing the uh these plugins 323 00:11:19,140 --> 00:11:23,459 so the RTP Source depacketizers on until 324 00:11:22,320 --> 00:11:25,200 you finally get to the app sync which 325 00:11:23,459 --> 00:11:27,120 allows you to extract images off the off 326 00:11:25,200 --> 00:11:28,620 the pipeline the pipeline has a 327 00:11:27,120 --> 00:11:30,300 reference to a bus which has a reference 328 00:11:28,620 --> 00:11:32,160 to the messages which is a stream across 329 00:11:30,300 --> 00:11:33,360 the bus these are the gstream that 330 00:11:32,160 --> 00:11:35,519 buffers 331 00:11:33,360 --> 00:11:37,260 uh and what so that's the first episode 332 00:11:35,519 --> 00:11:39,060 is to set that pipeline up we do that by 333 00:11:37,260 --> 00:11:40,620 just like creating the pipelines and 334 00:11:39,060 --> 00:11:42,720 then adding adding the pipeline and 335 00:11:40,620 --> 00:11:43,920 linking them together this thing so the 336 00:11:42,720 --> 00:11:46,800 first step is we'll create the input 337 00:11:43,920 --> 00:11:49,260 PowerPoint which um takes the rtsp video 338 00:11:46,800 --> 00:11:51,180 and uh and it gives you an application 339 00:11:49,260 --> 00:11:52,440 sync to get your images the second thing 340 00:11:51,180 --> 00:11:54,000 we'll do on the right is we'll create 341 00:11:52,440 --> 00:11:56,279 the output pipeline which takes to your 342 00:11:54,000 --> 00:11:57,720 app appsource plugin which is how you 343 00:11:56,279 --> 00:12:01,500 take images from Python and get them 344 00:11:57,720 --> 00:12:03,120 back onto the g-strip um uh pipeline and 345 00:12:01,500 --> 00:12:05,040 then I would and then that takes good 346 00:12:03,120 --> 00:12:07,860 piercing so you can pop a pump out it 347 00:12:05,040 --> 00:12:10,140 RTP at the end the third step is to 348 00:12:07,860 --> 00:12:12,000 create a callback function called sample 349 00:12:10,140 --> 00:12:14,220 image um that takes they'll take images 350 00:12:12,000 --> 00:12:16,140 from the uh we'll take buffers from the 351 00:12:14,220 --> 00:12:17,459 G stripper Pipeline and just converts 352 00:12:16,140 --> 00:12:20,220 them into images which we can then sit 353 00:12:17,459 --> 00:12:22,140 in the loop and um process processing 354 00:12:20,220 --> 00:12:25,380 and then then we then we push those out 355 00:12:22,140 --> 00:12:26,339 to that pipeline again so how that looks 356 00:12:25,380 --> 00:12:29,220 in code 357 00:12:26,339 --> 00:12:30,480 there's a bit of full screen is that the 358 00:12:29,220 --> 00:12:33,600 very top where you will construct 359 00:12:30,480 --> 00:12:35,640 construct that pipeline uh the SEC the 360 00:12:33,600 --> 00:12:36,839 Second Step box is where we uh we both 361 00:12:35,640 --> 00:12:39,660 reach the elements with using element 362 00:12:36,839 --> 00:12:42,060 Factory we create create those elements 363 00:12:39,660 --> 00:12:43,620 we start with the the rtsp source and 364 00:12:42,060 --> 00:12:45,420 then we end up finally having a sync so 365 00:12:43,620 --> 00:12:47,459 we're just using element factory make 366 00:12:45,420 --> 00:12:48,959 the second box that we add out all those 367 00:12:47,459 --> 00:12:51,000 pipelines together 368 00:12:48,959 --> 00:12:52,560 uh into the pipeline and the third is 369 00:12:51,000 --> 00:12:55,019 with boxes where we link them together 370 00:12:52,560 --> 00:12:57,420 so it basically steps up you create you 371 00:12:55,019 --> 00:12:58,980 create your elements you add them to the 372 00:12:57,420 --> 00:13:01,500 pipeline and then you link them together 373 00:12:58,980 --> 00:13:03,660 that gives you a pipeline 374 00:13:01,500 --> 00:13:05,459 so this is also doing the same thing for 375 00:13:03,660 --> 00:13:07,560 the output pipeline uh the difference 376 00:13:05,459 --> 00:13:10,200 here is we are whereas in the input 377 00:13:07,560 --> 00:13:11,880 pipeline we start with a an RTP Source 378 00:13:10,200 --> 00:13:13,800 at the very bottom end up with an app 379 00:13:11,880 --> 00:13:16,260 sync in this case we're going to start 380 00:13:13,800 --> 00:13:18,600 with an app source which are and then we 381 00:13:16,260 --> 00:13:22,380 add we add the elements to um to 382 00:13:18,600 --> 00:13:24,660 re-encode that as h.264 we do the 383 00:13:22,380 --> 00:13:27,260 payload for RTP and then we link all 384 00:13:24,660 --> 00:13:27,260 these elements together 385 00:13:27,980 --> 00:13:32,700 doing this is uh just use the GST method 386 00:13:31,380 --> 00:13:34,200 called parse launcher it takes your 387 00:13:32,700 --> 00:13:36,180 string and it just immediately gives you 388 00:13:34,200 --> 00:13:37,920 a pipeline in this case I want to create 389 00:13:36,180 --> 00:13:39,660 that balance myself until I can access 390 00:13:37,920 --> 00:13:41,639 the properties so for example to set 391 00:13:39,660 --> 00:13:45,120 properties like the host and port on the 392 00:13:41,639 --> 00:13:47,880 um on the UDP sync for example 393 00:13:45,120 --> 00:13:49,620 once uh once we've got our pipeline 394 00:13:47,880 --> 00:13:52,320 we'll create this function called sample 395 00:13:49,620 --> 00:13:54,480 by image the sample image chart will be 396 00:13:52,320 --> 00:13:56,160 as a callback and so we'll uh we'll get 397 00:13:54,480 --> 00:13:58,260 it will get pulled by these the sync 398 00:13:56,160 --> 00:14:01,380 which which is the end of the input 399 00:13:58,260 --> 00:14:03,060 pipeline this sync um we can give us a 400 00:14:01,380 --> 00:14:04,740 sample we pull out the buffer the buff 401 00:14:03,060 --> 00:14:07,680 has information like the resolution it 402 00:14:04,740 --> 00:14:09,420 also has timing information and we can 403 00:14:07,680 --> 00:14:11,519 then turn that buffer into a into a 404 00:14:09,420 --> 00:14:15,120 numpy image which we then we then put 405 00:14:11,519 --> 00:14:17,639 into a queue uh and that once once we've 406 00:14:15,120 --> 00:14:19,860 had that sample um image show a callback 407 00:14:17,639 --> 00:14:23,160 we can then just attach the sync to that 408 00:14:19,860 --> 00:14:24,720 by using the sync set property method so 409 00:14:23,160 --> 00:14:26,399 so both stage we should now have an 410 00:14:24,720 --> 00:14:27,839 input pipeline our Port Pipeline and 411 00:14:26,399 --> 00:14:29,639 we've connected the sync to our sample 412 00:14:27,839 --> 00:14:31,139 image pullback 413 00:14:29,639 --> 00:14:34,200 and then the final step is now to run 414 00:14:31,139 --> 00:14:36,420 the pipeline with a pipeline input in 415 00:14:34,200 --> 00:14:38,940 and pipeline objects we can start them 416 00:14:36,420 --> 00:14:41,519 start them playing and then um from the 417 00:14:38,940 --> 00:14:44,160 pipeline we get the bus we uh we're 418 00:14:41,519 --> 00:14:46,199 going to timeout uh every um every uh 419 00:14:44,160 --> 00:14:48,660 second that's a thousand five thousand 420 00:14:46,199 --> 00:14:50,399 nanoseconds and then we've done system 421 00:14:48,660 --> 00:14:53,279 Loop we pick sit and loop we pick up the 422 00:14:50,399 --> 00:14:55,320 messages off the bus and uh we uh I've 423 00:14:53,279 --> 00:14:56,699 cheated a bit here but I've I've got the 424 00:14:55,320 --> 00:14:58,500 code that shows we're pulling the image 425 00:14:56,699 --> 00:15:00,000 from a queue but we we're about to pull 426 00:14:58,500 --> 00:15:02,760 the image from a queue we do have python 427 00:15:00,000 --> 00:15:04,320 processing we uh turn that in that 428 00:15:02,760 --> 00:15:06,060 processed image back and divides into a 429 00:15:04,320 --> 00:15:08,699 buffer and then we emit that back onto 430 00:15:06,060 --> 00:15:10,500 the uh the Puffin output source and we 431 00:15:08,699 --> 00:15:11,940 can also check messages to see if um if 432 00:15:10,500 --> 00:15:13,980 we've received an end of end of stream 433 00:15:11,940 --> 00:15:16,019 for example the network I have failed 434 00:15:13,980 --> 00:15:18,180 all the district we move on to the end 435 00:15:16,019 --> 00:15:20,760 of file so after all that you can see 436 00:15:18,180 --> 00:15:23,699 it's a lot more work than um opencv or 437 00:15:20,760 --> 00:15:25,199 ffmpeg which is where I started but um 438 00:15:23,699 --> 00:15:26,579 once you've done it you can basically 439 00:15:25,199 --> 00:15:28,199 put it yeah put in a little library and 440 00:15:26,579 --> 00:15:29,760 forget about it but it gives you access 441 00:15:28,199 --> 00:15:32,279 to every Pipeline and the ability to 442 00:15:29,760 --> 00:15:34,019 change properties and to um really uh 443 00:15:32,279 --> 00:15:36,120 delve deeply into what's going on on the 444 00:15:34,019 --> 00:15:41,339 bus 445 00:15:36,120 --> 00:15:44,459 okay so now for um web HC uh so webrtc 446 00:15:41,339 --> 00:15:46,019 was a standard um it was a long time 447 00:15:44,459 --> 00:15:47,519 coming that's what I think it's about 10 448 00:15:46,019 --> 00:15:48,720 10 or more years until it appeared in 449 00:15:47,519 --> 00:15:51,540 all the major browsers but fortunately 450 00:15:48,720 --> 00:15:53,220 we're there and it's um basically ends 451 00:15:51,540 --> 00:15:54,720 up being in your browser a library that 452 00:15:53,220 --> 00:15:55,740 you can access from JavaScript which 453 00:15:54,720 --> 00:15:57,120 allows you to stop your media 454 00:15:55,740 --> 00:16:00,720 connections to like your web camera and 455 00:15:57,120 --> 00:16:02,820 uh and so on your microphone and uh and 456 00:16:00,720 --> 00:16:04,680 then it involves a lot of a number of 457 00:16:02,820 --> 00:16:05,880 servers uh that you need to have out in 458 00:16:04,680 --> 00:16:07,980 the net to help you with tuning through 459 00:16:05,880 --> 00:16:09,959 firewalls but then you have to provide 460 00:16:07,980 --> 00:16:12,360 you with um multiple channels of video 461 00:16:09,959 --> 00:16:15,180 audio and data they're bi-directional 462 00:16:12,360 --> 00:16:17,519 low latency which means about around a 463 00:16:15,180 --> 00:16:19,079 second or less and it's secure and it 464 00:16:17,519 --> 00:16:20,339 works through firewalls if you're trying 465 00:16:19,079 --> 00:16:22,079 to set up a network camera behind a 466 00:16:20,339 --> 00:16:24,180 firewall you'd be messing on a lot of 467 00:16:22,079 --> 00:16:27,480 port forwarding all that stuff uh 468 00:16:24,180 --> 00:16:29,820 whereas uh as we've seen web RTC is um 469 00:16:27,480 --> 00:16:31,440 Nat trends 470 00:16:29,820 --> 00:16:33,000 that reverse order basically get through 471 00:16:31,440 --> 00:16:35,760 all those problems for you so the parts 472 00:16:33,000 --> 00:16:38,220 that make up the webrtc uh standards 473 00:16:35,760 --> 00:16:40,139 you'll uh your a couple things on power 474 00:16:38,220 --> 00:16:41,459 savings you'll need a web server uh of 475 00:16:40,139 --> 00:16:43,259 your own which will provide HTML and 476 00:16:41,459 --> 00:16:46,019 JavaScript to your to your web browser 477 00:16:43,259 --> 00:16:47,880 so um that basically formula um the 478 00:16:46,019 --> 00:16:50,100 client side of your website application 479 00:16:47,880 --> 00:16:52,079 you need a signaling server that allows 480 00:16:50,100 --> 00:16:53,100 you to communicate between your web rcps 481 00:16:52,079 --> 00:16:56,519 because 482 00:16:53,100 --> 00:16:57,959 um uh yeah at first you um you have two 483 00:16:56,519 --> 00:17:00,300 two um 484 00:16:57,959 --> 00:17:02,220 to a part two systems like your clients 485 00:17:00,300 --> 00:17:03,779 don't know how to talk to each other and 486 00:17:02,220 --> 00:17:05,220 they don't if they get through firewalls 487 00:17:03,779 --> 00:17:07,079 so you have an external service they can 488 00:17:05,220 --> 00:17:09,480 talk to them you'll have your your 489 00:17:07,079 --> 00:17:11,699 python G streamer code in a back-end 490 00:17:09,480 --> 00:17:13,740 server and then you'll use probably a 491 00:17:11,699 --> 00:17:15,240 commercially available uh or freely 492 00:17:13,740 --> 00:17:16,679 available start and turn server to 493 00:17:15,240 --> 00:17:22,140 actually help you get through firewalls 494 00:17:16,679 --> 00:17:23,939 so a stunt server helps to to um two two 495 00:17:22,140 --> 00:17:24,959 programs or two applications behind 496 00:17:23,939 --> 00:17:27,480 files to find out what their IP 497 00:17:24,959 --> 00:17:29,280 addresses are and uh and uh so they can 498 00:17:27,480 --> 00:17:30,660 talk to each other and for some reason 499 00:17:29,280 --> 00:17:32,280 you can't form a direct connection 500 00:17:30,660 --> 00:17:33,780 between them you can use a turn server 501 00:17:32,280 --> 00:17:36,240 which allows you which basically allows 502 00:17:33,780 --> 00:17:38,280 you to transfer media streaming through 503 00:17:36,240 --> 00:17:39,720 through that external server and it's 504 00:17:38,280 --> 00:17:41,580 not it's not it's not always necessary 505 00:17:39,720 --> 00:17:43,140 you might be on the same land or you 506 00:17:41,580 --> 00:17:45,240 might be um you know using tail scales 507 00:17:43,140 --> 00:17:46,679 like that so you might be able to um we 508 00:17:45,240 --> 00:17:48,419 might or you're also you can use um 509 00:17:46,679 --> 00:17:50,700 natural reversal to negotiate getting 510 00:17:48,419 --> 00:17:52,080 through your firewall 511 00:17:50,700 --> 00:17:54,000 um at the very bottom put a link to the 512 00:17:52,080 --> 00:17:57,059 g-string webrtc demonstration code so 513 00:17:54,000 --> 00:17:58,140 about four or five years ago uh one of I 514 00:17:57,059 --> 00:17:59,700 guess from the team put up with some 515 00:17:58,140 --> 00:18:02,280 demonstration code 516 00:17:59,700 --> 00:18:04,020 um the main issue that code is um it it 517 00:18:02,280 --> 00:18:05,820 has some documentation it needs a lot 518 00:18:04,020 --> 00:18:08,400 more and also you need to fill in a lot 519 00:18:05,820 --> 00:18:09,539 of the details so so 520 00:18:08,400 --> 00:18:11,880 like 521 00:18:09,539 --> 00:18:14,700 okay so here's a bit of a diagram about 522 00:18:11,880 --> 00:18:16,080 about well how I feel I think web.c 523 00:18:14,700 --> 00:18:18,600 works there's a lot of a lot of other 524 00:18:16,080 --> 00:18:20,520 explanations on that but there's the um 525 00:18:18,600 --> 00:18:22,980 there's the servers the the web server 526 00:18:20,520 --> 00:18:24,780 the signaling server sunturn server and 527 00:18:22,980 --> 00:18:26,940 the first thing we need to do is get uh 528 00:18:24,780 --> 00:18:29,160 ht1 JavaScript to the browser so that 529 00:18:26,940 --> 00:18:32,520 the browser can now start to initiate 530 00:18:29,160 --> 00:18:36,299 the webrtc session once the web browsers 531 00:18:32,520 --> 00:18:38,700 are up and up and going it'll it'll send 532 00:18:36,299 --> 00:18:41,580 a peer ID like a random number to the 533 00:18:38,700 --> 00:18:42,840 signaling server and uh it's obviously 534 00:18:41,580 --> 00:18:46,140 to establish a sessional signaling 535 00:18:42,840 --> 00:18:48,299 server and uh in the approach I've taken 536 00:18:46,140 --> 00:18:50,220 the bread browsers also will connect to 537 00:18:48,299 --> 00:18:52,440 your um your back end your um your 538 00:18:50,220 --> 00:18:54,960 webrtc python code and also pass it to 539 00:18:52,440 --> 00:18:57,240 peer ID and then the fourth step is your 540 00:18:54,960 --> 00:18:58,980 your web RC colon talking signal and 541 00:18:57,240 --> 00:19:00,059 server with that same pure ID and now 542 00:18:58,980 --> 00:19:01,980 you've established a connection between 543 00:19:00,059 --> 00:19:03,780 the web browser and your backend through 544 00:19:01,980 --> 00:19:05,520 the signaling server 545 00:19:03,780 --> 00:19:07,020 um one of the surprising things about 546 00:19:05,520 --> 00:19:08,880 the web RC standard is the signaling 547 00:19:07,020 --> 00:19:11,100 server which is an essential initiation 548 00:19:08,880 --> 00:19:12,840 part of the process isn't actually um 549 00:19:11,100 --> 00:19:15,539 standardized it's up to you know 550 00:19:12,840 --> 00:19:16,919 everyone to create their own or maybe 551 00:19:15,539 --> 00:19:18,480 agree a bit 552 00:19:16,919 --> 00:19:20,240 um so probably one of the biggest 553 00:19:18,480 --> 00:19:24,120 problems about trying to hook up to um 554 00:19:20,240 --> 00:19:26,520 third-party uh webrtc vendors so I'd 555 00:19:24,120 --> 00:19:30,380 like it could be like Zoom which mostly 556 00:19:26,520 --> 00:19:33,840 follows where I see or Jitsu or or um 557 00:19:30,380 --> 00:19:35,220 Google meet is um is it everyone's 558 00:19:33,840 --> 00:19:36,840 signaling server protocol might be 559 00:19:35,220 --> 00:19:39,960 different which is um 560 00:19:36,840 --> 00:19:41,460 uh interesting uh something once you've 561 00:19:39,960 --> 00:19:42,960 got the signals once once about your web 562 00:19:41,460 --> 00:19:44,940 browser and your your path and peers 563 00:19:42,960 --> 00:19:47,039 talking to the to the signaling server 564 00:19:44,940 --> 00:19:49,260 you do what's called a service 565 00:19:47,039 --> 00:19:51,900 um service description protocol 566 00:19:49,260 --> 00:19:54,600 uh negotiation and ice is a product of 567 00:19:51,900 --> 00:19:56,340 help you Traverse the um firewall and 568 00:19:54,600 --> 00:19:58,440 you've got negotiate whether you want 569 00:19:56,340 --> 00:20:01,799 audio video or data channels not 570 00:19:58,440 --> 00:20:03,960 resolutions yours and once you negotiate 571 00:20:01,799 --> 00:20:05,460 that you then begin trying to establish 572 00:20:03,960 --> 00:20:08,340 a connection 573 00:20:05,460 --> 00:20:10,020 um use the sun sun server which appears 574 00:20:08,340 --> 00:20:11,400 to find out their their IP addresses are 575 00:20:10,020 --> 00:20:13,140 and see if they can corrupt your shell 576 00:20:11,400 --> 00:20:15,539 directly and if they can't they use the 577 00:20:13,140 --> 00:20:17,700 they'll use the turn the turn server to 578 00:20:15,539 --> 00:20:19,679 um basically afford their traffic back 579 00:20:17,700 --> 00:20:22,380 to the forwards through 580 00:20:19,679 --> 00:20:24,120 um so how does that diagram looks at the 581 00:20:22,380 --> 00:20:26,640 endpoint just have these purple dashed 582 00:20:24,120 --> 00:20:27,900 lines is the effect of like a I feel 583 00:20:26,640 --> 00:20:31,200 like a logical connection between the 584 00:20:27,900 --> 00:20:33,960 browser and the uh your python code to 585 00:20:31,200 --> 00:20:35,760 um do the negotiation with over the 586 00:20:33,960 --> 00:20:37,559 signaling server and the dashed line for 587 00:20:35,760 --> 00:20:39,960 the media transfer is to indicate that 588 00:20:37,559 --> 00:20:42,059 um if the end result is to establish a 589 00:20:39,960 --> 00:20:44,640 bi-directional media between your your 590 00:20:42,059 --> 00:20:46,740 browser and your code which may or may 591 00:20:44,640 --> 00:20:48,360 not use the the turn server so let's 592 00:20:46,740 --> 00:20:50,520 hopefully gives you a bit of an idea 593 00:20:48,360 --> 00:20:53,640 about what we're trying to 594 00:20:50,520 --> 00:20:56,640 so to set up webrt you see 595 00:20:53,640 --> 00:20:57,960 um you know the starting point for me 596 00:20:56,640 --> 00:20:59,280 was to use this RG streamer 597 00:20:57,960 --> 00:21:01,380 demonstration code and over a lot of 598 00:20:59,280 --> 00:21:03,179 time I've flush that out to um be more 599 00:21:01,380 --> 00:21:04,740 complete uh everything we're about to 600 00:21:03,179 --> 00:21:07,440 see is uh when you're going to 601 00:21:04,740 --> 00:21:08,760 production needs to be um SSL setting to 602 00:21:07,440 --> 00:21:10,500 grab some certificates 603 00:21:08,760 --> 00:21:11,940 um it's a bit of obviously paid a bit of 604 00:21:10,500 --> 00:21:13,440 pain when you're developing do that but 605 00:21:11,940 --> 00:21:15,360 um it's good to have a secure result 606 00:21:13,440 --> 00:21:16,740 forced upon you 607 00:21:15,360 --> 00:21:17,880 um I do read a few years ago there was a 608 00:21:16,740 --> 00:21:19,679 problem with um we're about to see 609 00:21:17,880 --> 00:21:21,059 leakage we could possibly leak out uh 610 00:21:19,679 --> 00:21:22,919 information about your IP address which 611 00:21:21,059 --> 00:21:24,360 is a it wasn't a bug was actually or 612 00:21:22,919 --> 00:21:26,160 wasn't implementation but I seem to be a 613 00:21:24,360 --> 00:21:27,600 design but I better simply workarounds 614 00:21:26,160 --> 00:21:30,900 around that so it is meant to be 615 00:21:27,600 --> 00:21:33,299 entirely secure but there's still issues 616 00:21:30,900 --> 00:21:35,760 um you know enough about Civics uh then 617 00:21:33,299 --> 00:21:38,580 you uh set up your web server to run to 618 00:21:35,760 --> 00:21:41,159 provide the HTML JavaScript uh you then 619 00:21:38,580 --> 00:21:42,840 run your python code look in a sec and 620 00:21:41,159 --> 00:21:44,940 then you can configure some stun and 621 00:21:42,840 --> 00:21:47,400 turn servers what we did is we just used 622 00:21:44,940 --> 00:21:49,080 a twilio because they um they're cheap 623 00:21:47,400 --> 00:21:52,260 and very reliable we tried using a few 624 00:21:49,080 --> 00:21:53,640 free ones but found it um uh they 625 00:21:52,260 --> 00:21:56,760 weren't they weren't reliable you could 626 00:21:53,640 --> 00:21:58,679 have much bandwidth so um for at least 627 00:21:56,760 --> 00:22:00,059 for commercial production purpose so 628 00:21:58,679 --> 00:22:02,340 suspenders to pay a small amount of 629 00:22:00,059 --> 00:22:04,919 money too hard yeah not not pay a lot 630 00:22:02,340 --> 00:22:07,380 more in time and effort and then first 631 00:22:04,919 --> 00:22:08,460 you just run your code so it's a little 632 00:22:07,380 --> 00:22:10,440 bit more complicated than it's talking 633 00:22:08,460 --> 00:22:11,220 to a network camera but um obviously got 634 00:22:10,440 --> 00:22:11,820 more 635 00:22:11,220 --> 00:22:13,740 um 636 00:22:11,820 --> 00:22:16,919 capability 637 00:22:13,740 --> 00:22:18,299 so when you um when you run your um your 638 00:22:16,919 --> 00:22:20,100 webass C code 639 00:22:18,299 --> 00:22:22,380 um this is sort of the phases it'll go 640 00:22:20,100 --> 00:22:25,440 through it'll the first 641 00:22:22,380 --> 00:22:26,940 um Euro so let's assume the browsers are 642 00:22:25,440 --> 00:22:28,620 up and running by this point 643 00:22:26,940 --> 00:22:31,679 um you'll receive from the browser a 644 00:22:28,620 --> 00:22:34,020 peer ID obviously the golden ratio 1618 645 00:22:31,679 --> 00:22:36,780 is the PID and you then connect to the 646 00:22:34,020 --> 00:22:39,360 signal server that sets up the call so 647 00:22:36,780 --> 00:22:41,580 you can now talk between the browser and 648 00:22:39,360 --> 00:22:43,620 your your back end through the signaling 649 00:22:41,580 --> 00:22:45,120 server to negotiate at that point you 650 00:22:43,620 --> 00:22:48,720 can start you you start your pipelines 651 00:22:45,120 --> 00:22:50,220 because the webrtc pipeline uh the 652 00:22:48,720 --> 00:22:52,200 plugin element called red RC bin 653 00:22:50,220 --> 00:22:55,559 includes all the negotiation and all the 654 00:22:52,200 --> 00:22:57,720 the all the under the underlying webrtc 655 00:22:55,559 --> 00:22:59,400 um back and forth like the sdp 656 00:22:57,720 --> 00:23:00,600 negotiation nice and so on and start 657 00:22:59,400 --> 00:23:02,640 until you have to worry about that at 658 00:23:00,600 --> 00:23:06,720 all you basically just set up a number 659 00:23:02,640 --> 00:23:08,880 of um uh python callbacks which are C 660 00:23:06,720 --> 00:23:11,340 plus plus hopes into um work after c-bin 661 00:23:08,880 --> 00:23:12,299 and so you'll be told that um uh the 662 00:23:11,340 --> 00:23:13,679 channel at the data channel is being 663 00:23:12,299 --> 00:23:17,280 created that 664 00:23:13,679 --> 00:23:18,620 Association is needed you can create the 665 00:23:17,280 --> 00:23:21,299 STP 666 00:23:18,620 --> 00:23:24,240 packet which says uh whether you want 667 00:23:21,299 --> 00:23:25,860 audio video and what resolution that the 668 00:23:24,240 --> 00:23:28,080 offer goes back to the browser and it 669 00:23:25,860 --> 00:23:31,260 either is accepted or rejected and then 670 00:23:28,080 --> 00:23:33,059 ice is used to set up the uh the media 671 00:23:31,260 --> 00:23:35,820 transfer and then after that your um 672 00:23:33,059 --> 00:23:38,640 your channels are open and your your own 673 00:23:35,820 --> 00:23:40,380 videos up and running so it's so I 674 00:23:38,640 --> 00:23:42,179 wasn't expecting this to be um 675 00:23:40,380 --> 00:23:43,380 completely clear we'll give a sense 676 00:23:42,179 --> 00:23:44,520 there is actually quite a bit of backers 677 00:23:43,380 --> 00:23:46,980 and forth which is sort of done on your 678 00:23:44,520 --> 00:23:48,840 behalf by the underlying g-string of 679 00:23:46,980 --> 00:23:50,700 plugins and you can put callbacks to 680 00:23:48,840 --> 00:23:52,080 capture them 681 00:23:50,700 --> 00:23:53,340 um it's also nice to know what this is 682 00:23:52,080 --> 00:23:55,559 what this looks like 683 00:23:53,340 --> 00:23:57,299 okay so 684 00:23:55,559 --> 00:24:00,659 um the main thing to do is um is the 685 00:23:57,299 --> 00:24:02,580 pipeline creation and um the uh the way 686 00:24:00,659 --> 00:24:03,539 you create the pipelines for webrtc is 687 00:24:02,580 --> 00:24:06,120 exactly what you spent through before 688 00:24:03,539 --> 00:24:08,940 and um with variety spu you create the 689 00:24:06,120 --> 00:24:11,780 pipelines adding these g-stroke plugins 690 00:24:08,940 --> 00:24:14,039 together you can see from these these um 691 00:24:11,780 --> 00:24:15,659 pipelines here there's a lot more a lot 692 00:24:14,039 --> 00:24:17,640 more elements in it but um you can 693 00:24:15,659 --> 00:24:18,799 pretty much copy that by by Road really 694 00:24:17,640 --> 00:24:21,240 um 695 00:24:18,799 --> 00:24:22,860 all the details 696 00:24:21,240 --> 00:24:24,500 um you can see it's using vp8 is the 697 00:24:22,860 --> 00:24:28,140 codec 698 00:24:24,500 --> 00:24:29,940 you create the pipelines you um you link 699 00:24:28,140 --> 00:24:31,500 them together you run you run you run 700 00:24:29,940 --> 00:24:33,000 the pipelines and then after that you 701 00:24:31,500 --> 00:24:34,860 handle the handle the pullbacks that 702 00:24:33,000 --> 00:24:37,559 come back from the robust C bin and 703 00:24:34,860 --> 00:24:40,860 they're they're the two Pipelines 704 00:24:37,559 --> 00:24:43,260 um and uh so left some link there for 705 00:24:40,860 --> 00:24:45,179 the um the demonstration code 706 00:24:43,260 --> 00:24:46,500 um plus also the uh the frame we're 707 00:24:45,179 --> 00:24:48,960 probably working on the type and source 708 00:24:46,500 --> 00:24:51,720 so I've got um uh more more fleshed Out 709 00:24:48,960 --> 00:24:53,520 versions of this um so that we actually 710 00:24:51,720 --> 00:24:55,559 use in production so let's put more for 711 00:24:53,520 --> 00:24:58,440 that flushed out 712 00:24:55,559 --> 00:24:59,880 all right so I've just got a link in 713 00:24:58,440 --> 00:25:01,620 here to some of the local hacker spaces 714 00:24:59,880 --> 00:25:05,280 I hope to visit the uh the Adelaide 715 00:25:01,620 --> 00:25:06,240 hacker space uh if we get time and uh oh 716 00:25:05,280 --> 00:25:07,260 it shows it said there's gonna be no 717 00:25:06,240 --> 00:25:08,820 time for questions so I finished it 718 00:25:07,260 --> 00:25:10,980 finished with early and I thought so um 719 00:25:08,820 --> 00:25:12,360 uh I'm gonna be here for it until the 720 00:25:10,980 --> 00:25:14,340 end of Tuesday so if anyone wants to 721 00:25:12,360 --> 00:25:15,419 chat more about this because it's um I 722 00:25:14,340 --> 00:25:16,980 had to get through a lot in a very short 723 00:25:15,419 --> 00:25:18,600 period of time but I'm happy to show you 724 00:25:16,980 --> 00:25:20,700 code and show you working stuff please 725 00:25:18,600 --> 00:25:22,200 please come contact me on Discord or 726 00:25:20,700 --> 00:25:24,120 however so thank you thanks very much 727 00:25:22,200 --> 00:25:26,100 for listening and uh 728 00:25:24,120 --> 00:25:29,120 um Jack if um 729 00:25:26,100 --> 00:25:29,120 if there's any questions 730 00:25:31,799 --> 00:25:37,320 pretty much uh now while Chris looks for 731 00:25:35,640 --> 00:25:39,179 your hands in the audience to pass a mic 732 00:25:37,320 --> 00:25:40,679 for questions I'd just like to say thank 733 00:25:39,179 --> 00:25:42,659 you because it was fascinating to see 734 00:25:40,679 --> 00:25:45,480 behind the scenes of the tech that's 735 00:25:42,659 --> 00:25:46,620 made Pike online possible and also 736 00:25:45,480 --> 00:25:49,140 what's helping all of our remote 737 00:25:46,620 --> 00:25:51,059 presenters come into our Halls today 738 00:25:49,140 --> 00:25:52,620 so thank you that was fascinating uh 739 00:25:51,059 --> 00:25:54,600 Chris we have a question 740 00:25:52,620 --> 00:25:57,600 hi hello 741 00:25:54,600 --> 00:25:59,159 um uh maybe not specifically regarding 742 00:25:57,600 --> 00:26:02,100 the uh 743 00:25:59,159 --> 00:26:06,179 webrtc python thing itself um but in 744 00:26:02,100 --> 00:26:08,820 terms of the free uh servers for uh 745 00:26:06,179 --> 00:26:11,820 standard slash turn yep like what did 746 00:26:08,820 --> 00:26:14,159 you look into that didn't work 747 00:26:11,820 --> 00:26:15,659 um I looked into the free ones about I 748 00:26:14,159 --> 00:26:18,059 mean nearly three years ago now so I'm 749 00:26:15,659 --> 00:26:19,559 not up to date and I just just found a 750 00:26:18,059 --> 00:26:20,580 couple we're using those these they sort 751 00:26:19,559 --> 00:26:22,200 of they sort of worked was enough to 752 00:26:20,580 --> 00:26:23,880 develop but soon we started to try and 753 00:26:22,200 --> 00:26:25,200 demonstrate the code and what we're 754 00:26:23,880 --> 00:26:27,299 doing um they just went basement 755 00:26:25,200 --> 00:26:29,640 reliable sometimes they'd be up 756 00:26:27,299 --> 00:26:32,659 um their bandwidth was not great 757 00:26:29,640 --> 00:26:36,120 um and so we found twilio was um 758 00:26:32,659 --> 00:26:38,880 but uh uh we've found the price went too 759 00:26:36,120 --> 00:26:40,799 bad and uh they're super reliable 760 00:26:38,880 --> 00:26:42,419 um typically because of course stun 761 00:26:40,799 --> 00:26:44,880 tuners don't use much bandwidth they 762 00:26:42,419 --> 00:26:46,020 just find the IP addresses and your 763 00:26:44,880 --> 00:26:47,400 reachability 764 00:26:46,020 --> 00:26:48,900 um quite often you can find free once in 765 00:26:47,400 --> 00:26:50,460 Google runs a free stun service that's 766 00:26:48,900 --> 00:26:52,200 not a problem the problem is once you 767 00:26:50,460 --> 00:26:53,940 make your connection is um if you're if 768 00:26:52,200 --> 00:26:55,740 you're too um peers can't talk directly 769 00:26:53,940 --> 00:26:57,120 each other use this thing to turn server 770 00:26:55,740 --> 00:26:58,200 and then you're moving gigabytes of the 771 00:26:57,120 --> 00:27:01,799 head over the network until someone 772 00:26:58,200 --> 00:27:04,799 else's dying so yeah yeah I was just 773 00:27:01,799 --> 00:27:07,080 curious because we use uh uh co-turn for 774 00:27:04,799 --> 00:27:09,779 our Beverly button okay turn this over 775 00:27:07,080 --> 00:27:11,640 so uh yeah we haven't had any 776 00:27:09,779 --> 00:27:13,200 limitations around that just yet but 777 00:27:11,640 --> 00:27:14,940 maybe we need more people getting it 778 00:27:13,200 --> 00:27:17,400 well I started before before 779 00:27:14,940 --> 00:27:19,799 um the lock you know Clover lockdown and 780 00:27:17,400 --> 00:27:20,520 so um I think over the last two or three 781 00:27:19,799 --> 00:27:21,779 years 782 00:27:20,520 --> 00:27:23,520 um obviously it's a lot more use of 783 00:27:21,779 --> 00:27:27,900 teleconferencing so maybe that's um 784 00:27:23,520 --> 00:27:30,360 changed the change the world a bit 785 00:27:27,900 --> 00:27:32,880 I'm not sure we've got time for uh any 786 00:27:30,360 --> 00:27:34,260 other questions but uh as Andy said time 787 00:27:32,880 --> 00:27:36,059 for the hallway track and have a 788 00:27:34,260 --> 00:27:38,400 wonderful conference to talk about other 789 00:27:36,059 --> 00:27:39,840 side I was hoping this to be a data 790 00:27:38,400 --> 00:27:41,220 source today but I didn't get on then 791 00:27:39,840 --> 00:27:44,220 onto the network but I'll be playing 792 00:27:41,220 --> 00:27:46,200 around this robot um with uh some uh AI 793 00:27:44,220 --> 00:27:48,840 if he wants to talk robots and machine 794 00:27:46,200 --> 00:27:51,480 learning thanks awesome well thank you 795 00:27:48,840 --> 00:27:54,120 very much for your talk and here's a 796 00:27:51,480 --> 00:27:56,480 thank you gift from the conference 797 00:27:54,120 --> 00:27:56,480 oh