1
00:00:06,320 --> 00:00:11,499
[Music]

2
00:00:15,519 --> 00:00:20,560
hello everyone welcome back to the very

3
00:00:18,560 --> 00:00:23,039
last talk for this year's linux

4
00:00:20,560 --> 00:00:25,760
conference

5
00:00:23,039 --> 00:00:29,920
all right to wrap things up on a

6
00:00:25,760 --> 00:00:32,640
fascinating note um we have indu pagat

7
00:00:29,920 --> 00:00:34,480
nick alcock telling us about compact c

8
00:00:32,640 --> 00:00:36,000
type format in the

9
00:00:34,480 --> 00:00:38,879
uh neutral

10
00:00:36,000 --> 00:00:40,879
tool chain uh tool chain was like the

11
00:00:38,879 --> 00:00:43,600
easiest word to pronounce in that entire

12
00:00:40,879 --> 00:00:46,480
title and i messed it up okay

13
00:00:43,600 --> 00:00:48,960
uh indo is part of the linux tool chain

14
00:00:46,480 --> 00:00:52,239
group at oracle in the recent few years

15
00:00:48,960 --> 00:00:55,360
her focus has been on ctf btf support in

16
00:00:52,239 --> 00:00:56,480
the neutral chain uh nick who is not

17
00:00:55,360 --> 00:00:59,280
able to

18
00:00:56,480 --> 00:01:01,760
join us sadly due to time zones

19
00:00:59,280 --> 00:01:05,680
is a free software developer currently

20
00:01:01,760 --> 00:01:08,880
working on dtrace ctf new bin utils and

21
00:01:05,680 --> 00:01:11,680
other tool cheney things at oracle

22
00:01:08,880 --> 00:01:13,119
so yes nick is not able to join us but

23
00:01:11,680 --> 00:01:15,360
indo is here

24
00:01:13,119 --> 00:01:18,000
this is a pre-recorded talk so indy will

25
00:01:15,360 --> 00:01:19,439
be joining you all in the chat

26
00:01:18,000 --> 00:01:21,600
um

27
00:01:19,439 --> 00:01:23,759
so feel free to

28
00:01:21,600 --> 00:01:26,320
have a nice time chatting away during

29
00:01:23,759 --> 00:01:28,880
the talk um in duty would you like to

30
00:01:26,320 --> 00:01:31,920
say a few words

31
00:01:28,880 --> 00:01:33,920
uh i welcome everyone and i'm grateful

32
00:01:31,920 --> 00:01:37,360
to be here presenting this

33
00:01:33,920 --> 00:01:42,079
exciting talk about compact c type debug

34
00:01:37,360 --> 00:01:42,079
format um i hope you guys enjoy the talk

35
00:01:48,560 --> 00:01:53,360
hello everyone welcome to the talk on

36
00:01:50,560 --> 00:01:55,759
cdf the compact c type format and gnu

37
00:01:53,360 --> 00:01:58,640
tool chain i and nick will be doing the

38
00:01:55,759 --> 00:02:01,200
talk jointly today

39
00:01:58,640 --> 00:02:04,159
so let's get started for those of you

40
00:02:01,200 --> 00:02:07,200
who have not heard of ctf before

41
00:02:04,159 --> 00:02:10,080
hopefully this is an exciting and useful

42
00:02:07,200 --> 00:02:12,000
talk for you we will talk about

43
00:02:10,080 --> 00:02:15,200
what cdf is all about and how to

44
00:02:12,000 --> 00:02:17,599
generate and use it if you have heard of

45
00:02:15,200 --> 00:02:19,920
and played with ctf before

46
00:02:17,599 --> 00:02:20,879
i hope this talk will still be exciting

47
00:02:19,920 --> 00:02:22,959
for you

48
00:02:20,879 --> 00:02:24,800
because we will talk about adding useful

49
00:02:22,959 --> 00:02:28,160
new extensions to the format which will

50
00:02:24,800 --> 00:02:30,959
be included in ctfv4

51
00:02:28,160 --> 00:02:34,400
so let's get started by first looking at

52
00:02:30,959 --> 00:02:37,680
what is debug information

53
00:02:34,400 --> 00:02:40,160
debug information is the metadata that

54
00:02:37,680 --> 00:02:42,400
conveys relationship between the

55
00:02:40,160 --> 00:02:43,840
executable version of your program and

56
00:02:42,400 --> 00:02:46,080
the original

57
00:02:43,840 --> 00:02:48,800
source code of your program it includes

58
00:02:46,080 --> 00:02:50,560
information around

59
00:02:48,800 --> 00:02:52,160
types of program construct source

60
00:02:50,560 --> 00:02:54,319
location information

61
00:02:52,160 --> 00:02:56,400
information for virtual stack unwinding

62
00:02:54,319 --> 00:02:59,040
and so on so forth

63
00:02:56,400 --> 00:03:01,360
there have existed many debug formats

64
00:02:59,040 --> 00:03:03,920
and many of you would have heard of or

65
00:03:01,360 --> 00:03:04,879
even used dwarf it is the most commonly

66
00:03:03,920 --> 00:03:09,120
used

67
00:03:04,879 --> 00:03:11,599
debug format it is very effective

68
00:03:09,120 --> 00:03:12,640
for elf executables

69
00:03:11,599 --> 00:03:15,120
dwarf

70
00:03:12,640 --> 00:03:16,239
debug information is available

71
00:03:15,120 --> 00:03:20,080
via

72
00:03:16,239 --> 00:03:22,000
the debug underscore sections

73
00:03:20,080 --> 00:03:24,959
dwarf is designed for expressiveness and

74
00:03:22,000 --> 00:03:27,680
is very powerful it works for a variety

75
00:03:24,959 --> 00:03:29,040
of platforms apis and languages it

76
00:03:27,680 --> 00:03:31,599
encodes

77
00:03:29,040 --> 00:03:32,640
types source location information on

78
00:03:31,599 --> 00:03:34,799
wine

79
00:03:32,640 --> 00:03:38,959
information call site information and

80
00:03:34,799 --> 00:03:38,959
much more it's quite comprehensive

81
00:03:39,680 --> 00:03:45,519
but there are some issues with dwarf

82
00:03:42,799 --> 00:03:48,239
it is large and in practice it is

83
00:03:45,519 --> 00:03:50,879
voluminous enough that most of the time

84
00:03:48,239 --> 00:03:53,519
it is stripped out or shipped as a

85
00:03:50,879 --> 00:03:55,920
distinct package or not shipped at all

86
00:03:53,519 --> 00:03:58,720
uh it is complex interpreting some

87
00:03:55,920 --> 00:04:01,760
aspects of dwarf need you to implement a

88
00:03:58,720 --> 00:04:04,080
small stack machine or have access to a

89
00:04:01,760 --> 00:04:06,640
small stack machine where

90
00:04:04,080 --> 00:04:08,319
these dwarf op codes need to be executed

91
00:04:06,640 --> 00:04:10,400
and dwarf expressions need to be

92
00:04:08,319 --> 00:04:12,080
evaluated

93
00:04:10,400 --> 00:04:12,959
and it's not just the

94
00:04:12,080 --> 00:04:15,200
the

95
00:04:12,959 --> 00:04:18,479
it's not just the dwarf expressions or

96
00:04:15,200 --> 00:04:20,160
the need to have a

97
00:04:18,479 --> 00:04:22,479
stack machine that adds to the

98
00:04:20,160 --> 00:04:25,040
complications there are other aspects of

99
00:04:22,479 --> 00:04:28,880
dwarf that sometimes make it too bulky a

100
00:04:25,040 --> 00:04:31,759
format to get the job done it is slow

101
00:04:28,880 --> 00:04:34,080
and as we will talk uh later today

102
00:04:31,759 --> 00:04:36,720
for a specific use case of online

103
00:04:34,080 --> 00:04:39,120
debugging uh dwarf based methods are not

104
00:04:36,720 --> 00:04:41,600
the first choice

105
00:04:39,120 --> 00:04:42,720
so here comes cdf with its niche

106
00:04:41,600 --> 00:04:45,919
offerings

107
00:04:42,720 --> 00:04:47,360
cdf is compact and fast ctf can

108
00:04:45,919 --> 00:04:49,919
represent types

109
00:04:47,360 --> 00:04:52,160
for all c constructs it can be used to

110
00:04:49,919 --> 00:04:54,840
provide a c application

111
00:04:52,160 --> 00:04:57,600
full runtime visibility into its type

112
00:04:54,840 --> 00:05:00,080
system the second most important feature

113
00:04:57,600 --> 00:05:01,680
which is currently work in progress and

114
00:05:00,080 --> 00:05:04,400
we will talk about this in the latter

115
00:05:01,680 --> 00:05:05,840
half of the talk is generating back

116
00:05:04,400 --> 00:05:09,440
traces

117
00:05:05,840 --> 00:05:11,600
cdf um aims to provide a fast compact

118
00:05:09,440 --> 00:05:14,000
way to generate back traces

119
00:05:11,600 --> 00:05:16,320
the support is being targeted to be

120
00:05:14,000 --> 00:05:18,880
present in the tool chain so putting all

121
00:05:16,320 --> 00:05:21,039
this together ctf can help alleviate

122
00:05:18,880 --> 00:05:24,479
some of those uh issues

123
00:05:21,039 --> 00:05:27,280
uh with dwarf by providing

124
00:05:24,479 --> 00:05:30,880
some means of debugging if dwarf methods

125
00:05:27,280 --> 00:05:32,400
are not preferred or not possible

126
00:05:30,880 --> 00:05:35,759
so

127
00:05:32,400 --> 00:05:38,080
what is ctf cdf stands for compact c

128
00:05:35,759 --> 00:05:40,720
type format and as the name goes it

129
00:05:38,080 --> 00:05:42,720
describes c types of a program it

130
00:05:40,720 --> 00:05:45,199
originated in the solaris kernel in the

131
00:05:42,720 --> 00:05:47,360
mid 2000s to describe the kernel debug

132
00:05:45,199 --> 00:05:48,400
information and was later ported to

133
00:05:47,360 --> 00:05:51,280
linux

134
00:05:48,400 --> 00:05:52,639
what we have today is the linux ports

135
00:05:51,280 --> 00:05:55,039
version 3

136
00:05:52,639 --> 00:05:57,840
so ctf v3

137
00:05:55,039 --> 00:05:59,520
dtrace on linux has been a consumer of

138
00:05:57,840 --> 00:06:01,759
ctf

139
00:05:59,520 --> 00:06:05,440
but but cdf in general is a debug format

140
00:06:01,759 --> 00:06:07,520
it is independent of d trace

141
00:06:05,440 --> 00:06:08,560
the ctf spec is available at this

142
00:06:07,520 --> 00:06:10,319
location

143
00:06:08,560 --> 00:06:13,440
and also in

144
00:06:10,319 --> 00:06:13,440
bit neutrals master

145
00:06:14,400 --> 00:06:19,360
now because ctf and dwarf are debug

146
00:06:16,960 --> 00:06:21,199
formats sometimes this question of how

147
00:06:19,360 --> 00:06:23,280
do the two for debug formats compare

148
00:06:21,199 --> 00:06:25,520
does come up and to that question my

149
00:06:23,280 --> 00:06:26,639
opinion is that this is not a meaningful

150
00:06:25,520 --> 00:06:28,880
comparison

151
00:06:26,639 --> 00:06:31,840
ctf and dwarf serve

152
00:06:28,880 --> 00:06:34,400
different set of use cases dwarf is the

153
00:06:31,840 --> 00:06:36,319
most comprehensive debug format that

154
00:06:34,400 --> 00:06:37,520
serves the need of most offline

155
00:06:36,319 --> 00:06:39,600
debuggers

156
00:06:37,520 --> 00:06:42,639
cdf on the other hand is type

157
00:06:39,600 --> 00:06:44,080
information for c only at this time

158
00:06:42,639 --> 00:06:46,160
it does not have any location

159
00:06:44,080 --> 00:06:48,080
information no complex expressions to be

160
00:06:46,160 --> 00:06:50,000
evaluated hence no stack machine in the

161
00:06:48,080 --> 00:06:52,800
reader etc

162
00:06:50,000 --> 00:06:55,360
so dwarf encodes a lot more information

163
00:06:52,800 --> 00:06:58,560
and the format itself is very different

164
00:06:55,360 --> 00:07:00,800
from ctf even for presenting just types

165
00:06:58,560 --> 00:07:03,360
and ctf on the other hand is a more

166
00:07:00,800 --> 00:07:05,919
compact simpler format

167
00:07:03,360 --> 00:07:07,840
which makes it faster to load so

168
00:07:05,919 --> 00:07:10,880
overall i would still say that this is

169
00:07:07,840 --> 00:07:12,400
not a meaningful comparison um but so

170
00:07:10,880 --> 00:07:15,280
let's see in the next few slides i will

171
00:07:12,400 --> 00:07:16,880
describe um the cdf formats so if you

172
00:07:15,280 --> 00:07:18,800
know dwarf you'll be able to appreciate

173
00:07:16,880 --> 00:07:20,720
the differences between the two formats

174
00:07:18,800 --> 00:07:23,680
and hopefully you'll have an opinion of

175
00:07:20,720 --> 00:07:23,680
your own on this matter

176
00:07:24,240 --> 00:07:30,240
so let's get started with

177
00:07:26,560 --> 00:07:32,800
some details on ctf debug format um

178
00:07:30,240 --> 00:07:36,319
apart from the spec ctf format is also

179
00:07:32,800 --> 00:07:38,560
documented in ctf.h header file

180
00:07:36,319 --> 00:07:40,800
installed by bennutils

181
00:07:38,560 --> 00:07:44,000
at the heart of cdf section is a ctf

182
00:07:40,800 --> 00:07:47,280
dictionary which is just um which is

183
00:07:44,000 --> 00:07:49,280
just a name for a set of cdf types

184
00:07:47,280 --> 00:07:51,120
these dictionaries can be arranged in a

185
00:07:49,280 --> 00:07:52,879
parent-child relationship called ctf

186
00:07:51,120 --> 00:07:55,360
archives

187
00:07:52,879 --> 00:07:57,759
for much of the discussion today we will

188
00:07:55,360 --> 00:08:01,039
focus on ctf

189
00:07:57,759 --> 00:08:03,520
as it is the crux of the cdf section

190
00:08:01,039 --> 00:08:06,479
so each cdf dictionary has multiple

191
00:08:03,520 --> 00:08:08,800
subsections where each subsection stores

192
00:08:06,479 --> 00:08:09,919
type information for a class of c

193
00:08:08,800 --> 00:08:12,960
constructs

194
00:08:09,919 --> 00:08:15,680
so if you if you can look up the if you

195
00:08:12,960 --> 00:08:18,080
if you want to look up the type of the

196
00:08:15,680 --> 00:08:19,680
variable in your program

197
00:08:18,080 --> 00:08:21,919
you can follow the

198
00:08:19,680 --> 00:08:23,759
variable subsection similarly you can

199
00:08:21,919 --> 00:08:27,360
know the return type and the argument

200
00:08:23,759 --> 00:08:28,960
type of um a function in your program by

201
00:08:27,360 --> 00:08:31,360
following the

202
00:08:28,960 --> 00:08:32,959
function subsection

203
00:08:31,360 --> 00:08:35,919
all the strings for variable names

204
00:08:32,959 --> 00:08:38,320
struct names etc they appear in the ctf

205
00:08:35,919 --> 00:08:41,839
string table which sits at the very end

206
00:08:38,320 --> 00:08:41,839
of the ctf dictionary

207
00:08:42,719 --> 00:08:49,360
the cdf types subsection is an array

208
00:08:47,600 --> 00:08:52,640
of variable

209
00:08:49,360 --> 00:08:55,680
length entries so

210
00:08:52,640 --> 00:08:57,839
each of these struct cdfs type is

211
00:08:55,680 --> 00:09:00,640
followed by variable optional variable

212
00:08:57,839 --> 00:09:02,720
length data and each type has an id

213
00:09:00,640 --> 00:09:05,440
which is simply its index in the array

214
00:09:02,720 --> 00:09:07,440
so type isd is not explicit in the ctf

215
00:09:05,440 --> 00:09:10,160
format

216
00:09:07,440 --> 00:09:12,640
ctfs type underscore t struct is used to

217
00:09:10,160 --> 00:09:15,040
encode the basic information for any

218
00:09:12,640 --> 00:09:15,760
type in cdf so let's see what it looks

219
00:09:15,040 --> 00:09:19,360
like

220
00:09:15,760 --> 00:09:22,399
ctt name here stores an offset to

221
00:09:19,360 --> 00:09:24,640
the name in the ctf string table

222
00:09:22,399 --> 00:09:26,880
each type needs to have either a size by

223
00:09:24,640 --> 00:09:29,200
itself or refer to another type so for

224
00:09:26,880 --> 00:09:32,480
example if it's a typedef it will refer

225
00:09:29,200 --> 00:09:34,839
to another type via the ctt type

226
00:09:32,480 --> 00:09:37,600
but it will not just have a size right

227
00:09:34,839 --> 00:09:40,000
here ctd info

228
00:09:37,600 --> 00:09:42,800
let's see cdv info is in effect a tiny

229
00:09:40,000 --> 00:09:43,760
struct itself which encodes the kind of

230
00:09:42,800 --> 00:09:45,600
the type

231
00:09:43,760 --> 00:09:49,680
uh so it tells you whether it's an array

232
00:09:45,600 --> 00:09:51,920
pointer type def struct integer

233
00:09:49,680 --> 00:09:54,959
and all that and the number of

234
00:09:51,920 --> 00:09:57,920
subsequent entries to expect after ctfs

235
00:09:54,959 --> 00:09:59,839
type struct right so there needs to be a

236
00:09:57,920 --> 00:10:03,040
way where each type

237
00:09:59,839 --> 00:10:05,920
where the reader of ctf type knows what

238
00:10:03,040 --> 00:10:07,839
follows cdfs type underscore t and ctt

239
00:10:05,920 --> 00:10:08,640
vlan is just that

240
00:10:07,839 --> 00:10:10,800
so

241
00:10:08,640 --> 00:10:12,880
the number of entries that follow a

242
00:10:10,800 --> 00:10:15,200
specific type depends on the kind of

243
00:10:12,880 --> 00:10:18,000
type some in some types like integers

244
00:10:15,200 --> 00:10:20,160
have a fixed number of bytes that follow

245
00:10:18,000 --> 00:10:22,000
other types like structs and functions

246
00:10:20,160 --> 00:10:23,440
have a variable number of struct members

247
00:10:22,000 --> 00:10:25,519
or you know function arguments

248
00:10:23,440 --> 00:10:27,519
respectively

249
00:10:25,519 --> 00:10:32,480
each of which gets an error element

250
00:10:27,519 --> 00:10:35,360
giving its name type and offset so

251
00:10:32,480 --> 00:10:39,120
maybe let's see some examples to to put

252
00:10:35,360 --> 00:10:40,399
it uh perhaps more clearly so let's

253
00:10:39,120 --> 00:10:43,440
with that background let's put it

254
00:10:40,399 --> 00:10:46,000
together for integer data type

255
00:10:43,440 --> 00:10:49,760
as you see ctd name points to the string

256
00:10:46,000 --> 00:10:52,240
int in ctf string table ctd info tells

257
00:10:49,760 --> 00:10:56,560
you that the kind of type is ctfk

258
00:10:52,240 --> 00:10:58,880
integer ctt vlan is set to 0

259
00:10:56,560 --> 00:11:01,440
and a cdf s type underscore t the first

260
00:10:58,880 --> 00:11:04,079
block is immediately followed by a

261
00:11:01,440 --> 00:11:06,399
variable length data which for end is a

262
00:11:04,079 --> 00:11:09,760
32-bit

263
00:11:06,399 --> 00:11:11,760
data data stream it encodes in this case

264
00:11:09,760 --> 00:11:14,320
whether the

265
00:11:11,760 --> 00:11:18,000
integer is assigned one a

266
00:11:14,320 --> 00:11:20,959
signed integer or not and other details

267
00:11:18,000 --> 00:11:22,000
okay let's work out another example for

268
00:11:20,959 --> 00:11:24,720
array

269
00:11:22,000 --> 00:11:28,000
representation in cdf and array type in

270
00:11:24,720 --> 00:11:29,279
cdf is represented by using a ctfs type

271
00:11:28,000 --> 00:11:32,000
underscore t

272
00:11:29,279 --> 00:11:35,200
followed by a cdf underscore array

273
00:11:32,000 --> 00:11:37,279
underscore t object so in this case the

274
00:11:35,200 --> 00:11:41,279
cdt name points to a null string in the

275
00:11:37,279 --> 00:11:42,480
ctf string table cdt info tells you

276
00:11:41,279 --> 00:11:45,519
this

277
00:11:42,480 --> 00:11:50,240
type is of kind ctfk array

278
00:11:45,519 --> 00:11:50,240
ctd vlan is again set to zero and unused

279
00:11:50,320 --> 00:11:57,120
the variable length data in this case is

280
00:11:53,279 --> 00:11:59,279
a struct cdf array underscore t which

281
00:11:57,120 --> 00:12:01,040
specifies a reference to the type of the

282
00:11:59,279 --> 00:12:03,760
array

283
00:12:01,040 --> 00:12:06,480
and the number of elements so

284
00:12:03,760 --> 00:12:09,200
that's it putting it together the cdf

285
00:12:06,480 --> 00:12:12,639
types refer so this pictorially shows

286
00:12:09,200 --> 00:12:14,959
you that for a sample uh c code this is

287
00:12:12,639 --> 00:12:17,440
the kind of types you will see in your

288
00:12:14,959 --> 00:12:18,240
ctf type subsection the

289
00:12:17,440 --> 00:12:21,200
and

290
00:12:18,240 --> 00:12:23,200
so the tcdf types reference

291
00:12:21,200 --> 00:12:24,720
so in this diagram here the ctf types

292
00:12:23,200 --> 00:12:26,880
reference each other to create a

293
00:12:24,720 --> 00:12:28,720
representation of the types in your high

294
00:12:26,880 --> 00:12:31,120
level program as shown

295
00:12:28,720 --> 00:12:33,120
and in this pictorial layout here on the

296
00:12:31,120 --> 00:12:36,079
right side the types in white are

297
00:12:33,120 --> 00:12:37,200
basically base types the pink ones are

298
00:12:36,079 --> 00:12:38,959
pointers

299
00:12:37,200 --> 00:12:41,519
and the yellow ones are sort of

300
00:12:38,959 --> 00:12:45,360
composite type structures and so on so

301
00:12:41,519 --> 00:12:48,000
all this helps give you an idea of how

302
00:12:45,360 --> 00:12:50,160
these types weave together to give you a

303
00:12:48,000 --> 00:12:53,320
true representation of all the types in

304
00:12:50,160 --> 00:12:53,320
your program

305
00:12:53,519 --> 00:12:57,279
so so far we've only been dealing with

306
00:12:55,360 --> 00:12:59,519
single object files but then no one runs

307
00:12:57,279 --> 00:13:00,720
a single object file executable

308
00:12:59,519 --> 00:13:02,000
and shared libraries what people are

309
00:13:00,720 --> 00:13:03,519
interested in

310
00:13:02,000 --> 00:13:05,200
so

311
00:13:03,519 --> 00:13:08,320
we did with this by having the link of

312
00:13:05,200 --> 00:13:10,720
spot ctf in in in incoming object files

313
00:13:08,320 --> 00:13:12,880
and duplicate duplicate it all together

314
00:13:10,720 --> 00:13:16,000
um so that so that no type appears more

315
00:13:12,880 --> 00:13:17,920
than once in the output um

316
00:13:16,000 --> 00:13:20,079
the layer also hunts around and finds

317
00:13:17,920 --> 00:13:22,000
symbols that okay and

318
00:13:20,079 --> 00:13:23,600
and find all the all the symbols that

319
00:13:22,000 --> 00:13:26,800
will appear on the output and if they

320
00:13:23,600 --> 00:13:29,519
have types and that have known types um

321
00:13:26,800 --> 00:13:32,320
it builds a table so that ctf users can

322
00:13:29,519 --> 00:13:34,959
associate the types without the um can

323
00:13:32,320 --> 00:13:36,720
ask given this symbol what is it what is

324
00:13:34,959 --> 00:13:38,480
its type i'm going to type back

325
00:13:36,720 --> 00:13:40,480
um this is all in the light implemented

326
00:13:38,480 --> 00:13:42,160
in the library the link it uses it

327
00:13:40,480 --> 00:13:43,920
anyone else can use this as a can use it

328
00:13:42,160 --> 00:13:46,079
as well it's not tied to generally in

329
00:13:43,920 --> 00:13:47,279
any way anyway except that it appears in

330
00:13:46,079 --> 00:13:49,120
video tools

331
00:13:47,279 --> 00:13:50,880
the duplication is fast

332
00:13:49,120 --> 00:13:52,160
most of the timing

333
00:13:50,880 --> 00:13:54,800
on the next slide it seems to be

334
00:13:52,160 --> 00:13:55,920
reliable it saves a lot of space

335
00:13:54,800 --> 00:13:58,079
um

336
00:13:55,920 --> 00:14:00,079
but there's a wrinkle

337
00:13:58,079 --> 00:14:01,920
c is annoying unlike c plus plus there

338
00:14:00,079 --> 00:14:03,680
is no one definition rule types can have

339
00:14:01,920 --> 00:14:04,880
different it can have the same name but

340
00:14:03,680 --> 00:14:06,800
completely different definitions in

341
00:14:04,880 --> 00:14:09,120
different translation units we spot

342
00:14:06,800 --> 00:14:11,279
these types and any types

343
00:14:09,120 --> 00:14:14,079
any types they relate to and put and put

344
00:14:11,279 --> 00:14:15,120
them into in separate child dictionaries

345
00:14:14,079 --> 00:14:17,279
um

346
00:14:15,120 --> 00:14:19,279
which have uh which which are connected

347
00:14:17,279 --> 00:14:22,160
which are connected to the uh to the

348
00:14:19,279 --> 00:14:23,920
parent and [ __ ] and and and and if you

349
00:14:22,160 --> 00:14:25,600
look at if you look for a type in the in

350
00:14:23,920 --> 00:14:26,880
the child you can see all the parents

351
00:14:25,600 --> 00:14:29,279
types as well

352
00:14:26,880 --> 00:14:31,120
um if child if ambiguous types are

353
00:14:29,279 --> 00:14:34,240
present we emit an archive of ctf

354
00:14:31,120 --> 00:14:36,320
dictionary dictionaries into the um

355
00:14:34,240 --> 00:14:38,320
output executable or shared library

356
00:14:36,320 --> 00:14:40,639
rather than a single dictionary but this

357
00:14:38,320 --> 00:14:43,199
is mostly transparent to the user

358
00:14:40,639 --> 00:14:45,120
um as in even if there is no even if

359
00:14:43,199 --> 00:14:46,800
there is no archive present the api

360
00:14:45,120 --> 00:14:48,240
makes it appear to be a single single

361
00:14:46,800 --> 00:14:49,680
member archive

362
00:14:48,240 --> 00:14:52,480
um

363
00:14:49,680 --> 00:14:54,720
we because we merge all the cts into a

364
00:14:52,480 --> 00:14:56,240
into a single section we do not track

365
00:14:54,720 --> 00:14:58,320
which translation units define which

366
00:14:56,240 --> 00:15:00,560
types debugging users mostly don't care

367
00:14:58,320 --> 00:15:03,040
and it would be an enormous

368
00:15:00,560 --> 00:15:05,760
expensive space because often most

369
00:15:03,040 --> 00:15:07,600
translation units define most types

370
00:15:05,760 --> 00:15:09,440
here's an example

371
00:15:07,600 --> 00:15:11,360
of ambiguous types in this situation

372
00:15:09,440 --> 00:15:13,920
structure the foo is a structure in one

373
00:15:11,360 --> 00:15:14,880
translation unit and and to give it 16t

374
00:15:13,920 --> 00:15:17,600
in the other

375
00:15:14,880 --> 00:15:19,240
um and and you can see and you can see

376
00:15:17,600 --> 00:15:20,800
here that all the types end up in the

377
00:15:19,240 --> 00:15:23,040
shared.ctf

378
00:15:20,800 --> 00:15:25,600
archive member except for except for the

379
00:15:23,040 --> 00:15:28,399
one which is varying which appear um

380
00:15:25,600 --> 00:15:32,079
which appears in in both um

381
00:15:28,399 --> 00:15:33,680
in the appropriate perfect um person you

382
00:15:32,079 --> 00:15:36,880
child you know that the co has

383
00:15:33,680 --> 00:15:38,480
disappeared entirely it hasn't um it has

384
00:15:36,880 --> 00:15:41,040
nothing which isn't present

385
00:15:38,480 --> 00:15:42,880
in other um it has nothing ambiguous and

386
00:15:41,040 --> 00:15:45,440
so it just all it types diffused into

387
00:15:42,880 --> 00:15:46,720
dot ctf and it vanishes ld can be asked

388
00:15:45,440 --> 00:15:48,079
to distribute types in other ways but

389
00:15:46,720 --> 00:15:49,519
this is uses far less space than the

390
00:15:48,079 --> 00:15:51,920
other approaches that people are likely

391
00:15:49,519 --> 00:15:54,240
to want to use almost all the time

392
00:15:51,920 --> 00:15:56,480
the result is pretty good as soon as as

393
00:15:54,240 --> 00:15:58,399
far as poses go all the phrases here are

394
00:15:56,480 --> 00:16:00,160
faces of the dot pts sections and

395
00:15:58,399 --> 00:16:02,240
they're functions of uncompressed dot

396
00:16:00,160 --> 00:16:04,000
ctf sections ctf is almost always

397
00:16:02,240 --> 00:16:06,160
compressed unless it's tiny currently

398
00:16:04,000 --> 00:16:08,320
with jesus um i'm only doing

399
00:16:06,160 --> 00:16:10,399
uncompressed places to make it uh to

400
00:16:08,320 --> 00:16:14,639
show you just how large the reduction is

401
00:16:10,399 --> 00:16:17,279
um so the ld drops with 3.2 meg of ctf

402
00:16:14,639 --> 00:16:20,480
on the uh on the input to 200k on the

403
00:16:17,279 --> 00:16:23,519
output and 60 70k after compression emex

404
00:16:20,480 --> 00:16:25,759
drops from seven meg um to 220 all the

405
00:16:23,519 --> 00:16:27,519
way down to 129k

406
00:16:25,759 --> 00:16:28,959
and in many cases the at the output if

407
00:16:27,519 --> 00:16:31,279
you look at lymphedema the output is

408
00:16:28,959 --> 00:16:32,000
smaller than the uh the

409
00:16:31,279 --> 00:16:34,399
the

410
00:16:32,000 --> 00:16:35,519
single object files worth of the input i

411
00:16:34,399 --> 00:16:36,399
think that's fairly acceptable

412
00:16:35,519 --> 00:16:38,880
compression

413
00:16:36,399 --> 00:16:40,079
um time-wise it takes almost no time at

414
00:16:38,880 --> 00:16:41,519
all even though it's single-printed and

415
00:16:40,079 --> 00:16:43,199
not really optimized there's a lot of

416
00:16:41,519 --> 00:16:44,639
low-hanging fruit to improve there as

417
00:16:43,199 --> 00:16:45,759
well

418
00:16:44,639 --> 00:16:46,959
um

419
00:16:45,759 --> 00:16:50,000
what's the support like in the tool

420
00:16:46,959 --> 00:16:51,199
chain it's on gcc 12 master plus n minus

421
00:16:50,000 --> 00:16:52,160
gcts

422
00:16:51,199 --> 00:16:54,000
um

423
00:16:52,160 --> 00:16:55,759
it's in canoe ld you don't need to pass

424
00:16:54,000 --> 00:16:58,079
anything if the tt is on the on the

425
00:16:55,759 --> 00:17:00,240
input you get you get compressed uh and

426
00:16:58,079 --> 00:17:02,160
um you duplicated ctf on the app but

427
00:17:00,240 --> 00:17:03,040
it's in object dump it's in readout in a

428
00:17:02,160 --> 00:17:05,280
moment

429
00:17:03,040 --> 00:17:08,079
um it's not in gold yet i'll have i'll

430
00:17:05,280 --> 00:17:09,039
have to add that at some point um and

431
00:17:08,079 --> 00:17:12,720
uh and

432
00:17:09,039 --> 00:17:14,720
and qld does not um support non-elf

433
00:17:12,720 --> 00:17:16,160
platforms with ctf at the moment it just

434
00:17:14,720 --> 00:17:18,240
hasn't been plumbed in

435
00:17:16,160 --> 00:17:20,079
um there's no support in llvm at all it

436
00:17:18,240 --> 00:17:21,679
just hasn't been added people have been

437
00:17:20,079 --> 00:17:22,799
asking occasionally

438
00:17:21,679 --> 00:17:24,480
it might get it should get done

439
00:17:22,799 --> 00:17:26,559
eventually it might get done eventually

440
00:17:24,480 --> 00:17:29,200
people have also been asking for c plus

441
00:17:26,559 --> 00:17:32,559
that might get done eventually as well

442
00:17:29,200 --> 00:17:34,480
gb we support uh we support cta um we

443
00:17:32,559 --> 00:17:36,160
support cdf dictionaries fully p type

444
00:17:34,480 --> 00:17:37,679
works obviously question obviously tv

445
00:17:36,160 --> 00:17:39,200
features don't work without dwarf but if

446
00:17:37,679 --> 00:17:39,919
you want insulin types we can give it to

447
00:17:39,200 --> 00:17:41,919
you

448
00:17:39,919 --> 00:17:43,200
um there are other things gdp users

449
00:17:41,919 --> 00:17:46,000
would be likely to want and that's what

450
00:17:43,200 --> 00:17:47,120
the main subject of this talk is about

451
00:17:46,000 --> 00:17:49,280
um

452
00:17:47,120 --> 00:17:51,760
there's more support glue poke can rea

453
00:17:49,280 --> 00:17:53,039
can read ctf um

454
00:17:51,760 --> 00:17:55,360
it's actually a

455
00:17:53,039 --> 00:17:57,679
very nice at this sort of thing um i use

456
00:17:55,360 --> 00:17:59,280
it in development these days um the

457
00:17:57,679 --> 00:18:00,880
abigail

458
00:17:59,280 --> 00:18:02,240
can take in executables that contain

459
00:18:00,880 --> 00:18:03,919
only cdf

460
00:18:02,240 --> 00:18:06,000
um

461
00:18:03,919 --> 00:18:07,600
and do api analysis of them and if we

462
00:18:06,000 --> 00:18:09,440
end up with most executables containing

463
00:18:07,600 --> 00:18:10,480
ctf which we might visit small this

464
00:18:09,440 --> 00:18:12,000
means you'll be able to see whether

465
00:18:10,480 --> 00:18:14,160
things are api compatible without

466
00:18:12,000 --> 00:18:15,440
installing any kind of debug info at all

467
00:18:14,160 --> 00:18:17,360
it also means that things might be able

468
00:18:15,440 --> 00:18:19,120
to might be able to check for api

469
00:18:17,360 --> 00:18:21,679
compatibility at runtime they just need

470
00:18:19,120 --> 00:18:23,440
to call on call and go to ctf and ask it

471
00:18:21,679 --> 00:18:26,080
or live

472
00:18:23,440 --> 00:18:29,400
so a quick demo

473
00:18:26,080 --> 00:18:29,400
turn on

474
00:18:31,200 --> 00:18:37,200
we have here two really exciting

475
00:18:33,520 --> 00:18:39,360
translation units um one to um

476
00:18:37,200 --> 00:18:41,200
they contain almost the same almost the

477
00:18:39,360 --> 00:18:43,360
same content except that except accepted

478
00:18:41,200 --> 00:18:46,400
in one case um

479
00:18:43,360 --> 00:18:48,080
struck foo has an extra member

480
00:18:46,400 --> 00:18:51,520
um structural too annoying because it

481
00:18:48,080 --> 00:18:51,520
points to itself um

482
00:18:51,679 --> 00:18:57,840
cycles are a known bane of

483
00:18:54,559 --> 00:18:57,840
systems like this

484
00:18:58,240 --> 00:19:03,320
what happens what happens if you link

485
00:18:59,440 --> 00:19:03,320
this together let's try it

486
00:19:16,160 --> 00:19:21,240
here we have some cts that's too bad

487
00:19:27,039 --> 00:19:32,240
we can't dump it

488
00:19:29,120 --> 00:19:32,240
just want to see what it looks like

489
00:19:36,000 --> 00:19:41,520
i think maybe i should make this

490
00:19:39,039 --> 00:19:42,720
window full size

491
00:19:41,520 --> 00:19:47,360
and

492
00:19:42,720 --> 00:19:48,960
this this is the the dot the cdf section

493
00:19:47,360 --> 00:19:50,400
the the the

494
00:19:48,960 --> 00:19:52,640
the main dictionary there's all sort

495
00:19:50,400 --> 00:19:55,200
there's all sorts of stuff in here

496
00:19:52,640 --> 00:19:57,120
most significantly there is an i o there

497
00:19:55,200 --> 00:19:59,520
is an i o file we use studio so there's

498
00:19:57,120 --> 00:20:02,559
a file star in there

499
00:19:59,520 --> 00:20:02,559
completely represented

500
00:20:05,520 --> 00:20:08,559
um

501
00:20:06,480 --> 00:20:10,720
all the pointers and so on i shared the

502
00:20:08,559 --> 00:20:12,640
only unshared things in the individual

503
00:20:10,720 --> 00:20:14,559
archive members down here

504
00:20:12,640 --> 00:20:15,440
are the actual structure which differed

505
00:20:14,559 --> 00:20:17,760
here

506
00:20:15,440 --> 00:20:20,080
here it is it won't see you with fubar

507
00:20:17,760 --> 00:20:21,520
and next

508
00:20:20,080 --> 00:20:24,159
it is and the other one with

509
00:20:21,520 --> 00:20:26,159
zubar there's two bar badly the the new

510
00:20:24,159 --> 00:20:27,679
extra integer and next it's pretty

511
00:20:26,159 --> 00:20:29,919
duplicated about as well about as well

512
00:20:27,679 --> 00:20:31,760
as you could hope and all that all the

513
00:20:29,919 --> 00:20:32,960
types have to be represented if you look

514
00:20:31,760 --> 00:20:34,640
here

515
00:20:32,960 --> 00:20:36,559
data object it knows that still there is

516
00:20:34,640 --> 00:20:38,320
a file star and it knows what sort of

517
00:20:36,559 --> 00:20:40,559
thing a file star is and it knows it

518
00:20:38,320 --> 00:20:41,840
turns into a struct i o file

519
00:20:40,559 --> 00:20:42,799
um despite the fact that we didn't

520
00:20:41,840 --> 00:20:44,559
bother

521
00:20:42,799 --> 00:20:47,799
to tell any any of that gtc and we did

522
00:20:44,559 --> 00:20:47,799
it for us

523
00:20:56,080 --> 00:21:00,240
so this is all it's still done with an

524
00:20:57,760 --> 00:21:02,080
underlying api um i think other people

525
00:21:00,240 --> 00:21:03,840
could use it it's divided into several

526
00:21:02,080 --> 00:21:05,440
sections you can query the properties of

527
00:21:03,840 --> 00:21:07,440
types you can open and click you can

528
00:21:05,440 --> 00:21:09,600
open you can open else objects and you

529
00:21:07,440 --> 00:21:11,520
get a ctf dictionary back or a cdf

530
00:21:09,600 --> 00:21:12,799
archive you can dig over the archives

531
00:21:11,520 --> 00:21:15,120
and find types in them and that sort of

532
00:21:12,799 --> 00:21:17,360
thing you can walk over struck members

533
00:21:15,120 --> 00:21:19,360
types anything anything in ctf you can

534
00:21:17,360 --> 00:21:20,880
integrate over it you can create you can

535
00:21:19,360 --> 00:21:22,320
you can add members you can create them

536
00:21:20,880 --> 00:21:24,960
the ctf knows everything about how to

537
00:21:22,320 --> 00:21:26,159
create ctf dictionaries so um

538
00:21:24,960 --> 00:21:27,440
so you never need you should never need

539
00:21:26,159 --> 00:21:29,760
to write your own code to do it or you

540
00:21:27,440 --> 00:21:31,760
can if you'd like to there is a

541
00:21:29,760 --> 00:21:33,360
um and you can link them together in

542
00:21:31,760 --> 00:21:34,960
about in about five lines of code which

543
00:21:33,360 --> 00:21:36,559
is which is quite nicely mean you can

544
00:21:34,960 --> 00:21:39,440
write plenty linkers in about in about

545
00:21:36,559 --> 00:21:42,240
200 lines um and there's plenty it would

546
00:21:39,440 --> 00:21:44,159
be about 10 lines if it hadn't been it

547
00:21:42,240 --> 00:21:45,600
wanted to do exactly what google d did

548
00:21:44,159 --> 00:21:47,600
but in that case you can just use google

549
00:21:45,600 --> 00:21:49,440
ld

550
00:21:47,600 --> 00:21:52,000
uh what's coming

551
00:21:49,440 --> 00:21:54,400
ctsd4 is being planned for uh the big

552
00:21:52,000 --> 00:21:56,559
thing of course backtrack support um

553
00:21:54,400 --> 00:21:58,720
which is halfway between ctf and the

554
00:21:56,559 --> 00:22:00,799
user of cts really because it has to

555
00:21:58,720 --> 00:22:03,200
work with note ct we have no need for

556
00:22:00,799 --> 00:22:04,720
the caller to read ctf at all

557
00:22:03,200 --> 00:22:06,880
um that's what the rest of this talk is

558
00:22:04,720 --> 00:22:08,000
about but also we're planning to improve

559
00:22:06,880 --> 00:22:09,360
expressiveness because there are some

560
00:22:08,000 --> 00:22:10,799
things in the pipe system that we're

561
00:22:09,360 --> 00:22:13,200
going to see type system that we can't

562
00:22:10,799 --> 00:22:15,360
represent yet all

563
00:22:13,200 --> 00:22:17,600
syntax extensions and that sort of thing

564
00:22:15,360 --> 00:22:19,200
um we can improve compactors in various

565
00:22:17,600 --> 00:22:21,600
ways reduce the space used to the string

566
00:22:19,200 --> 00:22:23,039
table by cutting it into chunks um use a

567
00:22:21,600 --> 00:22:27,440
better compressor than cheese if we're

568
00:22:23,039 --> 00:22:28,720
thinking since we're thinking zedsted um

569
00:22:27,440 --> 00:22:30,080
there's a bunch of things planned we

570
00:22:28,720 --> 00:22:31,360
even have some plans for each side

571
00:22:30,080 --> 00:22:33,679
because we don't want to do everything

572
00:22:31,360 --> 00:22:36,159
all at once in one gigantic format bump

573
00:22:33,679 --> 00:22:38,240
the existing ctf will always be readable

574
00:22:36,159 --> 00:22:40,480
even when new formats are

575
00:22:38,240 --> 00:22:42,080
even when new formats are generated um i

576
00:22:40,480 --> 00:22:43,679
don't see any reason to ever drop

577
00:22:42,080 --> 00:22:45,200
support for old versions

578
00:22:43,679 --> 00:22:46,720
um and there are more things there's a

579
00:22:45,200 --> 00:22:48,559
little there's a link here to them

580
00:22:46,720 --> 00:22:49,360
anyway actually

581
00:22:48,559 --> 00:22:50,400
um

582
00:22:49,360 --> 00:22:51,280
let's

583
00:22:50,400 --> 00:22:53,679
then

584
00:22:51,280 --> 00:22:56,960
kick off the discussion on bad traces by

585
00:22:53,679 --> 00:22:58,559
looking at some use cases and how their

586
00:22:56,960 --> 00:23:00,799
requirements vary

587
00:22:58,559 --> 00:23:03,760
i have classified them into two

588
00:23:00,799 --> 00:23:06,480
categories online and offline the

589
00:23:03,760 --> 00:23:08,480
category of online back tracing is

590
00:23:06,480 --> 00:23:11,200
basically all those

591
00:23:08,480 --> 00:23:13,520
use cases where the application is in a

592
00:23:11,200 --> 00:23:15,840
production like environment and the cost

593
00:23:13,520 --> 00:23:19,760
of generating bad traces is borne by the

594
00:23:15,840 --> 00:23:22,240
application the cost being um cpu cycles

595
00:23:19,760 --> 00:23:24,000
and program memory because the relevant

596
00:23:22,240 --> 00:23:25,760
debug sections need to be loaded in the

597
00:23:24,000 --> 00:23:27,760
memory

598
00:23:25,760 --> 00:23:31,280
conversely in case of offline back

599
00:23:27,760 --> 00:23:33,520
tracing the cost of back tracing is not

600
00:23:31,280 --> 00:23:35,440
draining on the application as they are

601
00:23:33,520 --> 00:23:38,000
often run after the fact like in a crash

602
00:23:35,440 --> 00:23:40,080
report analysis tool or a debugger

603
00:23:38,000 --> 00:23:43,120
so classic examples of online back

604
00:23:40,080 --> 00:23:44,480
tracers are profilers crash reporters

605
00:23:43,120 --> 00:23:47,679
exception

606
00:23:44,480 --> 00:23:50,240
handling and live patching

607
00:23:47,679 --> 00:23:51,279
now let's look at the requirements one

608
00:23:50,240 --> 00:23:54,240
by one

609
00:23:51,279 --> 00:23:56,640
online back tracers need to be fast

610
00:23:54,240 --> 00:23:58,960
right they need to be precise um

611
00:23:56,640 --> 00:24:00,559
sometimes extremely precise like in case

612
00:23:58,960 --> 00:24:02,960
of life patching they need to be

613
00:24:00,559 --> 00:24:04,480
extremely precise otherwise they are

614
00:24:02,960 --> 00:24:06,400
well

615
00:24:04,480 --> 00:24:10,320
it's not usable otherwise

616
00:24:06,400 --> 00:24:12,080
asynchronous unwind it's a capability

617
00:24:10,320 --> 00:24:15,600
which basically means that you need to

618
00:24:12,080 --> 00:24:18,159
be able to unwind uh from any pc now

619
00:24:15,600 --> 00:24:21,279
that capability is also a must-have for

620
00:24:18,159 --> 00:24:24,000
online back tracers um

621
00:24:21,279 --> 00:24:27,679
to achieve all these goals it is vital

622
00:24:24,000 --> 00:24:30,080
that the unwinder itself be small and

623
00:24:27,679 --> 00:24:34,000
that the debug information for unwinding

624
00:24:30,080 --> 00:24:34,960
itself as well be small right so

625
00:24:34,000 --> 00:24:37,360
so

626
00:24:34,960 --> 00:24:40,080
both so online back tracers need that

627
00:24:37,360 --> 00:24:43,520
the unwinder be small and the

628
00:24:40,080 --> 00:24:46,480
debug info as well be small some online

629
00:24:43,520 --> 00:24:48,880
back tracers may also need

630
00:24:46,480 --> 00:24:50,880
uh the original value of the arguments

631
00:24:48,880 --> 00:24:53,919
in back traces

632
00:24:50,880 --> 00:24:56,320
uh in contrast now for offline back

633
00:24:53,919 --> 00:24:58,159
tracing use cases all these features are

634
00:24:56,320 --> 00:25:01,520
in a good to have category as you can

635
00:24:58,159 --> 00:25:03,520
reason about them so the takeaway

636
00:25:01,520 --> 00:25:05,520
from this slide is that there are these

637
00:25:03,520 --> 00:25:08,080
two categories of back tracing and the

638
00:25:05,520 --> 00:25:11,120
requirements vary a lot

639
00:25:08,080 --> 00:25:14,880
and we are focusing on the case of

640
00:25:11,120 --> 00:25:17,360
online back pricing um for

641
00:25:14,880 --> 00:25:17,360
cdf

642
00:25:19,919 --> 00:25:25,520
so for online back tracing what is the

643
00:25:23,440 --> 00:25:28,640
state of the art how are these generated

644
00:25:25,520 --> 00:25:31,279
anyway right so in many if not most

645
00:25:28,640 --> 00:25:34,080
cases online back tracing is based on eh

646
00:25:31,279 --> 00:25:36,880
frame sections um

647
00:25:34,080 --> 00:25:38,799
these are dwarf based sections

648
00:25:36,880 --> 00:25:40,880
and these sections encode the

649
00:25:38,799 --> 00:25:42,559
fundamental

650
00:25:40,880 --> 00:25:45,679
information needed for back tracing

651
00:25:42,559 --> 00:25:49,200
which is given a pc what is the cfa and

652
00:25:45,679 --> 00:25:51,919
ra cfa is a short form for canonical

653
00:25:49,200 --> 00:25:53,760
frame address uh think of it as the

654
00:25:51,919 --> 00:25:54,960
frame address which is

655
00:25:53,760 --> 00:25:57,200
um

656
00:25:54,960 --> 00:25:58,720
which is at the before the function

657
00:25:57,200 --> 00:26:01,200
starts what is the frame address and see

658
00:25:58,720 --> 00:26:03,440
if it just points you to that

659
00:26:01,200 --> 00:26:04,720
each frame sections are the default for

660
00:26:03,440 --> 00:26:07,360
many targets

661
00:26:04,720 --> 00:26:10,080
and um there is this eh frame header

662
00:26:07,360 --> 00:26:12,480
sections as well which are basically

663
00:26:10,080 --> 00:26:14,960
which basically allow you a faster look

664
00:26:12,480 --> 00:26:17,840
up of the information in each frame

665
00:26:14,960 --> 00:26:18,799
so using read elf let's see the contents

666
00:26:17,840 --> 00:26:21,600
of

667
00:26:18,799 --> 00:26:24,640
a sample eh frame section as you see

668
00:26:21,600 --> 00:26:28,320
each frame gives us clearly for each pc

669
00:26:24,640 --> 00:26:30,640
a way to recover cfa ra and some other

670
00:26:28,320 --> 00:26:33,520
quality registers if they made it if

671
00:26:30,640 --> 00:26:35,039
they made it to stack um

672
00:26:33,520 --> 00:26:37,120
so

673
00:26:35,039 --> 00:26:39,679
as you also see sometimes if you go from

674
00:26:37,120 --> 00:26:42,640
one pc to the next there is not much

675
00:26:39,679 --> 00:26:44,720
change in some columns so look at rbp in

676
00:26:42,640 --> 00:26:47,600
the first ip it is undefined but there

677
00:26:44,720 --> 00:26:50,240
onwards for each ip it is defined but it

678
00:26:47,600 --> 00:26:53,200
is the same so it says cfa minus 16 it's

679
00:26:50,240 --> 00:26:55,440
present rpp is stored at a location cfa

680
00:26:53,200 --> 00:26:57,520
minus 16. now

681
00:26:55,440 --> 00:26:59,039
so there is a lot of repetition in this

682
00:26:57,520 --> 00:27:01,039
information right if stored in this

683
00:26:59,039 --> 00:27:03,840
tabular format

684
00:27:01,039 --> 00:27:06,960
and this is precisely the reason what

685
00:27:03,840 --> 00:27:08,240
each frame actually stores is some dwarf

686
00:27:06,960 --> 00:27:10,640
opcodes

687
00:27:08,240 --> 00:27:12,880
which are executed on a simple stack

688
00:27:10,640 --> 00:27:15,760
machine to generate the contents of the

689
00:27:12,880 --> 00:27:17,600
table we saw in the previous slide so

690
00:27:15,760 --> 00:27:19,840
this is a compact way of storing that

691
00:27:17,600 --> 00:27:22,720
kind of information but if you are the

692
00:27:19,840 --> 00:27:24,799
unwinder you need to execute these

693
00:27:22,720 --> 00:27:27,679
opcodes

694
00:27:24,799 --> 00:27:31,440
at runtime to figure out what this how

695
00:27:27,679 --> 00:27:32,559
the cfa recovery will be done right

696
00:27:31,440 --> 00:27:34,399
so

697
00:27:32,559 --> 00:27:35,840
that creates a problem

698
00:27:34,399 --> 00:27:38,480
for

699
00:27:35,840 --> 00:27:41,120
for any unwinders written for for

700
00:27:38,480 --> 00:27:41,919
basically eh frame based unwinders

701
00:27:41,120 --> 00:27:43,840
and

702
00:27:41,919 --> 00:27:45,919
they be they are slow because at the

703
00:27:43,840 --> 00:27:48,240
time of this this makes them slow

704
00:27:45,919 --> 00:27:50,480
because at the time of uh generating the

705
00:27:48,240 --> 00:27:54,080
bad traces you now need to execute these

706
00:27:50,480 --> 00:27:55,919
opcodes to figure out the cfa and ra etc

707
00:27:54,080 --> 00:27:57,840
they are large because of the same

708
00:27:55,919 --> 00:28:00,159
reason the unwinders need to implement a

709
00:27:57,840 --> 00:28:01,120
simple stack machine

710
00:28:00,159 --> 00:28:03,600
to

711
00:28:01,120 --> 00:28:05,679
exec to to run these opcodes and figure

712
00:28:03,600 --> 00:28:08,000
out the information at runtime now the

713
00:28:05,679 --> 00:28:10,080
second issue with each frame based

714
00:28:08,000 --> 00:28:13,520
approach is the maintenance of

715
00:28:10,080 --> 00:28:16,240
handwritten assembly now um

716
00:28:13,520 --> 00:28:18,240
so these these eh frames sections they

717
00:28:16,240 --> 00:28:20,080
are generated by the assembler and the

718
00:28:18,240 --> 00:28:23,039
simpler processes what are called the

719
00:28:20,080 --> 00:28:25,600
cfi directives and if you hand code

720
00:28:23,039 --> 00:28:28,640
parts of your application then you as a

721
00:28:25,600 --> 00:28:31,279
programmer are also responsible for hand

722
00:28:28,640 --> 00:28:32,960
coding these cfi directives as well cfi

723
00:28:31,279 --> 00:28:35,520
are call frame

724
00:28:32,960 --> 00:28:38,159
information directives and this can

725
00:28:35,520 --> 00:28:40,880
sometimes get quite complicated because

726
00:28:38,159 --> 00:28:44,399
if you're if your standard an assembly

727
00:28:40,880 --> 00:28:47,919
uses stack in a standard way this is

728
00:28:44,399 --> 00:28:51,120
relative this is arguably easy to do

729
00:28:47,919 --> 00:28:53,200
but if your handwritten assembly does

730
00:28:51,120 --> 00:28:54,960
non-standard stack usage then you may

731
00:28:53,200 --> 00:28:56,960
have to deal with what are called dwarf

732
00:28:54,960 --> 00:28:59,360
expressions now writing cfi directives

733
00:28:56,960 --> 00:29:01,520
with dwarf expression is not going to be

734
00:28:59,360 --> 00:29:03,360
very maintainable

735
00:29:01,520 --> 00:29:06,080
lastly if you

736
00:29:03,360 --> 00:29:07,600
if you do end up needing original value

737
00:29:06,080 --> 00:29:10,159
of your arguments for your online back

738
00:29:07,600 --> 00:29:12,080
tracing that's not going to be easy the

739
00:29:10,159 --> 00:29:14,240
information is stored in dwarf debug

740
00:29:12,080 --> 00:29:17,279
sections and which can be pretty large

741
00:29:14,240 --> 00:29:18,159
to consume at runtime

742
00:29:17,279 --> 00:29:20,559
so

743
00:29:18,159 --> 00:29:23,279
in reality because of these issues with

744
00:29:20,559 --> 00:29:25,600
eh frame based methods there exists a

745
00:29:23,279 --> 00:29:28,159
number of unwind formats so that's the

746
00:29:25,600 --> 00:29:30,960
state of the art right each frame is not

747
00:29:28,159 --> 00:29:31,919
the only one out in the field

748
00:29:30,960 --> 00:29:33,600
uh

749
00:29:31,919 --> 00:29:34,960
many applications that we have looked

750
00:29:33,600 --> 00:29:36,640
around um

751
00:29:34,960 --> 00:29:39,039
they have tried to come up with their

752
00:29:36,640 --> 00:29:41,360
own debug formats trying to

753
00:29:39,039 --> 00:29:43,360
you know steer away from eh frame based

754
00:29:41,360 --> 00:29:45,919
methods kernel for example has its own

755
00:29:43,360 --> 00:29:49,120
unwind format called the

756
00:29:45,919 --> 00:29:52,240
object rewind capability

757
00:29:49,120 --> 00:29:54,240
no sorry oops rewind capability perhaps

758
00:29:52,240 --> 00:29:56,880
anyway various other projects that we

759
00:29:54,240 --> 00:29:59,520
ran that we ran into also created their

760
00:29:56,880 --> 00:30:01,679
own ad hoc solutions around the problem

761
00:29:59,520 --> 00:30:03,600
uh the important bit here is that these

762
00:30:01,679 --> 00:30:05,679
ad hoc solutions and their unwind

763
00:30:03,600 --> 00:30:08,720
formats are not supported in the no tool

764
00:30:05,679 --> 00:30:11,919
chain and which makes all of these uh

765
00:30:08,720 --> 00:30:14,960
addox solutions a big ordeal

766
00:30:11,919 --> 00:30:16,799
and lastly well if you still want to

767
00:30:14,960 --> 00:30:19,200
recover if you still need means to

768
00:30:16,799 --> 00:30:22,080
recover original value of the arguments

769
00:30:19,200 --> 00:30:24,159
the state of the art remains that

770
00:30:22,080 --> 00:30:26,640
you need to get that information from

771
00:30:24,159 --> 00:30:28,480
dwarf call site information which lies

772
00:30:26,640 --> 00:30:30,159
in the debug info sections now remember

773
00:30:28,480 --> 00:30:32,399
all those problems with dwarf they mean

774
00:30:30,159 --> 00:30:34,159
the dwarf may be stripped out they are

775
00:30:32,399 --> 00:30:36,640
large and it's going to be slow just

776
00:30:34,159 --> 00:30:37,440
because of the size of the things

777
00:30:36,640 --> 00:30:42,159
so

778
00:30:37,440 --> 00:30:45,600
cdf backtrace format aims to solve

779
00:30:42,159 --> 00:30:48,880
these issues by providing um a solution

780
00:30:45,600 --> 00:30:50,399
um it tries to it the the premise here

781
00:30:48,880 --> 00:30:53,679
is that if you aim for the following

782
00:30:50,399 --> 00:30:55,679
requirements you will be able to solve

783
00:30:53,679 --> 00:30:57,200
uh the previously mentioned problems

784
00:30:55,679 --> 00:31:00,000
number one

785
00:30:57,200 --> 00:31:01,360
you must provide means for asynchronous

786
00:31:00,000 --> 00:31:03,519
track unwinding otherwise the back

787
00:31:01,360 --> 00:31:05,200
traces are not going to be useful number

788
00:31:03,519 --> 00:31:06,000
two

789
00:31:05,200 --> 00:31:08,000
yes

790
00:31:06,000 --> 00:31:10,720
have some extensions in the ctf format

791
00:31:08,000 --> 00:31:13,120
but keep the format compact and simple

792
00:31:10,720 --> 00:31:15,200
do not add complex expressions encoding

793
00:31:13,120 --> 00:31:18,080
no stack machines whatsoever

794
00:31:15,200 --> 00:31:19,840
and number three allow means to recover

795
00:31:18,080 --> 00:31:21,679
original value of the arguments to the

796
00:31:19,840 --> 00:31:23,760
function call

797
00:31:21,679 --> 00:31:25,519
some online back racers need this and

798
00:31:23,760 --> 00:31:28,919
it's a must have for them

799
00:31:25,519 --> 00:31:28,919
excuse me

800
00:31:36,799 --> 00:31:39,840
so

801
00:31:38,080 --> 00:31:40,799
we plan to introduce a new section

802
00:31:39,840 --> 00:31:42,559
called

803
00:31:40,799 --> 00:31:44,480
dot ctf frame

804
00:31:42,559 --> 00:31:46,960
we will only talk about high level

805
00:31:44,480 --> 00:31:49,039
details at this time

806
00:31:46,960 --> 00:31:50,159
the section stores information on how to

807
00:31:49,039 --> 00:31:53,039
recover

808
00:31:50,159 --> 00:31:54,880
cfa and ra given the pc

809
00:31:53,039 --> 00:31:58,240
it does not store information on how to

810
00:31:54,880 --> 00:32:00,399
recover other quality saved registers

811
00:31:58,240 --> 00:32:02,399
conceptually it's going to be equivalent

812
00:32:00,399 --> 00:32:03,360
right to the table of interpreted dwarf

813
00:32:02,399 --> 00:32:05,760
data

814
00:32:03,360 --> 00:32:07,760
but not the full table as compared to

815
00:32:05,760 --> 00:32:10,559
each frame because we only need to know

816
00:32:07,760 --> 00:32:12,640
about cfa and ra right so we are

817
00:32:10,559 --> 00:32:15,440
starting out with uh

818
00:32:12,640 --> 00:32:17,360
support for amd 64 and air 64 but there

819
00:32:15,440 --> 00:32:20,000
is no reason why other

820
00:32:17,360 --> 00:32:21,840
arches and abis cannot be accommodated

821
00:32:20,000 --> 00:32:24,399
we are working to support for the

822
00:32:21,840 --> 00:32:26,799
generation of cdf batteries format in

823
00:32:24,399 --> 00:32:29,200
the new tool chain uh toolchain is the

824
00:32:26,799 --> 00:32:30,720
ideal place where such a format

825
00:32:29,200 --> 00:32:32,799
needs to be supported because we know

826
00:32:30,720 --> 00:32:34,640
from existing ad hoc solutions that

827
00:32:32,799 --> 00:32:38,559
anything other than tool chain becomes

828
00:32:34,640 --> 00:32:38,559
very difficult to maintain and extend

829
00:32:40,080 --> 00:32:44,399
so let's talk about what cdf backtrace

830
00:32:42,559 --> 00:32:48,159
format has to offer for recovering

831
00:32:44,399 --> 00:32:50,640
original value of the arguments

832
00:32:48,159 --> 00:32:53,519
in debug format jargon this is often

833
00:32:50,640 --> 00:32:57,440
called call site information uh let's

834
00:32:53,519 --> 00:33:00,399
first revisit the background again

835
00:32:57,440 --> 00:33:03,200
how are arguments passed to a function

836
00:33:00,399 --> 00:33:06,159
right so there is the there is these

837
00:33:03,200 --> 00:33:09,120
abi's that define the parameter passing

838
00:33:06,159 --> 00:33:11,760
rules um the first bit that an abi

839
00:33:09,120 --> 00:33:13,919
specifies as how to assign

840
00:33:11,760 --> 00:33:16,240
a storage class to an argument so given

841
00:33:13,919 --> 00:33:18,799
a high level language type if you write

842
00:33:16,240 --> 00:33:20,000
foo and it has two arguments in a

843
00:33:18,799 --> 00:33:21,919
in b

844
00:33:20,000 --> 00:33:24,240
it tells you so given these high level

845
00:33:21,919 --> 00:33:25,840
language types inc and floats and

846
00:33:24,240 --> 00:33:27,760
whatnot what is the corresponding

847
00:33:25,840 --> 00:33:31,279
machine type

848
00:33:27,760 --> 00:33:33,360
aka the storage class next

849
00:33:31,279 --> 00:33:36,640
excuse me given the storage class the

850
00:33:33,360 --> 00:33:39,120
abi rules the api defines rules on how

851
00:33:36,640 --> 00:33:42,559
to define how to assign

852
00:33:39,120 --> 00:33:45,960
appropriate register or stack location

853
00:33:42,559 --> 00:33:45,960
excuse me

854
00:33:53,360 --> 00:33:59,760
so the advantage of uh of of of

855
00:33:57,760 --> 00:34:01,760
approaching the problem in this way is

856
00:33:59,760 --> 00:34:03,600
that you can

857
00:34:01,760 --> 00:34:06,000
you can potentially find a way to

858
00:34:03,600 --> 00:34:08,800
relieve the debug format of specifying

859
00:34:06,000 --> 00:34:10,399
the exact register location

860
00:34:08,800 --> 00:34:13,040
right so

861
00:34:10,399 --> 00:34:15,919
in this case um the exact register

862
00:34:13,040 --> 00:34:17,679
location set is going to be a larger set

863
00:34:15,919 --> 00:34:20,399
whereas storage location set is going to

864
00:34:17,679 --> 00:34:22,720
be a smaller set across apis now each

865
00:34:20,399 --> 00:34:24,960
api as you know will offer you know

866
00:34:22,720 --> 00:34:26,399
different set of registers and

867
00:34:24,960 --> 00:34:28,639
but the storage classes are going to be

868
00:34:26,399 --> 00:34:31,520
pretty similar across avis or a smaller

869
00:34:28,639 --> 00:34:31,520
set rather right

870
00:34:33,520 --> 00:34:40,000
so uh as an example let's see amd 64

871
00:34:36,960 --> 00:34:41,839
looks like this integer ssc sse

872
00:34:40,000 --> 00:34:45,520
up memory and so on

873
00:34:41,839 --> 00:34:47,760
for ar 64 similar looking classes with

874
00:34:45,520 --> 00:34:49,679
some differences of course right so as

875
00:34:47,760 --> 00:34:52,240
you see the set of storage classes and

876
00:34:49,679 --> 00:34:54,240
amd 64 look similar so this is the main

877
00:34:52,240 --> 00:34:56,800
premise we are relying on

878
00:34:54,240 --> 00:34:59,040
the storage location of the argument is

879
00:34:56,800 --> 00:35:01,839
not encoded explicitly in the debug

880
00:34:59,040 --> 00:35:03,359
format the natural location is inferred

881
00:35:01,839 --> 00:35:05,760
from the position and class of the

882
00:35:03,359 --> 00:35:07,599
argument and the abi

883
00:35:05,760 --> 00:35:10,400
by the client right it's not encoded in

884
00:35:07,599 --> 00:35:13,760
the debug format so

885
00:35:10,400 --> 00:35:17,720
if it is a bit hazy still i hope it is

886
00:35:13,760 --> 00:35:17,720
clearer with this example

887
00:35:18,640 --> 00:35:22,560
so function one is a function with six

888
00:35:20,960 --> 00:35:25,760
integer arguments

889
00:35:22,560 --> 00:35:29,040
the amd64 avi mandates that the

890
00:35:25,760 --> 00:35:31,680
arguments be passed via rdi rsi

891
00:35:29,040 --> 00:35:33,920
rcx and so on in order

892
00:35:31,680 --> 00:35:36,720
now as far as the call side information

893
00:35:33,920 --> 00:35:40,079
is concerned it should be enough if the

894
00:35:36,720 --> 00:35:43,119
debug format conveys that the arguments

895
00:35:40,079 --> 00:35:44,800
are of integer class

896
00:35:43,119 --> 00:35:47,359
and the backtrace client with an

897
00:35:44,800 --> 00:35:48,240
understanding of the api can conclude

898
00:35:47,359 --> 00:35:52,079
that

899
00:35:48,240 --> 00:35:53,440
because the storage class was integer i

900
00:35:52,079 --> 00:35:56,320
know that these

901
00:35:53,440 --> 00:35:58,960
arguments will be passed via rdi rsi and

902
00:35:56,320 --> 00:36:01,520
so on so in interest of time again i

903
00:35:58,960 --> 00:36:03,200
will skip through the second example

904
00:36:01,520 --> 00:36:04,560
where one of the arguments is a small

905
00:36:03,200 --> 00:36:07,760
aggregate

906
00:36:04,560 --> 00:36:10,000
but as you see what we need is just an

907
00:36:07,760 --> 00:36:10,960
unambiguous way to convey the storage

908
00:36:10,000 --> 00:36:12,320
class

909
00:36:10,960 --> 00:36:14,880
and the rest can be done by the

910
00:36:12,320 --> 00:36:17,200
batteries client

911
00:36:14,880 --> 00:36:20,000
so the call site information edition is

912
00:36:17,200 --> 00:36:23,599
being planned in two phases so that the

913
00:36:20,000 --> 00:36:26,560
addition of the cdf format the additions

914
00:36:23,599 --> 00:36:29,200
to the cdf format are done in a

915
00:36:26,560 --> 00:36:31,119
data driven and thoughtful way in phase

916
00:36:29,200 --> 00:36:33,680
one we will focus on what we talked

917
00:36:31,119 --> 00:36:35,440
about previously that the original value

918
00:36:33,680 --> 00:36:37,359
of the argument can be recovered from

919
00:36:35,440 --> 00:36:40,000
its natural location if it is not

920
00:36:37,359 --> 00:36:42,480
clobbered and what you encode is just

921
00:36:40,000 --> 00:36:45,760
some information about storage classes

922
00:36:42,480 --> 00:36:47,680
now clearly there will be cases when

923
00:36:45,760 --> 00:36:50,800
these natural locations will be

924
00:36:47,680 --> 00:36:52,720
clobbered by the colleagues so an option

925
00:36:50,800 --> 00:36:55,599
so as an option there is to

926
00:36:52,720 --> 00:36:57,280
there is to fall back on

927
00:36:55,599 --> 00:37:00,640
is

928
00:36:57,280 --> 00:37:03,200
is explicit saving of argument values on

929
00:37:00,640 --> 00:37:05,760
stack if the user chooses to do so and

930
00:37:03,200 --> 00:37:06,640
the format facilitates the recovery

931
00:37:05,760 --> 00:37:08,480
to

932
00:37:06,640 --> 00:37:09,680
of the original value of the arguments

933
00:37:08,480 --> 00:37:10,560
from stack

934
00:37:09,680 --> 00:37:14,000
so

935
00:37:10,560 --> 00:37:15,520
to summarize ctf back traces um we

936
00:37:14,000 --> 00:37:17,520
discussed today that the format

937
00:37:15,520 --> 00:37:18,880
extensions

938
00:37:17,520 --> 00:37:21,280
uh

939
00:37:18,880 --> 00:37:23,359
to support online information and call

940
00:37:21,280 --> 00:37:25,440
site information and underway

941
00:37:23,359 --> 00:37:27,680
we looked at the requirements of online

942
00:37:25,440 --> 00:37:30,160
back tracing and to fulfill those

943
00:37:27,680 --> 00:37:31,200
requirements we have to keep the format

944
00:37:30,160 --> 00:37:32,079
minimal

945
00:37:31,200 --> 00:37:33,920
um

946
00:37:32,079 --> 00:37:36,160
we are working on these format changes

947
00:37:33,920 --> 00:37:38,720
and incorporating the support in gnu

948
00:37:36,160 --> 00:37:38,720
tool chain

949
00:37:38,880 --> 00:37:44,000
finally to wrap it all up

950
00:37:41,440 --> 00:37:47,040
ctf we introduced ctf

951
00:37:44,000 --> 00:37:50,760
today and a cdf is a debug format that's

952
00:37:47,040 --> 00:37:50,760
compact and fast

953
00:37:51,520 --> 00:37:57,200
at this time it is

954
00:37:54,560 --> 00:38:00,640
type information for c programs only and

955
00:37:57,200 --> 00:38:03,359
is fully supported in the gnu tool chain

956
00:38:00,640 --> 00:38:04,960
it is evolving to address unfulfilled

957
00:38:03,359 --> 00:38:07,359
requirements around debugging and back

958
00:38:04,960 --> 00:38:09,760
tracing if you are interested and if you

959
00:38:07,359 --> 00:38:12,160
want to participate have questions you

960
00:38:09,760 --> 00:38:14,800
can reach us on any of these

961
00:38:12,160 --> 00:38:18,040
public mailing lists

962
00:38:14,800 --> 00:38:18,040
thank you

963
00:38:19,599 --> 00:38:25,839
thank you so much indo uh and nick and

964
00:38:22,640 --> 00:38:29,599
edsentia for um for sharing all that

965
00:38:25,839 --> 00:38:32,240
with us wow um that was

966
00:38:29,599 --> 00:38:35,280
really interesting and thorough look at

967
00:38:32,240 --> 00:38:37,200
i guess the new possibilities um in the

968
00:38:35,280 --> 00:38:38,960
future so

969
00:38:37,200 --> 00:38:42,320
thanks um

970
00:38:38,960 --> 00:38:42,320
is there anything you'd like to say

971
00:38:42,480 --> 00:38:45,359
thanks

972
00:38:42,930 --> 00:38:47,599
[Music]

973
00:38:45,359 --> 00:38:49,920
well yes thank you so much round of

974
00:38:47,599 --> 00:38:53,200
applause for indo and nick um

975
00:38:49,920 --> 00:38:55,480
and that was our last talk for today

976
00:38:53,200 --> 00:38:58,160
everyone last talk for

977
00:38:55,480 --> 00:39:00,960
linuxconfigu2022 but don't close the

978
00:38:58,160 --> 00:39:04,880
window yet um so we will have the

979
00:39:00,960 --> 00:39:05,760
conference closing in this very room um

980
00:39:04,880 --> 00:39:07,920
in

981
00:39:05,760 --> 00:39:10,000
about 15 minutes because this talk is

982
00:39:07,920 --> 00:39:12,240
finishing up a little early

983
00:39:10,000 --> 00:39:15,040
so don't go anywhere conference closing

984
00:39:12,240 --> 00:39:17,440
is always worth watching um it'll be

985
00:39:15,040 --> 00:39:20,160
lovely to see the wrap up

986
00:39:17,440 --> 00:39:23,040
um but for me that's it from me so thank

987
00:39:20,160 --> 00:39:24,960
you for joining us all today with all of

988
00:39:23,040 --> 00:39:28,280
these wonderful talks

989
00:39:24,960 --> 00:39:28,280
bye everyone

990
00:39:34,160 --> 00:39:36,240
you