1
00:00:00,329 --> 00:00:06,269
Good, thanks. So I'll start by saying that
I won't say much about interpretability

2
00:00:06,269 --> 00:00:10,559
explicitly. But I will try to say
something about reliance on simulation.

3
00:00:10,679 --> 00:00:14,579
And I only have 15 minutes. So I'll be
brief, but maybe there'll be some

4
00:00:14,579 --> 00:00:17,819
discussion later. And just to very
quickly, you can confirm that that you can

5
00:00:17,819 --> 00:00:18,959
see the slide advance, right?

6
00:00:21,390 --> 00:00:23,130
Yes, it was sort of fine.

7
00:00:23,640 --> 00:00:28,050
Excellent. Very good. I never remember in
video, which full screen mode works, but

8
00:00:28,050 --> 00:00:31,440
let's just get started. So I thought I
would begin by reviewing briefly data

9
00:00:31,440 --> 00:00:35,130
analysis, and how can we rely on
simulation? So I think this is a clear

10
00:00:35,130 --> 00:00:39,060
path parallel path that we have many
analyses in our field where we want to

11
00:00:39,060 --> 00:00:43,320
infer something about nature, obviously.
And in order to do so we build giant

12
00:00:43,320 --> 00:00:48,510
experiments we are able to observe from
those experiments, we do some

13
00:00:48,510 --> 00:00:53,100
dimensionality reduction, we have sort of
a parallel path in the simulation world

14
00:00:53,100 --> 00:00:56,820
from our theory of everything which we'd
like to infer something about two physics

15
00:00:56,820 --> 00:01:00,960
simulators which are here drawn as a black
box and then to have produce the same

16
00:01:01,290 --> 00:01:07,950
complex n dimensional face space, we can
compare left and right. And this path of

17
00:01:07,950 --> 00:01:12,060
imprints has been accelerated in many
ways, but learning for basically every

18
00:01:12,060 --> 00:01:16,950
aspect. So I've drawn little red arrows
here for all the places where many places

19
00:01:16,950 --> 00:01:20,730
for deep learning has played a role and
try to indicate a few examples. And I

20
00:01:20,730 --> 00:01:25,560
clearly don't have time to talk about all
these things, or even the impact of

21
00:01:25,650 --> 00:01:30,960
simulation and reliability. But I will try
to say a few things about overall pictures

22
00:01:30,960 --> 00:01:38,790
about overall picture of how this in what
is the impact of uncertainty. Okay, so

23
00:01:39,960 --> 00:01:44,220
right, so today, in this brief talk, I'll
first talk about uncertainty for

24
00:01:44,220 --> 00:01:48,630
simulation based inference. And then I'll
very briefly describe the landscape of

25
00:01:48,630 --> 00:01:54,150
simulation independent inference. And so
just to be clear, when I say simulation

26
00:01:54,150 --> 00:01:59,370
based inference here here, I really mean
using a simulator to infer something about

27
00:01:59,610 --> 00:02:03,540
the parameters. The theory thing which of
course spans a wide range itself, from

28
00:02:04,500 --> 00:02:07,830
using simulation to very minimal way to
infer something to using simulation very

29
00:02:07,830 --> 00:02:12,000
maximally, and it's really quite exciting
is that we have very high fidelity

30
00:02:12,000 --> 00:02:17,670
simulators which allow us to do things now
that we couldn't do before with deep

31
00:02:17,670 --> 00:02:20,910
learning, and I would love to tell you
about some of those things. And many of

32
00:02:20,910 --> 00:02:27,630
those things are covered in other areas of
this dis track in the conference. But

33
00:02:27,630 --> 00:02:30,420
unfortunately, today, I only want to live
in uncertainty. And so I'm motivated in

34
00:02:30,420 --> 00:02:34,140
part by this question, which I think is a
question that basically everyone asks

35
00:02:34,140 --> 00:02:38,550
about uncertainty. So what are the
uncertainties on that neural network? And

36
00:02:38,580 --> 00:02:41,490
I'd like to sharpen this question a little
bit. And so in order to do so let's think

37
00:02:41,490 --> 00:02:47,190
about a search, for instance. I think that
will help frame the question a bit more

38
00:02:47,700 --> 00:02:52,950
concretely. So I'll tell you probably what
a search is like. Typical search using

39
00:02:52,950 --> 00:02:56,730
simulation is you have some simulation in
the background and citizen or Donaldson

40
00:02:56,730 --> 00:03:00,870
simulation of a particular signal, you
train a classifier To distinguish them,

41
00:03:00,900 --> 00:03:04,020
and then you might have a plot like this,
where you have a neural network output for

42
00:03:04,020 --> 00:03:08,340
some some signal doesn't matter so much
what it is, you define a control region on

43
00:03:08,340 --> 00:03:11,640
the left where the signal purity is low.
And that's an origin on the right, where

44
00:03:11,640 --> 00:03:15,150
if there was a signal that there'd be, you
know, an excess. So you can use a control

45
00:03:15,150 --> 00:03:18,390
region to modify the simulation. And then,
you know, the ultimate goal is to compare

46
00:03:18,390 --> 00:03:21,960
the data missing in a single region. And
if they're different, that's great,

47
00:03:22,050 --> 00:03:25,440
discover something new, and that the data
in the simulation agree, and you can set

48
00:03:25,440 --> 00:03:30,330
limits on the possible production cross
section of this new signal. Okay, so I

49
00:03:30,330 --> 00:03:34,500
like to think about uncertainties is
coming into sort of two categories, what I

50
00:03:34,500 --> 00:03:39,660
call precision or optimality, and accuracy
and bias. Of course, say that these are

51
00:03:39,660 --> 00:03:44,250
not the words that people in machine
learning world use. They use different

52
00:03:44,250 --> 00:03:48,090
words, but I think these will map more
directly onto how we use them in our field

53
00:03:48,090 --> 00:03:51,660
and I'll break them down even more in a
second. So high precision often only I

54
00:03:51,660 --> 00:03:55,440
mean, a bad use of our data time money,
but but something that's not wrong. So you

55
00:03:55,440 --> 00:03:59,280
have a procedure which is optimal, but
but, but you can but you may result in a

56
00:03:59,280 --> 00:04:03,510
procedure which Which can be accurate. And
so what I mean by optimal, so let's say

57
00:04:03,510 --> 00:04:09,450
you're doing this particular search, I
mentioned the previous slide. If there are

58
00:04:09,450 --> 00:04:11,760
no nuisance parameters, and you're
training a neural network to distinguish

59
00:04:11,760 --> 00:04:15,120
signal from background, then there is an
optimal test statistic, which is a

60
00:04:15,120 --> 00:04:21,570
likelihood ratio. So P s over B over PB.
And in that case, if you're not, for

61
00:04:21,570 --> 00:04:26,310
instance, it's not like the ratio, then
your procedures are optimal. Or it's not,

62
00:04:26,340 --> 00:04:29,430
you know, mom's manipulated to like the
ratio. And so that's what I mean by

63
00:04:29,430 --> 00:04:29,970
optimal.

64
00:04:31,529 --> 00:04:32,309
And

65
00:04:33,839 --> 00:04:37,439
on the accuracy side, so there it's about
computing p values. So if you have some

66
00:04:37,439 --> 00:04:41,459
particular network, whether or not it's
optimal, if your predicted tail

67
00:04:41,459 --> 00:04:47,339
probabilities are not the true one, then
your result will be incorrect. And so

68
00:04:47,549 --> 00:04:51,839
here, no network is fixed, and you're just
computing the probability. So probability

69
00:04:51,839 --> 00:04:58,109
of no network under some model, and even
if your procedures is optimal, can still

70
00:04:58,109 --> 00:05:03,479
be biased and vice versa. Okay, I like to
further decompose these into further

71
00:05:03,479 --> 00:05:06,809
categories. It's just a consistent I
concert nice. And so we have four boxes.

72
00:05:07,229 --> 00:05:12,539
And I'm going to spend a little bit of
time going over how we can characterize

73
00:05:12,539 --> 00:05:16,289
each of these boxes. So the top left box
is the limited statistics, this is

74
00:05:16,349 --> 00:05:19,379
uncertainty on the optionality. So this
you can accomplish some sort of

75
00:05:19,379 --> 00:05:24,989
straightforward way by training by
inserting sub sampling. And, and

76
00:05:25,769 --> 00:05:28,559
retraining. This is kind of painful,
because you may have to be training a lot

77
00:05:28,559 --> 00:05:31,829
of neural networks. But in principle, it's
sort of straightforward. And there might

78
00:05:31,829 --> 00:05:36,329
be some, you know, some tricks to do this
all in one go. But nonetheless, I think we

79
00:05:36,329 --> 00:05:40,379
sort of know what to do here. What's Of
course, trickier in general is systematic

80
00:05:40,379 --> 00:05:44,069
uncertainties are hard to quantify. So one
component of this is the modeling of P

81
00:05:44,069 --> 00:05:48,149
itself. So if your training sample does
not does not represent nature, then

82
00:05:48,149 --> 00:05:51,449
obviously your procedure can be optimal.
But there are other components of this as

83
00:05:51,449 --> 00:05:55,679
well. So if your network doesn't have
enough flexibility, then that may not only

84
00:05:55,679 --> 00:06:00,449
be a ratio, or the equivalent to the
optimal procedure in your case. So that

85
00:06:00,449 --> 00:06:04,589
means your your, your sensitivity and your
need to be probed by varying, you know,

86
00:06:04,589 --> 00:06:08,009
the setup of your network, for instance,
like the number of layers, nodes per

87
00:06:08,009 --> 00:06:11,429
layer, etc. Now, these are hard to
quantify in terms of nuisance parameters.

88
00:06:11,429 --> 00:06:14,639
But at least in principle, you sort of
know what knobs are available to the very

89
00:06:16,919 --> 00:06:20,909
bottom left, this is sister concert on the
on the on the bias that this is less

90
00:06:20,909 --> 00:06:23,759
painful because they're No, no, it gets
fixed, but in principle, you can estimate

91
00:06:23,759 --> 00:06:30,689
it in a very similar way. And then I think
the trickiest one is the uncertainty on p

92
00:06:30,689 --> 00:06:35,879
itself. So the modeling of the feature
space you're using for computing p values.

93
00:06:36,299 --> 00:06:40,859
And this is a challenge because the whole
point of machine learning here is that x

94
00:06:40,859 --> 00:06:44,939
is gonna be high dimensional. So in many
cases, you know, for now, people using

95
00:06:45,239 --> 00:06:48,239
started using machine learning or your x
might be like 10 dimensional or 20

96
00:06:48,239 --> 00:06:51,869
dimensional, but ultimately, you know,
with the problems of deep learning x can

97
00:06:51,869 --> 00:06:55,439
be thousand dimensional or million
dimensional or whatever. And so, in some

98
00:06:55,439 --> 00:06:58,319
cases, uncertainties factorize. So let's
say you have a final state with many

99
00:06:58,319 --> 00:07:01,649
particles and you know, the uncertainty In
each article that maybe you can decompose

100
00:07:01,649 --> 00:07:05,039
the uncertainty, however, in many cases,
we simply don't know the full uncertainty

101
00:07:05,039 --> 00:07:08,309
model. So that means we don't know what
the nuisance parameters are. And of

102
00:07:08,309 --> 00:07:10,769
course, if you don't know what they are,
we only know their distribution, their

103
00:07:10,769 --> 00:07:15,989
probability density. So that's a
challenge. And I think that this is

104
00:07:15,989 --> 00:07:19,499
particularly true for certain kinds of
uncertainties. So in the current paradigm

105
00:07:19,559 --> 00:07:23,339
for uncertainties, we often have cases
where we compare two different simulators.

106
00:07:23,549 --> 00:07:27,629
And that gives us an uncertainty on you
know, supposed to capture a lot of

107
00:07:27,689 --> 00:07:30,569
features, which are in fact a high
dimensional feature space. And that can be

108
00:07:30,569 --> 00:07:33,809
like one nuisance parameter. I think we
all agree that's probably inappropriate.

109
00:07:34,589 --> 00:07:40,949
And tickets become more acute in high
dimensions. So how can we even see how

110
00:07:40,949 --> 00:07:43,559
sensitive we are to these high dimensional
effects, I don't have a solution to tell

111
00:07:43,559 --> 00:07:48,569
you how to make this uncertainty smaller,
but at least I can give you a partial tool

112
00:07:48,599 --> 00:07:53,039
diagnostic tool to try identify some of
these effects. And it borrows from the

113
00:07:53,039 --> 00:07:57,509
topic of AI safety. It actually is a
really fascinating field of machine

114
00:07:57,509 --> 00:08:01,919
learning and one subfield of that is a
Something called adversarial examples.

115
00:08:02,399 --> 00:08:06,929
Maybe the idea is to see if you can take
your feature space, slowly perturb it in

116
00:08:06,929 --> 00:08:11,399
such a way that you maximally change the
output of neural network. This example

117
00:08:11,399 --> 00:08:15,119
here it is you change the stop sign change
a few pixels in the picture to make a stop

118
00:08:15,119 --> 00:08:19,289
sign look like a 45 mile per hour sign,
which is obviously bad for self driving

119
00:08:19,289 --> 00:08:22,499
cars, they should be slowing down instead,
they're speeding up as a huge problem.

120
00:08:23,309 --> 00:08:26,699
Now, we don't think nature is evil. So no
one's going to be changing our particles

121
00:08:26,699 --> 00:08:30,269
by some deterministic value given our
neural network. But there's others that

122
00:08:30,299 --> 00:08:33,779
have said worst case bounds maybe about
allows us at worst case bounds to see our

123
00:08:33,779 --> 00:08:38,369
sensitivity. If I can make a small change
in our in our collision event, and there's

124
00:08:38,369 --> 00:08:40,799
a big change in our neural network that's
exists, the general network is

125
00:08:40,799 --> 00:08:47,309
particularly sensitive to perturbations.
Some of this work. So imagine you have

126
00:08:47,309 --> 00:08:50,789
some collision event j, which could be
high dimensional. Imagine you train a

127
00:08:50,789 --> 00:08:56,789
classifier for universal background.
That's f it's fixed X on J. And suppose

128
00:08:56,789 --> 00:09:02,669
you want to learn perturbation g which
maps Jake To j plus delta J. And I want

129
00:09:02,669 --> 00:09:06,989
this perturbation to maximally affect the
classifier. So let's select this first one

130
00:09:06,989 --> 00:09:12,059
here. And furthermore, I'd like it to
preserve a set of features. So the O of

131
00:09:12,059 --> 00:09:16,409
J's here are some observables that we can
validate in the control region. And the

132
00:09:16,409 --> 00:09:20,549
last term here says I want those to be not
changed. So I'd like to learn a

133
00:09:20,549 --> 00:09:24,239
perturbation that changes the network, but
doesn't change those features.

134
00:09:25,590 --> 00:09:29,130
There's just a quick plot to show that,
okay, there's a big impact. So the y axis

135
00:09:29,130 --> 00:09:32,520
is the relative discovery significance. So
up is better than threshold and the

136
00:09:32,520 --> 00:09:35,520
classifier is what you might cut for a
particular classification task doesn't

137
00:09:35,520 --> 00:09:38,580
matter so much what this is, but to go
from the solid, the dash, that's the

138
00:09:38,580 --> 00:09:42,030
important impact of this unobservable
perturbation. And so you see, there might

139
00:09:42,030 --> 00:09:45,570
be some sensitivity that may not be
accounted for with the current uncertainty

140
00:09:45,570 --> 00:09:48,630
paradigm. So this is something that that
you know, maybe we didn't think about more

141
00:09:48,630 --> 00:09:53,220
deeply in the future. Alright, so now let
me just quickly say about who can reduce

142
00:09:53,220 --> 00:09:58,170
the various uncertainties. So for the
physical journey, obviously continue with

143
00:09:58,170 --> 00:10:00,720
more events that will reduce the
uncertainty that Although it's possible

144
00:10:00,720 --> 00:10:04,260
that maybe neural networks can help, and I
don't have time to tell you how we can do

145
00:10:04,260 --> 00:10:07,530
that, well, there's been a lot of
discussion about GaNS. And this this track

146
00:10:09,210 --> 00:10:13,050
for the systematic uncertainties, some of
them, it may be possible to reduce the

147
00:10:13,050 --> 00:10:16,770
uncertainty by alleviating or at least
alleviate analysis, complexity and making

148
00:10:16,770 --> 00:10:19,980
your neural network independent of certain
nuisance parameters. But it may also be

149
00:10:19,980 --> 00:10:22,920
better to explicitly depend on them and
then profile. And this just depends on the

150
00:10:22,920 --> 00:10:28,980
various trade offs in your data set. But
then, I think that the most challenging

151
00:10:28,980 --> 00:10:32,400
uncertainty really is the modeling of the
high dimensional bias uncertainties for

152
00:10:32,400 --> 00:10:36,270
them the modeling of the P itself. I think
this is the biggest challenge of deploying

153
00:10:36,360 --> 00:10:39,270
full network based analysis and
dimensional feature space. And this will

154
00:10:39,270 --> 00:10:42,660
require hard physics work. So I think it's
possible with them, the community working

155
00:10:42,660 --> 00:10:46,950
together. Of course, an alternative
solution is not to use simulation at all.

156
00:10:47,310 --> 00:10:50,340
And this is not always possible. And of
course, they're always assumptions.

157
00:10:50,340 --> 00:10:53,850
There's there's no free lunch, but the
right methods for doing various sorts of

158
00:10:53,850 --> 00:10:57,270
inference. In particular, I'm going to
talk about anomaly detection very briefly,

159
00:10:57,870 --> 00:11:03,600
which can be can be illustrated On a
continuation. So that was very briefly the

160
00:11:03,600 --> 00:11:07,230
end of my talk, talk about the landscape
of model independence. So I like to think

161
00:11:07,230 --> 00:11:11,730
of this in having two areas. So there's
model dependence when you're gaining

162
00:11:11,730 --> 00:11:16,080
signal sensitivity. So that is to say, I
want to train a classifier. But there's

163
00:11:16,080 --> 00:11:18,930
also model dependence and how you
calibrate the background specificity. So

164
00:11:18,930 --> 00:11:22,800
how you basically know what the p value is
the tail probability. And so I'm gonna

165
00:11:22,830 --> 00:11:26,790
just briefly fill in these planes. But I
think most searches live in the bottom

166
00:11:26,790 --> 00:11:31,050
left of this plane. So they depend on a
signal simulation and the background

167
00:11:31,050 --> 00:11:35,070
simulation to train a classifier, which
might be by hand by bearing some cuts or

168
00:11:35,070 --> 00:11:39,420
by no training, no network. But there's
already searches that try to fill out the

169
00:11:39,420 --> 00:11:43,650
rest of this plane. So some searches on
the top left here, where for instance, you

170
00:11:43,680 --> 00:11:48,750
you have a signal signal simulation, and
you train against, say, a sideband, or

171
00:11:48,750 --> 00:11:51,930
some other calibration data. So that would
be pretty independent of the background

172
00:11:51,930 --> 00:11:55,350
model because you're not using background
simulation, but you depend very strongly

173
00:11:55,350 --> 00:12:02,550
on the signal model. The other extreme is
to Train, for instance classifier to

174
00:12:02,550 --> 00:12:06,960
distinguish data versus simulation. So
there's the the non machine learning

175
00:12:06,960 --> 00:12:11,100
version of this has a long history. And in
fact, CMS dealt with the result this week,

176
00:12:11,370 --> 00:12:16,170
which I've linked here. And this can be
done with or without machine learning. And

177
00:12:16,170 --> 00:12:19,320
there's some links at the bottom right.
But this is a pretty powerful technique,

178
00:12:19,320 --> 00:12:24,060
because it allows you to basically be
totally independent of the signal model.

179
00:12:24,450 --> 00:12:27,660
But it's very dependent on the background
model. Because obviously, you need to have

180
00:12:27,660 --> 00:12:32,610
a simulation or to compare simulation
data. And then there's some methods that

181
00:12:32,610 --> 00:12:35,670
are trying to move into the top right of
this to reduce the dependence on both

182
00:12:35,670 --> 00:12:39,780
signal model and background model. Live a
variety of names. And I only put some

183
00:12:39,780 --> 00:12:43,590
acronyms here, but there are some links.
And if you want to read more, you can have

184
00:12:43,590 --> 00:12:47,700
a look. And also I put a link to a more
comprehensive list at the bottom right.

185
00:12:48,270 --> 00:12:51,450
And I'm missing your favorite algorithm in
the bottom right, please let me know and

186
00:12:51,450 --> 00:12:55,770
of course we'll add it. But the right
these methods basically they use a variety

187
00:12:55,770 --> 00:12:59,880
of supervised, weekly supervised,
unsupervised methods, I would love to talk

188
00:12:59,880 --> 00:13:07,050
about But unfortunately, I don't have time
to talk about that they tried to use

189
00:13:07,080 --> 00:13:12,120
various properties of the data in order to
rely less on on the signal model, as well

190
00:13:12,120 --> 00:13:15,990
as the background model. And the last
thing I want to say here is that it's not

191
00:13:15,990 --> 00:13:19,230
good enough to be sensitive to the signal.
So you can't have a method that just is

192
00:13:19,230 --> 00:13:23,520
signal sensitive, you also have to
calibrate to the background. And so the

193
00:13:23,520 --> 00:13:27,540
right plot has has a variety of methods
that should sometimes similar. So from

194
00:13:27,540 --> 00:13:31,650
pure Monte Carlo prediction to a very
common control region method to the ABCD

195
00:13:31,650 --> 00:13:36,540
method to side bending. And each of these
sort of have a variety of single model and

196
00:13:36,540 --> 00:13:40,470
backward model independence. And they're
all sort of shift to the left with respect

197
00:13:40,470 --> 00:13:44,100
to the left plane, because these are all
relatively signal model independent

198
00:13:44,100 --> 00:13:47,850
compared to the left ones, but they have
some sort of model dependence for

199
00:13:47,850 --> 00:13:52,290
instance, assuming that the signal is
resonant, or is localized somewhere or

200
00:13:52,320 --> 00:13:56,700
whatever. And you can mix and match
methods obviously, between left and right

201
00:13:56,700 --> 00:13:59,910
but there are some that are no more
natural. So for instance, some of these

202
00:14:01,560 --> 00:14:05,490
These new techniques like the koala
method, for instance, sort of well matched

203
00:14:05,490 --> 00:14:10,050
with a bump. And so it sort of naturally
goes with side banding, although it can

204
00:14:10,050 --> 00:14:11,460
also be extended to other methods as well.

205
00:14:14,400 --> 00:14:18,090
Alright, so I'm getting close to the end,
I think there's a really exciting future

206
00:14:18,120 --> 00:14:22,920
in this area of less model dependent
inference. I just very quickly put a shout

207
00:14:22,920 --> 00:14:26,850
out to the LFC Olympics, which is a to
help facilitate this rapidly developing

208
00:14:26,850 --> 00:14:30,330
area. If you look at like the timestamps
from the previous slide, a lot of these

209
00:14:30,330 --> 00:14:33,540
new methods were developed in the last
year and this really exciting growth area.

210
00:14:34,050 --> 00:14:39,540
And so this, this one is a event to help
facilitate this weather, we've made some

211
00:14:39,540 --> 00:14:43,830
black boxes and we send it to the public
to test out your methods. And I'll just

212
00:14:43,830 --> 00:14:48,150
advertise will be a Summer Olympics, which
will happen soon will happen in July,

213
00:14:48,240 --> 00:14:53,430
it'll be virtual or you can sign up at
this link on the right. Okay, I think I'm

214
00:14:53,430 --> 00:14:56,520
now basically out of time. So I think I
don't have to tell you that deep learning

215
00:14:56,520 --> 00:15:00,930
has a great potential to impact her
physics analysis. Haven't really said

216
00:15:00,930 --> 00:15:05,490
anything about specific topics in this
talk because of the time limitation. But I

217
00:15:05,490 --> 00:15:09,810
know there's a lot of interesting subjects
covered in other areas in this track. With

218
00:15:10,140 --> 00:15:13,200
simulation based inference and other
techniques. There's the full face space of

219
00:15:13,200 --> 00:15:16,080
our experiences. We're now explorable,
which is really quite exciting, but I

220
00:15:16,080 --> 00:15:19,590
think we need to be cautious about new
challenges from uncertainty quantification

221
00:15:19,650 --> 00:15:23,160
on particular dimensions. Right. Thank
you.

222
00:15:31,140 --> 00:15:33,120
Do we have a question for Ben?

223
00:15:45,929 --> 00:15:54,029
I don't see any any. Any question? I think
we can also, since we have an extra locker

224
00:15:54,029 --> 00:15:59,129
For Question, we can also have a longer
discussion at the end of the article on

225
00:15:59,129 --> 00:16:02,429
specific questions. Any of the
contributions, so

226
00:16:03,689 --> 00:16:05,339
that can also be a possibility.

227
00:16:07,230 --> 00:16:11,550
Okay, so I don't see any specific question
for Ben. thing.

228
00:16:12,809 --> 00:16:14,189
Oh, hey,

229
00:16:15,630 --> 00:16:21,360
I have a question about the, the
adversarial thing to get the worst case

230
00:16:21,360 --> 00:16:29,220
uncertainty. I just wanted to ask, do you
have a feeling for so like, let's say the

231
00:16:29,220 --> 00:16:34,200
worst case is really bad. But what does
that mean in terms of like a mean or

232
00:16:34,200 --> 00:16:40,710
median? Like, where is the worst case
might be really terrible, but it will

233
00:16:40,710 --> 00:16:45,090
actually the data is not going to try to
deliberately fool you. So do you have a

234
00:16:45,090 --> 00:16:49,320
feeling for how that might translate to
what you would what you can expect?

235
00:16:50,220 --> 00:16:55,230
Yeah, that's an excellent question. So
this doesn't give you an impact in terms

236
00:16:55,230 --> 00:16:58,200
of like a one sigma uncertainty, right?
This just tells you like, what's the

237
00:16:58,200 --> 00:17:02,130
absolute worst so it's like a uniform
uniformly in a box of all possibilities,

238
00:17:02,130 --> 00:17:06,810
what what do you get? So clearly that, you
know, if you're doing some statistical

239
00:17:06,810 --> 00:17:09,600
analysis, you wouldn't want to put this in
as your one segment profile, because it's

240
00:17:09,600 --> 00:17:13,590
going to be really bad. However, it gives
you a bounce. So clearly, if this

241
00:17:13,590 --> 00:17:17,580
uncertainty is small, you can feel good or
not uncertainty, if this is bound is

242
00:17:17,580 --> 00:17:20,640
small, you can feel good. And if it's not
small, then maybe you have to think a bit

243
00:17:20,640 --> 00:17:25,080
harder about how you construct your
analysis. Now, at the same time, this you

244
00:17:25,080 --> 00:17:28,440
know, there's some field, you know, the
ICT field that has developed these

245
00:17:28,440 --> 00:17:31,710
diagnostic methods that have also
developed methods for robust, making

246
00:17:31,740 --> 00:17:35,880
networks more robust. And so I don't have
any plots here. But basically, you can

247
00:17:35,880 --> 00:17:41,610
combine those methods and show that you
can build, you know, robustness sense to

248
00:17:41,610 --> 00:17:45,660
insensitivity to some of these attack
attacks. So you can you can imagine, at

249
00:17:45,660 --> 00:17:48,570
least that will help you bring yourself
closer to the regime where you're less

250
00:17:48,570 --> 00:17:51,780
sensitive to by dimensional perturbations.
But no, this is a very, it's a very

251
00:17:51,780 --> 00:17:54,600
interesting question about how you
quantify this uncertainty and we don't

252
00:17:54,600 --> 00:17:55,650
have a good solution at the moment.

253
00:17:57,090 --> 00:17:58,530
That's very interesting. Thank you.

254
00:18:02,159 --> 00:18:12,539
Okay, thanks a lot. Any other questions
for 10? If not, we can thank Ben again and

255
00:18:12,539 --> 00:18:16,559
then we move to the next speaker. That