Good, thanks. So I'll start by saying that I won't say much about interpretability
explicitly. But I will try to say something about reliance on simulation.
And I only have 15 minutes. So I'll be brief, but maybe there'll be some
discussion later. And just to very quickly, you can confirm that that you can
see the slide advance, right?
Yes, it was sort of fine.
Excellent. Very good. I never remember in video, which full screen mode works, but
let's just get started. So I thought I would begin by reviewing briefly data
analysis, and how can we rely on simulation? So I think this is a clear
path parallel path that we have many analyses in our field where we want to
infer something about nature, obviously. And in order to do so we build giant
experiments we are able to observe from those experiments, we do some
dimensionality reduction, we have sort of a parallel path in the simulation world
from our theory of everything which we'd like to infer something about two physics
simulators which are here drawn as a black box and then to have produce the same
complex n dimensional face space, we can compare left and right. And this path of
imprints has been accelerated in many ways, but learning for basically every
aspect. So I've drawn little red arrows here for all the places where many places
for deep learning has played a role and try to indicate a few examples. And I
clearly don't have time to talk about all these things, or even the impact of
simulation and reliability. But I will try to say a few things about overall pictures
about overall picture of how this in what is the impact of uncertainty. Okay, so
right, so today, in this brief talk, I'll first talk about uncertainty for
simulation based inference. And then I'll very briefly describe the landscape of
simulation independent inference. And so just to be clear, when I say simulation
based inference here here, I really mean using a simulator to infer something about
the parameters. The theory thing which of course spans a wide range itself, from
using simulation to very minimal way to infer something to using simulation very
maximally, and it's really quite exciting is that we have very high fidelity
simulators which allow us to do things now that we couldn't do before with deep
learning, and I would love to tell you about some of those things. And many of
those things are covered in other areas of this dis track in the conference. But
unfortunately, today, I only want to live in uncertainty. And so I'm motivated in
part by this question, which I think is a question that basically everyone asks
about uncertainty. So what are the uncertainties on that neural network? And
I'd like to sharpen this question a little bit. And so in order to do so let's think
about a search, for instance. I think that will help frame the question a bit more
concretely. So I'll tell you probably what a search is like. Typical search using
simulation is you have some simulation in the background and citizen or Donaldson
simulation of a particular signal, you train a classifier To distinguish them,
and then you might have a plot like this, where you have a neural network output for
some some signal doesn't matter so much what it is, you define a control region on
the left where the signal purity is low. And that's an origin on the right, where
if there was a signal that there'd be, you know, an excess. So you can use a control
region to modify the simulation. And then, you know, the ultimate goal is to compare
the data missing in a single region. And if they're different, that's great,
discover something new, and that the data in the simulation agree, and you can set
limits on the possible production cross section of this new signal. Okay, so I
like to think about uncertainties is coming into sort of two categories, what I
call precision or optimality, and accuracy and bias. Of course, say that these are
not the words that people in machine learning world use. They use different
words, but I think these will map more directly onto how we use them in our field
and I'll break them down even more in a second. So high precision often only I
mean, a bad use of our data time money, but but something that's not wrong. So you
have a procedure which is optimal, but but, but you can but you may result in a
procedure which Which can be accurate. And so what I mean by optimal, so let's say
you're doing this particular search, I mentioned the previous slide. If there are
no nuisance parameters, and you're training a neural network to distinguish
signal from background, then there is an optimal test statistic, which is a
likelihood ratio. So P s over B over PB. And in that case, if you're not, for
instance, it's not like the ratio, then your procedures are optimal. Or it's not,
you know, mom's manipulated to like the ratio. And so that's what I mean by
optimal.
And
on the accuracy side, so there it's about computing p values. So if you have some
particular network, whether or not it's optimal, if your predicted tail
probabilities are not the true one, then your result will be incorrect. And so
here, no network is fixed, and you're just computing the probability. So probability
of no network under some model, and even if your procedures is optimal, can still
be biased and vice versa. Okay, I like to further decompose these into further
categories. It's just a consistent I concert nice. And so we have four boxes.
And I'm going to spend a little bit of time going over how we can characterize
each of these boxes. So the top left box is the limited statistics, this is
uncertainty on the optionality. So this you can accomplish some sort of
straightforward way by training by inserting sub sampling. And, and
retraining. This is kind of painful, because you may have to be training a lot
of neural networks. But in principle, it's sort of straightforward. And there might
be some, you know, some tricks to do this all in one go. But nonetheless, I think we
sort of know what to do here. What's Of course, trickier in general is systematic
uncertainties are hard to quantify. So one component of this is the modeling of P
itself. So if your training sample does not does not represent nature, then
obviously your procedure can be optimal. But there are other components of this as
well. So if your network doesn't have enough flexibility, then that may not only
be a ratio, or the equivalent to the optimal procedure in your case. So that
means your your, your sensitivity and your need to be probed by varying, you know,
the setup of your network, for instance, like the number of layers, nodes per
layer, etc. Now, these are hard to quantify in terms of nuisance parameters.
But at least in principle, you sort of know what knobs are available to the very
bottom left, this is sister concert on the on the on the bias that this is less
painful because they're No, no, it gets fixed, but in principle, you can estimate
it in a very similar way. And then I think the trickiest one is the uncertainty on p
itself. So the modeling of the feature space you're using for computing p values.
And this is a challenge because the whole point of machine learning here is that x
is gonna be high dimensional. So in many cases, you know, for now, people using
started using machine learning or your x might be like 10 dimensional or 20
dimensional, but ultimately, you know, with the problems of deep learning x can
be thousand dimensional or million dimensional or whatever. And so, in some
cases, uncertainties factorize. So let's say you have a final state with many
particles and you know, the uncertainty In each article that maybe you can decompose
the uncertainty, however, in many cases, we simply don't know the full uncertainty
model. So that means we don't know what the nuisance parameters are. And of
course, if you don't know what they are, we only know their distribution, their
probability density. So that's a challenge. And I think that this is
particularly true for certain kinds of uncertainties. So in the current paradigm
for uncertainties, we often have cases where we compare two different simulators.
And that gives us an uncertainty on you know, supposed to capture a lot of
features, which are in fact a high dimensional feature space. And that can be
like one nuisance parameter. I think we all agree that's probably inappropriate.
And tickets become more acute in high dimensions. So how can we even see how
sensitive we are to these high dimensional effects, I don't have a solution to tell
you how to make this uncertainty smaller, but at least I can give you a partial tool
diagnostic tool to try identify some of these effects. And it borrows from the
topic of AI safety. It actually is a really fascinating field of machine
learning and one subfield of that is a Something called adversarial examples.
Maybe the idea is to see if you can take your feature space, slowly perturb it in
such a way that you maximally change the output of neural network. This example
here it is you change the stop sign change a few pixels in the picture to make a stop
sign look like a 45 mile per hour sign, which is obviously bad for self driving
cars, they should be slowing down instead, they're speeding up as a huge problem.
Now, we don't think nature is evil. So no one's going to be changing our particles
by some deterministic value given our neural network. But there's others that
have said worst case bounds maybe about allows us at worst case bounds to see our
sensitivity. If I can make a small change in our in our collision event, and there's
a big change in our neural network that's exists, the general network is
particularly sensitive to perturbations. Some of this work. So imagine you have
some collision event j, which could be high dimensional. Imagine you train a
classifier for universal background. That's f it's fixed X on J. And suppose
you want to learn perturbation g which maps Jake To j plus delta J. And I want
this perturbation to maximally affect the classifier. So let's select this first one
here. And furthermore, I'd like it to preserve a set of features. So the O of
J's here are some observables that we can validate in the control region. And the
last term here says I want those to be not changed. So I'd like to learn a
perturbation that changes the network, but doesn't change those features.
There's just a quick plot to show that, okay, there's a big impact. So the y axis
is the relative discovery significance. So up is better than threshold and the
classifier is what you might cut for a particular classification task doesn't
matter so much what this is, but to go from the solid, the dash, that's the
important impact of this unobservable perturbation. And so you see, there might
be some sensitivity that may not be accounted for with the current uncertainty
paradigm. So this is something that that you know, maybe we didn't think about more
deeply in the future. Alright, so now let me just quickly say about who can reduce
the various uncertainties. So for the physical journey, obviously continue with
more events that will reduce the uncertainty that Although it's possible
that maybe neural networks can help, and I don't have time to tell you how we can do
that, well, there's been a lot of discussion about GaNS. And this this track
for the systematic uncertainties, some of them, it may be possible to reduce the
uncertainty by alleviating or at least alleviate analysis, complexity and making
your neural network independent of certain nuisance parameters. But it may also be
better to explicitly depend on them and then profile. And this just depends on the
various trade offs in your data set. But then, I think that the most challenging
uncertainty really is the modeling of the high dimensional bias uncertainties for
them the modeling of the P itself. I think this is the biggest challenge of deploying
full network based analysis and dimensional feature space. And this will
require hard physics work. So I think it's possible with them, the community working
together. Of course, an alternative solution is not to use simulation at all.
And this is not always possible. And of course, they're always assumptions.
There's there's no free lunch, but the right methods for doing various sorts of
inference. In particular, I'm going to talk about anomaly detection very briefly,
which can be can be illustrated On a continuation. So that was very briefly the
end of my talk, talk about the landscape of model independence. So I like to think
of this in having two areas. So there's model dependence when you're gaining
signal sensitivity. So that is to say, I want to train a classifier. But there's
also model dependence and how you calibrate the background specificity. So
how you basically know what the p value is the tail probability. And so I'm gonna
just briefly fill in these planes. But I think most searches live in the bottom
left of this plane. So they depend on a signal simulation and the background
simulation to train a classifier, which might be by hand by bearing some cuts or
by no training, no network. But there's already searches that try to fill out the
rest of this plane. So some searches on the top left here, where for instance, you
you have a signal signal simulation, and you train against, say, a sideband, or
some other calibration data. So that would be pretty independent of the background
model because you're not using background simulation, but you depend very strongly
on the signal model. The other extreme is to Train, for instance classifier to
distinguish data versus simulation. So there's the the non machine learning
version of this has a long history. And in fact, CMS dealt with the result this week,
which I've linked here. And this can be done with or without machine learning. And
there's some links at the bottom right. But this is a pretty powerful technique,
because it allows you to basically be totally independent of the signal model.
But it's very dependent on the background model. Because obviously, you need to have
a simulation or to compare simulation data. And then there's some methods that
are trying to move into the top right of this to reduce the dependence on both
signal model and background model. Live a variety of names. And I only put some
acronyms here, but there are some links. And if you want to read more, you can have
a look. And also I put a link to a more comprehensive list at the bottom right.
And I'm missing your favorite algorithm in the bottom right, please let me know and
of course we'll add it. But the right these methods basically they use a variety
of supervised, weekly supervised, unsupervised methods, I would love to talk
about But unfortunately, I don't have time to talk about that they tried to use
various properties of the data in order to rely less on on the signal model, as well
as the background model. And the last thing I want to say here is that it's not
good enough to be sensitive to the signal. So you can't have a method that just is
signal sensitive, you also have to calibrate to the background. And so the
right plot has has a variety of methods that should sometimes similar. So from
pure Monte Carlo prediction to a very common control region method to the ABCD
method to side bending. And each of these sort of have a variety of single model and
backward model independence. And they're all sort of shift to the left with respect
to the left plane, because these are all relatively signal model independent
compared to the left ones, but they have some sort of model dependence for
instance, assuming that the signal is resonant, or is localized somewhere or
whatever. And you can mix and match methods obviously, between left and right
but there are some that are no more natural. So for instance, some of these
These new techniques like the koala method, for instance, sort of well matched
with a bump. And so it sort of naturally goes with side banding, although it can
also be extended to other methods as well.
Alright, so I'm getting close to the end, I think there's a really exciting future
in this area of less model dependent inference. I just very quickly put a shout
out to the LFC Olympics, which is a to help facilitate this rapidly developing
area. If you look at like the timestamps from the previous slide, a lot of these
new methods were developed in the last year and this really exciting growth area.
And so this, this one is a event to help facilitate this weather, we've made some
black boxes and we send it to the public to test out your methods. And I'll just
advertise will be a Summer Olympics, which will happen soon will happen in July,
it'll be virtual or you can sign up at this link on the right. Okay, I think I'm
now basically out of time. So I think I don't have to tell you that deep learning
has a great potential to impact her physics analysis. Haven't really said
anything about specific topics in this talk because of the time limitation. But I
know there's a lot of interesting subjects covered in other areas in this track. With
simulation based inference and other techniques. There's the full face space of
our experiences. We're now explorable, which is really quite exciting, but I
think we need to be cautious about new challenges from uncertainty quantification
on particular dimensions. Right. Thank you.
Do we have a question for Ben?
I don't see any any. Any question? I think we can also, since we have an extra locker
For Question, we can also have a longer discussion at the end of the article on
specific questions. Any of the contributions, so
that can also be a possibility.
Okay, so I don't see any specific question for Ben. thing.
Oh, hey,
I have a question about the, the adversarial thing to get the worst case
uncertainty. I just wanted to ask, do you have a feeling for so like, let's say the
worst case is really bad. But what does that mean in terms of like a mean or
median? Like, where is the worst case might be really terrible, but it will
actually the data is not going to try to deliberately fool you. So do you have a
feeling for how that might translate to what you would what you can expect?
Yeah, that's an excellent question. So this doesn't give you an impact in terms
of like a one sigma uncertainty, right? This just tells you like, what's the
absolute worst so it's like a uniform uniformly in a box of all possibilities,
what what do you get? So clearly that, you know, if you're doing some statistical
analysis, you wouldn't want to put this in as your one segment profile, because it's
going to be really bad. However, it gives you a bounce. So clearly, if this
uncertainty is small, you can feel good or not uncertainty, if this is bound is
small, you can feel good. And if it's not small, then maybe you have to think a bit
harder about how you construct your analysis. Now, at the same time, this you
know, there's some field, you know, the ICT field that has developed these
diagnostic methods that have also developed methods for robust, making
networks more robust. And so I don't have any plots here. But basically, you can
combine those methods and show that you can build, you know, robustness sense to
insensitivity to some of these attack attacks. So you can you can imagine, at
least that will help you bring yourself closer to the regime where you're less
sensitive to by dimensional perturbations. But no, this is a very, it's a very
interesting question about how you quantify this uncertainty and we don't
have a good solution at the moment.
That's very interesting. Thank you.
Okay, thanks a lot. Any other questions for 10? If not, we can thank Ben again and
then we move to the next speaker. That