Thanks very much everyone for attending and my pleasure to give this talk
about triggering and online calibration with machine learning techniques
covering material from the four collaborations and also from some
community initiatives on these topics. So, I wanted to briefly start by, because you
know that whenever we discuss sort of machine learning, including in these
contexts, it's important to sort of remember that
it's been with us somehow since, since the beginning, but it's just taken different,
different forms. And so I stole this, this picture to illustrate it.
And the point of the picture is that you're doing the same thing fundamentally
(which is banging your computer) but you're just banging in different ways.
So before run one,
it's not that the community didn't know about machine learning. But
we were preoccupied by saying, you know, can we trust our detector in the first
place?
And then, as run one happened and things worked, and I think it's fair to say that
the detectors worked better than we could have hoped for.
We started
increasingly adopting sort of at least BDTs as some kind of minimal machine
learning approach and increasingly fancier things. First in offline analyses but
also in real time classification.
And then in run two, as we got more and more confident, there was an increasing use of
kind of aligning and calibrating our detectors in real time and real time
analysis, whether it's called that in LHCb or scouting in CMS or trigger level
analysis in ATLAS and that was increasingly becoming a thing.
And so, in addition to classification, we started really
deploying ML to assist, kind of also feature building in generators and for
calibration.
And now going towards run three,
we are really, I would say, kind of starting to
consider ML methods and how they interleave with classical methods across,
you know, the full list of our use cases, whether real time or offline.
And so that's somehow a little bit the context. And now I'd like to go through
and start with classification.
Because that's really the one where we started earliest in the trigger or real
time context to deploy it. So this is a slide from from LHCb
On the right hand side, you can see a plot of essentially our when we optimized our
main BDT based trigger for for run two and you see the different signal modes that you tested
with and the efficiency of these different types of
BDTs and classifiers that we that we tried out.
And on on the on the left, this is a kind of nine year old plot now but this is from
the very first time we deployed the same BDT that that was reoptimized in a fairly
sophisticated way for run two, was first deployed in a much cruder way for already
2011.
And you see the
in gray the response of the minimum bias data and in red the response of
bbar MonteCarlo
2010 MonteCarlo and this very nice modeling actually of the higher
response area where you're having
because this classifier was classifying
proton-proton collisions which had bbar pairs in them. And you see how nicely the
simulation is following data. So it's just to say that within the LHCb trigger
you know, the main, the main, inclusive trigger was based on BDTs since almost the
beginning. And then different classifiers have been used widely throughout, there's
a lot of documentation on that.
One thing we put quite some work into,
over the years is finding ways to
flatten the efficiency curve. So here you see
the efficiency of the BDT. On the left and on the right here it's for a three body
meson decay as a function of position in the Dalitz plane. On the left is if you
don't explicitly train your classifier to be flat. Now on the right is if you
explicitly do train it to keep the efficiency flat, and this is
to minimize systematic. So that just to give you an idea of the kind of things we worry
about when we're doing this training.
And actually this connects to one lesson for me over the last decade when it
comes to classifiers is that unlike when we were saying, like, before we trust our
detector, so you know, ML black boxes, do we know what they're doing? Will they bias
us, but actually, well designed ML classifiers turned out to be less
biasing than, quote unquote simple cuts, in many cases
and similarly, it's just too...
This has been used in LHCb for fake or ghost track classification. Also since
since quite early on. This is a plot
you see the ROC curve on the left and the
plot on the right with real data where you see in blue, what is
selected as
as being real tracks and in red was rejected as being fakes.
And we actually gained quite a lot over a simple kind of track chi squared by
training a neural network to really use the information from all the different
tracking sub detectors. And this was then tuned, as all of these things have been, by
hand, the implementation was tuned for execution speed.
And there are many other classifiers used for particle identification, things like that,
that LHCb included in the trigger.
Let me move on from LHCb,
not to be too
too parochial.
This is the true classification at Atlas.
So there's three examples here on the left, you see the algorithms used in the
HLT to to like, identify taus for
tau triggers.
And neural network based election ID in the middle, and the online b-jet
tagging on the right. So you can see on the on the right the performance of
different online operating points with respect to the offline, b-tagging.
And, indeed, Atlas uses
different kinds of classifiers in parts of its online processing chain with I would
say it's fair to say with a gradual evolution from BDTs to kind of neural
networks and more complex things over time and
with large rate reductions at the same efficiency working point. And also and
this is also what we see in LHCb, I think it's generally seen that you can
reduce your trigger latency by reducing backgrounds earlier in the processing
chain.
Now for CMS, I picked out two examples.
One is the L1 muon endcap trigger on the left of this plot, where you see the
the physics performance of this what is called the upgraded trigger on the right.
And you see it's better a bit better in this turn on region than the previous
incarnation. But on the far left, you see that the trigger rate is really
substantially lower again for broadly the same physics performance with
with this BDT based endcap trigger. On the right you can see
the performance of this
neural network deep CSV
b-jet tagging algorithm.
And you see again, the reduction in the light flavor efficiency. If you look
between the
solid and the dashed lines of each color, the reduction
in light flavor jet efficiency with respect for given b-jet efficiency working point.
So,
I find it by the way personally interesting that all of us seem to
have converged on using ML of one kind or another to tag beauty hadrons or beauty
jets. So that's quite nice.
But then the other thing I wanted a little bit to, to highlight here is somehow some
of the approaches towards the future for classification. So one thing
very exciting at the moment is deploying classifiers on FPGAs. I tried to say that
since the beginning we've worked on optimizing these classifiers for
speed and their implementations and kind of a logical end point of that is to see
if you can compress your models to the extent that they can fit into FPGAs and
then this would allow you to use them in first level triggers or in coprocessor
based architectures.
And this is, I think, extremely interesting also for the future. And
there's a lot of work going on in this and you see
a plot of the
performance of these,
these implementations in this HLS4ml framework.
And then the other thing that I find interesting is this unsupervised trigger
classification. So this is the question we often get right that, Oh, well, what if
your trigger misses something you didn't predict and so now there's approaches on going to
have a kind of generalized anomaly detection based triggers. And you can see
from these
from these plots, that
they're actually robust against having SM backgrounds in your training sample. So
you can you can train on data, but
of course you don't get the same performance as fully supervised training. But still, I
think it's interesting area being pursued. Let me move on to two other things as
I am running a bit behind time.
As far as reconstruction goes, I think the main message here is that for the moment,
reconstructions are still primarily sort of classical based with machine learning
assistance. And then the community is trying to see how far the assistance part
could be pushed. So again, at LHCb the first steps were to add kind of
classifiers inside the classic kind of Hough transform based pattern recognition
for an earlier rejections of bad hit combinations.
But you're not making trajectories from hits in the neural network itself, right,
you just classify.
And then in the community more broadly, there are ongoing efforts. So there
was the TrackML challenge,
where many participants sort of competed on kaggle to
to come up with sort of ML based tracking algorithms and actually,
interestingly, the best approaches kind of mixed classical tracking, and some
physical models of the track path with machine learning aspects again, to reject fake
combinatorics as early as possible and also approaches to training to find the optimal
search parameters. And then there's more kind of even more ambitious things, let's
say like this Exa.TrkX project, which is trying to use neural networks to really
have a completely ML based tracking.
And that I think, is very, very interesting.
And in a similar vein,
just want to show some results for
a neural network based primary vertex reconstruction, where you see here, this
kind of picture of primary vertex distribution in Z in LHCb and
sort of the tracks obviously cluster at various Z values and
you can quite intuitively use a neural network to try and find these peaks. But
again, you have a classical algorithm to find the tracks first. So, still a hybrid
approach, but
that's the direction of travel.
We go towards calibration.
A couple of examples here.
In ALICE
here you have
a calibration of the TPC, which you wish to achieve, and you want to correct for
distortions in the position of TPC clusters
already online, but your analytical correction procedure is far too
slow for this.
And so what's being explored is to use convolution neural networks to correct for
this
using space charge
or currents as the input, and
you
you see here on the on the plot that the resolution of the predicted distortion is
about where you would expect from the intrinsic resolution, but is much faster
than analytical methods. So,
you have a lot of details in the backup slides about this.
And I also wanted to mention in CMS, there was some very interesting work to
calibrate jet energy with with DNNs and just very quickly, because I am runnig out of
time on the right you can see the kind of visually the clear improvement in the
resolution between the DNN and the baseline approach.
So, let me just conclude with a few kind of personal remarks
which is
Just to say that, I mean broadly speaking, over the last 10 years, the direction of
travel has been increasing deployment of ML based methods throughout online and
calibration. And we know by now that this can outperform classical methods in
physics terms certainly in classification, increasing hints of that in reconstruction
too,
but what I think is going to be crucial is to continue to accumulate our experience
of deplying ML across different architectures, especially if computing
keeps becoming more heterogeneous, kind of in the real world outside of HEP, and to
understand where you know the computational efficiency and where we should mix,
the classical and ml approaches to make things faster. And both the
algorithms and also the structures used to pass data around are critical to this.
Thank you Vava
for the talk, and for sticking to time and opening the floor for questions.
People if you want to raise your hands in the zoom interface
So far I don't see any raised hands
yes, Igor,
you should be able to unmute yourself. Can you hear me? Yes.
Okay, this is Igor Volobuev with Texas Tech University. There appears to be a
major effort in at least statistical community on interpretability of various
machine learning techniques. I wonder if this kind of effort could be translated to
high energy physics.
So,
when you say interpretability what I mean just to understand the question, well,
what specifically are you are you thinking of? Well, interpretability
means that
humans should not be viewing
the machine learning techniques as a black box, right? So some
understanding of what the techniques is doing should be transferable and
quantifiable.
Right, no, but right. So I think
if we if we talk about concrete, in terms of quantifying it, certainly in LHCb
but I think this is true across the experiments.
There's a lot of work that that's already gone into, and I think goes into
especially when we deploy these things in real time, because if you make a mistake,
you can't recover very easily,
to quantify its performance
as a function of all the different potential variables of interest. So if you
deploy a classifier in LHCb you would check typically the efficiency
turn on curve not only in PT, but in the lifetime of the particle. I mean, I showed
this example of flattening the efficiency across the Dalitz plane which is
A good example of something where we, we care about controlling the performance of
the classifier, not only in some absolute efficiency, but say the shape of that
efficiency explicitly because it helps you minimize systematics associated with the
simulation of this efficiency later on.
And certainly, as part of that, I mean, again, I, you know, in LHCb I can
speak directly, but I'm sure it's true for others as well, that there's been since
the beginning, we've always pruned our classifier. So you always as part of it,
basically, start with some set of features. And then we fairly aggressively
reduced that to some minimum set that are actually relevant. And that's also part of
understanding what it what it does, right? You
use or figure out which features it's using more and so on and so forth. So
but I don't think that this work, this is sort of being done in if you want in
a physicist way. I don't think this is formalized, in
some language, that the statistical community wouldn't necessarily
recognize generally speaking. So that may be something where
which, which could be interesting to, to understand actually, how much of this are
we doing and just talking about it in a different way.
I see Steve, Steven, you're supposed to be able to unmute yourself?
Yeah. Hi, this is Steven Schramm. So very nice talk Vava. Thank you very much. I was just
wondering actually about combining the two aspects in the triggering and the
calibration.
You spoke a lot about how we are using classification with machine learning in
the trigger. I'm wondering, because I haven't seen too much of it yet, have you
seen much in the form of regression or other forms of calibration being applied
in the trigger as well?
So, no, I mean, no I mean certainly not, not in
LHCb I mean, our
our focus has been on
automating the calibration workflows so that
so they can be performed in real time but but really the workflows themselves by and
large are still very much classical. Now this may be
you know, this, this may be something where in the future, particularly
for
calorimeter related things, there are some, you know, obvious points to tackle
there. For tracking or for something like alignment, it's, it's less obvious to
me how you would
Yeah, I guess I was thinking of, for example, you know, some object PT
cut or something like that, and you lose a lot of rate from the difference between
the trigger and offline selection. And I was wondering if you know of any studies
where you're trying
to basically turn that into a step function by having something which really
converges to the offline. I don't know, I haven't seen any studies. But I was just
curious if you had... No I hadn't I mean, I mean, LHCb focuses on on making sure that
if you want, the features that go in are the same. So we focused on getting the
reconstruction to the underlying inputs being identical between online and
offline.
But it's true that what we have have seen over time is no matter how much
you make the underlying inputs, the same, actually, even even much more primitive
things in machine learning, like packing and unpacking the data, when you ship it
around, actually ends up if you end up with a classifier later, because it's so
multi dimensional, right? If you have tiny changes due to even trivial things like
that in your input features, you can end up with non negligible tails coming out
for no good reason, particularly. So when we've tried our best to
suppress stuff like that. But it's an interesting point something we should look
at more systematically. Okay, thanks a lot.
Maybe you have related questions
is, would online training be able to help with this type of
variational, slow variational classification of
classification performance or whatever performance of the models
so, I think I think where online training could be could be interesting is if you
is actually in helping you
if you're if you expect your detector performance to change significantly over
time. So if you, for example, if part of your detector ages and then again until
now, you know, we've often taken crude approaches to this. So the calorimeter
does age over time. So we actually
had an automated procedure in run two which follows the occupancy of the
calorimeter and then directly adjusts the voltages to
keep the relationship between the ADC counts and the energy as
similar as far as possible. And you can for sure, think of doing more
sophisticated things with stuff like this. Yes. Okay. All right. Thank you Vava for the
talk and for the answers
Thank you for listening.
Next is Lucas on machine learning techniques for jet classifications.