Can you see my slides? Yes. Okay.
Great. So thank you very much for the opportunity to be here and to present,
let's say our recent results. And so we'll talk about part of the cities with deep
learning models. This is a paper we have published last year. And these it is in
the context of n three PDF. So for those that are not aware of the NPP f is a
research project that we have in Milan, led by Stefan authority, and it tries
somehow to construct the next generation of nn PDF feeds. So let's start this talk.
I wanted to start by saying a couple of words concerning the PDF challenges. So in
the last 20 years, the PDF community we have a lot of priority to treat asks the
data implementation. So try to collect as much as possible data and and from
different processes and experiments. So, here on the right side you see these
kinematic plot where we collect all the GIS fix a target Dragon hydronic data and
you know Elysee data that has for example, included in the NP f 2.1 data set. We had
also lots of developments in terms of steering wheel trying to have a fast
mechanism to compute the theoretical predictions for multiple processes in
order to allow the possibility to have a PDF regression to get a PDF feed from
them. And the third point, which is the most important one is the one that I
wanted to focus today is the methodology. So, we have to define a proper
methodological framework in order to obtain reliable PDS with reliable
uncertainty And somehow have full control of what we are doing. And this is
something very complicated that we spend a lot of time trying to, to tweak and to
understand. In terms of nn, PDF, no, we the latest release we had so far is then
PDF 3.1 was released a couple of years ago. And it implements neural network
architectures optimize it through genetic algorithm. So, that was the original idea.
And we have been using that idea for a long time. The implementation we had was
in c++, because was easier to implement our convolution kernels between the
theoretical predictions and VMs. And the third point, which characterizes these
release is the possibility to mean to the model by doing by applying a manual
approach so we try to fit We tried to understand the feet. We use it closure
tests, but the fine tuning of the neural network architecture and the optimizer
setup, and the other parameters of our system was performance manually, that we
had to retry several times times and to getting the perfect or miss a close a
perfect solution we could. So when you see these three points, I think there are at
least two natural questions that we may ask in terms of challenges. Now, the first
question is, can we have feats, which are very, very fast, and that can provide us a
better understanding of what's going on in this release, because so far in the nn PDF
framework, feats will usually take around 30 to 35 hours per applicant. And if you
consider 35 hours, this is a very long time. And you could have I mean, you could
try many other things during this time. So this is the first question What can we
improve before Can we maybe modify the idea of using a genetical optimizer and
changing the parameterization? And the second question is, can we generalize the
methodology and implement some sort of learning automatic learning of the
methodology to perform the selection criteria automatically, instead of doing
it manually? And then the question is, probably yes, we can, we can try to follow
the approaches which are adopted in the common by the deep learning community by
many, many projects that we are seeing here in physics too. So we need some sort
of general upgrade of our software technology, but also in terms of
methodology. Now, that was the missing step. And that's why we have the entity
PDF. So it's some sort of research and development team that will try to
implement new features and see if it is new features that are acceptable or better
And the things we had so far. So, we did that to the implementation from the
methodology code and software technology.
So, going towards a deep learning approach, eds are a supervised learning
problem. So, we have our data, we have a model, a cost function and optimizer and
all these four items enters the enter the training, and then we have somehow cross
validation methods that will provide us the the best possible model. So, if you
want to improve on that, we have to focus on at least three points, the model
definition to how we build the neural networks, how we decided the best
architecture, then we move to the optimizer. So shall we continue using
genetic optimizers? Or shall we change it and use it for example, gradient descent,
which is the most popular technique, use it so far? And then the third question is
how To deal with cross validation, what is the best approach for PDFs? How we do
that? Now, in order to fix the first two items, the modern optimizer, we have
designed a new fitting algorithm that we call industry fit model. And here you have
a schematic picture of this model how this model is built. And let's say the most
important features of that, of that particular model are the implementation.
Now, the implementation uses speidel, instead of pure c++, and relies on
TensorFlow, we decided to move to an external library, because this library can
provide us many features like moving and running the code on different hardware,
like CPUs, GPUs, and provide us the flexibility to change parameterization
without having to implement everything ourselves. So if you wanted to take some
new architecture, we can just import from an external code and try it The second
point is the modular approach. So each single block that you see in this picture
from the values in x that you feeding in a PDF, to the pre processing the neural
network architecture, defeating basis, the normalization, and then the next step, so
the rotation and the convolution with the radical predictions. So all these steps
are possible to be tuned automatically. And it's part now of our metadata. So, you
can really change it very, very easily without spending much time in coding. So,
we can vary all aspects of the methodology by modifying few coordinates of the code.
So this is then three feet by doing that, in terms of performance, so switching from
c++, pure implementation based on graph on stone, genetic optimization, to tensor
flow, using gradient descent, we managed to to observe incredible performance
improvement. So Global PDF fees usually visit the histogram of computation time in
hours. And based on the number of replicas, so you see that in orange, we
usually spend something like 35 to 40 hours per replica using the genetic
optimizer after switching to a gradient descent and using the Python framework.
This time went down to few few minutes, let's say one hour, you know in an overall
so 60 minutes. The same for the is feeds before it was taking something around 15
hours. Now it takes just a few minutes, we're talking about 10 to 15 minutes. So
here we see the difference between entry feet average or for a global data set the
same user Dean and MPs 3.1. So now we are seated around 70 minutes, and the D is
only which it takes just a few few minutes. We're talking about between five
to 15 in maximum So we gain a lot of efficiency by doing that. And that's very
nice, because opens the possibility to find you another model. Now, instead of
waiting 40 hours, we can really run 40 feet or close to that, right. And that's
exactly what we try to do. So, we try to move on. And we have implemented a hyper
optimization algorithm that automatically checks different configurations of our 15
fitting procedure. For example, here you see on the plot the loss function, so in
our case, the chi square of the fit as a function of different parameters, like the
number of layers, the optimizer in terms of recent the learning rate in the cell
initializers handbooks, so you have many, many parameters as they once used to be
stable and we can try to monitor the validation chi square as a possible metric
for our target hybrid optimization setup.
In the setup, we decided to also implement about Asian approach. So you're using
hyper opt for that. So you can use the three of Parson estimator in order to get
the next configuration for our feet. So by looking at this plot, we can really
somehow have an intuition about which is the best configuration for our specific
problem. Now, you see that for example, the number of layers is used just one
layer for the neural network, we are not doing so well we have many outliers, if
you select to have a better distribution, so lower Chi squares, so you can try to
correlate all these different parameters and extract the best configuration. Now,
when we tried for the first time, we decided just to use for the hybrid
optimization evaluation chi squared for the hyper organization, are waiting until
we delivered that number as the figure of merit for that, for that approach. And if
you compare the entry fee to so the green replicas, for example, the down PDF, to
then PDF 3.1 replicas, you see that both cases they have some wiggles in the case
of an N PDF 3.1 the wiggles are due to finite size effects. So in principle, if
we increase the number of replicas, these wiggles will disappear for a very, very
reduced wiling then three feet that was not the case. So what we are seeing here
is really overfeeding so you get extremely strong overfeeding and that's due to the
fact that the training invalidation split of the data, so if you just pick up
randomly the points, the data points in, that's not enough, the data is extremely
correlated. So you Finish by getting overfeeding just because you need a
clever, very clever way to separate the training and validation. And that's what
we did, we'd had to implement a new quality control detail. And what we had in
place is designed by these men shown by these by this plot. So, we had to define a
complete uncorrelated the test set, which is used as a quality control for the hyper
optimization algorithm. To insert inside of each fit, we have the PDF optimization,
it will look at the chi squared off the validation in order to be implemented the
early stopping or to monitor the quality or the generalization quality of your
model. But that was not enough to avoid overfitting. So we had to modify this
approach and include that as it and at the end of the day, we decided to use final
figure of merit our weighted average between Between the validation and the
test set. So the test set that I'm talking now is a subset of data sets, which have
already been included in the feet. So, we had for example, some process like jets,
we include a Jetta data set for training with a very large domain x for the for
tonic momentum, but then we take another jet data set and we select one which has a
smaller range. So we know that at least we are putting in the feet the required
information. So by doing that, this is what we get. So for the ubar PDF, we move
on from the oscillating and over learning green distribution to the nice and smooth
orange distribution. So, you see that there is no overfitting by say by looking
at the plot, you see that the complexity of the architecture is not so hard to you
have a very nice and smooth distributions. You can also compute that they are clamps
and you see They are clamps are pretty small. And you have great stability and
reduces uncertainties, including if you compare the overall square of them
received versus the previous technology, so they want in nn, PDF 3.1. Again, okay
squares are very close to each other, but to get much less complexity inside of the
architecture.
Now you may ask, well, is that enough? What can we do next? And the answer is
well, we can implement further tests, so have further quality control. And the
first thing we tried is to see if it's possible to get stable chronological
feeds. So the idea is to go back to the past in the bracket here, time when we
have a debris here a dataset, perform our hyper optimization using the procedure I
just presented, and then compare our predictions to the future data. So we did
that. And at it for example, we tried to remove all data after here. And here you
have the distributions for post header distributions like they have to, and SR
stt bar repetitively simulation. You can repeat this for different experiments and
also compare PDFs. And you will always see that we get uncertainties, and we get the
result, which are within the PDF uncertainty. So the results are always
competitive. So that's less a reassuring approach that makes we more confident
about the approach the limitation we have. The second question is how to define a
proper data set. So here we have in the case of PDF, it's a very limited data set
with 5000 data points. And if you decided to say the wrong data set, we may get
completely different answers. So these are the hypotheses. So what can we do? Well,
we can try to implement the K folding process validation, we tried and indicate
folding, we perform a rotation estimate of the test set, and then we send the average
value of the test set to the hyper optimization algorithm. And we did it and
here you have the comparison between doing the K folding versus not doing the K fold
just using the previews, manually selection of test set and also here you
see that the results are very, very compatible with being in both cases. So,
our level of stability based on the test set definition is quite strong. And now,
to conclude, I want to just to comment about the two points we are studying. The
first concern is extrapolation. So, today pdbs are the computed by using a
polynomial pre processing term which multiplies the neural networks and what we
observe is that if we remove pre processing for example, at small x, the
PDS saturates So we have a situation, we cannot deal with that particular behavior.
So then these options are two possible challenges. The first challenge is to
avoid situation and these we know how to do, we have to create custom input layers
that will modify the behavior of and avoid the situation in our network. So we can do
this pre processing the data level. But then you may ask to Well, now that we
don't have separation, what is the best approach to have a model in that region?
And then the answer to that is that we could create pseudo data based on some
process specific process. And for example, when approaches we can take the D is
observables. Like they have to have to charm whatever distribution we have at low
Q and low x, build a Gaussian process model for that data, and then include
pseudo data from that Gaussian prior into our fees. And here we have an example.
This is The left the adoption process for telecom distribution at very low x values.
If you include the yellow points into feet, then we are able to modify the PDFs
and the PDFs will include that information in extrapolation. So that's for example is
a possible solution for the extrapolation business problem. So, going to my final
comments, we are very close to achieve nn PDF 4.0 release and this release will
contain many methodological improvements like speed, the code will also be open
source, and we have the possibility to learn the methodology with hyper
optimization and better quality controls for the different aspects of the
methodology. So, thank you very much.
If you have questions, please let me know.
Okay, thanks a lot.
So do we have any questions from the room
Do actually this is a rock a penetration by hand. But so on slide 14, I think or
yes no this this one is
the the extrapolation of shooter data there is really
strongly
guided by the Gaussian processes, modeling of these no data. Yes, for example, yes.
So, which could be if, if nature is pretty smooth, then the Gaussian process will get
it but if it's not extrapolating properly, then then it's going to go bad, right. So,
indeed, how do you indeed Yeah, decision for example, if using a Gaussian a
Gaussian,
Gaussian process is because well, we start from there. conceptual idea that the data
derives from a Gaussian probability a normal distribution. But in this
particular case, we may have a couple of things to consider like the kernel of the
Gaussian process, the correlation length, the autocorrelation, and different
features, technical features inside of them on, we are planning to test all these
different approaches with different kernels, and then probably, at the end,
combine all the results together. So have a microscopic representation of the
uncertainty for that data and see what are these translating terms of PDF is
obviously is not the perfect solution because, unfortunately is extrapolation.
So we cannot do much, but we could try to combine these Gaussians processes with
different configurations for different process types at small x and see if the
final answer improves or not the description industry In this model x we,
but that's still a very, that's still very, very complicated problem to solve,
because unfortunately, we cannot rely on simple neural networks for that.
Okay, thanks. Are there any more questions because this is the last talk before we go
to the, to the coffee break?
I guess because we don't really have the ability to applaud people, but I can
unmute all of the panelists. So then maybe we can clap for them.