>> Hello. Welcome back to Curlyboi Theatre, everyone. Our next speaker, Joachim Jablon,
is joining us all the way from France, so almost the other side of the world from where I am.
Where it is very early in the morning for him. So, thank very much for joining us.
Joachim is a French Pythonista soft engineer and started engaging with our dear Curlyboi.
He's contributed to bits of the Python Index. He has four cats he was telling us before.
What's your cat's name? >> [Indiscernible].
[Laughter]. >>
He has also spoken at and/or organized multiple Python and Django events.
They live in Poitiers, with the top four cutest cats in the world according to us,
and yes, we do want to see pictures. Okay. So, Joachim's talk is prerecorded, everyone,
so he will be in the chat, answering questions live in Venueless. So, don't worry about the
questions tab this time. Just talk away and Joachim will be there with you. Is there
anything you'd like to say before we start? JOACHIM JABLON: I'm good. I'm good. We can go.
From Hat Import Rabbit: Hacking Your Way Through Python's Import Machinery for
Fun and Profit. Hacking your way through import machinery for fun and for profit.
Hi, I'm Joachim and I'd like to perform a magic trick. I have to admit that I'm far too clumsy
to be a real illusionist and my favorite is to explain how it works so I'm not a great magician.
I've been a member of the Python Packaging Authority and the Python Software Foundation
and Django Software Foundation since 2021. Let's dive into it. Let me take you on a
short hike, where I tell you a trick that I had to perform last year. By the way, we'll be taking
the scenic route, so kick back, relax and enjoy the ride. This magic trick begins with a dove
and as doves migrate to a warmer climate twice a year, so does SQL, which migrates to a schema,
release, after release. In case you're not familiar with what a SQL migration is,
let's make a quick reminder. Let's imagine that we're responsible for an application that has user
accounts so it stores usernames and passwords. Now we want to attach some new information
to profiles, such as whether the user gave their consent to store cookies for them or not.
This means we'll have to change the database to add a colon in a table,
which in turn means we're going to write some SQL code and run this code on the database.
If you're developing a website using Python, chances are, you've heard of the Django framework.
Because Django helps you with your database, it has a whole subsystem for writing SQL for you
and especially writing SQL migrations for you and the subsystem is based on Python file that look
quite like this. So, this is migration module, so it's a module that contains a single class named
"migration." And inherits Django's own inherent class. There is dependencies, with Django
lets it determine in what order to run the
migrations and the operations lists, which contains the AddField or RemoveModel or
many different object that Django knows how to turn into migration as SQL code.
There's still a way we can use the Django migration system if we already have the SQL code,
by using run SQL object. We can use it directly and wrap it inside the Django migration system.
Migrations will be stored in a folder that is named "migrations." Inside
each and every Django application, these migration will live in their own module.
And now, for something completely different. In the last years, I've been working on a project
named procrastinate. Procrastinate is a Task Queue. Maybe you know celery.
You want something to do done later, for example, a long task that needs computation time and you
need to respond to the customer quickly. You take notice of it and you respond, yep, I'll do that
later. That's the name, "procrastinate." I'm sure that you will be doing it later.
What makes procrastinate unique is that many Task Queue, such as Celery, use it as a broker and it
is backed by a PostgreSQL database. The mascot is an elephant so the magic of procrastinate
is turning a rabbit into an elephant. As you imagine, in order to procrastinate to work,
it needs specific SQL schema and this schema can sometimes evolve. They can become complex
so I'm showing part of the file, I'm not going to explain it. It's just to give you an idea.
The procrastinate SQL migration files live in a migration folder, which is located within
the procrastinate source code. It looks like the Django migration folder, except
those files are SQL files and not Python modules.
So, on the one hand, procrastinate helps you when you have long tasks in a website
and on the other hand, Django helps you when you're building your website so quite logically,
people may want to use procrastinate with Django and for this reason, we've built a
Django compatibility app within procrastinate. But you might wonder, what about migrations?
The procrastinate Django application doesn't have a migration application model. If it did, it would
mean that we would have to manually duplicate the code from the SQL migrations into the Django
migration and it's something we didn't want to do. What we wanted to do is to make Django believe
that for each SQL migration file, as we can see on the left side,
there was a corresponding Django module on the right, in red. But this file wouldn't actually
exist under this so we would want to make Django believe that there are modules here,
although the migration modules don't exist and the migration folder, itself, doesn't even exist.
So, what we wanted were a virtual migration files. At first, I tried, for quite a while,
to see if there was such an existing hook for doing this within Django and it turns out,
there just isn't. By Python is a dynamic language so I thought maybe I could take an unofficial way,
monkey patch to get it. All I found was this was tricky and error prone
so I didn't like the approach. Markus Holtermann, who was, by the way,
on stage yesterday, for a talk on authentication, he gave me, from the top of his head, the idea
that I was missing, which was to use the Python import machinery to trick Django. I only know the
Python import machinery existed but I absolutely did not know what was inside it. So, I had to
take a route and explore the Python importer secret mechanism and I'm going to give you
some insight from what I learned at the time. So, what's the structure of the import machinery?
Let me start by a diagram showing you the interactions between the different parts.
Whoops. Wrong slide. Oh, yeah, that's the one. The import mechanism contains many moving parts
and we're going to introduce just a few elements today. If you find this topic interesting
and you want to dive deeper, I will give some pointers to some resources at the end. So,
let's dive in with the part that you probably already know, which is the import statement.
Actually, chances are that you even know multiple forms of the import statement, such as
import A to B, from A Import B, from Import A, Import B. This import statement can be
translated to assignments and function calls. An import statement always does two things.
First, it will import one module using the built in function and assign this module to a variable.
By the way, when I say "dunder," it's short for "double underscore." You'll hear "dunder."
The dunder import function does a few important things. You may know of syst.modules.
It will start any module that's been imported. Dunder import will help the cache for modules
that have been imported and add modules to this cache and they have been successfully imported.
Second thing, when you import A.B.C, it will import A first. The parent modules are always
imported before the children. When the import of Module C is successful, it will assign it. Thanks
to that, if you have a Module A.B in your code and import A.B C, you'll be able to access and
see that you have imported. And then the important part is the importing the module, itself.
So, this is a three steps operation. First will be looking for a MetaPathFinders. It is an object
which, given the import pass, says whether or not it's able to find a corresponding module
in order to import it. Python keeps a list of MetaPathFinders objects in sys.meta_path.
The idea is that Python will ask ever finder, from sys.meta_path,
whether they think they can import a path until one says "yes."
Asking MetaPathFinders object is done by calling it find_spec method, passing the
spec as parameter. It returns none if it cannot load the path or it returns a ModuleSpec object.
A ModuleSpec object has a few properties explaining how to load the module.
One is ModuleSpec.loader. It has an exec module method and it will
create using the info it gets from the ModuleSpec and call the exec_module and
it is expected to attach attribute to the module resulting in the final module imported.
A normal Python modules that are loaded from a source file on the disk,
this would be the part where Python will open the file and then run eval on all the contents,
store everything that was created and attach everything on the module object.
So, one of the places we'll want to explore, in order to add our customization,
will be the sys.meta_path list. Let's first have a look at the MetaPathFinders that we could find in
classical sys.meta_path. If you open sys.meta_path in Python, you'll find the built in importer and
the FrozenImporter, those are what Python needs in order to work. By the way, when a class is both
implementing the finder interface and the loader interface, we call it an importer.
Before going deeper in the next element, let me add a quick note on sys.meta_path. So, it's just
a simple list and you can actually add your own finders in there. The only requirement is that
anything you add must have find Spec Method. And then there's domain finder, that Python
implements, which is the one that's used most of the time. You need a PathFinder. The PathFinder
will look in multiple directories to see if we can find a module that we're trying to load. For
[Indiscernible] module, the list of directories to search is given by sys.path, which is a list of
strings. This is the in famous sys.path that you may or may not have had to tinker within the past.
Sys.path contains a list of folders on the disk to explore in order to find a module. When the module
is not a top level package, then, logically, it has a parent and the parent module has a dunder
path attribute, which is also going to be a list of strings and similarly, this list of string
is the list of places on the disk where one would expect to find submodules of the parent's package.
So, whether you're importing from whether you're importing a top level package or a submodule,
you'll end up with a list of folders to explore on the disk
and this folders are called path entries. We're going to iterate on each path entries we have.
The PathFinder will use another attribute which is sys.path_hooks. Sys.path_hooks is a list of
callables. We will iterate each callable from sys.path_hooks so we're doing double iteration,
both on the path entries and on the path hooks so it's just kind of a [Indiscernible] loop.
Each path hook receives the current one and iterative one. It receives a
importer or it returns a Path Entry Finder. It may return a module spec. At this point,
you may experience a very strange d j vu. It is close from the meta PathFinder we talked about.
I'll do my best to make it explicit, what the difference is between a path entry finder
and a meta PathFinder. Meta PathFinder tells you whether or not it can find a module A.B.
path entry finder tells you whether or not you can find a module A.B Inside one very specific folder,
which is the path entry that the path hook was called. Path entry finders are very specialized
meta PathFinders that restrict themselves to searching on the disk in one directory.
In order to determine if the path entry finder, we'll have the find_spec method. So,
kind of same as for the meta PathFinder. Same as for the PathFinder, it returns none, we could go
on and try another path hook or another path entry or return a spec and, hurray, we found our module.
So, what would be the default path hooks provided by Python? One, the first one is zipimporter path
hook. So, it's the one that lets you run Python modules that are inside zip files. By the way,
yes, you don't have to unzip archives to run Python files that are inside zip archives.
One of the hooks the single one is FileFinder. So, it will look for classical Python
files and folders on the disk, such as .py files and as well as folders containing dunder in it,
.py. You can add your own callables to the path hook. So one way to do that would be to implement
your path entry finder class and you would give it a constrictor dunder in it that receives
the path entry. So, you'd register the class, itself, as the path hook. Python will call the
restricter and pass the path entry as parameter so we may store this as an instance attribute
and Python, with path entry finder instance, it will call the find_spec method on it so this way,
you'll be able to access both the path entry and the import path that's being searched.
So, we're approaching the end of the wonderful land of Python imports. If we remind ourselves
of the initial goal, we were looking for ways to trick Django into believing that there is
a migration folder and that this folder contains plenty of migration files. So, Django is going to
dynamically import our applications migrations module but because Django
doesn't know the name of our application, it would be complicated to use an import statement directly
because the name will be inside a variable. Django could use the dunder import function
directly and it would work, but this function is made for Python to call
and there's actually a better interface for programmers to use, which is the Importly module.
This function's goal is to import modules when their path is dynamic. So, Import Module calls. So
Django will call Import Module and we'll expect to receive the Migrations module.
Now that Django has the migration, how does it discover all the ones inside the module? For now,
we've talked about how to import a module given its name, but not how to browse existing modules.
This is something we can achieve with pkgutil.iter_modules() which receives
the module path and returns a list of objects that reference the submodules below that path
so it taps directly into the PathFinder and the path entry finder
and it will call a method that's named modules on all the path entry finders that we found.
Then, Django will know the existing name of the migration submodules and it will call import
module again on every submodule that was found. So, it will read and hopefully use the migrations.
So, if you want to take a screenshot of the graph, this might be the right time.
So, what does it look like in procrastinate?
Let's have a look at the code and see the magic trick for real. As a reminder, we're trying to
generate both the migration module, the dunder.py and the individual submodules for each migration.
Also, here's what a migration submodule look like. It contains a single class name migration
that has a few attributes. So, creating all the migration classes dynamically by inspecting
existing parts of SQL is not the hardest part of the problem. It's a function that takes
a migration written in SQL as a string and generates a Django migration class. So, this
will be the class we want to attach inside the virtual dedicated migration that we're creating.
In order to tap into the import machinery, we'll need to create an importer, which, as you
remember, is both a MetaPathFinders and a Loader. In the constrictor, we'll generate and then store
all the migration class that we're going to want to dispatch into our virtual modules.
Our importer is a finder, so it needs find_spec method that will only work
for the migration module and the migration submodules. If we receive the right pass as
parameter, then we will return a spec instance. On our module spec, the loader is set to "self"
because it is the finder and the loader. The loader part will be on the same object.
Then, let's look at the loading part. We receive an module as argument and our
job is to attach things to this module. The loader must be able to load both the
migrations module and the individual submodules. If we are loading the migrations module, itself,
then the only important thing is the dunder path attributes, because the string we add in there is
the one that will later be checked by path hook. We want to make sure to put something that will
not be recognized as a valid path by other hooks, thus using angle brackets, as you can see, here,
which is unlikely to be a path on the disk. If we're using a submodule, that's the moment
where we finally get to do the magic. We figure out the name of the module we're building,
using module and the domain and we attach the class, the proper class inside the module.
Now, our importer will also have metas added to sys.path_hooks. So let's see what it looks like.
This is the meta you can see above. It only reacts to paths that match the attribute
we've set on the migration module. And it returns "self" as the loader the
finder for any path that matches our virtual path and it raises an importer for anything else.
There is the items module method. In here, you can see that we're leveraging that
we've already pre computed all the migrations and stored them in the finder the importer class.
And then, the last touch, we register importer in sys.meta_path and the path hook in sys.path hook.
We are being good citizens here. We're adding our stuff at the end of both iterables,
which means that our [Indiscernible] importer will be called but only if the other
conventional import mechanism didn't succeed. In other words, we are not slowing every import down,
but we're just slowing a little bit, the unsuccessful imports.
So, showtime. Now, it's time to ask Django to apply our migrations and everything kicks in from
all over the place. So, Django will try to find the migrations module and then it will export all
the submigrations. It will the submodules, it will find them and then execute them and everything
runs smoothly. Presto. Whew, what a journey. Before leaving the stage, let me share a couple
of [Indiscernible]. First, yes, this might another engineered solution. We could have a static
migration generator that would write normal Python files. In all honesty, I'm happy I get to learn
a ton of stuff doing this. To this day, I feel like hooking the import machinery is the cleanest
solution to the problem we had. It doesn't really add a lot of complicated code in the end.
If you plan to use this knowledge to do some good, you should definitely play with it
and make it into a nice library. If you do so, please feel free to send it my way. But,
remember, smart code goes into libraries and boring codes goes into projects because no
one wants to debug a very custom, meta, meta path entry PathFinder that went wild into production.
Even for procrastinate, I still think it might be a good idea, someday, to custom
the Django process into a dedicated library. And finally, the import machinery is a very
interesting part of Python and it could give birth to very innovative users.
Knowledge of this might not change your day to day practice but it can be more than one tool in
your built. As a general advice, Python is very accessible and both quite rewarding and useful
to take some time to learn how how one specific Python subsystem works and so, yeah. Be curious
with language and experiment and do some magic. Thank you. I'll be glad to take some questions,
but while this is a prerecorded video, the live question session will be featuring
the live version of me, just woken up at 6:00 a.m., so please be gentle. Thank you very much,
Goulwen, the illustration. Have a nice day and see you around.
>> All right. Thank you so much, Joachim, for that fascinating talk. Do you have any
any final words, closing remarks? JOACHIM JABLON: Uhhh. Yeah. Well,
no. I'm really happy that everyone was able to watch and I'll be available for questions in
the chat. Please feel free to reach out if you have any questions, even after the conference.
>> Thanks, again, for joining us so early in the morning and, yeah, thank you for the
talk. All right, everyone, we've got another 15 minute break and we'll see you back after that.