1 00:00:11,070 --> 00:00:19,040 MODERATOR: Hello, everyone. Welcome  back to the Platypus Hallway. Next up,   2 00:00:19,040 --> 00:00:24,320 we have Vinayak Mehta and fortunately it  is very early in the morning so he is not   3 00:00:25,280 --> 00:00:32,880 entering the video but he is in the chat. He is  giving a talk on "A tale of Python C extensions   4 00:00:32,880 --> 00:00:40,560 and cross-platform wheels." I hope you enjoy it. >> Hello, everyone. I am Vinayak. Today we   5 00:00:40,560 --> 00:00:44,480 will talk about how we can add Python C  extensions and package them into wheels   6 00:00:44,480 --> 00:00:47,440 so they are installable on  any major operator system.   7 00:00:50,880 --> 00:00:56,080 We will write a basic C extension using the  Python C-API and see how we can use Pybind 11,   8 00:01:01,360 --> 00:01:16,720 rook into libraries using, and rebuilding  using cibuildwheel. The tale begins at the   9 00:01:16,720 --> 00:01:21,200 ecosystem which is a directed educational  retreat for programmers in New York City.   10 00:01:22,080 --> 00:01:27,680 Last year in August, I started with the goal  of removing platform specific from camelot. It   11 00:01:34,560 --> 00:01:41,440 so you can do line recognition and identify  tables. The problem with the ecosystem, which   12 00:01:41,440 --> 00:02:01,440 is a large base, is it isn't available to install  from PyPI. In the past, there have been instances   13 00:02:01,440 --> 00:02:06,640 where users would install incorrect script version  for their system architecture using to seg fault   14 00:02:06,640 --> 00:02:13,440 or camelot wouldn't be able to find the executable  on the path. This led me to search for a pure   15 00:02:13,440 --> 00:02:21,280 Python PDF to PNG converter. I wasn't able to  find one in pure Python but found a C++ library.   16 00:02:26,320 --> 00:02:32,800 I was able to write a wrapper taking  a single PDF and converting to a PNG.   17 00:02:36,000 --> 00:02:43,520 I built that for every operating system  and included it so it can be installed   18 00:02:43,520 --> 00:02:52,000 from just PyPI. I will talk about the steps  I followed depending on the popper library.   19 00:02:53,600 --> 00:03:02,000 If you are interested in other things I did, you  can check out my blog where I wrote a blog post   20 00:03:02,000 --> 00:03:09,520 every day of the batch. CPython, the programming  language, as the name suggest is written in C.   21 00:03:10,080 --> 00:03:18,960 It exposes a rich API. The main use-cases are  writing extension modules so we can use C or C++   22 00:03:18,960 --> 00:03:24,880 modules in the Python interpreter and embedding  Python in an another application. We will look   23 00:03:24,880 --> 00:03:33,040 at the first use case and how we can write an  example module letting us add two numbers. Start   24 00:03:33,040 --> 00:03:40,080 with the header file which gives all functions  and macro definitions to use the Python C-API.   25 00:03:41,360 --> 00:03:50,560 Since all object types are treated the same  they are represented in CPython. Everything   26 00:03:50,560 --> 00:04:01,040 is a py object. It takes the pointer and applies  to the file object containing all the arguments   27 00:04:01,040 --> 00:04:13,200 passed from Python. Since we get all the passed  in to the function, we can use the stop function   28 00:04:13,200 --> 00:04:20,400 to parse tools, parameters into individual  readables. Here we pass and store the values in A   29 00:04:20,400 --> 00:04:26,640 and B. Here is the format unit for the parameter  we want to parse and there are two of them. LL.   30 00:04:29,120 --> 00:04:33,120 The string after the colon after LL is  used as a function name in error messages.   31 00:04:33,840 --> 00:04:39,760 If it is not able to parse that and returns false,  we need to return null to get an exception and   32 00:04:39,760 --> 00:04:48,160 pass to the call stack. Then we define two py  objects and use the pylong to convert C long   33 00:04:48,960 --> 00:04:56,960 to two long projects. We create another py  object called R which adds the function.   34 00:04:59,520 --> 00:05:04,480 In the end, we return R which is the result of  the iteration. Before we do that, we need to   35 00:05:04,480 --> 00:05:19,440 use py for this. If you are writing an extension  module you need to be aware of the counting method   36 00:05:20,960 --> 00:05:25,200 so you don't leak memory. You can check out  the resources in the last slide to learn more.   37 00:05:27,840 --> 00:05:31,920 We need to create a Pymethod structure  defining the list of functions.   38 00:05:33,520 --> 00:05:37,360 In that list, we specify the name  our Python callable should have,   39 00:05:37,360 --> 00:05:41,920 the C function, a flag that says the C  function should be called with self and org   40 00:05:41,920 --> 00:05:53,440 and a fangs -- function doc string. You can  add the name and list of functions here.   41 00:05:55,360 --> 00:06:01,680 Finally, we need to clear the module with the  function that needs to follow the convention here.   42 00:06:03,280 --> 00:06:10,160 In it, we call the Pymodule create function  and that's most of the C extension code.   43 00:06:11,440 --> 00:06:16,240 To package our extension module we will use  setup tools. Start by importing extension from   44 00:06:16,240 --> 00:06:21,200 setup tools, pass in the name of the module, the  path to the C code, and call the set of function   45 00:06:21,200 --> 00:06:31,680 with the ex t module pist. Pip picks it up  and installs it in side packages. If we put   46 00:06:31,680 --> 00:06:39,520 the module in the Python, you can see the Pip  tools use it in the side package on this bar.   47 00:06:41,920 --> 00:06:46,000 And we should be able to use the add callable now.   48 00:06:53,280 --> 00:07:03,760 Now let's look at how we can create the seamless  extension with Python 11. We can use it to easily   49 00:07:03,760 --> 00:07:11,040 create extension models on top of existing C++  code. We create an example and create the header.   50 00:07:13,760 --> 00:07:19,040 We create an add function. It returns  the sum which is a long integer.   51 00:07:20,320 --> 00:07:26,800 Finally, we use a pybind 11 macro to initialize.  Pass it in as the second parameter and the second   52 00:07:26,800 --> 00:07:34,560 parameter is the main interface to bind the  C++ functions. We design the module doc string   53 00:07:35,600 --> 00:07:42,080 and add all the C++ functions we want to use as  Python callables with m.dev. The first name is the   54 00:07:42,080 --> 00:07:47,280 Python callable and section is the reference  to the function, and third argument is the   55 00:07:47,280 --> 00:07:52,160 function doc string and the rest of the argument  is the parameters the Python extension accepts.   56 00:07:53,440 --> 00:08:03,840 That's most of the code. We get to setup.Py. After  importing this from extension tools, we pass in   57 00:08:03,840 --> 00:08:14,560 the name of the module, the c -- C++ source files,  and specify the language as C++. In the setup.Py,   58 00:08:14,560 --> 00:08:21,200 we call the function with the list of modules we  just created. We need to create a payproject.toml   59 00:08:23,200 --> 00:08:33,280 so Pip can install and use at build time. Finally,  we can install our extension module using Pip   60 00:08:33,280 --> 00:08:38,560 which then uses setup tools to build it and  install it on site packages. If we import it,   61 00:08:38,560 --> 00:08:42,640 we can see the example extension module was  combined into a shared library just like before.   62 00:08:45,360 --> 00:08:54,000 And it works just like before too. With  Pybind11 it is easy to wrap a library   63 00:08:54,000 --> 00:09:04,640 or create a new library. The API to write  an ex tension module is a bit more verbose.   64 00:09:14,880 --> 00:09:23,680 Pybind11 can take care of all this. In  my case, I could just take C++ code,   65 00:09:24,640 --> 00:09:33,920 identifying the main function, and turn it into  PDF into PNG and pass into a single page PDF and   66 00:09:35,520 --> 00:09:42,880 initialize with the Pybind11_macro. After  building it for different operating systems,   67 00:09:42,880 --> 00:09:48,880 I could access the code in the Python like this.  Let's look at how we can build extension modules   68 00:09:48,880 --> 00:09:54,560 depending on shared libraries. Let's look  into shared libraries and dynamic linking.   69 00:09:55,280 --> 00:10:01,360 To understand this, let's go to an example of  how a program might execute on Linux. Let's   70 00:10:01,360 --> 00:10:07,680 say we have a C program that prints hello world.  When we compile, we get an excutable and run it.   71 00:10:09,600 --> 00:10:17,360 The code needs the standard and since it does not  have the standard library, it asks the dynamic   72 00:10:17,360 --> 00:10:24,480 linker where it is stored. The dynamic linker  looks for it, finds it in one of the default pads,   73 00:10:24,480 --> 00:10:27,600 and gives it back to the program which  can then finally finish executing.   74 00:10:29,040 --> 00:10:37,360 Libc.so is shared library loaded into a memory the  first time the program requires it is executed.   75 00:10:38,000 --> 00:10:45,440 If another program requires this, it can use a  copy already loaded into memory. On Linux, shared   76 00:10:45,440 --> 00:10:55,200 libraries calling this naming convention. This  is a shared library search on Linux. The dynamic   77 00:10:55,200 --> 00:10:58,400 linker looks into the default directories and then  all directories in the config file and then looks   78 00:10:58,400 --> 00:11:10,640 for any parts on the LD library variable and so  on. On Windows, shared libraries with called dll   79 00:11:10,640 --> 00:11:19,360 or dynamic link libraries and have dll extension.  It doesn't search for a dll with the same name   80 00:11:21,200 --> 00:11:28,560 if loaded in memory. Otherwise, it  will search for a dll in this order.   81 00:11:29,280 --> 00:11:33,120 It will look at the directory from which had  application was loaded, the system directory,   82 00:11:33,120 --> 00:11:37,760 the Windows directory, and current directly  and then the directory listed on the path.   83 00:11:43,520 --> 00:11:49,440 Let's say we have three files that contain  three functions that sprint strings.   84 00:11:50,320 --> 00:11:58,800 They can be combined into a shared library.  Let's say we have a program expected to call   85 00:11:58,800 --> 00:12:10,160 the functions one by one at run-time and print  hello world. We can create an excutable and the   86 00:12:10,160 --> 00:12:19,280 dynamic linker can't find it. That's because it is  not one of the default directories it looks in.   87 00:12:19,920 --> 00:12:28,880 We can add this to the environment variable like  we saw in the Linux short order. Yay it works.   88 00:12:28,880 --> 00:12:35,440 All of the strings we wanted to print have been  printed. Let's see how we can use this information   89 00:12:35,440 --> 00:12:40,720 to package our extension with shared library  dependencies and wheels. We look at building   90 00:12:40,720 --> 00:12:45,600 shared libraries that our extension module depends  on. You might not need to build one because most   91 00:12:45,600 --> 00:12:52,400 C++ shared libraries might be directly installable  to a system package manager but there could be   92 00:12:52,400 --> 00:12:58,640 instances where you want to a wrap a large C++  codebase and might need to build a shared library.   93 00:12:59,600 --> 00:13:04,240 The instructions could be different for  each C++ project you are looking to build.   94 00:13:05,520 --> 00:13:10,400 We look at the tools we can use to build shared  library on each operating system. We will go over   95 00:13:10,400 --> 00:13:18,560 the examples of building Popler. Let's start with  Linux. Each Linux has its own package manager and   96 00:13:18,560 --> 00:13:23,760 own version of shared libraries. That's a problem  because you can get version mismatches when you   97 00:13:25,040 --> 00:13:31,920 use some shared libraries and then install it on a  different Linux that might have different versions   98 00:13:31,920 --> 00:13:40,640 of the shared libraries. The Python community  came up with a set of shared libraries. If you   99 00:13:40,640 --> 00:13:51,440 combine and link with this, it is guaranteed  to work on manyLinux. This is a shared library   100 00:13:51,440 --> 00:13:56,480 subset you can link with and stop worrys about the  libraries not being present on the Linux system.   101 00:13:58,000 --> 00:14:06,720 It pushes docker versions. You can build the image  for the architecture you want and run a docker   102 00:14:06,720 --> 00:14:14,320 container and build a shared library. In the  case of popler, we need to install using yam and   103 00:14:14,320 --> 00:14:21,520 build the shared library we need. Import all the  steps in a shell script so we can use them later.   104 00:14:24,240 --> 00:14:30,000 For building a shared library on macOS we  either need a Mac computer or we can use   105 00:14:30,000 --> 00:14:37,680 fastMac giving us access to a server. It is  useful for debugging potential build issues.   106 00:14:39,120 --> 00:14:43,600 On macOS we can install packages using  blue and build a shared library like   107 00:14:43,600 --> 00:14:48,080 before and again put all the steps in  a shell script which we will use later.   108 00:14:49,280 --> 00:14:56,000 For building a C++ project on Windows machine  we will need to install the visual studio 2019   109 00:14:57,040 --> 00:15:05,360 with Python developer tools. It comes with  a system package manager. One option is vc   110 00:15:05,360 --> 00:15:11,520 package which is a C++ library manager. Here  is how we can use to install shared libraries.   111 00:15:13,360 --> 00:15:20,320 Since it won't install shared libraries on  a default path, we also need to specify the   112 00:15:20,320 --> 00:15:27,840 installation tool directly to cmake. It is  also installed on the GitHub action runner.   113 00:15:33,840 --> 00:15:39,920 Again, report all those steps in a batch script  for use later. After installing shared libraries,   114 00:15:39,920 --> 00:15:44,720 or building them on our own, we need to make  sure our extension is linked with shared files.   115 00:15:46,480 --> 00:15:53,920 We can use setup.py to do that. We will create  a list of directories to search at link time   116 00:15:54,880 --> 00:15:58,080 and create a list of library names we want  to list it. It is popular in this case.   117 00:16:02,000 --> 00:16:09,440 If it is built on Windows we need to add  the directory and create a library list with   118 00:16:09,440 --> 00:16:15,440 all there names of the libraries we need to link  with. There are a lot more on this slide because   119 00:16:15,440 --> 00:16:28,480 we are using vcpkg. If you need header files, you  can create a list of all director files where it   120 00:16:28,480 --> 00:16:40,080 might be. We need to include the GIT include  function as well. We can create a list of ext   121 00:16:40,080 --> 00:16:49,600 modules and pass in the name, the source code,  library directories and the libraries themselves   122 00:16:49,600 --> 00:16:55,440 and specify the language as c plus plus. We call  the setup function with that list of modules   123 00:16:56,000 --> 00:17:00,400 and install it using Pip and should be  able to use the convert function from   124 00:17:00,400 --> 00:17:05,760 the extension model in our Python code. We can  also create a view for extension module using   125 00:17:07,120 --> 00:17:13,360 a current working directory. If we look at one of  the views, we will see our extension module has   126 00:17:13,360 --> 00:17:20,160 been built and the shared libraries it depends  on aren't there. That means if we ship it users   127 00:17:30,000 --> 00:17:32,080 need to install the shared libraries using the  system package manager bringing us to the next   128 00:17:32,080 --> 00:17:36,240 section on bundling shared libraries. On Linux,  we can use auditwheel another tool put out.   129 00:17:36,240 --> 00:17:41,520 It can check if the wheels are compliant  with the shared libraries we saw earlier   130 00:17:41,520 --> 00:17:51,760 and bundle them into the Linux wheel. We run audit  wheel repair and the wheel we need to repair is   131 00:17:54,160 --> 00:18:03,600 in quotes. We attempt to add the directory because  it is not one of the default directories. If we   132 00:18:03,600 --> 00:18:10,080 look, we can see all the shared libraries are  bundled into the wheel itself. Their names have   133 00:18:10,720 --> 00:18:19,520 hash for uniqueness. On macOS we can use delegate.  We can list all the dependencies for a wheel using   134 00:18:19,520 --> 00:18:25,280 delegate steps and run delegate wheel with the  directory and wheel we need to repair as input.   135 00:18:26,960 --> 00:18:34,800 We add the builder again to the path which  is the same as ld but for macOS. We can see   136 00:18:34,800 --> 00:18:38,560 all of the shared libraries our extension  needs are bundled into the wheel itself.   137 00:18:39,680 --> 00:18:44,960 On Windows because of the data and search  order, we could basically place our dll in   138 00:18:44,960 --> 00:18:52,000 the same directory from where the extension loads  and we should be good to go which means we can   139 00:18:52,000 --> 00:18:58,640 copy the dll from the directory and specify the  keyword argument for the setup tools and function.   140 00:18:59,920 --> 00:19:08,320 We make a minor tweak to setup.py by adding a copy  dll function which installs them and copies them   141 00:19:08,320 --> 00:19:13,520 to the correctry where there build extension is  present. If the platform is Windows, we call the   142 00:19:14,160 --> 00:19:20,480 function and add the star dll to the package  so the data files are included in the build.   143 00:19:21,440 --> 00:19:29,360 If you look at the wheel, all are present and our  extension should work. In the case where it is in   144 00:19:29,360 --> 00:19:38,800 another directory, you can use windll loadlibrary  to load the dll before anything else happens.   145 00:19:39,760 --> 00:19:46,720 Since Python 3.8, you can add the dll directory  to the search path using the OS.adddll function.   146 00:19:48,640 --> 00:19:55,120 When bundling shared libraries with generic names  you can run into DLL hell. If you remember the   147 00:19:55,840 --> 00:20:02,640 search order it will not search if the name is  loaded into memory meaning it is possible the   148 00:20:02,640 --> 00:20:09,360 DLL versions ship with a generic name might  not play with the other sets and vice versa.   149 00:20:12,400 --> 00:20:24,080 You can go the DLL mangling way. To do this, you  can write a script which unpacks your Windows   150 00:20:24,080 --> 00:20:32,080 wheel, looks for dependencies for the extension  module and directory, mangles using the hash   151 00:20:32,720 --> 00:20:38,560 and copies them into the same extension module,  and then modify the input tables for extension   152 00:20:38,560 --> 00:20:47,440 module and each dll using a library called  match -- this is what the Windows wheel looks   153 00:20:47,440 --> 00:20:54,320 like after all those steps. You can see the  names are mangled and the tables are updated.   154 00:20:56,000 --> 00:21:03,920 You can check out delvewheel which works on  Windows and does all the steps I just described.   155 00:21:06,880 --> 00:21:15,360 Let's look into how we can automate. Cibuildwheel  helps build wheels for all platform with minimal   156 00:21:15,360 --> 00:21:24,160 CI configuration. We create a workflow,  in the environment variable section you   157 00:21:24,160 --> 00:21:31,760 specify the build with cp3 question mark  which is used on Linux, Mac and Windows   158 00:21:32,320 --> 00:21:38,240 except 3.5 which we specified in the skip  variable. You can specify all the commands   159 00:21:40,320 --> 00:21:48,240 using the forward build Linux variable and pass in  the script we created earlier. We specify how it   160 00:21:48,800 --> 00:21:55,680 should be repaired with the wheel command  which contains the same command we saw earlier.   161 00:21:57,760 --> 00:21:59,109 We do the same for our macOS build script and  specify how we want to use this to build macOS   162 00:21:59,109 --> 00:22:09,520 builds and same for the Windows build script  and specify how we want to use delvewheel.   163 00:22:12,720 --> 00:22:19,920 Within the matrix of operating systems, check  out our GitHub repository, install Python.   164 00:22:20,880 --> 00:22:27,280 On Windows we need to set-up the developer command  from visual C++ which we can do with this GitHub   165 00:22:27,280 --> 00:22:33,760 action. We install and run it like this where  we specify wheel house is the output directory.   166 00:22:38,000 --> 00:22:42,960 In the final step, we upload the build fields  as a build artifact and we can manually download   167 00:22:44,880 --> 00:22:51,840 or add a job that can do that for  us automatically. Run from the job,   168 00:22:52,800 --> 00:23:00,000 upload to PyPI using the GitHub action  from PyPI. We need to configure the token.   169 00:23:02,080 --> 00:23:07,280 When the build and upload job finishes, we should  be able to see our wheels on PyPI. That's mostly   170 00:23:07,280 --> 00:23:17,680 it. I was playing around with this and I also made  an extension for the Snake came and tweaked it so   171 00:23:17,680 --> 00:23:26,880 it resembles Australia. I built it for Linux  and macOS so it can be installed using Pip.   172 00:23:27,520 --> 00:23:30,800 If you are interested in the source and  build process, you can check it out here.   173 00:23:32,240 --> 00:23:36,320 Here is all the code you can check out to  learn more. Here are some awesome talks   174 00:23:36,320 --> 00:23:41,840 you can watch to learn more about Python  C. Thank you for listening and you can   175 00:23:41,840 --> 00:23:45,920 reach out to me on these links. >> Hello, everyone. Thank you   176 00:23:45,920 --> 00:23:51,040 so much for your talk, Vinayak. It was really  interesting. We had good conversation in the chat.   177 00:23:51,040 --> 00:23:55,120 Vinayak has posted his slides in  the chat as well but, hopefully,   178 00:23:55,120 --> 00:24:01,600 we will get them into the text theater chat after  this as well. We have got some spare time back   179 00:24:02,400 --> 00:24:07,200 but you are very free to keep chatting in the  text chat available to us. The next talk is   180 00:24:07,200 --> 00:24:13,840 at 12:00 Ainks -- AEST from from Alan Green  talking about CFU-playground. See you then.