Skip to main content

Enaml Native - New build system for cross compiling Python for iOS and Android coming

Besides speed, I think the biggest barrier preventing python from making it's way onto mobile is the lack of a good way of cross compiling python and all the extension modules that are commonly required for mobile devices.



This is not exactly straightforward, nor standardized (at all), but it's actually not too bad if you have a lot of time and don't mind waking up in the morning with CFLAGS and LDFLAGS flying through your head.  Still, it's extremely time intensive and is not an option for someone with deadlines nor for the person trying to learn python.

I think it's time for something different.



Existing options


While there have been many scattered projects here and there for cross compiling Python for Android or iOS, most don't support building 3rd party extensions.  The only projects that I know that do provide this are kivy's subprojects python-for-android and kivy-ios.

While these are great and very popular, they were written close to 7+ years ago before python's package management was good (before pip, pipenv, wheel, conda, etc..) and are missing some of the features that new package managers provide (such as version management for recipes).

These projects target old versions of python, kivy-ios targets python 2.7.1 and python-for-android uses either 2.7.2 or relies on the crystax NDK which is also dated (2.7.10).  I do see some effort being put in for python 3.6 on the crystax NDK which is great but I still think there's room for improvements.

A major issue that will soon cause problems for python-for-android is that Google will start requiring new apps to have 32-bit and 64-bit binaries installed (see https://www.engadget.com/2017/12/19/android-apps-must-have-64-bit-support-by-august-2019/). I don't believe p4a currently supports multi arch builds in the same app as extensions are copied in the assets folder (but please correct me if I'm wrong!).

For kivy-ios, everything is built as a static lib. This is a major issue for building python extensions as all the tooling is aimed at making shared libraries. As of iOS 8, dynamic libraries are allowed which means this is no longer necessary.

Finally, python-for-android and kivy-ios are separate projects that do almost the same thing. This means managing two toolchains, and one is already complicated enough!

A new approach

I want enaml-native to be as easy as "regular" python, where you can just "pip install <package>" and press build in Android Stuido or Xcode and have a working app.  For that to happen, the problem of cross compiling has to exit the picture (for most users), and that means a new build system.

So for the past few weeks, I've been reading through as many different options as I could possibly find on what's available for python. Pip, pipenv, pipsi, conda, fades,  milksnake, the list goes on and on.

Goals
  • Support iPhone and Android (devices and simulators) with all arches
  • Have separate environments for each app that can be frozen, shared, and recreated
  • Recipes that have specific versions that can be individually installed, removed, or upgraded
  • Automatically installs dependencies for recipes AND pure python packages
  • Supports Windows, Mac, and Linux
  • Independent of any app framework (kivy, enaml-native) so it can be used in other projects
  • Uses the "standard" SDK/NDK for each platform (no crystax)
I didn't want to write something new, so pulled one of the package managers off the shelf that happens to fit perfectly (although I'm sure a few others could work as well) and got started.  

Results

After about three weeks of long nights and weekends, I ported my previously patched and hacked up kivy-ios and python-for-android to the new build system and have a fresh working version of Python 2.7.14 with all the dependencies for enaml-native (python, atom, enaml, msgpack), ctypes, openssl (1.0.2n) all cross compiled in separate versioned packages that can be used to run on Android and iOS devices and simulators.

 Bootstrapping

The new build system is much easier to use. You create an environment for your app, install the dependencies from a custom repo (to be announced), then add some minor bootstrapping to bundle and use it in your app.

For iOS

  1. Create an env for your app
  2. Install the requirements (ex. enaml-native install ios-python)
  3. Add a build phase script (about 4-5 lines) 
  4. Add libpython to the list of Linked libraries
  5. Change a few build settings (4 to be exact)
  6. Add the code to start Python using the C-API
  7. And your python script

For Android

  1. Create an env for your app
  2. Install the requirements (ex. enaml-native install android-python)
  3. Add a jni bootstrap to start Python using the C-API
  4. Add an Android.mk that includes libpython and your jni bootstrap
  5. Add a Gradle copy files task to bundle all the native libs and python
  6. And your python script 
More details to come on this when I release it.

Creating recipes

Recipes are all independent and the actual cross compiling can be done however you like, but most python extensions (including cython) can simply be copied from an existing android or ios recipe and updated as needed (change the name and download url).

The recipes are generally small and easy to understand. The recipe for cross-compiling msgpack for Android is ~60 lines and ~80 lines for iOS.

Building is done in one command. Which automatically installs a clean environment with only the build dependencies you define and is very reproducible.

Performance

I'm still working out some of the kinks and analyzing the performance differences that come from using different compilers/flags etc... Currently this it starts up slower for a simple app as shown below.



The current recipe compiles Python is with only core modules.  All extensions (ssl, ctypes, io, expat, etc..)  are all built as shared libs and included, so you can remove ones that your app doesn't need which reduces the size, but this may be why the speed is different.

It's around 10-20% slower which is noticeable even for this trivial app. It could be that compiling python with clang 5 is slower than gcc 5.3 from crystax.  It could also be because atom, enaml, and ply have all been updated. So this is an area to explore.


Edit: This was due to enaml 0.10.2 using the "future" library.  I've added a PR to enaml and atom to move the needed pieces into the compat modules and the speed is back to normal in 0.10.3.

Summary

Exact details on how to get and use it will follow shortly. You can also expect full integration in the enaml-native-cli to replace the existing customized forks of p4a and kivy-ios althogether if all goes according to plan.

Also, I'm thinking of doing a kickstarter, patron, or something support this (as I only have a part time contract at the moment) if anyone think it's a worthy cause.

Edit: See https://github.com/codelv/conda-mobile





Comments

  1. Just exploring all enaml-native resources and want to say thank you for your effort.

    It's hard for me to express how much on this blog there are opinions that coincide with mine.

    I began my professional career in '90s, and with enaml I see some basic similarity to Delphi's DFM files. Since then I think that front-end over-complicated.
    I think that front-end should be simple, intuitive, WYSIWYG as possible - not to block art expression with tech strangeness...Enaml is moving into right way.

    I am definitively back-end developer, mostly Python, and I lack art skills to design front-end properly and with taste. But even back-end dev sometimes need to write simple front-end...
    So after playing a little with React, I will definitively try Enaml-Native.

    ReplyDelete

Post a Comment

Popular posts from this blog

Kivy vs React-Native for building cross platform mobile apps

I've built three apps now using Kivy and one with React-Native, just wanted to share my thoughts on both. Just a warning, I am strongly biased towards python and this is all based on opinion and experience and is thus worth what you pay for it. I don't claim to be an expert in either of these, just have worked with each for several months.  If something is incorrect I'd love to hear advice. Kivy Demo of one of the apps Pros: Nice to be able to run natively on the desktop WITHOUT a simulator Python is easy to work with Use (almost) any python library Very easy to create custom widgets Kivy properties and data binding just work. Way nicer than React's "state" / flux / redux whatever you want to call it (stupid?).  Native interfaces (pyjnius) and (pyobjc) Runs and feels pretty smooth Cons: Default widget toolkit looks like Android 4.4. Requiring you use your own widgets or a theming kit like KivyMD  if styling bothers you Creating dy

Control Systems in Python - Part 2 - Routh Hurwitz

In my last post Control Systems in Python Part 1 , i described how to setup and use Python for doing some basic plotting of transfer functions. One of the biggest benefits of using sympy vs numeric packages like matlab/numpy/scipy is the fact that you can use symbolic variables. This post includes a function for computing the Routh Hurwitz table (Note: It does not work for row's of zeros). Lets do my control systems design homework problem together :) (Warning: I have not verified if this answer is right so please correct me if it’s not!) The Problem The problem is DP 9.11 from Dorf & Bishop’s Modern Control Systems. ISBN 0136024580. Basically we have to design a controller to compensate for a system with a time delay. The controller is: And the system is: First we approximate the exponential term with a 2nd order polynomial using pade(0.4,2) such that: Thus the approximated system is: Using frequency response methods, design the controller so that th

Control Systems in Python - Part 1 - Bode and Step Response

I hate matlab with passion, yet sadly, nearly everyone uses it.  I'm a fan of Python and open source stuff so here's a simple article on how to do some common control systems stuff in Python. First we need to make sure the environment is setup. Install IPython (or you can use any other python shell, but a unicode supported shell is preferred) Install python-control (numpy, scipy) Install sympy These should do if your on Ubuntu/debian: sudo apt - get install python - sympy python-numpy python-scipy python-matplotlib ipython Then you need to install python control, see How to download and install python-control Intro to using Sympy Open ipython and run the following: import sympy from sympy import * sympy.init_printing() s = Symbol('s') Now we can do things like define transfer functions using the symbolic variable s. We can expand the bottom using the .simplify() method and we can do something more complex like... which is really nice because it