Enaml Native - New build system for cross compiling Python for iOS and Android coming

Besides speed, I think the biggest barrier preventing python from making it's way onto mobile is the lack of a good way of cross compiling python and all the extension modules that are commonly required for mobile devices.

This is not exactly straightforward, nor standardized (at all), but it's actually not too bad if you have a lot of time and don't mind waking up in the morning with CFLAGS and LDFLAGS flying through your head.  Still, it's extremely time intensive and is not an option for someone with deadlines nor for the person trying to learn python.

I think it's time for something different.

Existing options

While there have been many scattered projects here and there for cross compiling Python for Android or iOS, most don't support building 3rd party extensions.  The only projects that I know that do provide this are kivy's subprojects python-for-android and kivy-ios.

While these are great and very popular, they were written close to 7+ years ago before python's package management was good (before pip, pipenv, wheel, conda, etc..) and are missing some of the features that new package managers provide (such as version management for recipes).

These projects target old versions of python, kivy-ios targets python 2.7.1 and python-for-android uses either 2.7.2 or relies on the crystax NDK which is also dated (2.7.10).  I do see some effort being put in for python 3.6 on the crystax NDK which is great but I still think there's room for improvements.

A major issue that will soon cause problems for python-for-android is that Google will start requiring new apps to have 32-bit and 64-bit binaries installed (see https://www.engadget.com/2017/12/19/android-apps-must-have-64-bit-support-by-august-2019/). I don't believe p4a currently supports multi arch builds in the same app as extensions are copied in the assets folder (but please correct me if I'm wrong!).

For kivy-ios, everything is built as a static lib. This is a major issue for building python extensions as all the tooling is aimed at making shared libraries. As of iOS 8, dynamic libraries are allowed which means this is no longer necessary.

Finally, python-for-android and kivy-ios are separate projects that do almost the same thing. This means managing two toolchains, and one is already complicated enough!

A new approach

I want enaml-native to be as easy as "regular" python, where you can just "pip install <package>" and press build in Android Stuido or Xcode and have a working app.  For that to happen, the problem of cross compiling has to exit the picture (for most users), and that means a new build system.

So for the past few weeks, I've been reading through as many different options as I could possibly find on what's available for python. Pip, pipenv, pipsi, conda, fades,  milksnake, the list goes on and on.

Goals

• Support iPhone and Android (devices and simulators) with all arches
• Have separate environments for each app that can be frozen, shared, and recreated
• Recipes that have specific versions that can be individually installed, removed, or upgraded
• Automatically installs dependencies for recipes AND pure python packages
• Supports Windows, Mac, and Linux
• Independent of any app framework (kivy, enaml-native) so it can be used in other projects
• Uses the "standard" SDK/NDK for each platform (no crystax)

I didn't want to write something new, so pulled one of the package managers off the shelf that happens to fit perfectly (although I'm sure a few others could work as well) and got started.  

Results

After about three weeks of long nights and weekends, I ported my previously patched and hacked up kivy-ios and python-for-android to the new build system and have a fresh working version of Python 2.7.14 with all the dependencies for enaml-native (python, atom, enaml, msgpack), ctypes, openssl (1.0.2n) all cross compiled in separate versioned packages that can be used to run on Android and iOS devices and simulators.

 Bootstrapping

The new build system is much easier to use. You create an environment for your app, install the dependencies from a custom repo (to be announced), then add some minor bootstrapping to bundle and use it in your app.

For iOS

1. Create an env for your app
2. Install the requirements (ex. enaml-native install ios-python)
3. Add a build phase script (about 4-5 lines) 
4. Add libpython to the list of Linked libraries
5. Change a few build settings (4 to be exact)
6. Add the code to start Python using the C-API
7. And your python script

For Android

1. Create an env for your app
2. Install the requirements (ex. enaml-native install android-python)
3. Add a jni bootstrap to start Python using the C-API
4. Add an Android.mk that includes libpython and your jni bootstrap
5. Add a Gradle copy files task to bundle all the native libs and python
6. And your python script 

More details to come on this when I release it.

Creating recipes

Recipes are all independent and the actual cross compiling can be done however you like, but most python extensions (including cython) can simply be copied from an existing android or ios recipe and updated as needed (change the name and download url).

The recipes are generally small and easy to understand. The recipe for cross-compiling msgpack for Android is ~60 lines and ~80 lines for iOS.

Building is done in one command. Which automatically installs a clean environment with only the build dependencies you define and is very reproducible.

Performance

I'm still working out some of the kinks and analyzing the performance differences that come from using different compilers/flags etc... Currently this it starts up slower for a simple app as shown below.

The current recipe compiles Python is with only core modules.  All extensions (ssl, ctypes, io, expat, etc..)  are all built as shared libs and included, so you can remove ones that your app doesn't need which reduces the size, but this may be why the speed is different.

It's around 10-20% slower which is noticeable even for this trivial app. It could be that compiling python with clang 5 is slower than gcc 5.3 from crystax.  It could also be because atom, enaml, and ply have all been updated. So this is an area to explore.


Edit: This was due to enaml 0.10.2 using the "future" library.  I've added a PR to enaml and atom to move the needed pieces into the compat modules and the speed is back to normal in 0.10.3.

Summary

Exact details on how to get and use it will follow shortly. You can also expect full integration in the enaml-native-cli to replace the existing customized forks of p4a and kivy-ios althogether if all goes according to plan.

Also, I'm thinking of doing a kickstarter, patron, or something support this (as I only have a part time contract at the moment) if anyone think it's a worthy cause.

Edit: See https://github.com/codelv/conda-mobile