Besides speed, I think the biggest barrier preventing python from making it's way onto mobile is the lack of a good way of cross compiling python and all the extension modules that are commonly required for mobile devices.
This is not exactly straightforward, nor standardized (at all), but it's actually not too bad if you have a lot of time and don't mind waking up in the morning with CFLAGS and LDFLAGS flying through your head. Still, it's extremely time intensive and is not an option for someone with deadlines nor for the person trying to learn python.
I think it's time for something different.
While there have been many scattered projects here and there for cross compiling Python for Android or iOS, most don't support building 3rd party extensions. The only projects that I know that do provide this are kivy's subprojects python-for-android and kivy-ios.
While these are great and very popular, they were written close to 7+ years ago before python's package management was good (before pip, pipenv, wheel, conda, etc..) and are missing some of the features that new package managers provide (such as version management for recipes).
These projects target old versions of python, kivy-ios targets python 2.7.1 and python-for-android uses either 2.7.2 or relies on the crystax NDK which is also dated (2.7.10). I do see some effort being put in for python 3.6 on the crystax NDK which is great but I still think there's room for improvements.
A major issue that will soon cause problems for python-for-android is that Google will start requiring new apps to have 32-bit and 64-bit binaries installed (see https://www.engadget.com/2017/12/19/android-apps-must-have-64-bit-support-by-august-2019/). I don't believe p4a currently supports multi arch builds in the same app as extensions are copied in the assets folder (but please correct me if I'm wrong!).
For kivy-ios, everything is built as a static lib. This is a major issue for building python extensions as all the tooling is aimed at making shared libraries. As of iOS 8, dynamic libraries are allowed which means this is no longer necessary.
Finally, python-for-android and kivy-ios are separate projects that do almost the same thing. This means managing two toolchains, and one is already complicated enough!
So for the past few weeks, I've been reading through as many different options as I could possibly find on what's available for python. Pip, pipenv, pipsi, conda, fades, milksnake, the list goes on and on.
Goals
The recipes are generally small and easy to understand. The recipe for cross-compiling msgpack for Android is ~60 lines and ~80 lines for iOS.
Building is done in one command. Which automatically installs a clean environment with only the build dependencies you define and is very reproducible.
The current recipe compiles Python is with only core modules. All extensions (ssl, ctypes, io, expat, etc..) are all built as shared libs and included, so you can remove ones that your app doesn't need which reduces the size, but this may be why the speed is different.
It's around 10-20% slower which is noticeable even for this trivial app. It could be that compiling python with clang 5 is slower than gcc 5.3 from crystax. It could also be because atom, enaml, and ply have all been updated. So this is an area to explore.
Edit: This was due to enaml 0.10.2 using the "future" library. I've added a PR to enaml and atom to move the needed pieces into the compat modules and the speed is back to normal in 0.10.3.
Also, I'm thinking of doing a kickstarter, patron, or something support this (as I only have a part time contract at the moment) if anyone think it's a worthy cause.
Edit: See https://github.com/codelv/conda-mobile
This is not exactly straightforward, nor standardized (at all), but it's actually not too bad if you have a lot of time and don't mind waking up in the morning with CFLAGS and LDFLAGS flying through your head. Still, it's extremely time intensive and is not an option for someone with deadlines nor for the person trying to learn python.
I think it's time for something different.
Existing options
While there have been many scattered projects here and there for cross compiling Python for Android or iOS, most don't support building 3rd party extensions. The only projects that I know that do provide this are kivy's subprojects python-for-android and kivy-ios.
While these are great and very popular, they were written close to 7+ years ago before python's package management was good (before pip, pipenv, wheel, conda, etc..) and are missing some of the features that new package managers provide (such as version management for recipes).
These projects target old versions of python, kivy-ios targets python 2.7.1 and python-for-android uses either 2.7.2 or relies on the crystax NDK which is also dated (2.7.10). I do see some effort being put in for python 3.6 on the crystax NDK which is great but I still think there's room for improvements.
A major issue that will soon cause problems for python-for-android is that Google will start requiring new apps to have 32-bit and 64-bit binaries installed (see https://www.engadget.com/2017/12/19/android-apps-must-have-64-bit-support-by-august-2019/). I don't believe p4a currently supports multi arch builds in the same app as extensions are copied in the assets folder (but please correct me if I'm wrong!).
For kivy-ios, everything is built as a static lib. This is a major issue for building python extensions as all the tooling is aimed at making shared libraries. As of iOS 8, dynamic libraries are allowed which means this is no longer necessary.
Finally, python-for-android and kivy-ios are separate projects that do almost the same thing. This means managing two toolchains, and one is already complicated enough!
A new approach
I want enaml-native to be as easy as "regular" python, where you can just "pip install <package>" and press build in Android Stuido or Xcode and have a working app. For that to happen, the problem of cross compiling has to exit the picture (for most users), and that means a new build system.So for the past few weeks, I've been reading through as many different options as I could possibly find on what's available for python. Pip, pipenv, pipsi, conda, fades, milksnake, the list goes on and on.
Goals
- Support iPhone and Android (devices and simulators) with all arches
- Have separate environments for each app that can be frozen, shared, and recreated
- Recipes that have specific versions that can be individually installed, removed, or upgraded
- Automatically installs dependencies for recipes AND pure python packages
- Supports Windows, Mac, and Linux
- Independent of any app framework (kivy, enaml-native) so it can be used in other projects
- Uses the "standard" SDK/NDK for each platform (no crystax)
Results
After about three weeks of long nights and weekends, I ported my previously patched and hacked up kivy-ios and python-for-android to the new build system and have a fresh working version of Python 2.7.14 with all the dependencies for enaml-native (python, atom, enaml, msgpack), ctypes, openssl (1.0.2n) all cross compiled in separate versioned packages that can be used to run on Android and iOS devices and simulators.Bootstrapping
The new build system is much easier to use. You create an environment for your app, install the dependencies from a custom repo (to be announced), then add some minor bootstrapping to bundle and use it in your app.For iOS
- Create an env for your app
- Install the requirements (ex. enaml-native install ios-python)
- Add a build phase script (about 4-5 lines)
- Add libpython to the list of Linked libraries
- Change a few build settings (4 to be exact)
- Add the code to start Python using the C-API
- And your python script
For Android
- Create an env for your app
- Install the requirements (ex. enaml-native install android-python)
- Add a jni bootstrap to start Python using the C-API
- Add an Android.mk that includes libpython and your jni bootstrap
- Add a Gradle copy files task to bundle all the native libs and python
- And your python script
Creating recipes
Recipes are all independent and the actual cross compiling can be done however you like, but most python extensions (including cython) can simply be copied from an existing android or ios recipe and updated as needed (change the name and download url).The recipes are generally small and easy to understand. The recipe for cross-compiling msgpack for Android is ~60 lines and ~80 lines for iOS.
Building is done in one command. Which automatically installs a clean environment with only the build dependencies you define and is very reproducible.
Performance
I'm still working out some of the kinks and analyzing the performance differences that come from using different compilers/flags etc... Currently this it starts up slower for a simple app as shown below.The current recipe compiles Python is with only core modules. All extensions (ssl, ctypes, io, expat, etc..) are all built as shared libs and included, so you can remove ones that your app doesn't need which reduces the size, but this may be why the speed is different.
It's around 10-20% slower which is noticeable even for this trivial app. It could be that compiling python with clang 5 is slower than gcc 5.3 from crystax. It could also be because atom, enaml, and ply have all been updated. So this is an area to explore.
Edit: This was due to enaml 0.10.2 using the "future" library. I've added a PR to enaml and atom to move the needed pieces into the compat modules and the speed is back to normal in 0.10.3.
Summary
Exact details on how to get and use it will follow shortly. You can also expect full integration in the enaml-native-cli to replace the existing customized forks of p4a and kivy-ios althogether if all goes according to plan.Also, I'm thinking of doing a kickstarter, patron, or something support this (as I only have a part time contract at the moment) if anyone think it's a worthy cause.
Edit: See https://github.com/codelv/conda-mobile
Just exploring all enaml-native resources and want to say thank you for your effort.
ReplyDeleteIt's hard for me to express how much on this blog there are opinions that coincide with mine.
I began my professional career in '90s, and with enaml I see some basic similarity to Delphi's DFM files. Since then I think that front-end over-complicated.
I think that front-end should be simple, intuitive, WYSIWYG as possible - not to block art expression with tech strangeness...Enaml is moving into right way.
I am definitively back-end developer, mostly Python, and I lack art skills to design front-end properly and with taste. But even back-end dev sometimes need to write simple front-end...
So after playing a little with React, I will definitively try Enaml-Native.