Thali

  • Fork me on GitHub

Notes on Node-Gyp for Android

WARNING

These are my raw notes trying to figure out how to get Node-Gyp on Linux to build for Android. As I learn more I’ll clean these up and eventually (if things work) turn them into instructions or (if things don’t work) explain the problem. But for now, they are just a mess. You have been warned!

The Notes

The immediate goal is to build node-leveldown.

I started simply on my Linux box and ran “npm install leveldown”. That worked just fine, including building leveldb which is a C program.

Then I decided to get fancy and tried “npm install leveldown –dist-os=android”. This ‘worked’ in the sense that stuff got built what what got built was Linux code, no Android code.

So clearly I wasn’t passing on the right ‘stuff’ so NPM would tell node-gyp to build for Android. But how to fix this?

I couldn’t find anything on how to use NPM so it would tell node-gyp to build binaries for Android so I decided to pop down a level and try to understand node-gyp. I figure if I can grok that then maybe I can understand the issues.

So first I installed node-gyp directly, sudo npm install node-gyp -g. To use node-gyp directly I needed all the leveldown files to be available (normally NPM would download those files automatically but I’m not using npm for the moment, just node-gyp). So I cloned leveldown and in the root ran: npm install. This was just to make sure that things were there. It built just fine, including dev dependencies.

So then, still in the leveldown clone, I tried “node-gyp configure” and “node-gyp build” and they ran fine. But I think that’s just because I had already built them. So I decided to push my luck and try “node-gyp rebuild” which includes a cleaning step. That worked just fine as well.

Just to test things I decided to delete the node_modules and build directories and try “node-gyp rebuild” again. That failed. I got a nan on finding a module. That actually makes a lot of sense because once I deleted those directories none of the dev dependencies were there. So the important lesson is that I do need to start with ‘npm install’ and then run ‘node-gyp clean’ to get to the point where at least in theory I could build for Android.

Now in theory the way one should, I think, get node-gyp to build for Android is something along the lines of ‘node-gyp rebuild –dist-os=android’. That does appear to work but it’s actually just building for Linux. The –dist-os=android argument is passed to the Python build scripts but the command line for python includes “-f make” which won’t trigger any of the Android logic. So it’s effectively a no-op.

In truth I dug into the node-gyp code and looked at the JS for handling the call to node-gyp and it will ignore –dist-op. Or, to be more exact, it won’t use it for creating the command line arguments to call python. Instead it will pack up the argument in a structure it passes to the Python code via a config file but by then it’s apparently too late. I think.

I actually tried a little experiment, I issued the following:

yaron@yaron-elementary-VirtualBox:/tmp/node-leveldown$ /usr/lib/node_modules/node-gyp/gyp/gyp_main.py binding.gyp -f android -I /tmp/node-leveldown/build/config.gypi -I /usr/lib/node_modules/node-gyp/addon.gypi -I /home/yaron/.node-gyp/0.10.32/common.gypi -Dlibrary=shared_library -Dvisibility=default -Dnode_root_dir=/home/yaron/.node-gyp/0.10.32 -Dmodule_root_dir=/tmp/node-leveldown --depth=. --no-parallel --generator-output build -Goutput_dir=.

The bad news is that it blew up because of some Android specific issues but the good news is that it blew up because of Android specific issues!!!!! But I admit that now I’m just trying voodoo. The previous is the command line generated by calling node-gyp on Linux where I changed the ‘-f’ argument to be android.

To make things more reasonable I need to understand the code better. The fund beings in the node file node-gyp but after creating the command line args and config file it calls gyp_main.py -> which imports the gyp library and calls gyp/pylib/gyp and then calls gyp.script_main() which calls init.py which then calls main() which calls gyp_main() which seems to be where the real fun is at. This is where we process all the command line args and start launching things. What I’m particularly looking out for is the ability to launch android.py since that appears to be where the android logic is.

For now I can force the use of android.py by using the ‘-f android’ list in the command line above. So I’m going to roll with that for a second and see what android.py wants from life. So I’m going to try and set up the python debugger to run the command line and then walk into the resulting code and see what the heck is going on.

One of the first things I leanred is that the python code will also look for an environmental variable ‘GYP_GENERATORS’. So we could slip in Android that way.

In addition to the -D command to set variables there is also a GYP_DEFINES environmental variable looked at in line 41 of init.py. I’m not sure of the details of how it works but it seems potentially useful if we are going to start setting things up via environmental variables. In line 478 a similar mechanism is used for submitting GYP_GENERATOR_FLAGS. I have a feeling we’ll need that at least for the ndk if not for the tool chain we will have to generate ourselves (a la how we build Node.js for Android).

Right now I’m getting an exception from snappy when input.py tries to call LoadTargetBuildFile on binding.gyp. I already know this issue. Snappy doesn’t have an os_include for Android and so it fails. In the past I worked around this by manually editing snappy to include an os_include for Android that maps to Linux. But I’m not sure if that is right.

However something caught my attention. Which is https://github.com/joyent/libuv#android. Those are the build instructions for libuv for Android. Notice how it calls Gyp! Could this finally be the example I need to figure out how to get leveldown building? So the command line is:

source ./android-configure NDK_PATH gyp

The script starts with the traditional android-toolchain script we see in the node.js build script. The interesting part is the bottom where it calls:

./gyp_uv.py -Dtarget_arch=arm -DOS=android

So obviously this requires a visit to https://github.com/joyent/libuv/blob/master/gyp_uv.py

As someone who doesn’t actually know anything about gyp the most interesting part is line 69-71 where the various gyp config files are submitted. But other than that. I just don’t see much that is really fundamentally different from what we are doing. Sigh….

O.k. so I guess we’ll have to go back to fixing snappy. My sleezy little fix is to go to /deps/snappy and edit snappy.gyp to add in an entry:

, ['OS=="android"', {'os_include': 'linux'}]

So once we make that one line change then that eventually results in android.py being called and getting an error on line 950 where we find out that ANDROID_BUILD_TOP isn’t set.

Now, whose job is it to really set that? And why do they need it anyway? My understanding (flawed I’m sure) is that ANDROID_BUILD_TOP (output by lunch) is used to build all of Android, not just an Android app. Or is it? I honestly don’t know.

I hate being ignorant. Here is one of the few articles I found that even mention ANDROID_TOP_DIR. But that is to build the entire Android environment. That’s nuts.

Well first of all I discovered something that explains things. android.py isn’t from node-gyp, it’s from gyp! It was written by Google. That’s probably why no one is answering any questions about it in node-gyp land, they don’t know! They just sucked it in from gyp!

Sigh… o.k. let’s step back again. I really have no desire to learn every possible detail about the Android build system. And I clearly don’t understand what’s actually going on. Which isn’t good.

So right now I have a specific goal. Let’s see if I can get to that more modest goal.

I want to run pouchdb-server on Android.

Most of pouchd-server is good ole Javascript. Except for level-down which has a set of native C APIs intended to talk to LevelDB.

This means we need to compile two different bits of native code. We need to compile levelDB and we need to compile the .node file generated by level-down.

Now LevelDB just loves Android, here is its android.mk file. So presumably we can make LevelDB compile by just pointing the NDK at it.

But what about the level-down wrapper? It’s a whole mess of native code.

Now I do have some hope if I’m willing to adopt this much more limited goal of just getting pouchdb-server running rather than figuring out how to get node-gyp to generally play well with Android. That hope comes from here which are instructions for building node-sqlite3 directly into node.js and then being able to call it.

Now what I could do (and what the PouchDB folks suggested I should add) is to not use Level-Down at all. But instead use the SQLite adapter in Level-Up. The idea being that SQlite is native to Android so if I can just get sqlite working then we have a working solution since PouchDB will support it via LevelUp.

BTW, if I was feeling full on evil there is an ever better plan. Which is that LevelUp supports MemDown which is a pure in-memory store solution. That isn’t useful for my Thali goals in the long term but right now I’m making some assumptions. The key assumption is that IF Thali is moving to node.js THEN I will have resources to handle the ports of node.js to various platforms like Android and so don’t need to really worry about the details of Android. However, this really sucks in the sense that any time I play with the code as soon as it, for example, gets forced to turn off due to resource conflicts or whatever or if I just have to restart I lose all state. No fun. It’s livable for a short time but I’d like some persistence if possible.

So I think it’s worth at least a few days to figure out how to get some kind of real storage working.

Now, https://github.com/rvagg/node-levelup/wiki/Modules#storage lists a bunch of alternate backends for LevelUP. Let’s walk through the ones that are potentially interesting for our immediate goals:

  • LevelDOWN - This is the ‘native’ back end for PouchDB in Node land. It has a bunch of C apis that call LevelDown. I strongly suspect we can just compile LevelDOWN directly into node.js as part of our build (following the sqlite3 instructions) and then have it call our local LevelDB instance compiled for Android. This should give us, btw, the best perf.
  • localstorage-down - This explicitly says that it supports Android but via PhoneGap. It’s a browser blug in so it’s not going to work for us.
  • MemDOWN - I mentioned this above. A pure in-memory solution. Not the best choice for the previously described reasons.
  • jsonDown - This is a step up from MemDOWN for us and could work. It is an in memory database (which is fine, for playing purposes we don’t have much data) but it persists to a file. It’s pure Javascript so it should work.
  • SQLdown - This has all the issues of LevelDOWN in terms of native compilation except that SQLite ships in the box. But since LevelDB wants to run on Android if we are going down the NDK route I’d rather try first with LevelDOWN.
  • medeadown - This is a pure JS database, it uses local files to record things but it keeps all the key data in memory which is basically the JSON content.

Of these I suspect that for a non-native solution medeadown is our best choice (and was recommended by the PouchDB folks if I wanted a pure JS solution).

So the obviously sane thing for me to do now is to start without any native code and then go native.

So in the source code I need to put in:

var pouch = new PouchDB('myDB', {db: require('medeadown')});

The previous was taken from here.

My current (incomplete, non-functional) attempt at instructions

Setting up IntelliJ on Linux

  1. Run ‘npm -g install node-gyp’
  2. Clone node-leveldown (https://github.com/rvagg/node-leveldown.git)
  3. Open IntelliJ and open a project at the root directory, node-leveldown
  4. Open any .py file and see if IntelliJ prompts you to install Python support, if so, install and reload
  5. Go to File->Project Structure and hit new and add the Python SDK as the Project SDK
  6. Go to Run->Edit Configurations->Green Plus and select Python
    1. I had to hit the “X items more” entry at the end to make Python show up
  7. For Script navigate to node-leveldown/node_modules/node-gyp/gyp/gyp_main.py and set that

  8. For Script parameters use:

     binding.gyp -f android -I ./build/config.gypi -I ./node_modules/node-gyp/addon.gypi -I /home/yaron/.node-gyp/0.10.32/common.gypi -Dlibrary=shared_library -Dvisibility=default -Dnode_root_dir=/home/yaron/.node-gyp/0.10.32 -Dmodule_root_dir=. --depth=. --no-parallel --generator-output build -Goutput_dir=.
    1. Note that you have to replace /home/yaron with your own home path. I did try using ~ but it doesn’t resolve correctly.
  9. For Python Interpreter set to ‘Use specified interpreter’ and choose the one set for your project
  10. For working directory set it to the full path to wherever you cloned node-leveldown

  11. Now find your comment line and edit deps/snappy/snappy.gyp and add the following to targets/variables/conditions:

    , ['OS=="android"', {'os_include': 'linux'}]