bblfsh client does not work on macOs

[smacker]

$ pip install bblfsh

$ python
Python 3.6.5 (default, Jun 17 2018, 12:13:06)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bblfsh
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/smacker/tmp/json-uast/venv/lib/python3.6/site-packages/bblfsh/__init__.py", line 1, in <module>
    from bblfsh.client import BblfshClient
  File "/Users/smacker/tmp/json-uast/venv/lib/python3.6/site-packages/bblfsh/client.py", line 9, in <module>
    from bblfsh.result_context import ResultContext
  File "/Users/smacker/tmp/json-uast/venv/lib/python3.6/site-packages/bblfsh/result_context.py", line 2, in <module>
    from bblfsh.node import Node
  File "/Users/smacker/tmp/json-uast/venv/lib/python3.6/site-packages/bblfsh/node.py", line 5, in <module>
    from bblfsh.pyuast import NodeExt
ImportError: dlopen(/Users/smacker/tmp/json-uast/venv/lib/python3.6/site-packages/bblfsh/pyuast.cpython-36m-darwin.so, 2): Symbol not found: _UastDecode
  Referenced from: /Users/smacker/tmp/json-uast/venv/lib/python3.6/site-packages/bblfsh/pyuast.cpython-36m-darwin.so
  Expected in: flat namespace
 in /Users/smacker/tmp/json-uast/venv/lib/python3.6/site-packages/bblfsh/pyuast.cpython-36m-darwin.so

$ python -V
Python 3.6.5

$ pip list
Package          Version
---------------- --------
bblfsh           3.0.3
...

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.14.3
BuildVersion:	18D109

[creachadair]

So the basic issue here is that libuast.so is not being pulled in to resolve the undefined symbols in the shared library. I am able to make it work by manually opening that .so prior to import, e.g.,

>>> from ctypes import _dlopen as dlopen
>>> dlopen('libuast.so', 2)
>>> import bblfsh

This is obviously not a fix, but it suggests we might need to add some additional bits to the setup.py and/or the installation instructions for macOS to ensure we have that .so on the loader's path.

[creachadair]

I'm having trouble reproducing the exact environment we get from the pip3 installation, but it looks like the setup.py fetches libuast.so for the wrong architecture on macOS: Specifically, it fixes LIBUAST_ARCH = "linux-amd64" and does not update it from the ambient.

When building, this means we wind up linking against the wrong thing:

$ python3 setup.py build_ext -i
[…]
ld: warning: ignoring file bblfsh/libuast/libuast.a, file was built for archive 
  which is not the architecture being linked (x86_64): bblfsh/libuast/libuast.a

But I suppose ld considers this a "warning" because it's building a shared library, and it's possible that we have the "right" one installed elsewhere. (In this case, we do not).

So I think the proximal fix here is for us to derive the architecture string from the environment. I think we do not see this in CI because everything is built in Docker containers, all of which run variations of Linux.

[creachadair]

Setting the architecture from sys.platform gets us the correct release, but the Darwin release binary does not contain a static libuast.a. It does contain a libuast.so.

The setup script presently assumes there will be a libuast.a and includes it as data for the bblfsh.pyuast external package, but of course on macOS that doesn't exist. Including libuast.so instead doesn't work as-written; the linker doesn't include the file, so once the link is complete the installed .so is still missing symbols.

Distutils is correctly flagging the output .so as a bundle, at least. Next I will see if I can build a static libuast.a manually, and see if that helps. (This is a little tricky on macOS, but I will kludge it for the sake of the experiment).

[creachadair]

I manually built libuast.a via go build -mode=c-archive, and copied it into my build tree with GET_LIBUAST set to false. This permits a python3 setup.py install to produce a working extension.

I still have some work to do, because Apple's clang seems by default to prefer a .so if one is available. However, I think we're getting closer to a solution.

[creachadair]

I still have some work to do, because Apple's clang seems by default to prefer a .so if one is available

This turned out not to be a problem because our config explicitly names the .a files to be linked on the command line, rather than pulling them in by path. (We do also reference the path, but fortunately explicit arguments supersede library search).

[creachadair]

Things to fix:

• Build a static libuast.a for macOS releases (Link a static version of libuast for Darwin. libuast#101).
• Push a new release of libuast with that change (https://github.com/bblfsh/libuast/releases/tag/v3.3.1).
• Fix the platform signature issue and update client-python to depend on that release (Derive the libuast os/arch selector from sys.platform. #157).
• Deploy a new bblfsh PyPI package (https://github.com/bblfsh/client-python/releases/tag/v3.0.4).

[creachadair]

I can confirm this now works after release of v3.0.4.

$ pip3 install bblfsh
[…]
Successfully built bblfsh
Installing collected packages: bblfsh
Successfully installed bblfsh-3.0.4
$ python3
Python 3.7.3 (default, Mar 27 2019, 09:23:15) 
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bblfsh
>>>