Replacing gyp

[jbergstroem]

Lets find a long-term replacement for gyp. Backlog here: nodejs/node#133

[mikeal]

+1000

[kobalicek]

What about writing a build system that uses node.js. I know that it was discussed and it's a bad idea in sense to build node itself, however, I thought about it after years of using cmake and gyp. I think a completely new build system that is not based on an archaic language like cmake and that doesn't need python would be really nice to have. If I think of node it's a perfect dependency for a build system, because nearly everybody has it installed and JS is a language most people understand. As a bonus, the build system itself can contain extensions that can automatically download dependencies like npm does, etc.

Just my 2 cents, would really love build system for C++ based on node that "just works". I would definitely contribute to such project, but I would not be able to write the whole thing myself right now.

Practically speaking..

Reinvention of a build system is a hard problem. Why not extend an existing one?

First:

Please do consider cmake as first-class in addition to (or better instead of) gyp, gn etc.

Cmake has no dependency on python, is a C/C++ based implementation to build C/C++ projects, provides built-in system introspection (find_library etc.) and IMO it is one mature, robust, true cross-platform, industry-ready and most reliable build system available at present. We can send PR upstream to kitware if there are discrepancies / blockers to build node.js and native modules for node.js. I honestly believe that will improve node.js ecosystem and mitigate many (if not all) node-gyp frustrations spontaneously.

https://github.com/kitware/cmake
https://github.com/cmake-js/cmake-js

Second:

If the (macro-based) cmake language is really 'archaic' (which sounds like a stretch to a C dev), we can extend cmake and build a JSON based configuration wrapper for node.

[kobalicek]

I really don't understand why cmake authors invented such a crappy language. I don't think its success is because of the language itself, it's because it provided pretty good detection and integration with IDEs and build systems. I personally use it, but because there is no alternative, I hate the syntax and spent a lot of time writing my cmake files.

I agree that there can be a better language, but we should note that there is no comparison between gyp JSON-like configuration and cmake scripting language. The latter provides try-compile, and all system introspection, find_program, find_library, etc. and the ability to compute depends in makefiles etc. with single language syntax; as opposed to mere JSON configuration for project properties and for detection, expects users to hand-role sh, csh, python, ps1, bat scripts for varied operating systems, which make the lives reasonably more dreadful.

XML based syntax is not popular either for programmable project configurations and environment detection; msbuild is a prime example, related discussion by unsatisfied users: dotnet/msbuild#16.

Although there might be a better way to describe the intent with all the introspection features cmake has to offer in compiler-neutral and OS-neutral manner; I think cmake had no choice but to coin their own scripting language while keeping all the dissimilarities under the hood (aka good architecture!). At this point, Google provides millions of search result if we go looking "how to do X in cmake", this is a must-have point to heed for rapid development.

If their API is extendable, we can probably implement it and add a parallel support for JavaScript which would bind to same internal implementations mimicking cmake scripting lang. From consumer standpoint, it would mean:

For cmake "list file", CMakeLists.txt:

# CMakeLists.txt
if(PROJECT_CMAKE_BUILD_TESTS)
  add_subdirectory(tests)
endif(PROJECT_CMAKE_BUILD_TESTS)

would become:

// CMakeLists.txt.js
if(cmake.PROJECT_CMAKE_BUILD_TESTS)
  cmake.add_subdirectory('tests');

(same for the cmake "script file", aka xxx.cmake file would become xxx.js or xxx.cmake.js)

IMO, JavaScript support can be added as an auxiliary addition to cmake scripting, as this is not super urgent and cmake syntax in its existing form is honestly not very hard to deal with and more like C compared to sh, python, ps1 etc. Although the JavaScript support would be most welcome, but that would be just a cherry on top.

[formula1]

Here. What I was doing before was going through every file and checking what needed to be changed. There hadn't been much that node-gyp does that javascript can't do so we can go through each and figure out what can be changed. Before I ended up trying to do it in regex because I got tired of rewriting manually, but honestly this would be much nicer crowd sourced than alone. Its pretty tedious. Heres a start

Heres another dirty one that I just did https://gist.github.com/formula1/eeba31bd42d9edbb94dd
This is completely untested and I am confident made some mistakes but its something to consider. Our options are

• Twiddle our thumbs for another year or three until an intern does it for free
• Move on to a new standard and limit support for old modules
• Start figuring out the best way to translate this thing
  • hack away and try to translate everything as literally as possible
  • Go through each file understand its purpose and rewrite it fluently
  • Find a method to automate it such as one of these

I'm assuming we'll twiddle our thumbs some more

Below is the easiest of what we can expect

gyp/gyp_main.py

#!/usr/bin/env python

# Copyright (c) 2009 Google Inc. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

import os
import sys

# Make sure we're using the version of pylib in this repo, not one installed
# elsewhere on the system.
sys.path.insert(0, os.path.join(os.path.dirname(sys.argv[0]), 'pylib'))
import gyp

if __name__ == '__main__':
  sys.exit(gyp.script_main())

gyp/gyp_main.js

#!/usr/bin/env nodejs

// Copyright (c) 2009 Google Inc. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

var os = require('os');
var path = require('path');

//below is not necessary
// sys.path.insert(0, path.join(path.dirname(process.argv[0]), 'pylib'))

var gyp = require('./jslib/gyp');

if(!module.parent){
  gyp.script_main()
  process.exit();
}

[jeduden]

Have a look at https://github.com/bkaradzic/genie (forked from premake). Produces project files for xcode + make + visual studio. Based on LUA. Binaries small.

[i5ting]

nice job with https://github.com/indutny/gyp.js

[john-yan]

Giving the v8 might dump gyp in the feature, continue to use gyp on nodejs really has some risk of incompatibility in feature v8 version, which gyp.js isn't going to solve. Instead of doing gyp.js, I wonder why not doing gn.js, which will solve both problems.

[refi64]

@john-yan Does GN even require Python? IIRC the binaries don't.

However, to build pretty much anything, you need Python, since GN only allows you to execute Python scripts.

[kobalicek]

WAF, GYP, GN, what would be next?

[john-yan]

@kirbyfan64 GN doesn't require python to run.