Frameworks with symlinks inside are not uploaded correctly [zlib] [common-crypto]

[c0diq]

Issue

Upload fails for certain types of frameworks where the binary inside the framework is a symlink to a version inside the framework.
For example, Starscream for Mac (https://github.com/daltoniam/Starscream). Here's a screenshot:
http://cloud.plutinosoft.com/oKcN

$ ls -la Carthage/Build/Mac/Starscream.framework
total 48
drwxr-xr-x   8 srebaud  staff   272 Dec 14 10:57 .
drwxr-xr-x  21 srebaud  staff   714 Dec 14 09:52 ..
-rw-r--r--@  1 srebaud  staff  6148 Dec 14 10:57 .DS_Store
lrwxr-xr-x   1 srebaud  staff    24 Dec 14 09:52 Headers -> Versions/Current/Headers
lrwxr-xr-x   1 srebaud  staff    24 Dec 14 09:52 Modules -> Versions/Current/Modules
lrwxr-xr-x   1 srebaud  staff    26 Dec 14 09:52 Resources -> Versions/Current/Resources
lrwxr-xr-x   1 srebaud  staff    27 Dec 14 09:52 Starscream -> Versions/Current/Starscream
drwxr-xr-x   5 srebaud  staff   170 Dec 14 10:57 Versions

Steps which explain the enhancement

1. Add github "daltoniam/Starscream" ~> 3.0.3 to your Cartfile
2. carthage bootstrap --platform macos
3. Use a local in your Romefile
4. rome upload

Current and suggested behavior
The Starscream.framework is not published in the local folder.

Rome version: 0.13.0.33

OS and version: ('Darwin', 'Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64', 'x86_64')

[tmspzz]

@c0diq Thanks for reporting. Can you paste the relevant part of your Romefile ?

[c0diq]

Romefile

[Cache]
    local = ~/dropbox/foo

Cartfile

github "daltoniam/Starscream" ~> 3.0.3

Cartfile.resolved

github "daltoniam/Starscream" "3.0.3"
github "daltoniam/common-crypto-spm" "1.1.0"
github "daltoniam/zlib-spm" "1.1.0"

Starscream dependencies include these two projects which don't have frameworks but are packaged with spm. When trying to import Starscream, I would get the following error:

missing required modules: 'SSCZLib', 'SSCommonCrypto

which both are defined in Starscream modulemap.
Eventually, somehow, Carthage stopped creating Mac frameworks with internal symlinks and Rome was able to upload it to the cache, but that still didn't solve the error.

I am not sure how these two dependencies get linked into Starscream but it appears that when I build manually Starscream with Carthage, the error goes away. If I use Rome to populate Carthage/Build/Mac, the error reappears.

The issue was reported at socketio/socket.io-client-swift#798 but the solution is not possible when using pre-built binaries.

[tmspzz]

@c0diq Unfortunately this is not a bug.

common-crypto-spm and zlib-spm are not frameworks. They are just llvm module.modulemap wrappers for C libraries so that they can imported into Swift.

If you look closely to the output of carthage bootstrap you will see that carthage does not build them. The author of Starscream is (ab)using carthage just to have the repos checked out.

I have the same issue with zlib in a work project. Fortunately on iOS since iOS > 10, zlib is now exposed to the system as a swift module so the module.modulemap dance is not needed anymore. I suspect that this is the case on MacOS too.

If you can't work around this, no matter what you do, Starscream can't be packaged as a binary dependency.

As a desperate attempt you can try to edit Versions/Current/Modules/module.modulemap after downloading the Starscream binary and patch it like so:

framework module Starscream {
  umbrella header "Starscream.h"

  export *
  module * { export * }
}

module Starscream.Swift {
    header "Starscream-Swift.h"
    requires objc
}

module SSCZLib [system] {
    header "include.h"
    link "z"
    export *
}
module SSCommonCrypto [system] {
    header "include.h"
    export *
}

Also you have to make sure that the header file include.h from Starscream/zlib is copied into the Versions/Current/Headers folder of Starscream's framework bundle .

[tmspzz]

@c0diq is this answer satisfactory for you? I'd like to close the issue if you don't mind.

[c0diq]

Your fix made it work. What I don't understand is why Carthage (Xcode) doesn't set the modulemap like you suggest.
Actually after retrying without your fix, it works too. I am so confused :-(

[tmspzz]

Yes this is a big problem and you're not alone. I am sorry but I can't assist you further as I am also affected by the same issue. If you find a solution please let me know :(

[c0diq]

I'd like to reopen this because I think there's still an issue. The framework that Rome packages and uploads to cache is not what Carthage built when it contains symlinks. If you run
carthage build archive you can see that the zip file generated for the (OSX) framework contains symlinks (Starscream). However the version uploaded to Rome cache rome upload does not.

[c0diq]

Ooh it appears to be a known issue #112! Great.

[tmspzz]

@c0diq #112 is closed now. Please let me know if you find any issues with https://github.com/blender/Rome/releases/tag/v0.14.0.38

[Max0u]

Hi Blender, sorry it's still me.

I'm on 0.17.2.50 and still get problem with Starscream.

Here is my Romefile :

cache: # required
  local: ~/Library/Caches/Rome
repositoryMap: # optional
- Moya:
  - name: Moya
  - name: RxMoya
  - name: ReactiveMoya
- RxSwift:
  - name: RxSwift
  - name: RxCocoa
  - name: RxTest
  - name: RxBlocking
- RxDataSources:
  - name: RxDataSources
  - name: Differentiator
- GoogleSignIn-iOS:
  - name: GoogleSignIn
  - name: GoogleToolboxForMac
  - name: GoogleUtilities
  - name: GTMSessionFetcher
- IHKeyboardAvoiding:
  - name: KeyboardAvoiding
- swift-protobuf:
  - name: Protobuf
  - name: SwiftProtobuf
- pusher-websocket-swift:
  - name: PusherSwift
  - name: Reachability
- swift-sodium:
  - name: Sodium
- twilio-video-ios:
  - name: TwilioVideo
- realm-cocoa:
  - name: Realm
  - name: RealmSwift
- CocoaLumberjack:
  - name: CocoaLumberjack
  - name: CocoaLumberjackSwift
- R.swift.Library:
  - name: Rswift
- FirebaseAnalyticsBinary:
  - name: Firebase
  - name: FirebaseAnalytics
  - name: nanopb
- FirebasePerformanceBinary:
  - name: FirebaseCore
  - name: FirebaseCoreDiagnostics
  - name: FirebasePerformance
  - name: GoogleAppMeasurement
- FirebaseMessagingBinary:
  - name: FirebaseInstanceID
  - name: FirebaseMessaging
- Starscream:
  - name: Starscream

Once I upload then download I got this :

missing required modules: 'SSCZLib', 'SSCommonCrypto