Skip to content

Commit a0734dc

Browse files
authored
Make it possible to run 'make benchmark-%' using jvm 17+ (apache#823)
When using jvm 17+ spark needs extra jvm args to avoid getting errors like ``` [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.2.0:java (default-cli) on project comet-spark-spark3.4_2.12: An exception occurred while executing the Java class. class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x35b75242) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x35b75242 -> [Help 1] ``` These args are already present inside the main pom.xml. To avoid duplicating the args we using maven to extract them. In order avoid slowing down the Makefile the args are defined as a function and therefore only evaluated when they are needed.
1 parent 9d4afc1 commit a0734dc

1 file changed

Lines changed: 6 additions & 2 deletions

File tree

Makefile

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@
1717

1818
.PHONY: all core jvm test clean release-linux release bench
1919

20+
define spark_jvm_17_extra_args
21+
$(shell ./mvnw help:evaluate -Dexpression=extraJavaTestArgs | grep -v '\[')
22+
endef
23+
2024
all: core jvm
2125

2226
core:
@@ -80,7 +84,7 @@ release:
8084
release-nogit:
8185
cd native && RUSTFLAGS="-Ctarget-cpu=native" cargo build --release
8286
./mvnw install -Prelease -DskipTests $(PROFILES) -Dmaven.gitcommitid.skip=true
83-
benchmark-%: clean release
84-
cd spark && COMET_CONF_DIR=$(shell pwd)/conf MAVEN_OPTS='-Xmx20g' ../mvnw exec:java -Dexec.mainClass="$*" -Dexec.classpathScope="test" -Dexec.cleanupDaemonThreads="false" -Dexec.args="$(filter-out $@,$(MAKECMDGOALS))" $(PROFILES)
87+
benchmark-%: release
88+
cd spark && COMET_CONF_DIR=$(shell pwd)/conf MAVEN_OPTS='-Xmx20g ${call spark_jvm_17_extra_args}' ../mvnw exec:java -Dexec.mainClass="$*" -Dexec.classpathScope="test" -Dexec.cleanupDaemonThreads="false" -Dexec.args="$(filter-out $@,$(MAKECMDGOALS))" $(PROFILES)
8589
.DEFAULT:
8690
@: # ignore arguments provided to benchmarks e.g. "make benchmark-foo -- --bar", we do not want to treat "--bar" as target

0 commit comments

Comments
 (0)