Skip to content

ARROW-1379: [Java] adding maven-dependency-plugin and fixing all reported dependency errors #978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
antonymayi wants to merge 4 commits into from
Closed

ARROW-1379: [Java] adding maven-dependency-plugin and fixing all reported dependency errors #978

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
dc833bb
adding maven-dependency-plugin and fixing all reported dependency errors
Aug 19, 2017
8cbfe5f
maven-dependency-plugin: ignoring dependencies of generated sources i…
antonymayi Aug 20, 2017
ec72717
removing unused apache.commons.lang3, fixing pom
antonymayi Aug 21, 2017
d7f081e
moving `copy-flatc` to initialize phase and `analyze` execution to
antonymayi Aug 22, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions java/format/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,20 +23,16 @@
<name>Arrow Format</name>

<properties>
<fbs.version>1.2.0-3f79e055</fbs.version>
<flatc.download.skip>false</flatc.download.skip>
<flatc.executable>${project.build.directory}/flatc-${os.detected.classifier}-${fbs.version}.exe</flatc.executable>
<flatc.generated.files>${project.build.directory}/generated-sources/flatc</flatc.generated.files>
<maven-compiler-plugin.version>3.3</maven-compiler-plugin.version>

@jacques-n jacques-n Aug 21, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why remove

@antonymayi antonymayi Aug 21, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any references to this property anywhere so thought this is some leftover garbage?

<maven-dependency-plugin.version>2.10</maven-dependency-plugin.version>

@jacques-n jacques-n Aug 21, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here. I think we added both of these previously since some versions of mvn 3 caused issues without these.

@antonymayi antonymayi Aug 21, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re the dependency plugin - I defined the dependency plugin globals in the pluginManagement in the parent pom, this way it gets propagate to all instances of this plugin in child poms.

re the compiler - see my separate answer above. if it is desirable to stick it to particular version I would recommend putting it into the pluginManagement section of the parent similarly to what I did with the dependency plugin.

@laurentgo laurentgo Aug 21, 2017 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it already is in pluginManagement (as I updated it for Java9 support) in the parent pom

@antonymayi antonymayi Aug 21, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I knew I saw it somewhere

<os-maven-plugin.version>1.5.0.Final</os-maven-plugin.version>
</properties>

<dependencies>
<dependency>
<groupId>com.vlkan</groupId>
<artifactId>flatbuffers</artifactId>
<version>${fbs.version}</version>
</dependency>
</dependencies>

Expand All @@ -54,7 +50,6 @@
<plugin> <!-- download the flatbuffer compiler -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>${maven-dependency-plugin.version}</version>
<executions>
<execution>
<id>copy-flatc</id>
Expand Down
25 changes: 20 additions & 5 deletions java/memory/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,33 @@
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
<version>3.0.2</version>
</dependency>

<dependency>
<groupId>com.carrotsearch</groupId>
<artifactId>hppc</artifactId>
<version>0.7.2</version>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-buffer</artifactId>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-common</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
</dependencies>


<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
</plugin>
</plugins>
</build>


Expand Down
172 changes: 113 additions & 59 deletions java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,10 @@
<dep.junit.version>4.11</dep.junit.version>
<dep.slf4j.version>1.7.25</dep.slf4j.version>
<dep.guava.version>18.0</dep.guava.version>
<dep.netty.version>4.0.49.Final</dep.netty.version>
<fbs.version>1.2.0-3f79e055</fbs.version>
<forkCount>2</forkCount>
<jackson.version>2.7.9</jackson.version>
<dep.jackson.version>2.7.9</dep.jackson.version>
<hadoop.version>2.7.1</hadoop.version>

@jacques-n jacques-n Aug 21, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missed "dep." here

@antonymayi antonymayi Aug 21, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you referring to the hadoop.version? I didn't touch that one actually. I was tempted to prepend it with dep. to make it consistent with the rest but thought it is out of scope of my PR... happy to do that though.

@antonymayi antonymayi Aug 21, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hadoop.version now renamed to dep.hadoop.version. this still leaves the fbs.version in its original form without the prefix, not sure if that should be renamed too but it is kind of special one so I am leaving it

<checkstyle.failOnViolation>false</checkstyle.failOnViolation>
</properties>
Expand Down Expand Up @@ -271,55 +273,70 @@
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>2.17</version>
<dependencies>
<dependency>
<groupId>com.puppycrawl.tools</groupId>
<artifactId>checkstyle</artifactId>
<version>6.19</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${dep.guava.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>1.7.5</version>
</dependency>
</dependencies>
<executions>
<execution>
<id>validate</id>
<phase>validate</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
<configuration>
<configLocation>google_checks.xml</configLocation>
<encoding>UTF-8</encoding>
<consoleOutput>true</consoleOutput>
<failsOnError>${checkstyle.failOnViolation}</failsOnError>
<failOnViolation>${checkstyle.failOnViolation}</failOnViolation>
<violationSeverity>warning</violationSeverity>
<format>xml</format>
<format>html</format>
<outputFile>${project.build.directory}/test/checkstyle-errors.xml</outputFile>
<linkXRef>false</linkXRef>
</configuration>
</plugin>
</plugins>

<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>2.17</version>
<dependencies>
<dependency>
<groupId>com.puppycrawl.tools</groupId>
<artifactId>checkstyle</artifactId>
<version>6.19</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${dep.guava.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>1.7.5</version>
</dependency>
</dependencies>
<artifactId>maven-dependency-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>validate</id>
<phase>validate</phase>
<id>analyze</id>

@jacques-n jacques-n Aug 21, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please describe the purpose for all the checkstyle changes

@antonymayi antonymayi Aug 21, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually no changes to the checkstyle plugin configuration. this is just the git diff making it to look very confusing as it is mixing lines from different part in the diff report... these changes actually belong to the dependency plugin, not the checkstyle. see the full pom.

@laurentgo laurentgo Aug 21, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's any reason not to run the dependency plugin in the parent pom, so maybe the plugin could be configured in the parent (and inherited by children) instead of each child declaring the plugin?

@antonymayi antonymayi Aug 21, 2017 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that was actually my very first approach as well but the arrow-format is already using the maven-dependency-plugin and its goal copy to fetch the flatc.exe. if I run the dependency plugin in the parent pom (with the analyze-only goal) then it shadows the copy goal instance in the arrow-format child.

I was trying to configure it so that the arrow-format is just extending the configuration adding/merging the copy goal execution with the inherited analyze-only global execution but had no luck with any sensible approach (it is only possible to reduce/activate some executions in child pom by overriding its phase to/from none but not adding completely new one; so the only possible way would be defining the plugin in the root pom with both analyze-only execution (for global dependency enforcements as per tis subj) and also the copy execution (ie with phase being none so that it can be used in the arrow-format child (setting the phase only in that module to generate-source) - but specifying the copy-flatc execution in the parent pom would be ugly, hence my final approach with defining the analyze-only execution in pluginManagement and letting each module to instantiate (and override) it itself).

so if you know a way to move it to the parent while having arrow-format still pulling the flatc.exe correctly that would indeed be my preferred way also...

@laurentgo laurentgo Aug 21, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect a maven bug here: when adding an execution for dependency plugin in the parent pom, the execution copy-flatc is moved after script-chmod. The merge is okay, but the order is wrong (Maven is supposed to use declaration order).

One way to workaround this would be move copy-flatc to an earlier phase like initialize.

@antonymayi antonymayi Aug 22, 2017 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this approach - having the analyze-only execution directly in the parent pom is much better and moving the copy-flatc to initialize looks almost like the right phase anyway.

so this is now in place in this PR.

<goals>
<goal>check</goal>
<goal>analyze-only</goal>
</goals>
<configuration>
<ignoreNonCompile>true</ignoreNonCompile>
<failOnWarning>true</failOnWarning>
</configuration>
</execution>
</executions>
<configuration>
<configLocation>google_checks.xml</configLocation>
<encoding>UTF-8</encoding>
<consoleOutput>true</consoleOutput>
<failsOnError>${checkstyle.failOnViolation}</failsOnError>
<failOnViolation>${checkstyle.failOnViolation}</failOnViolation>
<violationSeverity>warning</violationSeverity>
<format>xml</format>
<format>html</format>
<outputFile>${project.build.directory}/test/checkstyle-errors.xml</outputFile>
<linkXRef>false</linkXRef>
</configuration>
</plugin>


</plugins>
<pluginManagement>

<plugins>
<plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
Expand Down Expand Up @@ -449,26 +466,63 @@
</plugins>
</pluginManagement>
</build>
<dependencies>

<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
<version>4.0.49.Final</version>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${dep.guava.version}</version>
</dependency>

<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${dep.slf4j.version}</version>
</dependency>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.vlkan</groupId>
<artifactId>flatbuffers</artifactId>
<version>${fbs.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${dep.guava.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
<version>${dep.netty.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-buffer</artifactId>
<version>${dep.netty.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-common</artifactId>
<version>${dep.netty.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>${dep.jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${dep.jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${dep.jackson.version}</version>
</dependency>
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
<version>3.0.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${dep.slf4j.version}</version>
</dependency>
</dependencies>
</dependencyManagement>

<dependencies>
<!-- Test Dependencies -->
<dependency>
<groupId>org.slf4j</groupId>
Expand Down
26 changes: 18 additions & 8 deletions java/tools/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,6 @@
<name>Arrow Tools</name>

<dependencies>
<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-format</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-memory</artifactId>
Expand All @@ -36,9 +31,8 @@
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.6</version>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
Expand All @@ -51,6 +45,18 @@
<version>1.2.3</version>
<scope>run</scope>

@laurentgo laurentgo Aug 21, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you fix the scope too? (it's runtime, not run)

@antonymayi antonymayi Aug 21, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
</dependencies>

<build>
Expand All @@ -73,6 +79,10 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
</plugin>
</plugins>
</build>

Expand Down
44 changes: 42 additions & 2 deletions java/vector/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,15 +36,17 @@
<artifactId>joda-time</artifactId>
<version>2.9.9</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.carrotsearch</groupId>
Expand All @@ -61,6 +63,30 @@
<artifactId>commons-codec</artifactId>
<version>1.10</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-buffer</artifactId>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-common</artifactId>
</dependency>
<dependency>
<groupId>com.vlkan</groupId>
<artifactId>flatbuffers</artifactId>
</dependency>
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
</dependencies>

<pluginRepositories>
Expand Down Expand Up @@ -129,6 +155,20 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>analyze</id>
<configuration>
<ignoredUnusedDeclaredDependencies>
<ignoredUnusedDeclaredDependency>org.apache.commons:commons-lang3</ignoredUnusedDeclaredDependency>

@laurentgo laurentgo Aug 21, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead, let's fix vector/src/main/codegen//includes/vv_imports.ftl to remove the unnecessary import

@antonymayi antonymayi Aug 21, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 nice catch, done.

</ignoredUnusedDeclaredDependencies>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<pluginManagement>
<plugins>
Expand Down