Workspace Gradle conversion #676

ialarmedalien · 2023-02-21T21:07:06Z

To make things easier, I intend to do the conversion in stages -- it can either be done in separate PRs or as commits in this PR.

Proposed steps:
1: recapitulate ant tasks. (this one)
2: replace local archives with jars from Maven.
3: trim down dependency list as some deps are supplied as part of other deps
4. ensure that test files are not being included in the ServiceJar build (as is currently the case with the ant-built jar)
5. split out tests into Mongo vs non-Mongo tests
6. debug test failures when junit is not set to fork each process
7. etc.

I've been writing up some documentation as I've gone along; I've included it as comments on this PR and will move it all into the appropriate doc files.

codecov · 2023-02-21T21:58:16Z

Codecov Report

Merging #676 (b22777a) into develop (cdf7705) will increase coverage by 0.08%.
Report is 8 commits behind head on develop.
The diff coverage is n/a.

❗ Current head b22777a differs from pull request most recent head bdd877f. Consider uploading reports for the commit bdd877f to get more accurate results

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #676      +/-   ##
=============================================
+ Coverage      86.70%   86.78%   +0.08%     
  Complexity      5399     5399              
=============================================
  Files            242      242              
  Lines          18288    18271      -17     
  Branches        2677     2674       -3     
=============================================
  Hits           15857    15857              
+ Misses          1962     1945      -17     
  Partials         469      469

Goal is to completely remove ant and make from the build and have a single build tool to rule them all, namely gradle Currently the build compiles but tests don't pass. Will also need a lot of doc updates.

…ile. Fix error in build.xml

ialarmedalien · 2023-02-23T13:45:00Z

.github/workflows/test.yml

-            wired_tiger: 'false'
-            ant_test: 'test_quick_coverage'
+        mongo: ['3.6.13', '3.6.23']
+        test_cmd: ['test_quick_coverage', 'NoLongTests']


test_quick_coverage is the old ant tests, NoLongTests is the new gradle tests

ialarmedalien · 2023-02-23T13:45:34Z

.github/workflows/test.yml

+          export MINIO_VER='2019-05-23T00-29-34Z'
+          export MONGO_VER='mongodb-linux-x86_64-${{matrix.mongo}}'
+          # 3.6.13 has wired_tiger set to 'false', 3.6.23 has wired_tiger 'true'
+          if [ "${{matrix.mongo}}" = "3.6.13" ]; then
+            export WIRED_TIGER="false"
+          else
+            export WIRED_TIGER="true"
+          fi
+          export ARANGODB_VER=3.9.1
+          export ARANGODB_V=39
+          export HS_COMMIT=08e18379817e16db920501b66ba62b66598f506c
+          export LOG_COMMIT=b549c557e3c519e0a55eadf7863a93db25cd6806
+          export SAMPLE_COMMIT=6813fb148e95db2b11db6eea04f4d1d45cbb7119


put all the version stuff at the top so it's easy to change

ialarmedalien · 2023-02-23T13:46:33Z

.github/workflows/test.yml

+      - name: build the war file (requires compileJava, javadoc)
+        run: gradle war --info


This step is to ensure that the gradle build processes are working properly

ialarmedalien · 2023-02-23T13:47:32Z

.github/workflows/test.yml

+      - name: upload jacoco report
+        uses: actions/upload-artifact@v3
+        if: ${{ matrix.test_cmd }} != 'test_quick_coverage'
+        with:
+          name: coverage${{matrix.test_cmd}}-${{matrix.mongo}}.exec
+          path: build/
+          if-no-files-found: warn


temporarily add a step to make the build directory available for download so that the test and coverage reports can be seen

ialarmedalien · 2023-02-23T13:53:24Z

build.gradle

+	}
+}
+
+tasks.withType(JavaCompile) {


Compilation

Run gradle compileJava to compile the source code excluding tests for the workspace, or gradle compileTestJava to compile everything. The results can be found in the directory build/classes/.

ialarmedalien · 2023-02-23T13:53:50Z

build.gradle

+	destinationDirectory = file("$dist")
+}
+
+task buildJars {


Build the Jar files

The client and service jars can be built individually or can be built at the same time using the task gradle buildJars.

build the client jar:

gradle clientJar

resulting file in dist/client/WorkspaceClient.jar

build the service jar:

gradle serviceJar

resulting file in dist/WorkspaceService.jar

to test:

run ant clean && ant compile
move or rename the resultant jar files in the dist directory so that they don't get overwritten by the gradle task
run gradle clean buildJars
download jarcomp_03.jar from https://activityworkshop.net/software/jarcomp/index.html

compare each of the jar files to the ant equivalent, e.g.

java -jar jarcomp_03.jar ant_WorkspaceClient.jar dist/client/WorkspaceClient.jar

Client jar is almost identical apart from the manifest file
Service jar: the ant service jar contains the test classes and their resources, plus a couple of package-info.class files

ialarmedalien · 2023-02-23T13:55:22Z

build.gradle

+}
+
+// by default, the files go into build/docs/javadoc
+tasks.javadoc {


Build the Javadocs

This task is run by the Gradle Java plugin. It compiles the main source set, adds classes from the kbase-common jar, and then generates javadocs for a select set of classes. The resulting documentation can be found in the directory build/docs/javadoc.

to test:

run ant clean
run ant javadoc, which will generate documentation in the directory docs/javadoc
run gradle clean javadoc
run diff -r docs build/docs
Note any differences

ialarmedalien · 2023-02-23T13:56:00Z

build.gradle

+			"https://www.javadoc.io/doc/com.fasterxml.jackson.core/jackson-core/2.9.9/",
+			"https://www.javadoc.io/doc/com.fasterxml.jackson.core/jackson-databind/2.9.9/"


updated URLs

ialarmedalien · 2023-02-23T13:56:52Z

build.gradle

+	}
+}
+
+tasks.war {


Build the war file

Building the war file involves compiling the main classes, generating documentation, packaging everything into the appropriate directories, and then generating the war file from that.

to test:

run ant clean && ant compile && ant buildwar
run gradle expandWar to expand the resultant file into its constituent parts, and rename the directory war_contents to something like ant_war_contents

run gradle clean buildWar
then expand the war file with gradle expandWar

The ant version of the war contains the test classes and dependencies.

❯ diff -r war_contents/WEB-INF/lib war_contents_ant/WEB-INF/lib Only in war_contents_ant/WEB-INF/lib: arangodb-java-driver-6.7.2.jar Only in war_contents_ant/WEB-INF/lib: byte-buddy-1.9.10.jar Only in war_contents_ant/WEB-INF/lib: byte-buddy-agent-1.9.10.jar Only in war_contents_ant/WEB-INF/lib: commons-collections-3.2.1.jar Only in war_contents_ant/WEB-INF/lib: commons-lang-2.4.jar Only in war_contents_ant/WEB-INF/lib: equalsverifier-3.1.10.jar Only in war_contents_ant/WEB-INF/lib: java-semver-0.9.0.jar Only in war_contents_ant/WEB-INF/lib: kbase-auth2test-0.2.4.jar Only in war_contents_ant/WEB-INF/lib: mockito-core-3.0.0.jar Only in war_contents_ant/WEB-INF/lib: objenesis-2.6.jar Only in war_contents_ant/WEB-INF/lib: velocity-1.7.jar Only in war_contents_ant/WEB-INF/lib: velocypack-2.4.0.jar ❯ diff -r war_contents/WEB-INF/classes/us war_contents_ant/WEB-INF/classes/us Only in war_contents_ant/WEB-INF/classes/us/kbase/common: test Only in war_contents_ant/WEB-INF/classes/us/kbase/typedobj/db: test Only in war_contents_ant/WEB-INF/classes/us/kbase/typedobj: test Only in war_contents_ant/WEB-INF/classes/us/kbase/workspace/listener: package-info.class Only in war_contents_ant/WEB-INF/classes/us/kbase/workspace/modules: package-info.class Only in war_contents_ant/WEB-INF/classes/us/kbase/workspace: test

ialarmedalien · 2023-02-23T13:57:47Z

build.gradle

+}
+
+// test reports are saved to build/reports/tests/<testName>
+tasks.withType(Test) {


Running Tests

Currently there is a single test task that runs all tests except the *LongTest tests. To execute it, run gradle testNoLongTests (or gradle coverageNoLongTests to generate coverage files). HTML test results are saved to the directory build/reports/tests/<testName> and coverage data (in CSV, XML, and HTML formats) goes to build/reports/jacoco/<taskName>.

to test:

download one of the coverageNoLongTests-3.6.n3 archives from GitHub Actions to see an example of the resulting build directory and the test results/coverage.

NOTE

Gradle is currently set to fork for each test run as there are two tests that fail if that setting is NOT on. To be continued!

Assuming I'm reading this correctly we'll def want a target that runs all the tests, long tests included

I have it set up so that in GitHub Actions, we would run the long tests, the mongo tests, and the quick tests (all tests minus long tests and mongo tests) in parallel, and then combine the coverage data. Locally it would be possible to run any of those sets on their own or all tests.

ialarmedalien · 2023-02-23T13:58:54Z

build.gradle

+	executionData(testNoLongTests)
+}
+
+task createScript {


Create the CLI script

Run the gradle command gradle createScript to generate the CLI script.

to test:

run ant clean && ant script
rename the resulting file (update_workspace_database_schema)
run gradle clean createScript

replace : with \n and sort lines in both files:

sed 's/:/\n/g' update_workspace_database_schema_ant | sort > script_ant.txt sed 's/:/\n/g' update_workspace_database_schema | sort > script_gradle.txt

diff the two files

diff script_ant.txt script_gradle.txt 1d0 <

ialarmedalien · 2023-02-23T14:09:58Z

build.xml

@@ -242,7 +242,8 @@
    <javadoc access="protected" author="false" classpathref="compile.classpath"
      destdir="${doc}" nodeprecated="false" nodeprecatedlist="false"
      noindex="false" nonavbar="false" notree="false"
-      source="11" splitindex="true" use="true" version="true">
+      source="11" splitindex="true" use="true" Windowtitle="Workspace 0.14.1 API"


add in the window title to make the output from this command more similar to the settings that that the gradle javadoc task uses.

MrCreosote · 2023-02-24T19:13:46Z

I'd like to review at a point when build.xml and Makefile (and any other files that are no longer needed based on the elimination of those files) are gone and gradle handles all the functionality from those files (maybe with some simplifications / deletions for things we no longer need), so that looks like through item 3. After that, improvements and optimizations can be handled piecemeal.

The other thing that, IMO, needs to be in the first PR is that test classes and individual tests need to be runnable in Eclipse (or whatever). Right now that's done by maintaining the .classpath file with paths into the Jars repo, so that's no longer workable. Ideally the IDE would be able to Just Do The Right Thing and run a gradle build if needed when tests are triggered, but I think the minimum is that if a gradle build has been run the tests should run from the IDE.

2: replace local archives with jars from Maven.
3: trim down dependency list as some deps are supplied as part of other deps

I already did this in 6bde7be ?

ialarmedalien · 2023-04-17T13:33:32Z

@MrCreosote which Makefile directives are still used (so I know what to replace and what can be dumped)?

MrCreosote · 2023-04-17T18:27:09Z

I think the deploy endpoints can all go, as well as the undeploy endpoint. The rest need to stay (or at least their functionality needs to stay) but they might be able to be simplified, possibly?

Obviously test that the docker image can build and run without those endpoints though.

In particular any /kb/deployment or /kb/runtime stuff should go

ialarmedalien · 2023-04-17T18:38:09Z

build.gradle

+	useJUnit()
+	systemProperty "test.cfg", testConfig
+	maxHeapSize = "3G"
+	testLogging {


test logging

add gradle to dockerfile use gradle for GHA tests/builds remove unused server admin scripts

add in workflow caching adding minio since it is not always available from minio server fix incorrect if statements

ialarmedalien force-pushed the dev-gradle branch from 57a9e4c to 081c3ea Compare February 22, 2023 01:01

MrCreosote and others added 2 commits February 22, 2023 06:13

WIP on converting from ant/make to gradle

6bde7be

Goal is to completely remove ant and make from the build and have a single build tool to rule them all, namely gradle Currently the build compiles but tests don't pass. Will also need a lot of doc updates.

Add standard tasks--jars, wars, javadoc, tests, coverage--to gradle f…

22ee5c3

…ile. Fix error in build.xml

ialarmedalien force-pushed the dev-gradle branch from b87b5b7 to 22ee5c3 Compare February 22, 2023 14:33

ialarmedalien self-assigned this Feb 22, 2023

ialarmedalien commented Feb 23, 2023

View reviewed changes

ialarmedalien requested a review from MrCreosote February 24, 2023 16:19

ialarmedalien and others added 2 commits February 24, 2023 11:48

Merge branch 'develop' into dev-gradle

a97d823

remove redundant jars, use downloadJars task to get kbase jars

da6ec77

ialarmedalien force-pushed the dev-gradle branch from 940c1a8 to da6ec77 Compare April 16, 2023 17:57

ialarmedalien and others added 2 commits April 16, 2023 12:43

add jna

0fa79a8

Merge branch 'develop' into dev-gradle

d9a55e7

ialarmedalien commented Apr 17, 2023

View reviewed changes

build.gradle

useJUnit()

systemProperty "test.cfg", testConfig

maxHeapSize = "3G"

testLogging {

Copy link

Collaborator Author

ialarmedalien Apr 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test logging

removing commands from Makefile that are now run by gradle

9442ea3

add gradle to dockerfile use gradle for GHA tests/builds remove unused server admin scripts

ialarmedalien force-pushed the dev-gradle branch from 6bc700c to 9442ea3 Compare May 4, 2023 14:32

ialarmedalien marked this pull request as ready for review August 16, 2023 13:29

ialarmedalien added 2 commits September 6, 2023 08:00

Try a panoply of test type and OS combinations

4d9230c

add in workflow caching adding minio since it is not always available from minio server fix incorrect if statements

reinstitute tests

cefbfec

ialarmedalien force-pushed the dev-gradle branch from 8ef4bc1 to cefbfec Compare September 6, 2023 15:03

ialarmedalien added 4 commits September 6, 2023 09:56

try moving set up to separate workflow job

e1de684

fix workflow call

6f552e5

fix error

c4fb794

fix error

bdd877f

MrCreosote mentioned this pull request Feb 18, 2024

[WIP] Convert to Gradle build #716

Closed

ialarmedalien closed this Apr 23, 2024

MrCreosote deleted the dev-gradle branch May 10, 2024 17:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Workspace Gradle conversion #676

Workspace Gradle conversion #676

ialarmedalien commented Feb 21, 2023 •

edited

Loading

codecov bot commented Feb 21, 2023 •

edited

Loading

ialarmedalien Feb 23, 2023

ialarmedalien Feb 23, 2023

ialarmedalien Feb 23, 2023

ialarmedalien Feb 23, 2023

ialarmedalien Feb 23, 2023

ialarmedalien Feb 23, 2023 •

edited

Loading

ialarmedalien Feb 23, 2023

ialarmedalien Feb 23, 2023

ialarmedalien Feb 23, 2023

ialarmedalien Feb 23, 2023

MrCreosote Feb 24, 2023

ialarmedalien Feb 24, 2023

ialarmedalien Feb 23, 2023

ialarmedalien Feb 23, 2023

MrCreosote commented Feb 24, 2023 •

edited

Loading

ialarmedalien commented Apr 17, 2023

MrCreosote commented Apr 17, 2023

ialarmedalien Apr 17, 2023

		- name: build the war file (requires compileJava, javadoc)
		run: gradle war --info

		"https://www.javadoc.io/doc/com.fasterxml.jackson.core/jackson-core/2.9.9/",
		"https://www.javadoc.io/doc/com.fasterxml.jackson.core/jackson-databind/2.9.9/"

Workspace Gradle conversion #676

Workspace Gradle conversion #676

Conversation

ialarmedalien commented Feb 21, 2023 • edited Loading

codecov bot commented Feb 21, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Compilation

ialarmedalien Feb 23, 2023 • edited Loading

Choose a reason for hiding this comment

Build the Jar files

to test:

Choose a reason for hiding this comment

Build the Javadocs

to test:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Build the war file

to test:

Choose a reason for hiding this comment

Running Tests

to test:

NOTE

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Create the CLI script

to test:

Choose a reason for hiding this comment

MrCreosote commented Feb 24, 2023 • edited Loading

ialarmedalien commented Apr 17, 2023

MrCreosote commented Apr 17, 2023

Choose a reason for hiding this comment

ialarmedalien commented Feb 21, 2023 •

edited

Loading

codecov bot commented Feb 21, 2023 •

edited

Loading

ialarmedalien Feb 23, 2023 •

edited

Loading

MrCreosote commented Feb 24, 2023 •

edited

Loading