Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore(dist): improve validate-release.sh #329

Merged
merged 5 commits into from
Mar 11, 2024
Merged

chore(dist): improve validate-release.sh #329

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f8c3e37
Update validate-release.sh
javeme Feb 25, 2024
1b2c963
Update validate-release.sh
javeme Feb 25, 2024
8f30986
tiny improve
javeme Mar 3, 2024
78a3ec7
revert pushd quota
javeme Mar 3, 2024
7a34eb7
Update validate-release.sh
imbajin Mar 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 90 additions & 82 deletions dist/validate-release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,38 +21,45 @@
# 3. Compile the source package & run server & toolchain
# 4. Run server & toolchain in binary package

URL_PREFIX="https://dist.apache.org/repos/dist/dev/incubator/hugegraph/"
# if we don't want to exit after '|', remove "-o pipefail"
set -euxo pipefail
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-o pipefail will exit when COUNT=$(grep -E "$CATEGORY_X" LICENSE NOTICE | wc -l)

Start to check the package content: apache-hugegraph-ai-incubating-1.3.0-src
+ [[ ! -f LICENSE ]]
+ [[ ! -f NOTICE ]]
+ [[ ! -f DISCLAIMER ]]
++ grep -E '\bGPL|\bLGPL|Sleepycat License|BSD-4-Clause|\bBCL\b|JSR-275|Amazon Software License|\bRSAL\b|\bQPL\b|\bSSPL|\bCPOL|\bNPL1|Creative Commons Non-Commercial' LICENSE NOTICE
++ wc -l
+ COUNT='       0'


# release version (input by committer)
RELEASE_VERSION=$1
JAVA_VERSION=$2
RELEASE_VERSION=$1 # like 1.2.0
JAVA_VERSION=$2 # like 11
USER=$3

# this URL is only valid during the release process
SVN_URL_PREFIX="https://dist.apache.org/repos/dist/dev/incubator/hugegraph"

# git release branch (check it carefully)
#GIT_BRANCH="release-${RELEASE_VERSION}"

RELEASE_VERSION=${RELEASE_VERSION:?"Please input the release version behind script"}
RELEASE_VERSION=${RELEASE_VERSION:?"Please input the release version, like 1.2.0"}
imbajin marked this conversation as resolved.
Show resolved Hide resolved
USER=${USER:-"imbajin"}
WORK_DIR=$(
cd "$(dirname "$0")" || exit
cd "$(dirname "$0")"
pwd
)

cd "${WORK_DIR}" || exit
Comment on lines -34 to -38
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we remove || exit, when the input/exec path is not as expected, some files may be deleted or move to wrong space?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can remove all the || exit since added set -e

Copy link
Member

@imbajin imbajin Mar 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can remove all the || exit since added set -e

Get it, thanks, I check & test the usage about it, find some points we need to care (when use set -e):

set -e
ls not-exist | echo 2 # set -e won't influence the '|"(pipe)
echo "Not Exit"

but use set -eo pipefail we could also solve the problem:

set -eo pipefail
ls not-exist | echo 2  # exit after "echo 2"
echo "Not Exit"

And set -e will influence the whole script, if some cmd return non-zero but we don't want it exit, could use set +e & set -e to deactivate it temporarily~

set -e

.....
set +e
cmd_that_shouldnt_exit()
set -e

# Or we could use "cmd || ture"
cmd_that_shouldnt_exit() || ture # bypass -e
....

And in our case, use set -euxo pipefail seems better? (Remember to Test it)

cc @VGalaxies @Pengzna @liuxiaocs7 @msgui

Refer: https://www.ruanyifeng.com/blog/2017/11/bash-set.html (recommend)

cd "${WORK_DIR}"
echo "Current work dir: $(pwd)"

################################
# Step 1: Download SVN Sources #
################################
rm -rf "$WORK_DIR"/dist/"$RELEASE_VERSION"
svn co ${URL_PREFIX}/"$RELEASE_VERSION" "$WORK_DIR"/dist/"$RELEASE_VERSION"
rm -rf "${WORK_DIR}/dist/${RELEASE_VERSION}"
mkdir -p "${WORK_DIR}/dist/${RELEASE_VERSION}"
cd "${WORK_DIR}/dist/${RELEASE_VERSION}"
svn co "${SVN_URL_PREFIX}/${RELEASE_VERSION}" .

##################################################
# Step 2: Check Environment & Import Public Keys #
##################################################
cd "$WORK_DIR"/dist/"$RELEASE_VERSION" || exit
shasum --version 1>/dev/null
gpg --version 1>/dev/null
imbajin marked this conversation as resolved.
Show resolved Hide resolved

shasum --version 1>/dev/null || exit
gpg --version 1>/dev/null || exit

wget https://downloads.apache.org/incubator/hugegraph/KEYS || exit
wget https://downloads.apache.org/incubator/hugegraph/KEYS
echo "Import KEYS:" && gpg --import KEYS
# TODO: how to trust all public keys in gpg list, currently only trust the first one
echo -e "5\ny\n" | gpg --batch --command-fd 0 --edit-key $USER trust
Expand All @@ -65,18 +72,18 @@ done
########################################
# Step 3: Check SHA512 & GPG Signature #
########################################
cd "$WORK_DIR"/dist/"$RELEASE_VERSION" || exit
cd "${WORK_DIR}/dist/${RELEASE_VERSION}"

for i in *.tar.gz; do
echo "$i"
shasum -a 512 --check "$i".sha512 || exit
eval gpg "${GPG_OPT}" --verify "$i".asc "$i" || exit
shasum -a 512 --check "$i".sha512
eval gpg "${GPG_OPT}" --verify "$i".asc "$i"
done

####################################
# Step 4: Validate Source Packages #
####################################
cd "$WORK_DIR"/dist/"$RELEASE_VERSION" || exit
cd "${WORK_DIR}/dist/${RELEASE_VERSION}"

CATEGORY_X="\bGPL|\bLGPL|Sleepycat License|BSD-4-Clause|\bBCL\b|JSR-275|Amazon Software License|\bRSAL\b|\bQPL\b|\bSSPL|\bCPOL|\bNPL1|Creative Commons Non-Commercial"
CATEGORY_B="\bCDDL1|\bCPL|\bEPL|\bIPL|\bMPL|\bSPL|OSL-3.0|UnRAR License|Erlang Public License|\bOFL\b|Ubuntu Font License Version 1.0|IPA Font License Agreement v1.0|EPL2.0|CC-BY"
Expand All @@ -89,9 +96,10 @@ for i in *src.tar.gz; do
echo "The package name $i should include incubating" && exit 1
fi

tar xzvf "$i" || exit
pushd "$(basename "$i" .tar.gz)" || exit
echo "Start to check the package content: $(basename "$i" .tar.gz)"
tar -xzvf "$i"
imbajin marked this conversation as resolved.
Show resolved Hide resolved
MODULE_DIR=$(basename "$i" .tar.gz)
pushd ${MODULE_DIR}
echo "Start to check the package content: ${MODULE_DIR}"

# 4.2: check the directory include "NOTICE" and "LICENSE" file and "DISCLAIMER" file
if [[ ! -f "LICENSE" ]]; then
Expand Down Expand Up @@ -147,68 +155,68 @@ for i in *src.tar.gz; do
# 4.8: test compile the packages
if [[ $JAVA_VERSION == 8 && "$i" =~ "computer" ]]; then
echo "skip computer module in java8"
popd || exit
popd
continue
fi
# TODO: consider using commands that are entirely consistent with building binary packages
mvn package -DskipTests -Papache-release -ntp -e || exit
mvn package -DskipTests -Papache-release -ntp -e
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compile hugegraph-core error:

Caused by: org.eclipse.aether.resolution.DependencyResolutionException:
 Could not find artifact org.apache.hugegraph:hugegraph-common:jar:1.3.0 in central (https://repo.maven.apache.org/maven2)

Copy link
Member

@imbajin imbajin Mar 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need use mvn xxx -P stage to get the latest dependency （this has been done in ci by replacing settings.xml so the script doesn't do it）

refer:
https://github.com/apache/incubator-hugegraph-doc/blob/master/.github/workflows/validate-release.yml#L63

ls -lh

popd || exit
popd
done

###########################################
# Step 5: Run Compiled Packages In Server #
# Step 5: Run Compiled Packages of Server #
###########################################
cd "$WORK_DIR"/dist/"$RELEASE_VERSION" || exit
cd "${WORK_DIR}/dist/${RELEASE_VERSION}"

ls -lh
pushd ./*hugegraph-incubating*src/hugegraph-server/*hugegraph*"${RELEASE_VERSION}" || exit
bin/init-store.sh || exit
pushd ./*hugegraph-incubating*src/hugegraph-server/*hugegraph*"${RELEASE_VERSION}"
bin/init-store.sh
sleep 3
bin/start-hugegraph.sh || exit
popd || exit
bin/start-hugegraph.sh
popd

#######################################################################
# Step 6: Run Compiled Packages In ToolChain (Loader & Tool & Hubble) #
# Step 6: Run Compiled Packages of ToolChain (Loader & Tool & Hubble) #
#######################################################################
cd "$WORK_DIR"/dist/"$RELEASE_VERSION" || exit
cd "${WORK_DIR}/dist/${RELEASE_VERSION}"

pushd ./*toolchain*src || exit
pushd ./*toolchain*src
ls -lh
pushd ./*toolchain*"${RELEASE_VERSION}" || exit
pushd ./*toolchain*"${RELEASE_VERSION}"
ls -lh

# 6.1: load some data first
echo "test loader"
pushd ./*loader*"${RELEASE_VERSION}" || exit
pushd ./*loader*"${RELEASE_VERSION}"
bin/hugegraph-loader.sh -f ./example/file/struct.json -s ./example/file/schema.groovy \
-g hugegraph || exit
popd || exit
-g hugegraph
popd

# 6.2: try some gremlin query & api in tool
echo "test tool"
pushd ./*tool*"${RELEASE_VERSION}" || exit
bin/hugegraph gremlin-execute --script 'g.V().count()' || exit
bin/hugegraph task-list || exit
bin/hugegraph backup -t all --directory ./backup-test || exit
popd || exit
pushd ./*tool*"${RELEASE_VERSION}"
bin/hugegraph gremlin-execute --script 'g.V().count()'
bin/hugegraph task-list
bin/hugegraph backup -t all --directory ./backup-test
popd

# 6.3: start hubble and connect to server
echo "test hubble"
pushd ./*hubble*"${RELEASE_VERSION}" || exit
pushd ./*hubble*"${RELEASE_VERSION}"
# TODO: add hubble doc & test it
cat conf/hugegraph-hubble.properties
bin/start-hubble.sh || exit
bin/stop-hubble.sh || exit
popd || exit
bin/start-hubble.sh
bin/stop-hubble.sh
popd

popd || exit
popd || exit
popd
popd
# stop server
pushd ./*hugegraph-incubating*src/hugegraph-server/*hugegraph*"${RELEASE_VERSION}" || exit
bin/stop-hugegraph.sh || exit
popd || exit
pushd ./*hugegraph-incubating*src/hugegraph-server/*hugegraph*"${RELEASE_VERSION}"
bin/stop-hugegraph.sh
popd

# clear source packages
#rm -rf ./*src*
Expand All @@ -217,7 +225,7 @@ popd || exit
####################################
# Step 7: Validate Binary Packages #
####################################
cd "$WORK_DIR"/dist/"$RELEASE_VERSION" || exit
cd "${WORK_DIR}/dist/${RELEASE_VERSION}"

for i in *.tar.gz; do
if [[ "$i" == *-src.tar.gz ]]; then
Expand All @@ -232,10 +240,11 @@ for i in *.tar.gz; do
echo "The package name $i should include incubating" && exit 1
fi

tar xzvf "$i" || exit
pushd "$(basename "$i" .tar.gz)" || exit
tar -xzvf "$i"
MODULE_DIR=$(basename "$i" .tar.gz)
pushd ${MODULE_DIR}
ls -lh
echo "Start to check the package content: $(basename "$i" .tar.gz)"
echo "Start to check the package content: ${MODULE_DIR}"

# 7.2: check root dir include "NOTICE"/"LICENSE"/"DISCLAIMER" files & "licenses" dir
if [[ ! -f "LICENSE" ]]; then
Expand Down Expand Up @@ -268,57 +277,56 @@ for i in *.tar.gz; do
echo "The package $i shouldn't include empty file: $EMPTY_FILE is empty" && exit 1
done

popd || exit
popd
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

The package apache-hugegraph-commons-incubating-1.3.0-src.tar.gz shouldn't include empty directory: ./hugegraph-rpc/target/generated-sources/annotations is empty

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-gpg-plugin:1.6:sign (sign-release-artifacts) on project hugegraph-commons: Exit code: 2 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-gpg-plugin:1.6:sign (sign-release-artifacts) on project hugegraph-commons: Exit code: 2

[ERROR] Failed to execute goal on project hugegraph-loader: Could not resolve dependencies for project org.apache.hugegraph:hugegraph-loader:jar:1.3.0: Failed to collect dependencies at org.apache.hbase:hbase-mapreduce:jar:2.2.3 -> org.apache.hbase:hbase-server:jar:2.2.3 -> org.glassfish.web:javax.servlet.jsp:jar:2.3.2 -> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Failed to read artifact descriptor for org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to jvnet-nexus-snapshots (https://maven.java.net/content/repositories/snapshots): transfer failed for https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same reason

done

# TODO: skip the following steps by comparing the artifacts built from source packages with binary packages
#########################################
# Step 8: Run Binary Packages In Server #
# Step 8: Run Binary Packages of Server #
#########################################
cd "$WORK_DIR"/dist/"$RELEASE_VERSION" || exit
cd "${WORK_DIR}/dist/${RELEASE_VERSION}"

pushd ./*hugegraph-incubating*"${RELEASE_VERSION}" || exit
bin/init-store.sh || exit
sleep 30
bin/start-hugegraph.sh || exit
popd || exit
pushd ./*hugegraph-incubating*"${RELEASE_VERSION}"
bin/init-store.sh
sleep 3
bin/start-hugegraph.sh
popd

#####################################################################
# Step 9: Run Binary Packages In ToolChain (Loader & Tool & Hubble) #
# Step 9: Run Binary Packages of ToolChain (Loader & Tool & Hubble) #
#####################################################################
cd "$WORK_DIR"/dist/"$RELEASE_VERSION" || exit
cd "${WORK_DIR}/dist/${RELEASE_VERSION}"

pushd ./*toolchain*"${RELEASE_VERSION}" || exit
pushd ./*toolchain*"${RELEASE_VERSION}"
ls -lh

# 9.1: load some data first
echo "test loader"
pushd ./*loader*"${RELEASE_VERSION}" || exit
bin/hugegraph-loader.sh -f ./example/file/struct.json -s ./example/file/schema.groovy \
-g hugegraph || exit
popd || exit
pushd ./*loader*"${RELEASE_VERSION}"
bin/hugegraph-loader.sh -f ./example/file/struct.json -s ./example/file/schema.groovy -g hugegraph
popd

# 9.2: try some gremlin query & api in tool
echo "test tool"
pushd ./*tool*"${RELEASE_VERSION}" || exit
bin/hugegraph gremlin-execute --script 'g.V().count()' || exit
bin/hugegraph task-list || exit
bin/hugegraph backup -t all --directory ./backup-test || exit
popd || exit
pushd ./*tool*"${RELEASE_VERSION}"
bin/hugegraph gremlin-execute --script 'g.V().count()'
bin/hugegraph task-list
bin/hugegraph backup -t all --directory ./backup-test
popd

# 9.3: start hubble and connect to server
echo "test hubble"
pushd ./*hubble*"${RELEASE_VERSION}" || exit
pushd ./*hubble*"${RELEASE_VERSION}"
# TODO: add hubble doc & test it
cat conf/hugegraph-hubble.properties
bin/start-hubble.sh || exit
bin/stop-hubble.sh || exit
popd || exit
bin/start-hubble.sh
bin/stop-hubble.sh
popd

popd || exit
popd
# stop server
pushd ./*hugegraph-incubating*"${RELEASE_VERSION}" || exit
bin/stop-hugegraph.sh || exit
popd || exit
pushd ./*hugegraph-incubating*"${RELEASE_VERSION}"
bin/stop-hugegraph.sh
popd

echo "Finish validate, please check all steps manually again!"
Loading