Sandbox, Import/Export tool, and Syn toggling #95

dannylamb · 2020-12-08T15:32:31Z

Does the above^^

Plus it guts the startup script and I'm pushing that stuff into the Makefile in isle-dc. Will link PR when it's ready.

nigelgbanks · 2020-12-08T15:34:51Z

drupal/rootfs/etc/confd/templates/setup-environment.sh.tmpl

@@ -33,7 +33,7 @@ multisubstitute
  define DB_USER "drupal_default"
  define EMAIL "webmaster@localhost.com"
  define INSTALL "true"
-  define INSTALL_EXISTING_CONFIG "false"
+  define INSTALL_EXISTING_CONFIG "true"


Maybe override it in the demo image as there is no configuration in the base Drupal image.

nigelgbanks · 2020-12-08T17:14:58Z

demo/Dockerfile

-        /var/www/drupal \
-    && \
+    # Get the Drupal codebase
+    git clone https://github.com/dannylamb/islandora-sandbox && \


There is a helper https://github.com/Islandora-Devops/isle-buildkit/blob/main/crayfish/Dockerfile#L8 git-clone-cached.sh which might be helpful in reducing build times.

into post-sprint-4

…ring init scripts

noahwsmith · 2021-02-22T02:54:19Z

The only thing I found is that GEMINI needs to be remove here:

https://github.com/Islandora-Devops/isle-buildkit/blob/post-sprint-4/drupal/rootfs/etc/islandora/utilities.sh#L80

dannylamb · 2021-02-22T15:00:55Z

🙇‍♂️ Thx for the commit.

I think it's time this becomes a real pull request.

dannylamb · 2021-02-22T16:07:19Z

Missed a default. I've confirmed this works with and without secrets now.

noahwsmith · 2021-02-23T14:37:09Z

Pulling all the images into our work, this seems to work well without secrets. I must have missed something in my merge because secrets don't work anymore with all of this merged into our work. But they seem to work independently...

dannylamb · 2021-02-24T17:10:13Z

@noahwsmith I think I changed one of the secrets from being Fedora specific to Tomcat specific. Maybe it's that? I'm pretty sure I can revert that and we can test.

nigelgbanks

Lot had changed since the last time I gave a look so I thought I would look in again, it looks like it has come along way. I just mainly had some concerns around the environment variables and what looks like potential issues with defaults.

nigelgbanks · 2021-02-25T17:21:48Z

alpaca/Dockerfile

@@ -76,3 +75,5 @@ ENV \
    JAVA_OPTS="-Dfile.encoding=UTF-8 -Dnet.sf.ehcache.skipUpdateCheck=true -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:MaxPermSize=128m -Xms512m -Xmx8g"

 COPY rootfs /
+
+#    bin/client -r 10 -d 5  "feature:install fcrepo-indexing-triplestore" && \


Looks like a bit of dead code to be removed?

nigelgbanks · 2021-02-25T17:26:14Z

drupal/rootfs/etc/islandora/utilities.sh

 }

 # After enabling and importing features a number of configurations need to be updated.
 function configure_islandora_default_module {
+    if ! drush pm-list --pipe --type=module --status=enabled --no-core | grep -q islandora_defaults; then
+	echo "islandora_defaults is not installed.  Skipping configuration"
+        return 0


🤓 to be pedantic inconsistent indents (feel free to ignore).

nah, i'll fix 'em. dang tabs...

nigelgbanks · 2021-02-25T17:26:28Z

drupal/rootfs/etc/islandora/utilities.sh

@@ -403,9 +404,13 @@ function configure_search_api_solr_module {

 # Enables and sets carapace as the default theme.
 function set_carapace_default_theme {
+    if ! drush pm-list --pipe --type=theme --status=enabled --no-core | grep -q carapace; then
+	echo "carapace is not available. Skipping configuration."
+        return 0


🤓 to be pedantic inconsistent indents (feel free to ignore).

nigelgbanks · 2021-02-25T17:26:46Z

drupal/rootfs/etc/islandora/utilities.sh

@@ -452,20 +457,29 @@ function create_solr_core_with_default_config {

 # Install matomo and configure.
 function configure_matomo_module {
+    if ! drush pm-list --pipe --type=module --status=enabled --no-core | grep -q matomo; then
+	echo "matomo is not installed.  Skipping configuration"
+        return 0


🤓 to be pedantic inconsistent indents (feel free to ignore).

nigelgbanks · 2021-02-25T17:29:58Z

fcrepo/Dockerfile

@@ -27,18 +33,19 @@ ENV \
    FCREPO_ACTIVEMQ_QUEUE=fedora \
    FCREPO_ACTIVEMQ_TOPIC=fedora \
    FCREPO_BINARYSTORAGE_TYPE=file \
-    FCREPO_DB_MYSQL_HOST=mariadb \
-    FCREPO_DB_MYSQL_PORT=3306 \


These are still in use and provide defaults when choosing a driver. See the references that exist in a confd template here:

isle-buildkit/fcrepo/rootfs/etc/confd/templates/repository.json.tmpl

Line 20 in 449c34a

"connectionUrl": "jdbc:mysql://{{ getv "/db/mysql/host" (getenv "FCREPO_DB_MYSQL_HOST") }}:{{ getv "/db/mysql/port" (getenv "FCREPO_DB_MYSQL_PORT") }}/{{ getv "/db/name" (getenv "FCREPO_DB_NAME") }}?createDatabaseIfNotExist=true",

What should probably be changed instead is the usage in 03-fcrepo-setup.sh from:

function execute_sql_file { execute-sql-file.sh \ --driver "${FCREPO_PERSISTENCE_TYPE}" \ --host "${FCREPO_DB_HOST}" \ --port "${FCREPO_DB_PORT}" \ --user "${FCREPO_DB_ROOT_USER}" \ --password "${FCREPO_DB_ROOT_PASSWORD}" \ "${@}" }

To

function execute_sql_file { local FCREPO_DB_HOST="" local FCREPO_DB_PORT="" if [[ "${FCREPO_PERSISTENCE_TYPE}" == "mysql" ]]; then FCREPO_DB_HOST="${FCREPO_DB_MYSQL_HOST}" FCREPO_DB_PORT="${FCREPO_DB_MYSQL_PORT}" else FCREPO_DB_HOST="${FCREPO_DB_POSTGRESQL_HOST}" FCREPO_DB_PORT="${FCREPO_DB_POSTGRESQL_PORT}" fi execute-sql-file.sh \ --driver "${FCREPO_PERSISTENCE_TYPE}" \ --host "${FCREPO_DB_HOST}" \ --port "${FCREPO_DB_PORT}" \ --user "${FCREPO_DB_ROOT_USER}" \ --password "${FCREPO_DB_ROOT_PASSWORD}" \ "${@}" }

and remove FCREPO_DB_HOST and FCREPO_DB_PORT from the Dockerfile and README.md

That may have been my problem- Fedora couldn't find its database when I was using secrets.

nigelgbanks · 2021-02-25T17:41:49Z

fcrepo6/rootfs/etc/confd/templates/context.xml.tmpl

@@ -4,5 +4,7 @@
  <!-- web application will be reloaded.                                   -->
  <WatchedResource>WEB-INF/web.xml</WatchedResource>
  <WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>
+{{ if eq (getv "/disable/syn" "false") "false" }}


Should be:

{{ if eq (getv "/disable/syn" (getenv "FCREPO_DISABLE_SYN")) "false" }}

nigelgbanks · 2021-02-25T17:43:13Z

fcrepo6/rootfs/etc/confd/templates/setenv.sh.tmpl

+export CATALINA_OPTS="${CATALINA_OPTS} -Dfcrepo.db.password={{ getv "/db/password" (getenv "FCREPO_DB_PASSWORD") }}"
+export CATALINA_OPTS="${CATALINA_OPTS} -Dfcrepo.activemq.directory=file:///data/home/data/Activemq"
+export CATALINA_OPTS="${CATALINA_OPTS} -Dfcrepo.activemq.configuration=file:///opt/tomcat/conf/activemq.xml"
+{{ if eq (getv "/disable/syn" (getenv "FCREPO_DISABLE_SYN")) "true" }}


Same as above it doesn't need to be a template.

export CATALINA_OPTS="${CATALINA_OPTS} -Dfcrepo.activemq.configuration=file:///opt/tomcat/conf/activemq.xml" if [[ "${FCREPO_DISABLE_SYN}" == "true" ]]; then export CATALINA_OPTS="${CATALINA_OPTS} -Dfcrepo.properties.management=relaxed" fi

nigelgbanks · 2021-02-25T18:15:03Z

fcrepo6/README.md

+| FCREPO_BINARYSTORAGE_TYPE      | /fcrepo/binarystorage/type      | file                              |             |
+| FCREPO_CATALINA_OPTS           | /fcrepo/catalina/opts           |                                   |             |
+| FCREPO_DB_HOST                 | /fcrepo/db/host                 | mariadb                           |             |
+| FCREPO_DB_NAME                 | /fcrepo/db/name                 | fcrepo                            |             |


See notes above about environment variables.

nigelgbanks · 2021-02-25T18:16:21Z

fcrepo6/rootfs/etc/cont-init.d/03-fcrepo-setup.sh

+
+function execute_sql_file {
+    execute-sql-file.sh \
+        --driver "${FCREPO_PERSISTENCE_TYPE}" \


See notes above for fcrepo4, May or may not be relevant to this since repository.json is no longer in use.

nigelgbanks · 2021-02-25T18:20:57Z

fcrepo6/rootfs/etc/confd/templates/setenv.sh.tmpl

+export CATALINA_OPTS="${CATALINA_OPTS} -Dfcrepo.jms.baseUrl=http://{{ index (lookupIP (getenv "HOSTNAME")) 0 }}/fcrepo/rest"
+export CATALINA_OPTS="${CATALINA_OPTS} -Dfcrepo.external.content.allowed=/opt/tomcat/conf/allowed-external-content.txt"
+export CATALINA_OPTS="${CATALINA_OPTS} -Dfcrepo.autoversioning.enabled=false"
+export CATALINA_OPTS="${CATALINA_OPTS} -Dfcrepo.db.url=jdbc:{{ getv "/persistence/type" (getenv "FCREPO_PERSISTENCE_TYPE") }}://{{ getv "/db/host" (getenv "FCREPO_DB_HOST") }}:{{ getv "/db/port" (getenv "FCREPO_DB_PORT") }}/fcrepo"


I assume there is no repository.json in fcrepo6 seems like the defaults here would cause problems? Like the default for FCREPO_PERSISTENCE_TYPE is file?

dannylamb · 2021-02-25T20:33:38Z

Thanks for the feedback @nigelgbanks . I agree, the defaults are problematic for fcrepo6, but I wasn't sure how best to handle that, as well as splitting apart the mysql vs postgres variables. I could've sworn I grepped for their usage and came up short, which is why I just straight up overwrote them, but I guess not. One of those bad decisions made during "make it work" mode.

I'll make another pass at this and try to re-work things... I"m 🤦 'ing hard just looking at those templates which need not be templates....

nigelgbanks · 2021-02-25T22:11:43Z

You probably had them as templates before the rebase and changes to the confd system which makes sense.

Pulling the sandbox in as a demo makes a lot of sense, let me know if/when you want me to setup default_content in https://github.com/dannylamb/islandora-sandbox.

nigelgbanks · 2021-02-26T22:03:33Z

@dannylamb I merged main into your branch to unblock @zacanbot and his work on Kubernetes.

nigelgbanks · 2021-03-01T00:39:43Z

alpaca/Dockerfile

@@ -29,7 +29,6 @@ RUN bin/start && \

 # Triple indexing
 RUN bin/start && \
-    bin/client -r 10 -d 5  "feature:install fcrepo-indexing-triplestore" && \


This service is keeping me from being able to run on Arm for some reason I cannot sort out (Karaf makes me wanna pull my hair out), is it not needed on main either?

dannylamb · 2021-03-05T20:11:42Z

Ok, I've finally wrapped up all the code/test stuff for our php code and can circle back to this. Thank you for your patience 🙇 . We'll get there.

…sle-buildkit into post-sprint-4

dannylamb · 2021-03-15T18:14:25Z

@nigelgbanks @noahwsmith I've made another pass. Any 👀 on this would be appreciated.

noahwsmith · 2021-03-16T14:20:23Z

Hi Danny - I'm pretty deep underwater at the moment, but can test over the weekend if this is still open.

nigelgbanks · 2021-03-16T17:17:43Z

Changes look good, and everything appears to work for me locally

Sandbox, Import/Export tool, and Syn toggling

5984233

nigelgbanks reviewed Dec 8, 2020

View reviewed changes

dannylamb added 13 commits January 6, 2021 13:06

Changes for reindexing and no gemini

15a1dd1

Undoing some wrong things i did

d0624e8

Merge branch 'main' of https://github.com/islandora-devops/isle-buildkit

569c13b

into post-sprint-4

Facepalming on no /rootfs

d82cb7e

Updating islandora-sandbox

bddbdd6

Updating islandora-sandbox

82557c1

Fedora 6!

bf3ee86

Updating utilities for local test case

cb827c0

Remote activemq with fcrepo6. Reworking the fedoraAdmin permission du…

6055ade

…ring init scripts

Re-triggering github action to push new images

298feb2

Merging in upstream changes

a9797ea

Adding in getenvs

b8ad278

Rebasing off of nigel's work

e36ddd5

Remove Gemini from list of require services

421eaf7

dannylamb marked this pull request as ready for review February 22, 2021 15:01

Missing FCREPO_DB_ROOT_PASSWORD

116e319

dannylamb mentioned this pull request Feb 22, 2021

Pushing init logic into Make. Focusing on 'demo', 'local', and (not q… Islandora-Devops/isle-dc#132

Merged

Reworking how I check if modules are enabled or not

449c34a

nigelgbanks reviewed Feb 25, 2021

View reviewed changes

Merge branch 'main' into post-sprint-4

171f43d

nigelgbanks reviewed Mar 1, 2021

View reviewed changes

dannylamb added 2 commits March 5, 2021 16:51

Updating to use newest code. Touching up recast

9da6228

Merge branch 'post-sprint-4' of https://github.com/islandora-devops/i…

d159262

…sle-buildkit into post-sprint-4

This was referenced Mar 7, 2021

Use the version 0.0.4 of the build plugin #118

Closed

Use the version 0.0.4 of the build plugin #119

Closed

dannylamb added 3 commits March 11, 2021 14:50

Updating crayfish commit

6c23a9d

Responding to feedback. README updates

ab03986

Touching up utilities.sh

21a28a8

Attempting to fix deploying on remote instance without traefik.me

b6f7e4e

nigelgbanks approved these changes Mar 16, 2021

View reviewed changes

nigelgbanks merged commit 18a6631 into main Mar 16, 2021

nigelgbanks deleted the post-sprint-4 branch March 19, 2021 11:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sandbox, Import/Export tool, and Syn toggling #95

Sandbox, Import/Export tool, and Syn toggling #95

dannylamb commented Dec 8, 2020 •

edited

Loading

nigelgbanks Dec 8, 2020

nigelgbanks Dec 8, 2020

noahwsmith commented Feb 22, 2021

dannylamb commented Feb 22, 2021

dannylamb commented Feb 22, 2021

noahwsmith commented Feb 23, 2021

dannylamb commented Feb 24, 2021

nigelgbanks left a comment

nigelgbanks Feb 25, 2021

nigelgbanks Feb 25, 2021

dannylamb Mar 11, 2021

nigelgbanks Feb 25, 2021

nigelgbanks Feb 25, 2021

nigelgbanks Feb 25, 2021

noahwsmith Feb 25, 2021

nigelgbanks Feb 25, 2021

nigelgbanks Feb 25, 2021

nigelgbanks Feb 25, 2021

nigelgbanks Feb 25, 2021

nigelgbanks Feb 25, 2021

dannylamb commented Feb 25, 2021

nigelgbanks commented Feb 25, 2021

nigelgbanks commented Feb 26, 2021

nigelgbanks Mar 1, 2021

dannylamb commented Mar 5, 2021

dannylamb commented Mar 15, 2021

noahwsmith commented Mar 16, 2021

nigelgbanks commented Mar 16, 2021

Sandbox, Import/Export tool, and Syn toggling #95

Sandbox, Import/Export tool, and Syn toggling #95

Conversation

dannylamb commented Dec 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noahwsmith commented Feb 22, 2021

dannylamb commented Feb 22, 2021

dannylamb commented Feb 22, 2021

noahwsmith commented Feb 23, 2021

dannylamb commented Feb 24, 2021

nigelgbanks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dannylamb commented Feb 25, 2021

nigelgbanks commented Feb 25, 2021

nigelgbanks commented Feb 26, 2021

Choose a reason for hiding this comment

dannylamb commented Mar 5, 2021

dannylamb commented Mar 15, 2021

noahwsmith commented Mar 16, 2021

nigelgbanks commented Mar 16, 2021

dannylamb commented Dec 8, 2020 •

edited

Loading