Update clojure packages

[crkrenn]

@patcon & @metasoarous, what is the interest and importance of updating clojure packages?

polis-math is using a postgres driver from 2010: PostgreSQL JDBC Driver » 8.4-702.jdbc4 (Oct 04, 2010). The latest one is from June 2020.

There is an argument to not fixing what ain't broke, but I imagine that there are some bugs fixed and features added that might be beneficial.

[patcon]

I'm just peanut gallery on this, so I'll leave it to @metasoarous :)

Fwiw, dependabot (which we're using to auto-update github-actions, npm, and docker versions) doesn't yet work for clojure -- otherwise we'd probably be talking about turning that on.

Related:
https://github.com/snyk/snyk/issues/503
dependabot/dependabot-core#572

@metasoarous how would you go about adding tests for the math component anyhow? Is that easy or worthwhile? (Happy to spin this out if it's a big convo)

[patcon]

Also, could run something like this in a github action check, to keep us honest:

• https://github.com/rm-hull/lein-nvd
• https://github.com/marketplace/actions/setup-clojure

[crkrenn]

https://github.com/xsc/lein-ancient is another tool to consider.

…

--- Sent from Workspace ONE Boxer<https://whatisworkspaceone.com/boxer> On July 3, 2020 at 10:46:03 PM PDT, Patrick Connolly <notifications@github.com> wrote: Also, could run something like this in a github action check, to keep us honest: * https://github.com/rm-hull/lein-nvd * https://github.com/marketplace/actions/setup-clojure — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#400 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABOJ5V7ODJHADPJ6T2H3P33RZ26XZANCNFSM4OQHEDUQ>.

[patcon]

Rough strategy to get lay of landscape:

1. Checked for updates via lein ancient
2. Upgraded all possible packages (not tested) https://gist.github.com/patcon/8ade9b4cb717b100d5d05682e36058a6
3. Ran lein nvd check to see if any critical vulns even at newest versions (nevermind code updates to get working)

There ended up being a few packages with many critical vulns, which means that they've become behind on maintenance, as even the newest versions have some vulnerable dependencies. These are the ones most worth considering moving away from imho, when considering pkg updates.

First two columns are output of nvd check, and other two are tracking originating package and remediation details.

Results

dependency	status	origin pkg	alt pkg
aws-java-sdk-opsworkscm-1.11.698.jar	CVE-2015-8559	amazonica
aws-java-sdk-sagemaker-1.11.698.jar	CVE-2007-0896, CVE-2006-4711	amazonica
batik-css-1.9.1.jar	CVE-2018-8013	incanter/incanter-svg
bcprov-jdk14-1.46.jar	CVE-2018-1000613, CVE-2018-5382, CVE-2016-1000352, CVE-2016-1000344, CVE-2015-794...	incanter/incanter-pdf
bcprov-jdk15on-1.52.jar	CVE-2018-1000613, CVE-2016-1000352, CVE-2016-1000344, CVE-2016-1000342, CVE-2016-...	metasoarous/oz
c3p0-0.9.5.2.jar	CVE-2018-20433	korma
dom4j-1.6.1.jar	CVE-2020-10683, CVE-2018-1000632	???
google-closure-library-0.0-20160609-f42b4a24.jar	CVE-2020-8910	semantic-csv
jackson-databind-2.6.7.3.jar	CVE-2018-14721, CVE-2018-14720, CVE-2018-19360, CVE-2018-19361, CVE-2018-19362	amazonica
jetty-server-9.2.10.v20150310.jar	CVE-2017-7657, CVE-2017-7658, CVE-2017-9735, CVE-2017-7656, CVE-2018-12536, CVE-2019-10247, CVE-2019-10241	ring-jetty-component
jetty-util-9.2.10.v20150310.jar	CVE-2019-10247, CVE-2019-10241	ring-jetty-component
maven-aether-provider-3.0.4.jar	CVE-2013-0253	metasoarous/oz
maven-model-3.0.4.jar	CVE-2013-0253	metasoarous/oz
maven-model-builder-3.0.4.jar	CVE-2013-0253	metasoarous/oz
maven-repository-metadata-3.0.4.jar	CVE-2013-0253	metasoarous/oz
msgpack-0.6.12.jar	CVE-2020-5234	metasoarous/oz
netty-transport-4.1.25.Final.jar	CVE-2020-11612, CVE-2019-20444, CVE-2019-20445, CVE-2019-16869	metasoarous/oz
plexus-utils-2.0.6.jar	CVE-2017-1000487, Directory traversal in org.codehaus.plexus.util.Expand, Possible XML Injection	metasoarous/oz
poi-3.9.jar	CVE-2017-5644, CVE-2017-12626, CVE-2014-9527, CVE-2014-3529, CVE-2014-3574, CVE-2016-5000, CVE-2019-12415	clj-excel	docjure
protobuf-java-2.6.1.jar	CVE-2015-5237	amazonica
react-16.3.2-0.jar	CVE-2018-6341	metasoarous/oz
react-dom-16.3.2-0.jar	CVE-2018-6341	metasoarous/oz
react-dom-server-16.3.2-0.jar	CVE-2018-6341	metasoarous/oz
sente-1.13.1.jar	CVE-2019-1000022	metasoarous/oz
vega-tooltip-0.17.0-0.jar	CVE-2019-10806	metasoarous/oz

• metasoarous/oz depends on aleph, which is having maintenance issues: Is this project abandoned? clj-commons/aleph#539
• ring-jetty-component 0.3.1 depends on an old version of ring/ring-jetty-adapter 1.4.0, which has an update that hasn't been tagged and released: weavejester/ring-jetty-component@7d68f84

[crkrenn]

Updating the java version from 1.7 to 1.8 in project.clj and system.properties would make deployment to heroku easier. Is there a reason not to do this?

[patcon]

No objection, but can you clarify what's easier about 1.8? I thought either one was just a line in system.properties

[patcon]

Ah, found your comment in #244 (comment) that explains 👍

math also seems to be working after I bumped the java version from "1.7" to "1.8". I'm trying the same change in docker right now.

[crkrenn]

1.8 works in heroku. 1.7 does not because some supporting packages in the default heroku build are compiled under 1.8 and are not backward compatible. The math logs for the docker build with 1.8 look normal, but they are not very informative. C.

…

--- Sent from Workspace ONE Boxer<https://whatisworkspaceone.com/boxer> On August 4, 2020 at 10:06:17 AM PDT, Patrick Connolly <notifications@github.com> wrote: Ah, found your comment in #244 (comment)<#244 (comment)> that explains 👍 math also seems to be working after I bumped the java version from "1.7" to "1.8". I'm trying the same change in docker right now. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#400 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABOJ5V7JQAQVDQU5YT54BFDR7A5YHANCNFSM4OQHEDUQ>.

[patcon]

Ah, seems this is likely that the files in math are out-of-date, and we already use openjdk8-jre :)

https://github.com/pol-is/polis/blob/d7d24be8ed569c9f849e77431f707c94801db3e7/math/Dockerfile#L3-L5

[patcon]

@metasoarous in Gitter: https://gitter.im/pol-is/polisDeployment?at=5f2b0653028fac5e4d9ad609

a number of those packages (for incanter) should be removed. There's a tiny bit of code using them, but it needs to be stripped out.

[metasoarous]

Thanks for putting this out there @crkrenn. And @patcon for digging into the dep tree.

This is definitely something we should tackle. I can update those Oz dependencies. Also, Incanter should really be removed as it's defunct, and I think it's just one or two utility functions using it. We should be using the tech.ml stack now (see the analysis repo). I actually have a few of these changes in progress on a local checkout from some data-poking I was doing recently, so I'll try to wrap that up.

For the record though, a lot of these vulnerabilities don't really apply to the math worker as it never receives direct traffic from the web. But, that's not a good reason not to keep things up to date, and if someone got onto a machine, some of these vulnerabilities could presumably be problematic.

Thanks again!

[metasoarous]

Some work was recently done on this here which we should look at pulling in: DFE-Digital#51