Fixes for the gurobi interface and OSQP tolerances #235

cdiener · 2021-04-28T01:07:25Z

fix Gurobi models can not be pickled anymore #232
description of feature/fix
tests added/passed
add an entry to the next release

This PR fixes several issues with Gurobi and a bug where one could not clone from the OSQP interface.

Fixes

renaming variables was broken due to an obsolete workaround (fixed in Gurobi 9.1), now works
now correctly serializes the configuration in the Gurobi interface (fixes pickling)
added a pickling test for gurobi
fixes 2 broken tests for Gurobi
fixed the shim for the integrality tolerance in osqp_interface so it can be cloned to other interfaces

Tests on all interfaces

============================================================================ test session starts =============================================================================
platform linux -- Python 3.7.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /users/cdiener/optlang, configfile: tox.ini, testpaths: src/optlang/tests
plugins: anyio-2.2.0
collected 858 items                                                                                                                                                          

src/optlang/tests/test_change_solver.py .....                                                                                                                          [  0%]
src/optlang/tests/test_coinor_cbc_interface.py s.s...s........s.....s......................s.........................sss..........................s...                 [ 12%]
src/optlang/tests/test_container.py .......................                                                                                                            [ 15%]
src/optlang/tests/test_cplex_interface.py .......s...................................s...................s.............................................ss............. [ 29%]
.....                                                                                                                                                                  [ 30%]
src/optlang/tests/test_duality.py ................                                                                                                                     [ 32%]
src/optlang/tests/test_elements.py .................                                                                                                                   [ 34%]
src/optlang/tests/test_expression_parsing.py ...                                                                                                                       [ 34%]
src/optlang/tests/test_glpk_exact_interface.py ......s..............s....................s....................s....ssss.......s...............                         [ 45%]
src/optlang/tests/test_glpk_interface.py .......s..............................................................................................                        [ 57%]
src/optlang/tests/test_gurobi_interface.py .........s............................................................s.s................................s.s.               [ 70%]
src/optlang/tests/test_inspyred_interface.py s                                                                                                                         [ 70%]
src/optlang/tests/test_interface.py .............................                                                                                                      [ 73%]
src/optlang/tests/test_io.py ...                                                                                                                                       [ 74%]
src/optlang/tests/test_netlib_glpk_exact_interface.py x                                                                                                                [ 74%]
src/optlang/tests/test_netlib_glpk_interface.py x                                                                                                                      [ 74%]
src/optlang/tests/test_osqp_interface.py .......s.................................................................................................................     [ 88%]
src/optlang/tests/test_scipy_interface.py ..ss..s............s.s.................s..ss...sss..s.s..s..s.ssss.ssssss......sss........sss..s                             [ 99%]
src/optlang/tests/test_symbolics.py ..                                                                                                                                 [ 99%]
src/optlang/tests/test_util.py ..                                                                                                                                      [100%]

============================================================================== warnings summary ==============================================================================
../miniconda3/envs/micom-devel/lib/python3.7/site-packages/nose/importer.py:12
  /users/cdiener/miniconda3/envs/micom-devel/lib/python3.7/site-packages/nose/importer.py:12: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    from imp import find_module, load_module, acquire_lock, release_lock

src/optlang/tests/test_netlib_glpk_exact_interface.py:19
  src/optlang/tests/test_netlib_glpk_exact_interface.py:19: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_netlib will be ignored
    def test_netlib(netlib_tar_path=os.path.join(os.path.dirname(__file__), 'data/netlib_lp_problems.tar.gz')):

src/optlang/tests/test_netlib_glpk_interface.py:19
  src/optlang/tests/test_netlib_glpk_interface.py:19: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_netlib will be ignored
    def test_netlib(netlib_tar_path=os.path.join(os.path.dirname(__file__), 'data/netlib_lp_problems.tar.gz')):

src/optlang/tests/test_coinor_cbc_interface.py::ModelTestCase::test_clone_model_with_lp
src/optlang/tests/test_coinor_cbc_interface.py::ModelTestCase::test_clone_small_model_with_lp
src/optlang/tests/test_cplex_interface.py::ModelTestCase::test_clone_model_with_lp
src/optlang/tests/test_glpk_exact_interface.py::ModelTestCase::test_clone_model_with_lp
src/optlang/tests/test_glpk_interface.py::ModelTestCase::test_clone_model_with_lp
src/optlang/tests/test_gurobi_interface.py::ModelTestCase::test_clone_model_with_lp
  /users/cdiener/optlang/src/optlang/interface.py:1135: UserWarning: Cloning with LP formats can change variable and constraint ID's.
    warnings.warn("Cloning with LP formats can change variable and constraint ID's.")

src/optlang/tests/test_gurobi_interface.py: 5643 warnings
  /users/cdiener/optlang/src/optlang/interface.py:1474: DeprecationWarning: Deprecated, pass a TempConstr or use Model.addLConstr
    self._add_constraints(add_constr)

src/optlang/tests/test_gurobi_interface.py::ConstraintTestCase::test_setting_bounds
  /users/cdiener/optlang/src/optlang/gurobi_interface.py:316: DeprecationWarning: Deprecated, pass a TempConstr or use Model.addLConstr
    self._set_constraint_bounds(sense, rhs, range_value)

src/optlang/tests/test_gurobi_interface.py::ModelTestCase::test_add_constraints
  /users/cdiener/optlang/src/optlang/interface.py:1479: DeprecationWarning: Deprecated, pass a TempConstr or use Model.addLConstr
    self._add_constraints(add_constr_sloppy, sloppy=True)

src/optlang/tests/test_gurobi_interface.py: 73 warnings
  /users/cdiener/optlang/src/optlang/interface.py:1566: DeprecationWarning: Deprecated, pass a TempConstr or use Model.addLConstr
    self._add_constraints([constraint], sloppy=sloppy)

src/optlang/tests/test_osqp_interface.py: 64 warnings
  /users/cdiener/miniconda3/envs/micom-devel/lib/python3.7/site-packages/osqp/utils.py:74: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    np.zeros((0,), dtype=np.int),

src/optlang/tests/test_osqp_interface.py: 64 warnings
  /users/cdiener/miniconda3/envs/micom-devel/lib/python3.7/site-packages/osqp/utils.py:75: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    np.zeros((n+1,), dtype=np.int)),

-- Docs: https://docs.pytest.org/en/stable/warnings.html
==================================================== 792 passed, 64 skipped, 2 xfailed, 5855 warnings in 72.68s (0:01:12) ====================================================

Midnighter · 2021-05-01T14:59:34Z

renaming variables was broken due to an obsolete workaround (fixed in Gurobi 9.1), now works

Since I don't have access to an academic license any longer, does this mean it's fixed for 9.1 onward or also works for older versions of Gurobi now?

cdiener · 2021-05-01T20:06:23Z

Yeah not sure, would have to ask who added the comment before. The workaround is from 5 years ago though. It is fixed in 9.1 and there are some other part that do depend on Gurobi 9 I think so I would not expect the interface to work with Gurobi 8.

Midnighter · 2021-05-02T12:07:05Z

I guess anyone with a license can download the latest version but it also seems a bit brutal. I would only like to do that if we start adding the externally installed solvers to 'extras' so that we can require a minimum version. Maybe @phantomas1234 and @KristianJensen have some thoughts on this, too?

cdiener · 2021-05-02T20:29:41Z

I could check if it work with older supported Gurobi versions. The current interface does not work with Gurobi 7 anyways due to API incompatibilities so the effective minimum version right now is Gurobi 8.

…tolerances

cdiener · 2021-05-02T21:41:30Z

Here is a version that works with all supported Gurobi versions (>8.0.1) (https://support.gurobi.com/hc/en-us/articles/360048138771-Gurobi-release-and-support-history).

# packages in environment at /users/cdiener/miniconda3/envs/gurobi:                                                                                                           
gurobi                    8.0.1                    py36_0    gurobi 
pytest -k gurobi                                                                                                                            
================================================================================ test session starts =========================================================================
========                                                                                                                                                                      
platform linux -- Python 3.6.13, pytest-6.2.3, py-1.10.0, pluggy-0.13.1                                                                                                       
rootdir: /users/cdiener/optlang, configfile: tox.ini, testpaths: src/optlang/tests                                                                                            
collecting 1 item                                                                                                                                                             
collected 504 items / 395 deselected / 109 selected                                                                                                                           
                                                                                                                                                                              
                                                                                                                                                                              
src/optlang/tests/test_gurobi_interface.py .........s............................................................s.s................................s.s.                       [100%]

================================================================================== warnings summary ==================================================================================
../miniconda3/envs/gurobi/lib/python3.6/site-packages/nose/importer.py:12
  /users/cdiener/miniconda3/envs/gurobi/lib/python3.6/site-packages/nose/importer.py:12: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    from imp import find_module, load_module, acquire_lock, release_lock

src/optlang/tests/test_netlib_glpk_exact_interface.py:19
  src/optlang/tests/test_netlib_glpk_exact_interface.py:19: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_netlib will be ignored
    def test_netlib(netlib_tar_path=os.path.join(os.path.dirname(__file__), 'data/netlib_lp_problems.tar.gz')):

src/optlang/tests/test_netlib_glpk_interface.py:19
  src/optlang/tests/test_netlib_glpk_interface.py:19: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_netlib will be ignored
    def test_netlib(netlib_tar_path=os.path.join(os.path.dirname(__file__), 'data/netlib_lp_problems.tar.gz')):

src/optlang/tests/test_gurobi_interface.py::ModelTestCase::test_clone_model_with_lp
  /users/cdiener/optlang/src/optlang/interface.py:1135: UserWarning: Cloning with LP formats can change variable and constraint ID's.
    warnings.warn("Cloning with LP formats can change variable and constraint ID's.")

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================================================= 104 passed, 5 skipped, 395 deselected, 4 warnings in 3.76s =============================================================

Though I personally think supporting the current stable major version is hard enough and we should not try to support legacy versions since you could still use those by pinning an older version of optlang. However, the current version has been breaking MICOM (with gurobi) for at least a month and I would like to fix that.

Also had to disable safety in Python 2.7 since the last supported version of pip in 2.7 will be flagged now due to a CVE.

Midnighter

Changes look good to me. And sorry for causing you more trouble. I actually just wanted some more opinions but now you already added support for 8 🙂

cdiener · 2021-05-03T15:49:41Z

That's fine, I appreciate feedback 😁

cdiener added 4 commits April 27, 2021 12:30

fix config serialization in gurobi and tolerance errors in OSQP

97b9956

remove stream settings

4743406

fix gurobi name changing

d138de7

fix gurobi tests

02f9b70

cdiener requested review from carrascomj and Midnighter April 30, 2021 17:32

cdiener added 3 commits May 2, 2021 14:10

support legacy optlang versions

8925c8c

Merge branch 'fix/tolerances' of github.com:cdiener/optlang into fix/…

1f4a50c

…tolerances

support legacy versions of gurobi

6916ed1

cdiener force-pushed the fix/tolerances branch from 47e774d to 6916ed1 Compare May 2, 2021 22:06

cdiener added 2 commits May 2, 2021 15:12

try setting the pip version for safety

a5d75d0

looks better...

ede8c59

Midnighter approved these changes May 3, 2021

View reviewed changes

add changelog

c51c6f4

cdiener merged commit 47c0ded into opencobra:master May 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes for the gurobi interface and OSQP tolerances #235

Fixes for the gurobi interface and OSQP tolerances #235

cdiener commented Apr 28, 2021

Midnighter commented May 1, 2021

cdiener commented May 1, 2021 •

edited

Loading

Midnighter commented May 2, 2021

cdiener commented May 2, 2021 •

edited

Loading

cdiener commented May 2, 2021 •

edited

Loading

Midnighter left a comment

cdiener commented May 3, 2021

Fixes for the gurobi interface and OSQP tolerances #235

Fixes for the gurobi interface and OSQP tolerances #235

Conversation

cdiener commented Apr 28, 2021

Fixes

Tests on all interfaces

Midnighter commented May 1, 2021

cdiener commented May 1, 2021 • edited Loading

Midnighter commented May 2, 2021

cdiener commented May 2, 2021 • edited Loading

cdiener commented May 2, 2021 • edited Loading

Midnighter left a comment

Choose a reason for hiding this comment

cdiener commented May 3, 2021

cdiener commented May 1, 2021 •

edited

Loading

cdiener commented May 2, 2021 •

edited

Loading

cdiener commented May 2, 2021 •

edited

Loading