Make model compilation failure a warning only

[theobjectivedad]

A small update that will output a warning when model compilation fails instead of halting training.

Background

While testing on (unsupported) Python 3.11 + Torch 2.0.1 the following error is raised:

[2023-09-06 11:16:32,019] [INFO] [axolotl.train.train:84] [PID:9554] Compiling torch model
Traceback (most recent call last):
  File "/workspace/axolotl/scripts/finetune.py", line 278, in <module>
    fire.Fire(do_cli)
  File "/opt/resonance_ai/venv/lib/python3.11/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resonance_ai/venv/lib/python3.11/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/opt/resonance_ai/venv/lib/python3.11/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/scripts/finetune.py", line 274, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/workspace/axolotl/src/axolotl/train.py", line 85, in train
    model = torch.compile(model)
            ^^^^^^^^^^^^^^^^^^^^
  File "/opt/resonance_ai/venv/lib/python3.11/site-packages/torch/__init__.py", line 1444, in compile
    return torch._dynamo.optimize(backend=backend, nopython=fullgraph, dynamic=dynamic, disable=disable)(model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/resonance_ai/venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 457, in optimize
    check_if_dynamo_supported()
  File "/opt/resonance_ai/venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 413, in check_if_dynamo_supported
    raise RuntimeError("Python 3.11+ not yet supported for torch.compile")
RuntimeError: Python 3.11+ not yet supported for torch.compile

Since model compilation may not be required this change seems reasonable to increase compatibility.

[tmm1]

Note that the torch.compile here does nothing at all, because it's calling after the trainer is setup with the regular model.

see #516

[winglian]

fixed by #516