Updating how we load metrics in a state_dict so we don't add extra memory overhead

[bcui19]

What does this PR do?

When loading composer state dicts, we load in train and eval metrics. However, these metrics have a defined _device which is usually always the rank 0 device since that's what stored in the state_dict. As a result, we need to manually copy over each metric (so we don't override a user's current metrics) and manually set the devices.

Before our memory overhead was:
| 0 N/A N/A 3770555 C 31567MiB |
| 0 N/A N/A 3770556 C 965MiB |
| 1 N/A N/A 3770556 C 31567MiB

Now, our memory overhead when loading models is normal:
| 0 N/A N/A 3770555 C 31567MiB |
| 1 N/A N/A 3770556 C 31567MiB |

What issue(s) does this change relate to?

CO-1485

Before submitting

• [ x] Have you read the contributor guidelines?
• Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
• Did you update any related tests and add any new tests related to your change? (see testing)
• [x ] Did you run the tests locally to make sure they pass?
• [x ] Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

[dakinggg]

LGTM, could you add a test that loading a checkpoint doesn't use extra memory? I hope this is not hard and can just be done as part of this PR. If it is hard, feel free to make a JIRA and do it later.

[eracah]

LGTM. Is there an easy way to add a test?