Fix FSDP ckpt by filtering User Waring

[j316chuck]

What does this PR do?

Fix FSDP ckpt by filtering User Warning

What issue(s) does this change relate to?

Fix failing dailies
https://github.com/mosaicml/composer/actions/runs/9231304954

[snarayan21]

@j316chuck Can you run the daily tests on this branch and see if they pass now? will approve after that

[j316chuck]

@snarayan21 running daily: https://github.com/mosaicml/composer/actions/runs/9231304954

[mvpatel2000]

Hm automerge was on so it merged even though its not fixed on linked dailies...

[snarayan21]

sad...but the error isn't related to this PR, it seems related to the TP PR @mvpatel2000

[j316chuck]

@mvpatel2000 I think it fixed the UserWarning but now we are hitting a new failure.

tensor = ShardedTensor(ShardedTensorMetadata(shards_metadata=[ShardMetadata(shard_offsets=[0, 0], shard_sizes=[16, 32], placeme...pe=torch.float32, layout=torch.strided, requires_grad=False, memory_format=torch.contiguous_format, pin_memory=False)))
parent_mesh = None

    def _all_gather_dtensor(
        tensor: DTensor,
        parent_mesh: Optional[DeviceMesh],
    ) -> torch.Tensor:
        """
        All gather a DTensor in its sharded dimension and return the local tensor.
        """
        assert parent_mesh is None
    
>       placements = list(copy.deepcopy(tensor.placements))
E       AttributeError: 'ShardedTensor' object has no attribute 'placements'

Let's fix in a new PR!