BUG: Set_index fills in some NaN

[hayd]

In this example when setting the index of a DataFrame to columns with NaN this values is filled:

In [11]: df = pd.DataFrame({'id1': {0: '1a3', 1: '9h4'}, 'id2': {0: nan, 1: 'd67'},
                            'id3': {0: '78d', 1: '79d'}, 'value': {0: 123, 1: 64}})

In [12]: df
Out[12]: 
   id1  id2  id3  value
0  1a3  NaN  78d    123
1  9h4  d67  79d     64

In [13]: df.set_index(['id1', 'id2', 'id3'])
Out[13]: 
             value
id1 id2 id3       
1a3 d67 78d    123
9h4 d67 79d     64

I see this on '0.10.1.dev-f73128e', apologies if this has been mentioned before.
Note: This example is essentially from this SO question.

[ghost]

jeff fixed this.

[Marigold]

Unfortunately, if you specify append=True, the same thing happens.

df = pd.DataFrame({'id1': {0: '1a3', 1: '9h4'}, 'id2': {0: nan, 1: 'd67'}, 'id3': {0: '78d', 1: '79d'}, 'value': {0: 123, 1: 64}})
df.set_index(['id1', 'id2']).set_index('id3', append=True)

Output

             value
id1 id2 id3       
1a3 d67 78d    123
9h4 d67 79d     64

[jreback]

what version are u on?

this should be fixed on >= 0.11

[Marigold]

I'm on 0.12. It happens only with append=True, otherwise it's working fine.

[hayd]

@Marigold I confirm what you saw in 0.12, however it's fixed in master:

#master
In [10]: df.set_index(['id1', 'id2']).set_index('id3', append=True)
Out[10]: 
             value
id1 id2 id3       
1a3 NaN 78d    123
9h4 d67 79d     64

[2 rows x 1 columns]

I added in an additional test case for this.

[Marigold]

Confirm too. Thanks a lot.