Add PyTable Paths to HDFStore

[surbas]

Hello,
I not sure if this was the best way to suggest improvements, but here I go anyway :)

I really like the easy of the HDFStore (and the entire project for that matter), but I wanted the ability to store and retrieve DataFrames in Groups below the root. .i.e

h5 = HDFStore('test.h5')
h5['/groups/below/theroot'] = DataFrame(data, index)

df =  h5['/groups/below/theroot']

I made the following changes to HDFStore class to do this, and am now using it in production code. If you agree this is something useful then I would like this to become apart of the main code base, using your own approach or the below.

I changed the repr, getitem, and _write_group functions in pandas.io.pytables.

def __repr__(self):
    output = str(self.__class__) + '\n'

    #Exstract path and kind of all 'pandas_type' pytable Groups.
    keys, values = zip(*((x._v_pathname, x._v_attrs.pandas_type) for x in self.handle.walkGroups() if hasattr(x._v_attrs,'pandas_type')))

    output += adjoin(5, keys, values)
    return output



def __getitem__(self, key):

    if not key[0] == '/': #Then add root slash so we can use getNode belwo
        key = '/' + key

    group = self.handle.getNode(key)
    return _read_group(group)



def _write_group(self, key, value):
    root = self.handle.root


    if key[0] == '/': #Assume they want a nested pytable Group
        final_slash = key.rfind('/')
        where = key[:final_slash]
        name = key[final_slash + 1:]
    else:
        where = '/'
        name = key

    try:
        group = self.handle.getNode(key)
    except:
        group = self.handle.createGroup(where, name, createparents=True)


    kind = type(value)
    handler = self._get_write_handler(kind)

    try:
        handler(group, value)
    except Exception:
        raise

    group._v_attrs.pandas_type = kind.__name__
    return True

please let me know if there is questions or concerns... I made it so that it still works just like the original if the user doesn't need groups...

Thanks for a great project.

[wesm]

Hi Surbas,

I'm sorry this has taken me ages to get to. I just completed some major work on the HDFStore class. Let me see if I can incorporate your patch without breaking any of the unit tests, will get back to you.

[jreback]

@wesm I believe u can close this issue as 0.10 implements

[ghost]

@surbas, is there any functionality still missing in the latest git?
jreback has put in an enormous number of improvements into HDFS support
in 0.10.0.

[surbas]

Looks good to me. Thanks for the work.