Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PostgreSQL adapter #126

Closed
wants to merge 19 commits into from
Closed

Add PostgreSQL adapter #126

wants to merge 19 commits into from

Conversation

kyoujuro
Copy link

I added PostgreSQL extract and load adapters.
It may be insufficient because it is the first pull request, but please check it.
Thank you.

@cocoa-maemae
Copy link
Member

Thank you very much for your pull request. We will review it.

@bo-naoki-okamoto
Copy link
Contributor

First I must say we are really sorry for not providing user development guid to implement for the framework
We are now working on it.

So let me guild you how to create new cliboa classes

  1. Decide what ETL class you want to create (Load, Extract, or Transform)
    -> Now You already done this.

  2. Make sure if base class already exists in cliboa framework
    -> We still have not provided base of PostgreSql class or even base of Sql class(Only BaseSqliteClass)
    So you need to create a BaseSql class in "cliboa/scenario/base.py"(Or perhaps create a new file "cliboa/scenario/sql.py")
    By the way, looks like you are using BasePostgres, but I don't think I can find it, are you missing it?

  3. Create new class
    -> All base classes must extend BaseStep.
    Also base classes might have parameters or methods which can generally be used from sub classes.
    In this case, BaseSql class will be something like the below.

from cliboa.scenario.base import BaseStep

class BaseSql(BaseStep):
  def __init__(self):
      super().__init__()

      # These parameters are required for any type of databases. So define these in base class.
      self._host = None
      self._dbname = None
      self._user = None
      self._password = None

  def host(self, host):
      self._host = host
      
  def dbname(self, dbname):
      self._dbname = dbname
      
  def user(self, user):
      self._user = user

  def password(self, password):
      self._password = password
      
  def transaction():
      pass
  1. Create any ETL classes.
    -> Now you have base class for sql.
    Create PostgresRead with extending BaseSql class.
    Note: When you create a new class, plaese do NOT copy code from class which is deprecated currently
    we have a lot of deprecated classes.

Finaly this framework is still under updating.
We can provide BaseSql class sometime soon(maybe...)
If you are not in hurry, please wait untill our new update.

@kyoujuro
Copy link
Author

Thank you for confirmation.
And thank you for your polite explanation.
I am sorry that there are some inadequacies.
I learned about this OSS at OSS Conference in Kyoto.
It was very attractive, and I hope that you will continue to contribute with little effort.
I am waiting for the framework. Thank you.

cliboa/util/postgres.py Outdated Show resolved Hide resolved
cliboa/scenario/scenario.yml Outdated Show resolved Hide resolved
cliboa/scenario/extract/postgres.py Show resolved Hide resolved
cliboa/util/postgres.py Outdated Show resolved Hide resolved
cliboa/scenario/extract/postgres.py Show resolved Hide resolved
cliboa/scenario/load/postgres.py Show resolved Hide resolved
@cocoa-maemae
Copy link
Member

I truly apologize for late comment. I forgot to submit. Now I submitted. Best Regards,

psycopg library move into PostgresAdapter.
posycopg2 add this file.
I change double underscore to sigle underscore.
@kyoujuro
Copy link
Author

I changed this branch.
Please check it.
Best Regards.

@cocoa-maemae
Copy link
Member

Hello. Could you merge master commits to your development branch. There are many changes.
Best Regards,

@cocoa-maemae
Copy link
Member

Now I groomed development document. Would you check the page?
https://github.com/BrainPad/cliboa/wiki

@kyoujuro
Copy link
Author

kyoujuro commented Nov 6, 2020

I am very sorry for the late response.
Let me commit your master branch.
I apologize for the inconvenience for the first time.

@kyoujuro
Copy link
Author

kyoujuro commented Nov 9, 2020

I think I merged them.
I apologize for the inconvenience, but thank you for your understanding.

Best Regards,

@bo-naoki-okamoto
Copy link
Contributor

I will take care this pull request with the following ticket
#294

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants