Why:
- Small, Fast, Efficient, Easy, Funny
Important:
- The Detti DB handles string key, and many value types.
- The Detti Server handles only string type as key and value (
Dict[str, str]
).
>>> pip install detti-db-server
Create a project folder and a venv folder within:
>>> mkdir -p detti
>>> cd detti
Clone the source code from Git:
>>> git clone https://github.com/milanbalazs/detti_db_server.git
>>> cd detti_db_server
Create and activate the virtual environment:
>>> python3 -m venv venv
>>> source venv/bin/activate
Install the required modules from requirements.txt:
>>> pip install -r requirements.txt
OR
>>> python setup.py install
Requirements:
-
Interpreter
- Python3.6.x <
-
Python packages
- They can find in the requirements.txt file (pipreqs).
- The required packages can be installed with
pip
.
Tested system:
- Interpreter:
- Python 3.6.9
- Operation system:
- Linux Mint 19.1 Tessa
- Bash:
- 4.4.20(1)-release
- Curl
- curl 7.58.0 (x86_64-pc-linux-gnu)
The default config is detti_conf.ini
.
It is in the root folder (next to the detti_db.py
file).
The configuration file is a standard INI file format.
Default config:
[DETTI_DB]
# Path of the DB file. Recommended to define full path.
path_of_db = test.db
# Maximum length of the keys in DB (Avoid memory overload).
len_of_key = 100
# Maximum length of the values in DB (Avoid memory overload).
len_of_val = 100
# Level of the logger. Possible: DEBUG, INFO, WARNING, ERROR, CRITICAL
# IMPORTANT: The generated log file will contain all log level messages!
log_level = WARNING
Note:
- The default
detti_conf.ini
file contains more sections but only theDETTI_DB
section is related to the DB. Other sections are not used in case of DB. It is not problem if other (default) sections are not in the config file.
Use own config file:
- There is
config_file
argument (str
) ofDettiDB
class which set the used config file path. - Example:
-
detti_db = DettiDB(config_file="own_config.ini")
-
Owerwrite config file parameters:
- You can overwrite all config file parameters as instance variables.
- Example:
-
detti_db = DettiDB(len_of_val=50) # The parameter is set to 100 in config file but it will be overwrite to 50.
-
Import DettiDB
class from the detti_db
module:
from detti_db import DettiDB
Create instance from DettiDB
class (Using the default init values):
detti_db = DettiDB()
Supported types in DB:
- Key
str
- Value
str
float
int
list
dict
➡️ Setters:
Key: str
, Value: str
:
With dictionary like solution:
detti_db["test_str_key"] = "test_val" # Set the value as "test_val" (str)
:Return: True
if the setting is successful else False
Note:
- If you want to set a non-supported value type, you will get a warning message, and
the value won't be store to DB. Eg.:
-
detti_db["test_key"] = (1, 2, 3) # Try to store Tuple type >> [detti_db.py][WARNING] The getting value type is not supported (<class 'tuple'>). The value won't be stored.
-
With method usage:
detti_db.set("test_str_key_2", "test_val_2") # Set the value as "test_val_2" (str)
:Return: True
if the setting is successful else False
Note:
- The
set()
method tries to cast the getting value to string. Eg.: -
detti_db.set("test_str_key_3", 123) # Set the value as "123" (str)
Key: str
, Value: int
:
With dictionary like solution:
detti_db["test_int_key"] = 8 # Set the value as 8 (int)
:Return: True
if the setting is successful else False
With method usage:
detti_db.set_int("test_int_key_2", 9) # Set the value as 9 (int)
:Return: True
if the setting is successful else False
Note:
- The
set_int()
method tries to cast the getting value to integer. Eg.: -
detti_db.set_int("test_int_key_3", 123.123) # Set the value as 123 (int) detti_db.set_int("test_int_key_4", "888") # Set the value as 888 (int)
Key: str
, Value: float
:
With dictionary like solution:
detti_db["test_float_key"] = 8.8 # Set the value as 8.8 (float)
:Return: True
if the setting is successful else False
With method usage:
detti_db.set_float("test_float_key_2", 9.9) # Set the value as 9.9 (float)
:Return: True
if the setting is successful else False
Note:
- The
set_float()
method tries to cast the getting value to float. Eg.: -
detti_db.set_float("test_float_key_3", 123) # Set the value as 123.0 (float) detti_db.set_float("test_float_key_4", "888.888") # Set the value as 888.888 (float)
Key: str
, Value: List[Any]
:
With dictionary like solution:
detti_db["test_list_key"] = ["a", 1] # Set the value as ["a", 1] (list)
:Return: True
if the setting is successful else False
With method usage:
detti_db.set_list("test_list_key_2", ["a", 1]) # Set the value as ["a", 1] (list)
:Return: True
if the setting is successful else False
Note:
- The
set_list()
method tries to cast the getting value to float. Eg.: -
detti_db.set_list("test_list_key_3", ("a", 2)) # Set the value as ["a", 2] (list) detti_db.set_list("test_list_key_4", "abc") # Set the value as ["a", "b", "c"] (list))
Append Any
to list
:
detti_db.append_list("key", "value")
Example:
detti_db.set_list("test_list", ["a", 1])
detti_db.append_list("test_list", 666)
print(detti_db["test_list"])
>>> ["a", 1, 666]
:Return: True
if the appending is successful else False
Note:
- The return value is
False
if you try to append a new element to a not existing key in DB. - The return value is
False
if you try to append a new element to a key which value is not list type.
Key: str
, Value: dict
:
With dictionary like solution:
detti_db["test_dict_key"] = {"a": 1} # Set the value as {"a": 1} (dict)
:Return: True
if the setting is successful else False
With method usage:
detti_db.set_dict("test_dict_key_2", {"a": 1}) # Set the value as {"a": 1} (dict)
➡️ Getters:
Get element
With dictionary like solution:
detti_db["test_key"] # Return: "test_val"
:Return: The requested value if it exists in DB else None
With method usage:
detti_db.get("test_key_2") # Return: "test_val_2"
:Return: The requested value if it exists in DB else None
Note:
- The above getter solutions can return any types.
Using default value in get:
detti_db.get("not_exist_key", default_value=5) # Return: 5 (Due to the "not_exist_key" key is not in DB.)
:Return: The requested value if it exists in DB else the set default value.
Note:
- The above getter solutions can return any types.
- Any type can be set as default parameter.
Get all elements:
The get_all()
method provides the all key-value pairs from DB in Json format. It supports any types.
detti_db.get_all() # Return: {'test_key': 'test_val', 'test_key_2': 'test_val_2'}
:Return: The requested items in dict if any exists in DB else empty dict
Check if key exists in DB:
The is_exist()
method returns True
if the key is in DB else False
.
detti_db.is_exist("elem_key")
:Return: True
if the key is in DB else False
.
Deletion:
With dictionary like solution:
del detti_db["test_key"]
:Return: True
if the setting is successful else False
With method usage:
detti_db.delete("test_key_2") # Return True if it's success else False
:Return: True
if the setting is successful else False
Searching:
Search keys based on provided prefix (Returning a Dict[str, str]
):
detti_db.search_keys_in_db("my_") # Return: {'my_test_key_4': 'test_val_4'}
:Return: The requested items in dict if any exists in DB else empty dict
Search values based on provided prefix (Returning a Dict[str, str]
):
detti_db.search_values_in_db("my_") # Return: {'test_key_4': 'my_test_val_4'}
:Return: The requested items in dict if any exists in DB else empty dict
Get size of DB:
detti_db.size_of_db() # Return: 666 (int)
:Return: Size of the DB is bytes (int
).
Get all keys of DB:
detti_db.get_all_keys() # Return: ["test_key", "test"key_2"] (List[str])
:Return: All keys of the DB (List[str]
).
Dump the current loaded DB to Json file:
detti_db.dump_to_json("dump.json")
Possible parameters:
file_path
: Path of the destination file.force
: If it is given the existing file will be overwritten.- Default: False
permissions
: Set the permission of file. It is true for overwritten files! Octal!- Default: 0o600
Eg.:
detti_db.dump_to_json(file_path="dump.json", force=True, permissions=0o664)
Complete example code (With not existing DB):
import os
import sys
# Get the path of the directory of the current file.
PATH_OF_FILE_DIR: str = os.path.realpath(os.path.dirname(__file__))
# Append the path of the tools folder to find modules.
sys.path.append(os.path.join(PATH_OF_FILE_DIR, ".."))
from detti_db import DettiDB # noqa: E40
detti_db: DettiDB = DettiDB()
# setters
detti_db["test_key"] = "test_val"
detti_db.set("test_key_2", "test_val_2")
detti_db.set_int("test_int_key", 123)
detti_db.set_float("test_float_key", 123.123)
detti_db.set_list("test_list_key", ["a", 1])
detti_db.set_dict("test_dict_key", {"b": 2})
# getters
print("test_key -> {}".format(detti_db["test_key"]))
print("test_int_key -> {}".format(detti_db.get("test_int_key")))
print("test_list_key -> {}".format(detti_db["test_list_key"]))
print("test_dict_key -> {}".format(detti_db["test_dict_key"]))
print("All content: {}".format(detti_db.get_all()))
print("Number of elements in DB: {}".format(detti_db.get_number_of_elements()))
print("Size of DB: {}".format(detti_db.size_of_db()))
print("All keys in DB: {}".format(detti_db.get_all_keys()))
# deletions
del detti_db["test_key"]
detti_db.delete("test_key_2")
# Set some new items for searching
detti_db["test_key_3"] = "my_test_val_3"
detti_db["my_test_key_4"] = "test_val_4"
# Searching
print("'my_' key prefixes -> {}".format(detti_db.search_keys_in_db("my_")))
print("'my_' value prefixes -> {}".format(detti_db.search_values_in_db("my_")))
Output:
>>> python3 examples/db_example_1.py
test_key -> test_val
test_int_key -> 123
test_list_key -> ['a', 1]
test_dict_key -> {'b': 2}
All content: {'test_key': 'test_val', 'test_key_2': 'test_val_2', 'test_int_key': 123, 'test_float_key': 123.123, 'test_list_key': ['a', 1], 'test_dict_key': {'b': 2}}
Number of elements in DB: 6
Size of DB: 216
All keys in DB: ['test_key', 'test_key_2', 'test_int_key', 'test_float_key', 'test_list_key', 'test_dict_key']
'my_' key prefixes -> {'my_test_key_4': 'test_val_4'}
'my_' value prefixes -> {'test_key_3': 'my_test_val_3'}
The default config is detti_conf.ini
.
It is in the root folder (next to the detti_db.py
file).
The configuration file is a standard INI file format.
Default config:
[DETTI_DB]
# Path of the DB file. Recommended to define full path.
path_of_db = test.db
# Maximum length of the keys in DB (Avoid memory overload).
len_of_key = 100
# Maximum length of the values in DB (Avoid memory overload).
len_of_val = 100
# Level of the logger. Possible: DEBUG, INFO, WARNING, ERROR, CRITICAL
# IMPORTANT: The generated log file will contain all log level messages!
log_level = WARNING
[SERVER]
host = localhost
port = 5000
debug = True
# Setting the request limits in different unites. The most strict will be used!
sec_limit = 5
min_limit = 300
hour_limit = 18000
day_limit = 432000
# IMPORTANT
# If you set the user and password parameter the DB will be accessed with JWT Token!
user =
password =
Note:
- The default
detti_conf.ini
file contains more sections but theSERVER
andDETTI_DB
sections are related (and mandatory) to the Server running. Other sections are not used in case of DB. It is not problem if other (default) sections are not in the config file.
Use own config file:
- There is
config_file
CLI argument (str
).- Force to use a config file. Default:
detti_conf.ini
(In root folder)
- Force to use a config file. Default:
- Example:
>>> python3 detti_server --config_file my_own_config.ini
Start the server:
>>> python3 detti_server.py
Output in case of successful starting (with the default config):
* Serving Flask app "detti_server" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://localhost:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 217-599-780
The server status can be checked to send a GET request to /ping
end-point of the server.
Response if the server is up and running:
Curl:
>>> curl http://localhost:5000/ping
> "PONG"
Python:
import requests
requests.get("http://localhost:5000/ping") # Return: "PONG"
Response if the server is down (Status code: 7):
>>> curl http://localhost:5000/ping
> curl: (7) Failed to connect to localhost port 5000: Kapcsolat elutasítva
/get/<string:db_key>
Providing the key-value pair based on getting key. The status code is 201 in case of error.
Curl:
>>> curl http://localhost:5000/set -d "exist=value_of_exist_key" -X PUT
> {"STATUS": "OK"}
>>> curl http://localhost:5000/get/exist
> {"exist": "value_of_exist_key"}
>>> curl http://localhost:5000/get/doesnt_exist
> {"doesnt_exist": "The key doesn't exist in DB."}
Python:
import requests
# Set an element in the DB
put_resp = requests.put("http://localhost:5000/set", data={"exist": "value_of_exist_key"})
print(put_resp.json()) # Return: {"STATUS": "OK"}
# Get the element from DB
resp = requests.get("http://localhost:5000/get/exist")
print(resp.json()) # Return: {"exist": "value_of_exist_key"}
/set
Setting/updating key-value pair in the DB.
Curl:
>>> curl http://localhost:5000/set -d "test_key=test_val" -X PUT
> {"STATUS": "OK"}
>>> curl http://localhost:5000/get/test_key
> {"test_key": "test_val"}
Python:
import requests
# Set an element in the DB
put_resp = requests.put("http://localhost:5000/set", data={"test_key": "test_val"})
print(put_resp.json()) # Return: {"STATUS": "OK"}
# Get the element from DB
resp = requests.get("http://localhost:5000/get/test_key")
print(resp.json()) # Return: {"test_key": "test_val"}
/search_key/<string:key_prefix>
Searching keys in the DB based on provided key prefix.
Curl:
>>> curl http://localhost:5000/set -d "test_key=test_val" -X PUT
> {"STATUS": "OK"}
>>> curl http://localhost:5000/set -d "prod_key_1=prod_val_1" -X PUT
> {"STATUS": "OK"}
>>> curl http://localhost:5000/set -d "prod_key_2=prod_val_2" -X PUT
> {"STATUS": "OK"}
>>> curl http://localhost:5000/search_key/prod_
> {
"prod_key_1": "prod_val_1",
"prod_key_2": "prod_val_2"
}
>>> curl http://localhost:5000/search_key/not_exist
> {"not_exist": "Cannot find keys for prefix"}
Python:
import requests
# Set some elements in DB.
requests.put("http://localhost:5000/set", data={"dev_data": "dev"})
requests.put("http://localhost:5000/set", data={"prod_key_1": "prod_val_1"})
requests.put("http://localhost:5000/set", data={"prod_key_2": "prod_val_2"})
# Get the elements based on provided KEY prefix.
resp = requests.get("http://localhost:5000/search_key/prod_")
print(resp.json()) # Return: {"prod_key_1": "prod_val_1", "prod_key_2": "prod_val_2"}
# Try a KEY prefix which is not contained in keys.
resp = requests.get("http://localhost:5000/search_key/nonono")
print(resp.json()) # Return: {"nonono": "Cannot find keys for prefix"}
/search_val/<string:value_prefix>
Searching values in the DB based on provided value prefix.
Curl:
>>> curl http://localhost:5000/set -d "test_key=test_val" -X PUT
> {"STATUS": "OK"}
>>> curl http://localhost:5000/set -d "prod_key_1=prod_val_1" -X PUT
> {"STATUS": "OK"}
>>> curl http://localhost:5000/set -d "prod_key_2=prod_val_2" -X PUT
> {"STATUS": "OK"}
>>> curl http://localhost:5000/search_val/prod_
> {
"prod_key_1": "prod_val_1",
"prod_key_2": "prod_val_2"
}
>>> curl http://localhost:5000/search_val/not_exist
> {"not_exist": "Cannot find values for prefix"}
Python:
import requests
# Set some elements in DB.
requests.put("http://localhost:5000/set", data={"dev_data": "dev"})
requests.put("http://localhost:5000/set", data={"prod_key_1": "prod_val_1"})
requests.put("http://localhost:5000/set", data={"prod_key_2": "prod_val_2"})
# Get the elements based on provided VALUE prefix.
resp = requests.get("http://localhost:5000/search_val/prod_")
print(resp.json()) # Return: {"prod_key_1": "prod_val_1", "prod_key_2": "prod_val_2"}
# Try a VALUE prefix which is not contained in keys.
resp = requests.get("http://localhost:5000/search_val/nonono")
print(resp.json()) # Return: {"nonono": "Cannot find values for prefix"}
/delete/<string:db_key>
Deleting an element from the DB.
Curl:
>> curl http://localhost:5000/set -d "test_key=test_val" -X PUT
> {"STATUS": "OK"}
>> curl http://localhost:5000/get/test_key
> {"test_key": "test_val"}
>> curl http://localhost:5000/delete/test_key -X DELETE
> {"STATUS": "OK"}
>> curl http://localhost:5000/get/test_key
> {"test_key": "The key doesn't exist in DB."}
Python:
import requests
# Set some elements in DB.
requests.put("http://localhost:5000/set", data={"to_be_deleted": "dummy"})
# Delete the created item
del_resp = requests.delete("http://localhost:5000/delete/to_be_deleted")
print(del_resp.json()) # Return: {"STATUS": "OK"}
# Try to get the deleted item
resp = requests.get("http://localhost:5000/get/to_be_deleted")
print(resp.status_code) # Return: 201
print(resp.json()) # Return: {"to_be_deleted": "The key doesn't exist in DB."}
/ping
Checking if the server is running.
Success example:
>>> curl http://localhost:5000/ping
> "PONG"
Failed example (Status code: 7):
>>> curl http://localhost:5000/ping
> curl: (7) Failed to connect to localhost port 5000: Kapcsolat elutasítva
/getall
Providing all elements from the DB. {key: value, key: value}
Curl:
>>> curl http://localhost:5000/set -d "test_key=test_val" -X PUT
> {"STATUS": "OK"}
>>> curl http://localhost:5000/set -d "test_key_1=test_val_1" -X PUT
> {"STATUS": "OK"}
>>> curl http://localhost:5000/getall
> {
"test_key": "test_val",
"test_key_1": "test_val_1"
}
Python:
import requests
# Set some elements in DB.
requests.put("http://localhost:5000/set", data={"get_all_1": "dummy"})
requests.put("http://localhost:5000/set", data={"get_all_2": "dummy"})
# Getting all elements from DB.
resp = requests.get("http://localhost:5000/getall")
print(resp.json()) # Return: {"get_all_1": "dummy", "get_all_2": "dummy"}
Official page of JWT:
Important:
- The JWT authentication is not active with the default configuration.
The "user" and "password" parameters have to be set in the configuration file to activate the JWT Authentication.
For example:
user = test_user
password = test_password
Get the token from the server:
>>> curl -i -X POST -H "Content-Type: application/json" -d '{"username":"test_user","password":"test_password"}' http://localhost:5000/auth
> {
"access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2MTE4...
}
Use the JWT authentication for APIs:
>>> curl -H "Authorization: jwt eyJ0eXAiOiJKV..." http://localhost:5000/get/exist
> {
"exist": "exist_val"
}
If the token is not used, the APIs provide error message with 401 status code:
>>> curl http://localhost:5000/get/exist
> {
"description": "Request does not contain an access token",
"error": "Authorization Required",
"status_code": 401
}
Currently, the production line support is not implemented in this repo (But it is in the road-map)! You can run the server on the production line with Nginx and Gunicorn.
Tutorial:
- Introduce the multithreading/multiprocessing in searching methods
- In case of big data the multithreading/multiprocessing can reduce the execution time
- Add support for more data types on server side.
- Placeholder
- Pass the
**options
parameter inrun_server()
function.- The options to be forwarded to the underlying Werkzeug server.
See :func:
werkzeug.serving.run_simple
for more information.
- The options to be forwarded to the underlying Werkzeug server.
See :func:
- Add the
dump_to_json()
method to dump the current DB to Json file. - Add
Python Black
andFlake8
pre-commit hooks to repo for better contributions. - Fix TypeError in case of
dict
type setting. - Add Cross Origin Resource Sharing (CORS) for server to make cross-origin AJAX possible.
- Add
get_number_of_elements()
method to get number of elements of DB. - Add
get_all_keys()
method to get all keys of DB. - Add
set_dict()
method asdict
type setter. - Add
run_server()
function to server part for better integration.- Now the
detti_server.py
file can be imported and configurable before starting the server. - Eg.:
import detti_server; detti_server.run_server()
- Now the
- Get size of DB with
size_of_db()
method. - Implement
__contains__
magic method (Eg.:"a" in detti_db
). - Add
default_value
option toget
method (Eg.:detti_db.get("not_exist_key", default_value=5)
) - All parameters from config file can be overwritten as
__init__
argument (Eg.:detti_db = DettiDB(len_of_val=50)
)