Skip to content

jackdoe/sfs

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

*WARNING: experimental stage, work in progress, DOES NOT USE SSL, DO NOT USE IN PRODUCTION*

very basic mysql backed fuse file system.
0: abstract
	fuse based, mysql backed file system

1. requirements:
	fuse, mysql, pthreads, c compiler

2. install:
	download from https://github.com/jackdoe/sfs
	$EDOTOR common.h #put proper mysql settings into common.h
	make #DEBUG=1 if you want debug output
	./sfs /mnt/fuse

3. info:
	this is the table that is used:
	
	Q("CREATE TABLE IF NOT EXISTS `entries_%s` (	\
	`id` bigint(20) NOT NULL AUTO_INCREMENT,	\
	`path` varchar(255) NOT NULL,			\
	`data` longblob,				\
	`obj_type` int(11) NOT NULL,			\
	`uid` bigint(20) NOT NULL,			\
	`gid` bigint(20) NOT NULL,			\
	`mode` bigint(20) NOT NULL,			\
	`atime` bigint(20) NOT NULL,			\
	`mtime` bigint(20) NOT NULL,			\
	`ctime` bigint(20) NOT NULL,			\
	`btime` bigint(20) NOT NULL,			\
	PRIMARY KEY (`id`),				\
	UNIQUE KEY `path` (`path`))",identifier);
	
	identifier is a copy of the mysql username (for example root)
	and the table will be entries_root
	table is automaticaly created at start, root directory is auto inserted as well
	
	mysql user needs to select/update/delete/insert and maybe create if you dont have
	the table created.

	
4. idea:
	i made this as proof of concept to be used in diskless linux based routers,
	where i can edit their config files remotely - for example /etc or /usr/local/etc,
	with simple web based application.

5. supported calls:
	sfs_oper.getattr	= sfs_getattr;
	sfs_oper.readdir	= sfs_readdir;
	sfs_oper.open		= sfs_open;
	sfs_oper.write		= sfs_write;
	sfs_oper.truncate	= sfs_truncate;
	sfs_oper.read		= sfs_read;
	sfs_oper.create		= sfs_create;
	sfs_oper.unlink		= sfs_delete;
	sfs_oper.chmod		= sfs_chmod;
	sfs_oper.chown		= sfs_chown;
	sfs_oper.statfs		= sfs_statfs;
	sfs_oper.mkdir		= sfs_mkdir;
	sfs_oper.readlink	= sfs_readlink;
	sfs_oper.symlink	= sfs_symlink;
	sfs_oper.rename		= sfs_rename;
	sfs_oper.rmdir		= sfs_rmdir;
	sfs_oper.utimens	= sfs_utimens;

6. limitations:
	MAX_PATH_SIZE: 256 chars
	max data size: mysql's longblob 4gb
	read/write is extreamly slow on big files
	currently there is no locking

7. example usage:
	
	$ mkdir ~/mysql_backed_fs
	$ ./sfs ~/mysql_backed_fs/
	$ cd ~/mysql_backed_fs
	mysql_backed_fs $ mkdir test_directory
	mysql_backed_fs $ cd test_directory/
	test_directory $ ls -la
	total 0
	drwxr-xr-x  1 jack  staff  0 Dec 27 09:24 .
	drwxr-xr-x  1 jack  staff  0 Dec 24 15:35 ..
	test_directory $ touch empty_file
	test_directory $ cat empty_file	
	test_directory $ echo 'not empty anymore' >> empty_file
	test_directory $ cat empty_file 
	not empty anymore
	test_directory $ cd ..
	mysql_backed_fs $ ln -s test_directory/empty_file link
	mysql_backed_fs $ cat link
	not empty anymore
	mysql_backed_fs $ ln -s test_directory link2
	mysql_backed_fs $ cat link2/empty_file 
	not empty anymore
	mysql_backed_fs $ cd ..
	$ dd if=/dev/random of=blob.dd bs=1m count=1 && md5 blob.dd 
	1+0 records in
	1+0 records out
	1048576 bytes transferred in 0.080215 secs (13072073 bytes/sec)
	MD5 (blob.dd) = 7d84c26f72aa5440c778014b1c9d23fe
	$ cp blob.dd mysql_backed_fs/ && md5 mysql_backed_fs/blob.dd
	MD5 (mysql_backed_fs/blob.dd) = 7d84c26f72aa5440c778014b1c9d23fe
	$ cd mysql_backed_fs

as you can see write is very very slow

	mysql_backed_fs $ dd if=/dev/zero of=./speed.blob.dd bs=1m count=5 
	5+0 records in
	5+0 records out
	5242880 bytes transferred in 14.620257 secs (358604 bytes/sec)

reading is fine on not very large files tho, because it uses SUBSTRING() and 
gets only requested offset+size from the mysql

	mysql_backed_fs $ time cat speed.blob.dd > /dev/null
	real	0m0.005s
	user	0m0.001s
	sys	0m0.004s
	

mysql does not have a way for partial updates on blob fields so
there is download/modify (offset+size - usually 65k)/push the whole file and 
the speed drops like a rock in water :)
 
	mysql_backed_fs$ dd if=/dev/zero of=./speed.blob.dd bs=65k count=1
	1+0 records in
	1+0 records out
	66560 bytes transferred in 0.004575 secs (14548589 bytes/sec)
	mysql_backed_fs$ dd if=/dev/zero of=./speed.blob.dd bs=256k count=1
	1+0 records in
	1+0 records out
	262144 bytes transferred in 0.015075 secs (17389082 bytes/sec)
	mysql_backed_fs$ dd if=/dev/zero of=./speed.blob.dd bs=512k count=1
	1+0 records in
	1+0 records out
	524288 bytes transferred in 0.046395 secs (11300513 bytes/sec)
	mysql_backed_fs$ dd if=/dev/zero of=./speed.blob.dd bs=1024k count=1
	1+0 records in
	1+0 records out
	1048576 bytes transferred in 0.301245 secs (3480808 bytes/sec)
	mysql_backed_fs$ dd if=/dev/zero of=./speed.blob.dd bs=2048k count=1
	1+0 records in
	1+0 records out
	2097152 bytes transferred in 1.876782 secs (1117419 bytes/sec)

but for small files it is fine

About

fuse based mysql backed file system

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published