Memcache Support #2096

ilixiaocui · 2022-11-22T11:37:55Z

What problem does this PR solve?

Issue Number: #xxx

Problem Summary:

What is changed and how it works?

What's Changed:

How it Works:

Side effects(Breaking backward compatibility? Performance regression?):

Check List

Relevant documentation/comments is changed or added
I acknowledge that all my contributions will be made under the project's license

ilixiaocui · 2022-12-06T10:49:26Z

recheck

SeanHai · 2022-12-07T05:43:46Z

Makefile

@@ -59,7 +60,7 @@ list:
 	@bash util/build.sh --stor=$(stor) --list

 build:
-	@bash util/build.sh --stor=$(stor) --only=$(only) --dep=$(dep) --release=$(release) --os=$(os)
+	@bash util/build.sh --stor=${stor} --only=$(only) --dep=$(dep) --release=$(release) --ci=$(ci) --os=$(os)


wthat's the 'ci' option used for?

SeanHai · 2022-12-07T06:22:06Z

curvefs/src/client/fuse_s3_client.cpp

+    if (!mdsClient_->AllocOrGetMemcacheCluster(fsInfo_->fsid(),
+                                               &kvcachecluster)) {
+        LOG(ERROR) << "FLAGS_supportKVcache = " << FLAGS_supportKVcache
+                   << ", but AllocOrGetMemcacheCluster  fail";


more space before "fail"

SeanHai · 2022-12-07T06:23:05Z

curvefs/src/client/fuse_s3_client.cpp

+        return false;
+    }
+
+    // std::string ip[3] = {"10.182.2.46", "10.182.2.47", "10.182.2.48"};


Delete code needn't

SeanHai · 2022-12-07T06:49:35Z

curvefs/src/client/kvclient/kvclient_manager.cpp

+}
+
+bool KVClientManager::Get(std::shared_ptr<GetKvCacheContext> task) {
+    LatencyGuard guard(&g_kvClientMetric->kvClientGet.latency);


This 'Get' function can request the 'Get' function in line84.

assert(nullptr != task->value);
return Get(task->key, task->value, task->offset, task->length);

SeanHai · 2022-12-07T06:51:09Z

curvefs/src/client/rpcclient/mds_client.cpp

+
+bool MdsClientImpl::AllocOrGetMemcacheCluster(
+    uint32_t fsId, curvefs::mds::topology::MemcacheCluster *cluster) {
+    return false;


Why use a fake one?

it will be implemented by #2108

SeanHai · 2022-12-07T07:21:32Z

curvefs/src/client/s3/client_s3_cache_manager.cpp

+        GenerateKVReuqest(inodeWrapper, memCacheMissRequest, dataBuf,
+                          &kvRequest);
+
+        // read from s3


read from diskcache or s3？
The read order is kvcache -> diskcache -> s3?

localcache/remote kv cluster fail will not return error code.
Failure to read from s3 will eventually return failure.
done.

SeanHai · 2022-12-07T07:21:55Z

curvefs/src/client/s3/client_s3_cache_manager.cpp

+            // meaningful error code
+            //  read from s3 not exist
+            // 1. may be the metaserver compaction update inode is not
+            //    synchronized to the client. clear inodecache && get agin


Yes. ReadKVRequest is already handled with while.
The situations of s3 return error will retry：If the error is not exist, it may be that the data fragment has not been transmitted to s3, and it will wait for retry according to HandleReadS3NotExist. Other errors indicate that reading from s3 failed and will not be retried.

cw123 · 2022-12-07T06:08:57Z

util/image.sh

 copy_file ./thirdparties/aws/aws-sdk-cpp/build/aws-cpp-sdk-s3-crt/libaws-cpp-sdk-s3-crt.so $docker_prefix
+copy_file ./thirdparties/memcache/libmemcached-1.1.2/build-libmemcached/src/libmemcached/libmemcached.so $docker_prefix
+copy_file ./thirdparties/memcache/libmemcached-1.1.2/build-libmemcached/src/libmemcached/libmemcached.so.11 $docker_prefix


lrwxrwxrwx 1 xxxx 18 Dec 7 14:05 libmemcached.so -> libmemcached.so.11
lrwxrwxrwx 1 xxxx 22 Dec 7 14:05 libmemcached.so.11 -> libmemcached.so.11.0.0
-rwxr-xr-x 1 xxxx 225728 Dec 7 14:05 libmemcached.so.11.0.0

need copy libmemcached.so.11.0.0 ?

From the perspective of ldd, it directly depends on so.11, and the actual construction environment test only needs to rely on so.11.

cw123 · 2022-12-07T06:33:23Z

curvefs/src/client/fuse_s3_client.cpp

+        return false;
+    }
+
+    // std::string ip[3] = {"10.182.2.46", "10.182.2.47", "10.182.2.48"};


delete unused code

cw123 · 2022-12-07T06:34:25Z

curvefs/src/client/fuse_s3_client.cpp

+    }
+
+    // init kvcacheclient manager
+    // KVClientManagerOpt config;


delete unused code

cw123 · 2022-12-07T06:39:24Z

curvefs/src/client/rpcclient/mds_client.cpp

+
+bool MdsClientImpl::AllocOrGetMemcacheCluster(
+    uint32_t fsId, curvefs::mds::topology::MemcacheCluster *cluster) {
+    return false;


return false derectly?

it will be implemented by #2108

cw123 · 2022-12-07T07:23:44Z

curvefs/src/client/kvclient/kvclient_manager.cpp

+#define ONRETURN(TYPE, RES, KEY, ERRORLOG)                                     \
+    if (RES) {                                                                 \
+        g_kvClientMetric->kvClient##TYPE.qps.count << 1;                       \
+        VLOG(9) << "Set key = " << KEY << " OK";                               \


set key -> type key

cw123 · 2022-12-07T07:23:51Z

curvefs/src/client/kvclient/kvclient_manager.cpp

+        VLOG(9) << "Set key = " << KEY << " OK";                               \
+    } else {                                                                   \
+        g_kvClientMetric->kvClient##TYPE.eps.count << 1;                       \
+        LOG(ERROR) << "Set key = " << KEY << " error = " << ERRORLOG;          \


set key -> type key

cw123 · 2022-12-07T08:18:58Z

curvefs/src/client/s3/client_s3_cache_manager.cpp

+                                           char *databuf, uint64_t offset,
+                                           uint64_t length, int *ret) {
+    uint64_t start = butil::cpuwide_time_us();
+    *ret = s3ClientAdaptor_->GetS3Client()->Download(name, databuf, offset,


There is only the logic of sync download here, do you not consider async download?

Yes, it's a remaining problem and is fixed by #2080

cw123 · 2022-12-07T08:54:09Z

curvefs/src/client/common/config.h

@@ -80,6 +80,11 @@ struct SpaceAllocServerOption {
    uint64_t rpcTimeoutMs;
 };

+struct KVClientManagerOpt {
+    int setThreadPooln = 4;
+    int getThreadPooln = 4;


getThreadPooln where does this param use?

Here is a reserved parameter, libmemcached, the current overall reading process is synchronous. Also waiting for pr #2080

Signed-off-by: ilixiaocui <ilixiaocui@163.com>

cw123 · 2022-12-08T09:24:03Z

this pr should wait #2108 #2080 ?

ilixiaocui · 2022-12-08T09:31:36Z

this pr should wait #2108 #2080 ?

After #2108 is merged, we can test it.

#2080 #2100 involves performance issues，we can wait it.

ilixiaocui force-pushed the add_global_cache branch 15 times, most recently from e5a62fe to 81efaa0 Compare December 1, 2022 12:24

ilixiaocui force-pushed the add_global_cache branch 11 times, most recently from 676140c to 359e7ca Compare December 6, 2022 06:29

ilixiaocui force-pushed the add_global_cache branch 2 times, most recently from e7512e8 to dc4fb31 Compare December 6, 2022 12:27

SeanHai reviewed Dec 7, 2022

View reviewed changes

cw123 reviewed Dec 7, 2022

View reviewed changes

cw123 mentioned this pull request Dec 7, 2022

curvefs: support memcache cluster #2108

Merged

2 tasks

ilixiaocui force-pushed the add_global_cache branch from dc4fb31 to f347a8f Compare December 7, 2022 12:02

ilixiaocui marked this pull request as ready for review December 7, 2022 12:03

ilixiaocui force-pushed the add_global_cache branch 3 times, most recently from 2bdaf12 to f5a0193 Compare December 8, 2022 03:08

curvefs: support client to memcached cluster

ba71cc1

Signed-off-by: ilixiaocui <ilixiaocui@163.com>

ilixiaocui force-pushed the add_global_cache branch from f5a0193 to ba71cc1 Compare December 8, 2022 03:27

SeanHai approved these changes Dec 8, 2022

View reviewed changes

cw123 approved these changes Dec 8, 2022

View reviewed changes

ilixiaocui merged commit 2c851ac into opencurve:master Dec 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memcache Support #2096

Memcache Support #2096

ilixiaocui commented Nov 22, 2022

ilixiaocui commented Dec 6, 2022

SeanHai Dec 7, 2022

ilixiaocui Dec 7, 2022

SeanHai Dec 7, 2022

SeanHai Dec 7, 2022

ilixiaocui Dec 7, 2022

SeanHai Dec 7, 2022

ilixiaocui Dec 7, 2022

SeanHai Dec 7, 2022

ilixiaocui Dec 7, 2022

SeanHai Dec 7, 2022

ilixiaocui Dec 7, 2022

SeanHai Dec 7, 2022

ilixiaocui Dec 7, 2022

cw123 Dec 7, 2022

ilixiaocui Dec 7, 2022

cw123 Dec 7, 2022

ilixiaocui Dec 7, 2022

cw123 Dec 7, 2022

ilixiaocui Dec 7, 2022

cw123 Dec 7, 2022

ilixiaocui Dec 7, 2022

cw123 Dec 7, 2022

ilixiaocui Dec 7, 2022

cw123 Dec 7, 2022

ilixiaocui Dec 7, 2022

cw123 Dec 7, 2022

ilixiaocui Dec 7, 2022

cw123 Dec 7, 2022

ilixiaocui Dec 7, 2022

cw123 commented Dec 8, 2022

ilixiaocui commented Dec 8, 2022

Memcache Support #2096

Memcache Support #2096

Conversation

ilixiaocui commented Nov 22, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

ilixiaocui commented Dec 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cw123 commented Dec 8, 2022

ilixiaocui commented Dec 8, 2022