Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] single threaded nats client POC #769

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[DO NOT MERGE] single threaded nats client POC #769

wants to merge 1 commit into from

Conversation

levb
Copy link
Collaborator

@levb levb commented Jun 24, 2024 •
edited
Loading

PR for discussion only, not to be merged.

  • uses libevent for all read/write, still uses natsSock_ConnectTcp to connect
  • heap implementation abstracted, and the use consolidated
  • can connect, initial ping, responds to server pings, sends messages
  • no reconnect, TLS, auth, subs, requests, etc.
  • writes are not optimized at all, yet. Turn on DEV_MODE_CONN_TRACE to see the trace.

Example: https://github.com/nats-io/nats.c/blob/mininats/examples/libevent-pub.c
It connects and sends 5 messages to a server. The entire heap usage is logged (there are still leaks, please ignore, cleanup is not fully implemented).

To see more details, turn on more logging here

➜  build git:(mininats) ./bin/nats-libevent-pub
TRACE:   HEAP: allocated bytes:4096, ptr:0x153008800 (nats_log_createPool:mem_pool.c:600)
Hello, mini-nats! Will try to send a message to 'foo'
TRACE:   HEAP: allocated bytes:4096, ptr:0x15300b800 (nats_log_createPool:mem_pool.c:600)
TRACE:   HEAP: allocated bytes:4096, ptr:0x15300d800 (nats_log_createPool:mem_pool.c:600)
TRACE:   HEAP: allocated bytes:4096, ptr:0x154008200 (nats_log_createPool:mem_pool.c:600)
TRACE:   HEAP: allocated bytes:65536, ptr:0x158008000 (natsPool_getReadBuffer:mem_pool.c:216)
TRACE:   HEAP: allocated bytes:4096, ptr:0x154009200 (_allocSmall:mem_pool.c:135)
TRACE:   HEAP: allocated bytes:4096, ptr:0x154808200 (nats_log_createPool:mem_pool.c:600)
Message enqueued, text: Hello, NATS!
TRACE:   HEAP: allocated bytes:4096, ptr:0x154809200 (nats_log_createPool:mem_pool.c:600)
Message enqueued as a copy, freeing the data now: Hello, NATS! I was allocated on the heap, my bytes were copied into another buffer, and I was freed immediately by the app.
TRACE:   HEAP: allocated bytes:4096, ptr:0x15480a200 (nats_log_createPool:mem_pool.c:600)
Message enqueued, text: Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was freed by a cusom 'message published' callback
TRACE:   HEAP: allocated bytes:4096, ptr:0x15480b200 (nats_log_createPool:mem_pool.c:600)
Message enqueued, text: Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was 'free'-ed by the cleanup, no custom callback needed.
TRACE:   HEAP: allocated bytes:4096, ptr:0x15480c200 (nats_log_createPool:mem_pool.c:600)
FINAL message enqueued, text: Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was 'free'-ed by the cleanup, no custom callback needed.
TRACE:   HEAP: freeing ptr:0x154808200 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x154809200 (_free:mem_pool.c:84)
Message published, freeing the data: Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was freed by a cusom 'message published' callback
TRACE:   HEAP: freeing ptr:0x15480a200 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x15480b200 (_free:mem_pool.c:84)
Last message published, now disconnecting:
Connection closed, last error: ''
TRACE:   HEAP: freeing ptr:0x154008200 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x154009200 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x15300d800 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x15300b800 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x153008800 (_free:mem_pool.c:84)

server log:

[70659] 2024/06/24 08:52:14.781129 [TRC] [::1]:59103 - cid:85 - <<- [CONNECT {"verbose":false,"pedantic":false,"tls_required":false,"name":"","lang":"C","version":"4.0.0-mininats","protocol":1,"echo":true,"headers":true,"no_responders":true}]
[70659] 2024/06/24 08:52:14.781167 [TRC] [::1]:59103 - cid:85 - <<- [PING]
[70659] 2024/06/24 08:52:14.781169 [TRC] [::1]:59103 - cid:85 - ->> [PONG]
[70659] 2024/06/24 08:52:14.781292 [TRC] [::1]:59103 - cid:85 - <<- [PUB foo 12]
[70659] 2024/06/24 08:52:14.781296 [TRC] [::1]:59103 - cid:85 - <<- MSG_PAYLOAD: ["Hello, NATS!"]
[70659] 2024/06/24 08:52:14.781301 [TRC] [::1]:59103 - cid:85 - <<- [PUB foo 123]
[70659] 2024/06/24 08:52:14.781351 [TRC] [::1]:59103 - cid:85 - <<- MSG_PAYLOAD: ["Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was freed by a cusom 'message published' callback"]
[70659] 2024/06/24 08:52:14.781368 [TRC] [::1]:59103 - cid:85 - <<- [PUB foo 123]
[70659] 2024/06/24 08:52:14.781382 [TRC] [::1]:59103 - cid:85 - <<- MSG_PAYLOAD: ["Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was freed by a cusom 'message published' callback"]
[70659] 2024/06/24 08:52:14.781387 [TRC] [::1]:59103 - cid:85 - <<- [PUB foo 130]
[70659] 2024/06/24 08:52:14.781391 [TRC] [::1]:59103 - cid:85 - <<- MSG_PAYLOAD: ["Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was 'free'-ed by the cleanup, no custom callback needed."]
[70659] 2024/06/24 08:52:14.781398 [TRC] [::1]:59103 - cid:85 - <<- [PUB foo 0]
[70659] 2024/06/24 08:52:14.781402 [TRC] [::1]:59103 - cid:85 - <<- MSG_PAYLOAD: [""]
[70659] 2024/06/24 08:52:14.781409 [DBG] [::1]:59103 - cid:85 - Client connection closed: Read Error

@levb levb added the proposal Enhancement idea or proposal label Jun 24, 2024
@levb levb requested a review from kozlovic June 24, 2024 15:47
kozlovic
kozlovic reviewed Jun 28, 2024
View reviewed changes
Copy link
Member

@kozlovic kozlovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too much to review. This is obviously very early and more of a POC, but you should still try to run a little benchmark for the publish to see how it will perform. I know that early in the C client development the publish perf was really bad compared to the Go client because each publish ended-up result in the tcp write (because notifying the flusher would do the flush right away). He we copy/move the date to a chain but how often the chain will be flushed/processed will dictate the publish performance.

src/conn.c

// Initialize the connect-time memory pool.
s = nats_createPool(&nc->connectPool, &nc->opts->mem, "conn-connect");
if (s != NATS_OK)
return NATS_UPDATE_ERR_STACK(s);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the state set above remain the same or be updated on failure? Or maybe set the state after this possible men failure.

src/conn.c
s = natsSock_ConnectTcp(&(nc->sockCtx), nc->connectPool, nc->cur->url->host, nc->cur->url->port);
if (STILL_OK(s))
nc->sockCtx.fdActive = true;
if (STILL_OK(s))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed, falls under the above STILL_OK.

src/conn.c

return NATS_UPDATE_ERR_STACK(s);
}
natsConn_retain(nc);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment says that locking is done inside the function itself, but I don't see it happening (setting "nc->state", etc..).

@levb
Copy link
Collaborator Author

levb commented Jun 28, 2024

@kozlovic ACK that it is way too early to review, and I am focused on other things for the time being anyway. I marked you to get your "FYI"-level attention, feel free to un-assign if this is too noisy.

...each publish ended-up result in the tcp write

the plan was to optimize it to use writev, like in https://github.com/nginx/nginx/blob/e734df6664e70f118ca3140bcef6d4f1750fa8fa/src/os/unix/ngx_writev_chain.c#L14 but I didn't get to prototype it. So everything would still be buffered (into a chain), and then segments written out of the chain, 1 iovec at a time per write event. I just didn't get to the sockets at all, figured I'd wrap it on top of the existing code for now. Ditto establishing connections - that too would need to be event driven for re-connections.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kozlovic kozlovic kozlovic left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
proposal Enhancement idea or proposal
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@levb @kozlovic