Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: progress logging #290

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/client/src/blob.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ pub mod create;
pub mod decompress;
pub mod download;
pub mod read;
pub mod read_progress;

/// A blob kind.
#[derive(Clone, Copy, Debug)]
Expand Down
12 changes: 12 additions & 0 deletions packages/client/src/blob/read.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
use crate::{
self as tg,
blob::read_progress::ProgressReader,
handle::Ext as _,
util::serde::{BytesBase64, SeekFromString},
Client,
Expand Down Expand Up @@ -82,6 +83,13 @@ impl tg::Blob {
Reader::new(handle, self.clone()).await
}

pub async fn progress_reader<H>(&self, handle: &H) -> tg::Result<ProgressReader<H>>
where
H: tg::Handle,
{
ProgressReader::new(self.reader(handle).await.unwrap())
}

pub async fn bytes<H>(&self, handle: &H) -> tg::Result<Vec<u8>>
where
H: tg::Handle,
Expand Down Expand Up @@ -125,6 +133,10 @@ where
self.position
}

pub fn size(&self) -> u64 {
self.size
}

pub fn end(&self) -> bool {
self.position == self.size
}
Expand Down
85 changes: 85 additions & 0 deletions packages/client/src/blob/read_progress.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
use std::sync::atomic::AtomicU64;

use crate as tg;
use crate::blob::Reader;
use std::pin::Pin;
use std::sync::{atomic::Ordering, Arc};
use tokio::io::{AsyncBufRead, AsyncRead, AsyncSeek};

pub struct ProgressReader<H> {
inner: Reader<H>,
position: Arc<AtomicU64>,
}

impl<H> ProgressReader<H>
where
H: tg::Handle,
{
pub fn new(inner: Reader<H>) -> tg::Result<ProgressReader<H>> {
Ok(ProgressReader {
inner,
position: Arc::new(AtomicU64::new(0)),
})
}

pub fn position(&self) -> Arc<AtomicU64> {
self.position.clone()
}

pub fn size(&self) -> u64 {
self.inner.size()
}
}

impl<H> AsyncRead for ProgressReader<H>
where
H: tg::Handle,
{
fn poll_read(
self: Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
buf: &mut tokio::io::ReadBuf<'_>,
) -> std::task::Poll<std::io::Result<()>> {
let this = self.get_mut();
let poll_result = Pin::new(&mut this.inner).poll_read(cx, buf);
if let std::task::Poll::Ready(Ok(())) = &poll_result {
let read_bytes = buf.filled().len() as u64;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is ugly because we also keep track of progress in consume() below. In practice, we always use one or the other in extract/(de)compress. Taking suggestions on how to improve this.

this.position.fetch_add(read_bytes, Ordering::Relaxed);
}
poll_result
}
}

impl<H> AsyncSeek for ProgressReader<H>
where
H: tg::Handle,
{
fn start_seek(self: Pin<&mut Self>, position: std::io::SeekFrom) -> std::io::Result<()> {
Pin::new(&mut self.get_mut().inner).start_seek(position)
}

fn poll_complete(
self: Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<std::io::Result<u64>> {
Pin::new(&mut self.get_mut().inner).poll_complete(cx)
}
}

impl<H> AsyncBufRead for ProgressReader<H>
where
H: tg::Handle,
{
fn poll_fill_buf(
self: Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<std::io::Result<&[u8]>> {
Pin::new(&mut self.get_mut().inner).poll_fill_buf(cx)
}

fn consume(self: Pin<&mut Self>, amt: usize) {
let this = self.get_mut();
Pin::new(&mut this.inner).consume(amt);
this.position.fetch_add(amt as u64, Ordering::Relaxed);
}
}