Using liberofs

[cgwalters]

It came up in a discussion that perhaps instead of composefs having its own "write EROFS" code we could reuse liberofs from https://github.com/erofs/erofs-utils/

These are just the things I'm thinking about:

• API stability; the library isn't stable, but we could probably get around that with vendoring
• Licensing erofs-utils is GPLv2+ or ASL2.0+ but our lib is LGPLv2+ - I guess it works? Yeah, LGPLv2+ should be compatible with ASL2, I was just thinking of ASL2 and GPLv2 being classically incompatible
• Format stability: this is a big one. Effectively we promise that the input mkcomposefs accepts today will result in a bit for bit compatible output going forward. Now it seems unlikely mkfs.erofs would change what it does by default, but if it ever did we'd need an option to control that.
• block vs byte based: AFAICS liberofs operates in terms of blocks (which makes total sense) but the public API we've exposed here via lcfs_write_to effectively promises to operate in terms of streaming. I think it wouldn't be too hard to bridge these two, but still a concern.

Combining these things, I think it's potentially viable for us to investigate vendoring portions of liberofs to start.

[hsiangkao]

I could help for composefs after I sort out apis mess...
liberofs could be an alternative first since

• as you said: format stability
• I do need time to make sure no memory leak in liberofs itself. For original mkfs.erofs usage likewise, it was considered harmless..

[cgwalters]

I do need time to make sure no memory leak in liberofs itself. For original mkfs.erofs usage likewise, it was considered harmless..

Heh yeah, understood...as I like to say in C one needs to work hard not to leak memory, in Rust one needs to work hard to leak memory.

That said, any reason you're not using __attribute__((cleanup()))?

[hsiangkao]

I do need time to make sure no memory leak in liberofs itself. For original mkfs.erofs usage likewise, it was considered harmless..

Heh yeah, understood...as I like to say in C one needs to work hard not to leak memory, in Rust one needs to work hard to leak memory.

Rust is a great thing. Yet Rust is still not quite portable and syntax-stable compared to C and at an early stage of Linux distribution ecosystems (Rust crates vs C libararies).

I don't have enough time to maintain several language implementaions (some other people asked for Go implementation) mainly due to active development. But if any volunteer I'd happy to enbrace them.

That said, any reason you're not using __attribute__((cleanup()))?

Not for any particular reason (as long as --std=gnu89 supports it but not quite sure which version gcc started to support this), I could refactor code to use it.

[cgwalters]

I don't have enough time to maintain several language implementaions (some other people asked for Go implementation) mainly due to active development. But if any volunteer I'd happy to enbrace them.

No interest in Go from me, but certainly if I had started composefs it would have been in Rust and I maintain higher level Rust crates here.

That said, I did recently come across https://github.com/ToolmanP/erofs-rs - not sure if you saw that. I only very superficially skimmed the code, it looked decent.

[hsiangkao]

I don't have enough time to maintain several language implementaions (some other people asked for Go implementation) mainly due to active development. But if any volunteer I'd happy to enbrace them.

No interest in Go from me, but certainly if I had started composefs it would have been in Rust and I maintain higher level Rust crates here.

Yeah, I totally understand you aren't interested.
But in term to EROFS perspertive itself, we shouldn't ignore the requirement since EROFS is language-independent(neutral), so there is no difference between C, Rust, and Go.
Just because C is still the most portable language in the current world, but Rust and Go are definitely acceptable (although my leisure bandwidth is limited, life is hard.)

That said, I did recently come across https://github.com/ToolmanP/erofs-rs - not sure if you saw that. I only very superficially skimmed the code, it looked decent.

Yeah, I noticed that, but it still need some time to form a proper one for erofs-utils integration.