C-DNS format support

[nicki-krizek]

The C-DNS (RFC8618) format looks quite useful for storing and processing large amount of DNS data.

Are there any plans to add reader/writer for C-DNS?

[jelu]

No plans, unless you do it or fund it.

Are there any libraries yet?

[nicki-krizek]

There seems to be a C++ implementation in https://github.com/dns-stats/compactor , although it's not exactly a library.

[jelu]

@tomaskrizek I know about compactor and inspector.

If a linkable C library existed that has a call that returns the DNS payload then it's a couple of hours job to add read support for C-DNS, which I might be able to squeeze in before the end of the year.

Anything else is beyond my current available time.

[nicki-krizek]

On a related note, since file size is my main concern, perhaps using compression might be sufficient.

Looking at zstd, it looks quite fast even with a single thread. My input PCAP of 3.4 GB DNS queries takes 18s to compress, but more importantly, just 5s to decompress (on my laptop, with tmpfs).

$ time zstd --single-thread arekol_120s_800k.pcap -f
arekol_120s_800k.pcap : 16.86%   (3576432957 => 603152277 bytes, arekol_120s_800k.pcap.zst) 

real	0m18.363s
user	0m17.196s
sys	0m1.107s

$ time zstd --single-thread arekol_120s_800k.pcap.zst -f -d
arekol_120s_800k.pcap.zst: 3576432957 bytes                                    

real	0m5.471s
user	0m3.938s
sys	0m1.486s

I wonder how difficult would it be to support compressed PCAPs as an input...

[jelu]

I wonder how difficult would it be to support compressed PCAPs as an input...

It depends on the compression library's interfaces, PCAP is simple, fpcap and mmpcap parses the PCAP themselves.