Adds citation

CITATION.cff

@@ -0,0 +1,25 @@
+cff-version: 1.2.0
+title: 'DataTrove: large scale data processing'
+message: >-
+  If you use this software, please cite it using the metadata from this file.
+type: software
+authors:
+  - given-names: Guilherme
+    family-names: Penedo
+  - given-names: Alessandro
+    family-names: Cappelli
+  - given-names: Thomas
+    family-names: Wolf
+  - given-names: Mario
+    family-names: Sasko
+repository-code: 'https://github.com/huggingface/datatrove'
+abstract: "DataTrove is a library to process, filter and deduplicate text data at a very large scale. It provides a set of prebuilt commonly used processing blocks with a framework to easily add custom functionality. DataTrove processing pipelines are platform-agnostic, running out of the box locally or on a slurm cluster. Its (relatively) low memory usage and multiple step design makes it ideal for large workloads, such as to process an LLM's training data."
+keywords:
+  - deep-learning
+  - pytorch
+  - transformers
+  - llms
+  - data
+  - scale
+license: Apache-2.0
+version: 0.0.1

README.md

@@ -32,6 +32,7 @@ Local, remote and other file systems are supported through [fsspec](https://file
     + [Custom function](#custom-function)
     + [Custom block](#custom-block)
 - [Contributing](#contributing)
+- [Citation](#citation)
 
 <!-- tocstop -->
 
@@ -399,3 +400,16 @@ Run the tests:
 ```bash
 pytest -sv ./tests/
 ```
+
+## Citation
+
+```bibtex
+@misc{penedo2024datatrove,
+  author = {Penedo, Guilherme and Cappelli, Alessandro and Wolf, Thomas and Sasko, Mario},
+  title = {DataTrove: large scale data processing},
+  year = {2024},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  url = {https://github.com/huggingface/datatrove}
+}
+```