Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update 2021-05-31-distributed-wikipedia-mirror-update.md #532

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update 2021-05-31-distributed-wikipedia-mirror-update.md #532

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 2 additions & 10 deletions src/_blog/2021-05-31-distributed-wikipedia-mirror-update.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ To address this critical need, we [created a mirror of Myanmar Wikipedia](https:

## How to help co-hosting this?

You can run your own IPFS node and co-host a subset of Wikipedia, store a full copy, or even follow collaborative cluster to pull in future updates automatically.
You can run your own IPFS node and co-host a subset of Wikipedia, or store a full copy.

It is also possible to donate co-hosting costs by pinning specific CID to a remote service.

Expand All @@ -71,14 +71,6 @@ Be wary that the English one is far bigger than other ones, and pinning it requi

The size of a specific mirror can be read with `ipfs files stat /ipfs/{cid}`.

### Collaborative cluster

This is an advanced option aimed at server administrators and power users. The `wikipedia` cluster includes all language versions and its size only grows over time.

$ ipfs-cluster-follow wikipedia run --init wikipedia.collab.ipfscluster.io

See _Instructions_ at [collab.ipfscluster.io](https://collab.ipfscluster.io#instructions).

### Donate remote pins

When co-hosting with your own IPFS node is not possible, one can still help by pinning snapshot CIDs to a remote pinning service.
Expand Down Expand Up @@ -112,4 +104,4 @@ Below are areas that could use a helping hand, and ideas looking for someone to
* **Improving the way ZIM is represented on IPFS.** When we store an original ZIM on IPFS, the DAG is produced by `ipfs add --cid-version 1`. This works fine, but with additional research on customizing DAG creation, we may improve deduplication and speed when doing range requests for specific bytes. There are different stages to explore here: if any of them sounds interesting to you, please comment in [distributed-wikipedia-mirror/issues/42](https://github.com/ipfs/distributed-wikipedia-mirror/issues/42).
* Stage 1: Invest some time to benchmark parameter space to see if low hanging fruits exists.
* Stage 2: Create a DAG builder that understands ZIM format and maximizes deduplication of image assets by representing them as sub-DAGs with dag-pb files.
* Stage 3: Research augmenting or replacing ZIM with [IPLD](https://ipld.io/). How can we maximize block deduplication across all snapshots and languages? How would an IPLD-based search index work?
* Stage 3: Research augmenting or replacing ZIM with [IPLD](https://ipld.io/). How can we maximize block deduplication across all snapshots and languages? How would an IPLD-based search index work?