Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial version #1

Open
wants to merge 89 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
a3ae825
Initial commit
bludnic Apr 7, 2024
225b328
chore(config): add ipf2 and ipf3 multiaddresses
bludnic Apr 8, 2024
d2ed9c8
chore(libp2p, connectionManager): add IPFS nodes multiaddrs to whitelist
bludnic Apr 8, 2024
23fa684
chore: rm console.log
bludnic Apr 8, 2024
9339d9b
chore: add node name
bludnic Apr 8, 2024
6943e15
feat: auto peering
bludnic Apr 8, 2024
0db56a8
fix: fix ipfs1 multiaddr
bludnic Apr 8, 2024
c06b360
fix: fix peerId string type
bludnic Apr 8, 2024
618b479
feat(/file/:cid): timeout out after 30s
bludnic Apr 8, 2024
fdca00e
feat(/file/upload): upload multiple files
bludnic Apr 8, 2024
eb093c6
feat(/file/upload): automatically detect mimetype
bludnic Apr 8, 2024
2d1a6e1
fix: fix mimetype for encrypted files
bludnic Apr 9, 2024
2fcf3f4
feat: add findProviders endpoint
bludnic Apr 10, 2024
3745df0
feat: enable routing provide
bludnic Apr 10, 2024
0f6aae3
feat: switch DTH from client to server
bludnic Apr 10, 2024
e11b823
feat: switch DTH from server to client
bludnic Apr 10, 2024
54791b4
feat: use verifiedFetch
bludnic Apr 17, 2024
9ed4c5f
feat: typescript + todo fixes
PaulDremanovich Apr 20, 2024
b908fa7
feat: remove babel and other fixes
PaulDremanovich May 1, 2024
58976f8
fix: loglevel, creation store folder and etc
PaulDremanovich May 1, 2024
5f4dfdc
fix: using config log, nodemon autoreload
PaulDremanovich May 1, 2024
b1e4c30
fix: config and autopeering
PaulDremanovich May 2, 2024
f0b765f
Merge pull request #2 from Adamant-im/feat/ts
RealGoodProgrammer May 2, 2024
211d4f8
feat: healtcheck + cors
PaulDremanovich May 3, 2024
b092b46
fix: explicit Error type
PaulDremanovich May 3, 2024
edbc9f9
fix: remove cors for options
PaulDremanovich May 3, 2024
9138cbd
feat: streaming files
PaulDremanovich May 4, 2024
63c6d9c
feat: move findFileTimeout to config
PaulDremanovich May 4, 2024
2a727c4
fix: corsOptions + remove helia Version
PaulDremanovich May 6, 2024
e57f235
fix: logger
PaulDremanovich May 7, 2024
22c23e8
fix: multer 1.4.5 with UnixFsStorage
PaulDremanovich May 8, 2024
7eb763d
fix: comments from PR
PaulDremanovich May 9, 2024
6bf8a58
fix: dirSize reworked
PaulDremanovich May 9, 2024
b34fdec
fix: bump helia libs versions
PaulDremanovich May 9, 2024
c402370
Merge pull request #3 from Adamant-im/feat/helthcheck
PaulDremanovich May 9, 2024
679217a
Merge branch 'refs/heads/dev' into feat/streams
PaulDremanovich May 9, 2024
a63293d
Merge pull request #4 from Adamant-im/feat/streams
PaulDremanovich May 9, 2024
800ef0d
fix: url "api" prefix added, localhost in cors
PaulDremanovich May 13, 2024
02131ca
fix: renew helia config
PaulDremanovich May 16, 2024
2f5bd69
fix(config): fix ipfs2 addr
bludnic May 17, 2024
e8df5a1
fix(/api/file/:cid): fix uncatched error
bludnic May 17, 2024
0c7703c
fix(/api/file/upload): disable DHT (fix Query Aborted error)
bludnic May 17, 2024
993d26b
chore(config): decrease file not found timeout
bludnic May 17, 2024
c4a90d7
fix(/api/file/upload): fix uncatched timeout error
bludnic May 17, 2024
9b306d8
fix: update nodes multiaddr
bludnic May 18, 2024
e7170da
fix: use single datastore
bludnic May 19, 2024
257efc0
chore: increase find file timeout
bludnic May 19, 2024
7d20a70
chore: old way to find content by CID
bludnic May 19, 2024
e8973ac
chore: use default helia config
bludnic May 19, 2024
2588f9c
chore: update multiaddresses
bludnic May 19, 2024
c676361
chore(p2p): add allowed list
bludnic May 19, 2024
0378300
chore(p2p): update peer discovery list
bludnic May 19, 2024
345f9f6
chore(p2p): disable autopeering
bludnic May 19, 2024
502b3a4
refactor: split endpoints into routers
bludnic May 19, 2024
0ab2fc0
chore(helia): use verifiedFetch
bludnic May 19, 2024
cfdb003
chore(/file/:cid): use ReadableStream with AbortController
bludnic May 19, 2024
5203cbf
fix(/file/:cid): catch invalid CID errors
bludnic May 19, 2024
ee9c1bc
fix(/file/:cid): fix error messages
bludnic May 19, 2024
4f8c1ae
fix(/file/:cid): fix set headers multiple times
bludnic May 21, 2024
994fa09
fix: disk usage checker + quickly search for a cid on another node
PaulDremanovich May 23, 2024
cc36596
feat(file): resp content-length
RealGoodProgrammer May 23, 2024
c2b09ae
Merge pull request #7 from Adamant-im/feat/content-length
RealGoodProgrammer May 23, 2024
0c9b51b
fix(file): unixfs -> ifs
RealGoodProgrammer May 23, 2024
6e6ab7e
Merge branch 'refs/heads/hotfixes' into fix/checking-disk-usage
PaulDremanovich May 29, 2024
65ff30e
fix: remove adamant ipfs nodes from bootstrap
PaulDremanovich May 29, 2024
c01902e
Merge pull request #6 from Adamant-im/fix/checking-disk-usage
bludnic May 29, 2024
fc7401c
Merge pull request #5 from Adamant-im/hotfixes
bludnic May 29, 2024
1df662c
fix: prepare routes
PaulDremanovich Jun 8, 2024
1daef31
fix: move the maximum number of files to the configuration
PaulDremanovich Jun 8, 2024
6ad9579
fix: readme.md
PaulDremanovich Jun 8, 2024
c4481fc
fix: removed unused verified-fetch
PaulDremanovich Jun 8, 2024
b1821fa
fix: readme
PaulDremanovich Jun 25, 2024
fc6f44b
fix: project description
PaulDremanovich Jun 25, 2024
2cad236
Merge pull request #8 from Adamant-im/fix/prepare-to-main
PaulDremanovich Jun 26, 2024
6a32c7f
refactor: readme
confleux Jul 10, 2024
8deb29f
fix: user's root directory -> user's home directory
confleux Jul 10, 2024
4081738
refactor: remove extra response info
confleux Jul 10, 2024
c0a3565
refactor: format json
confleux Jul 10, 2024
2595191
chore: update `package-lock.json`
bludnic Aug 9, 2024
35d7b91
chore(package.json): remove unused dependencies
bludnic Aug 9, 2024
6b56705
chore(package.json): upgrade dependencies
bludnic Aug 9, 2024
4890d43
chore(package.json): fix `main` entry point
bludnic Aug 9, 2024
f22655d
chore(package.json): mark as private
bludnic Aug 9, 2024
b9a873b
chore: use one config
bludnic Aug 9, 2024
ea2235f
fix(`api/file/:cid`): not timing out when using `ifs.stat()`
bludnic Aug 9, 2024
b8cec35
chore: rollback blockstore/datastore versions due breaking changes
bludnic Aug 9, 2024
e627dcb
feat: bootstrap only with known nodes
bludnic Aug 9, 2024
85f4ec2
feat(file): return 408 when file not found
bludnic Aug 9, 2024
137ec73
Merge pull request #9 from Adamant-im/refactor/readme
bludnic Aug 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# EditorConfig is awesome: https://EditorConfig.org

# top-most EditorConfig file
root = true

# Unix-style newlines with a newline ending every file
[*]
indent_style = space
indent_size = 2
end_of_line = lf
insert_final_newline = true
1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@

3 changes: 3 additions & 0 deletions .eslintrc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"extends": "./node_modules/gts/"
}
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
node_modules
.idea
dist/
config.json5
4 changes: 4 additions & 0 deletions .husky/commit-msg
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#!/usr/bin/env sh
. "$(dirname -- "$0")/_/husky.sh"

npx --no -- commitlint --edit ${1}
3 changes: 3 additions & 0 deletions .husky/pre-commit
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/usr/bin/env sh
. "$(dirname -- "$0")/_/husky.sh"
npx lint-staged
1 change: 1 addition & 0 deletions .nvmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
20.*
8 changes: 8 additions & 0 deletions .prettierrc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"$schema": "https://json.schemastore.org/prettierrc",
"semi": false,
"tabWidth": 2,
"singleQuote": true,
"printWidth": 100,
"trailingComma": "none"
}
137 changes: 136 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,137 @@
# ipfs-node
IPFS decentralized file storage and transfers
ADAMANT ipfs-node. Designed for downloading and exchanging files in the ADAMANT Messenger.

Unlike the standard libraries (helia or kubo), this ipfs-node is equipped with a web server for performing REST requests for downloading and receiving files.

The plans also include the implementation of the Garbage Collector function, which will save disk space by removing unsent files.

## How to start
- You will need Node.js v20.11.1 (You can install via nvm: https://github.com/nvm-sh/nvm):
```bash
curl -o- https://github.com/raw/nvm-sh/nvm/v0.39.7/install.sh | bash
nvm install 20.11.1
```
- Cloning and building node:
```bash
git clone https://github.com/Adamant-im/ipfs-node.git
cd ipfs-node
npm i
npm run build
```
- Running node with [pm2](https://github.com/Unitech/pm2):
```bash
npm i -g pm2
pm2 start dist/index.js --name="IPFS node"
```


## How to configure
Using `config.default.json5` as a template, you can create various configuration files.

For example:
`node dist/index.js test1` — this command will launch the server with the configuration from the file config.test1.json5

```jsonc
{
// List of IPFS ADAMANT nodes interacting between each other
nodes: [
{
name: "ipfs1",
multiAddr: "/ip4/194.163.154.252/tcp/4001/p2p/12D3KooWSUCe86zWfas1Lo1UQzXzquZgS81d1DpPPYAuTNjSyniq"
},
...
],
storeFolder: '.adm-ipfs', // File storage directory (the directory is set from the user’s home directory)
logLevel: 'debug', // Logging level: fatal, error, warn, info, debug, trace
peerDiscovery: {
// IPFS network nodes that will be used to search for new nodes
// Details: https://github.com/libp2p/js-libp2p/tree/main/packages/peer-discovery-bootstrap
bootstrap: [
'/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN',
...
],
// Addresses that helia will listen to
listen: [
'/ip4/0.0.0.0/tcp/4001',
]
},
serverPort: 4000, // API server deployment port
autoPeeringPeriod: '*/10 * * * * *',
diskUsageScanPeriod: '*/30 * * * * *', // Disk space scanning period. Set in cron format: '* * * * * *'
uploadLimitSizeBytes: 268435456, // Maximum upload file size (in bytes)
maxFileCount: 5, // Maximum upload count of files per request
findFileTimeout: 20000, // Time limit for searching for a file on the IPFS network
cors: {
// Allowed URLs to make request to node. Set using regular expressions
originRegexps: ['.*\.adamant\.im', 'adm.im', '.*\.vercel\.app', '.*\.surge\.sh', 'localhost:8080']
}
}
```

## How to use

### Upload file
Request body type: `form-data`

It should contain a "files" field, which can accept an array of files up to 5 pieces at a time.

#### Request
```POST /api/file/upload```
```bash
curl -i --location 'http://localhost:4000/api/file/upload' --form 'files=@"file.txt"'
```

#### Response
```
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
...

{"filesNames":["file.txt"],"cids":["bafkreif7v2d2wdyh6pz5y2pwmrpegfpdgh5u7n5vomxnbofraqhuk2wapm"]}
```

### Get file

#### Request
```GET /api/file/:cid```
```bash
curl -i --location 'http://localhost:4000/api/file/bafkreif7v2d2wdyh6pz5y2pwmrpegfpdgh5u7n5vomxnbofraqhuk2wapm'
```

#### Response
```
HTTP/1.1 200 OK
Content-Type: application/octet-stream
...

Hello ipfs-node!
```

### Get node info

#### Request
```GET /api/node/info```
```bash
curl -i --location 'http://localhost:4000/api/node/info'
```

#### Response
```
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
...

{
"version": "0.0.1",
"timestamp": 1720614998797,
"heliaStatus": "started",
"peerId": "12D3KooWJSiMDfyDLK3EMe2567sSM1VKQVnUn2getimGqVTWqKX9",
"multiAddresses": [
"/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWJSiMDfyDLK3EMe2567sSM1VKQVnUn2getimGqVTWqKX9",
"/ip4/62.72.43.99/tcp/4001/p2p/12D3KooWJSiMDfyDLK3EMe2567sSM1VKQVnUn2getimGqVTWqKX9"
],
"blockstoreSizeMb": 0.0009489059448242188,
"datastoreSizeMb": 0.006007194519042969,
"availableSizeInMb": 2257731
}
```
1 change: 1 addition & 0 deletions commitlint.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export default { extends: ['@commitlint/config-conventional'] }
35 changes: 35 additions & 0 deletions config.default.json5
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
nodes: [
{
name: 'ipfs1',
multiAddr: '/ip4/194.163.154.252/tcp/4001/p2p/12D3KooWSUCe86zWfas1Lo1UQzXzquZgS81d1DpPPYAuTNjSyniq'
},
{
name: 'ipfs2',
multiAddr: '/ip4/154.26.159.245/tcp/4001/p2p/12D3KooWJw99nqrQ2L2joFuGCF9EN9EyF8ZrvGr1odQ61HoPrbXd'
},
{
name: 'ipfs3',
multiAddr: '/ip4/109.123.240.102/tcp/4001/p2p/12D3KooWHTvrWSTCXZFR2vS8Cbw17b5peK2aSBr8CmkyA96CDqiU'
}
],
storeFolder: '.adm-ipfs',
logLevel: 'debug',
peerDiscovery: {
bootstrap: [
'/ip4/194.163.154.252/tcp/4001/p2p/12D3KooWSUCe86zWfas1Lo1UQzXzquZgS81d1DpPPYAuTNjSyniq', // ipfs1
'/ip4/154.26.159.245/tcp/4001/p2p/12D3KooWJw99nqrQ2L2joFuGCF9EN9EyF8ZrvGr1odQ61HoPrbXd', // ipfs2
'/ip4/109.123.240.102/tcp/4001/p2p/12D3KooWHTvrWSTCXZFR2vS8Cbw17b5peK2aSBr8CmkyA96CDqiU' // ipfs3
],
listen: ['/ip4/0.0.0.0/tcp/4001']
},
serverPort: 4000,
autoPeeringPeriod: '*/10 * * * * *',
diskUsageScanPeriod: '*/30 * * * * *',
uploadLimitSizeBytes: 268435456, // 256 Mb.
maxFileCount: 5,
findFileTimeout: 20000,
cors: {
originRegexps: ['.*.adamant.im', 'adm.im', '.*.vercel.app', '.*.surge.sh', 'localhost:8080']
}
}
6 changes: 6 additions & 0 deletions nodemon.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"watch": ["src"],
"ext": "js, ts, json",
"ignore": ["dist/**/*.spec.js"],
"exec": "npm run build; node dist/index.js"
}
Loading