tutorial: Transfer Files #714

haadcode · 2017-01-16T16:31:22Z

This PR will add a basic interop example as per #711.

daviddias

The demo available here is not an interop demo, in fact, it doesn't touch go-ipfs at all, I know you wanted first to do js-ipfs to js-ipfs, but that is not the purpose.

The demo must be (as discussed during prep session):

User adds content using go-ipfs CLI
User accesses that content from a browser.

To overcome the challenge of not being blocked by the stream muxing issue, you can write the whole tutorial using js-ipfs CLI + js-ipfs in the Browser, then when muxed is fixed (cc @victorbjelkholm), we will just need to s/jsipfs/ipfs.

daviddias · 2017-01-17T10:00:26Z

examples/interop-examples/README.md

@@ -0,0 +1,102 @@
+# interop examples


s/interop examples/fetch content added by go-ipfs

daviddias · 2017-01-17T10:00:38Z

examples/interop-examples/README.md

+npm start
+```
+
+Open http://127.0.0.1:8080 in your favorite browser.


Chrome or Firefox

Both should work, thus "your favorite" :)

There is also Safari, Edge, Internet Explorer and Opera, not everyone's favourite is Chrome or Firefox :)

daviddias · 2017-01-17T10:02:21Z

examples/interop-examples/README.md

+Steps
+1. create IPFS instance
+
+First, we need an IPFS instance that we'll use throughout the tutorial. For the sake of convinience, we'll use `ipfs-daemon` module to create and IPFS instance.


We should not be recommending ipfs-daemon in order to instantiate js-ipfs, it just means that js-ipfs needs to improve its init process.

I'll rework this in a way that the init dance is in its own file and we include js-ipfs instead of ipfs-daemon.

I don't think we'll get to merge ipfs-daemon to js-ipfs within two weeks given how massive public API change that is. Let me know if you think we can.

I don't think we'll get to merge ipfs-daemon to js-ipfs within two weeks given how massive public API change that is. Let me know if you think we can.

We totally can, it is not actually that big of a change, really, just requires putting in the hours, it is pretty clear what we want out of it and we have a ton of tests creating js-ipfs daemons with different configurations and in Node.js + Browser, those that will ensure that the new API works.

It is, however, a low priority compared to the rest of the tasks for this Sprint, so, if we get to finish all the others, we will want ot tackle this.

Until then, the demo should be without an external module to create an instance of this module.

daviddias · 2017-01-17T10:02:50Z

examples/interop-examples/README.md

+})
+```
+
+2. listen for file drop


The goal of this demo is to "access files from go-ipfs", and not adding files inside the browser.

Understood.

Somewhere we discussed, and decided, that we do go-ipfs -> js-ipfs-browser -> js-ipfs-browser (stream a video or smth). I believe adding the support for adding files to the browser version is not that much out of scope in terms of work and it adds another important step to the tutorial for one of the most asked questions for someone who's starting with ipfs "how do I add files to IPFS?". So having that functionality gives a more fully fledged tutorial than just catting files from go-ipfs. Don't worry, we're still going for the same goal: access files from go-ipfs.

Thank you :)

haadcode · 2017-01-17T10:22:46Z

The demo available here is not an interop demo, in fact, it doesn't touch go-ipfs at all, I know you wanted first to do js-ipfs to js-ipfs, but that is not the purpose.

All of this is what the example will be. Perhaps the intent was dropped in the communication. If you take a look at my sprint updates, you'll see the progress being described (ie. it's WIP).

See this comment #714 (comment) for details to why there's browser-to-browser functionality too.

haadcode · 2017-01-17T10:23:11Z

To clarify, this PR will (eventually) implement the example as designed here: #711 (comment)

daviddias · 2017-01-17T10:24:24Z

@haadcode thank you for clarifying! :)

daviddias

@haadcode just tried the example and worked like a charm.

But I didn't really run through the Tutorial, because the example is the full blown thing already and I understand why, we need to have the thing done once, in order to know it works, so here is my suggestion to improve the UX.

~~create a folder called tutorials at the root of the project and move this example there~~
rename the folder to 'access-go-ipfs-files'
inside the folder have README.md, final/, start
inside the folder final, have the full built thing
inside the folder start, have just the starting files (package.json, http server step up, etc)
in the README, explain the purpose of this tutorial, how people can run the real deal and that they can do this as a tutorial.
Important to add to gitignore the files from the startfolder, with the exceptions of the ones you leave there :)

Apply this to other example as well.

Note that, things that are still missing from the goal of this example (which you are probably well aware, but just listing them now to keep track):

right now, it is still just doing browser to browser
There is no reference as to explain that a file should be added with js-ipfs/go-ipfs CLI
It needs to explain that a WebSockets address needs to added to go-ipfs/js-ipfs
or a webrtc-star address to js-ipfs

daviddias · 2017-01-20T10:02:19Z

examples/interop-examples/README.md

+
+## Run
+```
+cd examples/interop-examples


no need, people will be inside this folder already

daviddias · 2017-01-20T10:03:43Z

examples/interop-examples/package.json

+  "author": "",
+  "license": "ISC",
+  "dependencies": {
+    "ipfs-daemon": "^0.3.0-beta.16"


When ipfs-daemon is out of the equation, it will be so much faster to get this example to work because one less npm install of the whole thing is needed.

daviddias · 2017-01-20T10:03:52Z

examples/interop-examples/package.json

+    "start": "http-server -c-1"
+  },
+  "author": "",
+  "license": "ISC",


daviddias · 2017-01-23T08:00:55Z

@haadcode I was going to review this example again, but I see there hasn't been any other commit since my last review, so I will avoid listing out the points of feedback again.

I received a notification through IRC that you were preparing some handoff guide for the next person to take on this examples, however, I'm being unable to find it, I've scanned the IRC logs and opened all the related github issues, could you point me to it? Thank you.

haadcode · 2017-01-23T08:02:02Z

you were preparing some handoff guide for the next person to take on this examples

Working on it.

haadcode · 2017-01-23T09:23:03Z

Updated everything, fixed a bunch of stuff. This should be now ready to be picked up by others. What remains to be done is listed in: https://github.com/ipfs/js-ipfs/blob/e916a5d1d0c654a6111e5a4861aafb79b72b3e62/examples/access-go-ipfs-files/cat-a-file/README.md#todo. There's also the new signal server address that needs to be fixed as per libp2p/js-libp2p-webrtc-star#87.

The short is: add the UI for connecting to a go-ipfs address manually and make sure catting the file from go-ipfs-->js-ipfs(-browser) works.

daviddias · 2017-01-25T10:24:28Z

Seems that it is assumed that the user has ran npm run build before hand.

» npm i

> cat-a-file@1.0.0 postinstall /Users/koruza/code/js-ipfs/examples/access-go-ipfs-files/cat-a-file
> cp ../../../dist/index.js ./public/ipfs.js

cp: ../../../dist/index.js: No such file or directory

npm WARN cat-a-file@1.0.0 No repository field.
npm ERR! Darwin 16.1.0
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "i"
npm ERR! node v6.9.1

daviddias · 2017-01-25T10:30:07Z

The example doesn't have a way to signal a want to dial to any other node, assumes that webrtc is used. The purpose was to offer to the user a way to dial through websockets.

I rebased the commits

daviddias · 2017-01-25T10:30:27Z

working on this example today

using only webrtc-star as a transport

daviddias · 2017-01-26T12:15:49Z

Todo notes left by @haadcode for @RichardLitt

Add "connect to peer" input field and "connect" button under the "Peers" section in index.html
Add connectToPeer function which calls ipfs.swarm.connect(multiaddr). See

js-ipfs/examples/ipfm/src/DataStore.js

Line 82 in b0a7cd8

connectToPeer (multiaddr) {

and https://github.com/ipfs/js-ipfs/blob/b0a7cd83cbf146b0f147467dedc686f94a5f751f/examples/ipfm/README.md#start-an-interplanetary-file-exchange-daemon. The multiaddr to connect to looks like this: /ip4/127.0.0.1/tcp/9999/ws/ipfs/QmZGH8GeASSkSZoNLPEBu1MqtzLTERNUEwh9yTHLEF5kcW
Hook up "connect" button's click event to connectToPeer function
Add instructions to this README on how to add a file in go-ipfs
Add instructions to this README on how to cat it in the UI
Make sure the "Start a go-ipfs daemon" instructions are correct
Make sure go-ipfs daemon and the example connect to each other

victorb · 2017-01-26T14:27:44Z

Taking over this for now

victorb · 2017-01-29T11:52:45Z

Sorry @haadcode missed your comments before, reply inline:

It's better to use dynamic sizes (for fonts, margins, widths, etc.) instead of pixels, ie. px --> em / rem / etc.

I would like the layout to stay the same no matter the viewport so I'll go with px here.

When changing CSS, make sure to test the changes at least with Firefox and Chrome to make sure they both work (usually they don't quite render the same), and make sure to try it on a mobile browser(s) too to make sure the layouts and fonts scale.

Sure, no problem. I'm not gonna spend any time on trying it on mobile though, if people want it to work on mobile, they'll have to solve any issues themselves (or we can do another example for mobile in the future)

I believe that you're still working on this, but make sure the picture is actually shown: ea48e5b#diff-6db58e31a051cfc7a871d0fd2444b037R100 :)

Not really, I've removed the picture since I didn't see how it tied into the example. I think it's not necessary to show a picture inline in the webpage but rather when people add files, we show them in a download list.

victorb · 2017-01-29T17:25:52Z

Last update before weekend

Still missing some refactor, mainly splitting up the ipfs and UI logic, right now it's all wrapped together. Made just one quick pass for comments so probably there are things that can be better explained. I have not looked at the tutorial and I'm pretty sure it's unfinished. Did some improvements with the drag-and-drop but still there is a bug left.

Biggest one is that .get doesn't actually work at the moment. Purpose is to fetch the content of the file and set as an octet-stream as the download when clicking the link, but something is crashing with the wrong cid version. @diasdavid is working on fixing this.

Refactor code + add valuable comments
Have .get actually work with just one file and also maybe directories
Fix drag-and-drop not closing popup when ending up not dropping file in UI
Finish tutorial in readme.md

- refactor: reorg + fix lint - fix: build step, no longer fail on postinstall - docs: add diagram - docs: add the instructions steps - lint: fix all the things - note about signalling server

- nicer colors - include logo - fancier buttons - easier flow to start daemon - less information after starting daemon - use dns signal server - larger font size - removal of unused features - removal of duplicated script references - more explanation - and probably something more I forgot

daviddias · 2017-01-30T09:39:32Z

@victorbjelkholm I just fresh git cloned and fresh install both package.json and this is what I'm getting:

Updated: fixed in #31c6383

victorb · 2017-01-30T10:25:03Z

Final pass-through before leaving this move on to the CI/CD sprint

cd into /complete directory of the transfer files example
Run npm install
Run npm start
cd ../../../
run npm run build
cd examples/transfer-files/complete
npm start
navigate to localhost:12345
in js-ipfs, run IPFS_PATH=$HOME/.examples-test node src/cli/bin.js init
run IPFS_PATH=$HOME/.examples-test node src/cli/bin.js daemon
run IPFS_PATH=$HOME/.examples-test node src/cli/bin.js id
grab the address containing /ws
connect directly to it by putting /ip4/127.0.0.1/tcp/4003/ws/ipfs/QmPs14AG8igTwHKvZ2deVDrkYTtczghHTeyivUNsSydhpH in the multiaddr input and pressing "Connect to peer"
Address shows up in list of remote peers
Run IPFS_PATH=$HOME/.examples-test node src/cli/bin.js add package.json, save the hash
Paste the hash into multihash input field, press "Fetch file"
Date - Hash and size appears in the list of fetched files
Clicking on the item for the file triggers download
File finished downloading, opening up shows content of file being "content of file" since the .get is not fully working yet... See TODO comment in function catFile in app.js
Dragging a file into the UI shows wonky "drop file popup" that appears and disappears when moving the mouse around
Dropping the file shows "Added antigen.sh as Qm[...]" and changes the hash in the multihash input
Clicking "Fetch file" adds the file to the list of files with it's size and the date
Clicking on the added file item downloads the file but still with the file contents being "content of file"
Let's try with a go-ipfs node. Kill the current running nodejs node
Peer disappears from the UI in the complete example
Pull down latest master of go-ipfs and install it
Run IPFS_PATH=$HOME/.examples-test-go ipfs init
Run IPFS_PATH=$HOME/.examples-test-go ipfs daemon
No /ws address, need to add it to the configuration
Open $HOME/.examples-test-go/config and add /ip4/127.0.0.1/tcp/4004/ws to the list of swarm addresses
Run IPFS_PATH=$HOME/.examples-test-go ipfs daemon
Run IPFS_PATH=$HOME/.examples-test-go ipfs id, save the /ws address
Paste /ws address into multiaddr input and press "Connect to peer"
Peer address shows up in UI
Run IPFS_PATH=$HOME/.examples-test-go ipfs add appveyor.yml in go-ipfs repository
Save hash
Paste hash into multihash input and press "Fetch file"
Receive stream with files but it doesn't have any .content field as the response from nodejs process has

Conclusion

"content of file" is currently being set as the content of the file since we haven't fully got .get to work yet
Communication with go-ipfs nodes does not work the same way as when communicating with nodejs nodes
When connecting with the go-ipfs daemon, peer is listed in the browser node but not in the go-ipfs node

victorb · 2017-01-30T17:52:52Z

Discovered via @diasdavid doing a demo today with this.

Peers list have a comma between every peer that is not supposed to be there
Multiaddr input is too wide and goes outside the containing box

daviddias · 2017-02-07T08:12:11Z

@victorbjelkholm fixed the get problem. The stream was never resumed :)

843e227

daviddias · 2017-03-07T16:20:58Z

Follow here: #774 (comment)

* docs: transfer-files-example

haadcode added status/in-progress In progress labels Jan 16, 2017

haadcode self-assigned this Jan 16, 2017

haadcode mentioned this pull request Jan 17, 2017

Sprint: Browser accesses go-ipfs content ipfs/team-mgmt#310

Closed

50 tasks

daviddias suggested changes Jan 17, 2017

View reviewed changes

daviddias mentioned this pull request Jan 17, 2017

Browser + Node.js + Go Interop Examples/Tutorials #711

Closed

daviddias changed the title ~~(WIP) Add basic interop example~~ (WIP) Example: cat a go-ipfs added file from the browser Jan 19, 2017

daviddias suggested changes Jan 20, 2017

View reviewed changes

This was referenced Jan 20, 2017

js/go interop not working #718

Closed

browser/nodejs interop issues #720

Closed

daviddias changed the title ~~(WIP) Example: cat a go-ipfs added file from the browser~~ (WIP) Tutorial: cat a go-ipfs added file from the browser Jan 20, 2017

daviddias force-pushed the docs/interop-examples branch from e916a5d to ed49df6 Compare January 25, 2017 10:30

daviddias force-pushed the docs/interop-examples branch from ed49df6 to 4c5443a Compare January 25, 2017 10:31

docs(example): create an example of cat'ing files added from ipfs CLI

0bdd2ed

using only webrtc-star as a transport

daviddias force-pushed the docs/interop-examples branch 2 times, most recently from af0ff7a to 06aa615 Compare January 25, 2017 17:05

victorb assigned victorb and unassigned haadcode Jan 26, 2017

daviddias changed the title ~~(WIP) Tutorial: cat a go-ipfs added file from the browser~~ (WIP) Tutorial: Transfer Files Jan 27, 2017

victorb force-pushed the docs/interop-examples branch from c50e642 to 8ce1e98 Compare January 29, 2017 11:59

daviddias mentioned this pull request Jan 29, 2017

No peers and addresses #694

Closed

daviddias and others added 7 commits January 30, 2017 09:24

docs(example: transfer-files)

cbf7cf0

- refactor: reorg + fix lint - fix: build step, no longer fail on postinstall - docs: add diagram - docs: add the instructions steps - lint: fix all the things - note about signalling server

Modifications to transfer-files readme

c15370e

Latest changes

3cca07b

First pass

387ff7e

Added step 3

9d318d0

docs(example: transfer-files): Improve the UI

e88a73a

fix: default config fault, add MDNS.Enabled back

31c6383

daviddias force-pushed the docs/interop-examples branch from d622347 to 31c6383 Compare January 30, 2017 10:03

daviddias changed the title ~~(WIP) Tutorial: Transfer Files~~ tutorial: Transfer Files Jan 31, 2017

fix: get files now works

843e227

dignifiedquire mentioned this pull request Feb 24, 2017

Tutorial: transfer-files (previsously #714) #774

Merged

daviddias closed this Mar 7, 2017

daviddias deleted the docs/interop-examples branch March 7, 2017 16:21

daviddias removed the status/in-progress In progress label Mar 7, 2017

daviddias pushed a commit that referenced this pull request Mar 15, 2017

Tutorial: transfer-files (previsously #714) (#774)

2c31523

* docs: transfer-files-example

daviddias pushed a commit that referenced this pull request Mar 17, 2017

Tutorial: transfer-files (previsously #714) (#774)

026c801

* docs: transfer-files-example

daviddias pushed a commit that referenced this pull request Mar 21, 2017

docs: Tutorial: transfer-files (previsously #714) (#774)

f2ef1d9

daviddias pushed a commit that referenced this pull request Mar 21, 2017

docs: Tutorial: transfer-files (previsously #714) (#774)

3ee3acd

daviddias pushed a commit that referenced this pull request Mar 21, 2017

docs: Tutorial: transfer-files (previsously #714) (#774)

b2afbf4

daviddias pushed a commit that referenced this pull request Mar 21, 2017

docs: Tutorial: transfer-files (previsously #714) (#774)

8c243f1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tutorial: Transfer Files #714

tutorial: Transfer Files #714

haadcode commented Jan 16, 2017 •

edited

Loading

daviddias left a comment

daviddias Jan 17, 2017

daviddias Jan 17, 2017

haadcode Jan 17, 2017

daviddias Jan 20, 2017

daviddias Jan 17, 2017

haadcode Jan 17, 2017

daviddias Jan 20, 2017

daviddias Jan 17, 2017

haadcode Jan 17, 2017

daviddias Jan 17, 2017

haadcode commented Jan 17, 2017

haadcode commented Jan 17, 2017

daviddias commented Jan 17, 2017

daviddias left a comment •

edited

Loading

daviddias Jan 20, 2017

daviddias Jan 20, 2017

daviddias Jan 20, 2017

daviddias commented Jan 23, 2017

haadcode commented Jan 23, 2017

haadcode commented Jan 23, 2017

daviddias commented Jan 25, 2017

daviddias commented Jan 25, 2017 •

edited

Loading

daviddias commented Jan 25, 2017

daviddias commented Jan 26, 2017

victorb commented Jan 26, 2017 •

edited

Loading

victorb commented Jan 29, 2017 •

edited

Loading

victorb commented Jan 29, 2017 •

edited by daviddias

Loading

daviddias commented Jan 30, 2017 •

edited

Loading

victorb commented Jan 30, 2017 •

edited

Loading

victorb commented Jan 30, 2017

daviddias commented Feb 7, 2017

daviddias commented Mar 7, 2017

tutorial: Transfer Files #714

tutorial: Transfer Files #714

Conversation

haadcode commented Jan 16, 2017 • edited Loading

daviddias left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haadcode commented Jan 17, 2017

haadcode commented Jan 17, 2017

daviddias commented Jan 17, 2017

daviddias left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias commented Jan 23, 2017

haadcode commented Jan 23, 2017

haadcode commented Jan 23, 2017

daviddias commented Jan 25, 2017

daviddias commented Jan 25, 2017 • edited Loading

daviddias commented Jan 25, 2017

daviddias commented Jan 26, 2017

Todo notes left by @haadcode for @RichardLitt

victorb commented Jan 26, 2017 • edited Loading

victorb commented Jan 29, 2017 • edited Loading

victorb commented Jan 29, 2017 • edited by daviddias Loading

Last update before weekend

daviddias commented Jan 30, 2017 • edited Loading

victorb commented Jan 30, 2017 • edited Loading

Conclusion

victorb commented Jan 30, 2017

daviddias commented Feb 7, 2017

daviddias commented Mar 7, 2017

haadcode commented Jan 16, 2017 •

edited

Loading

daviddias left a comment •

edited

Loading

daviddias commented Jan 25, 2017 •

edited

Loading

victorb commented Jan 26, 2017 •

edited

Loading

victorb commented Jan 29, 2017 •

edited

Loading

victorb commented Jan 29, 2017 •

edited by daviddias

Loading

daviddias commented Jan 30, 2017 •

edited

Loading

victorb commented Jan 30, 2017 •

edited

Loading