Initial work #1

AuHau · 2019-04-04T23:31:18Z

A first draft version of the migration tool, that can be meaningfully feedbacked.
To see how migration looks like, see #2

This PR is written against the async PR's of dependencies (datastores mainly) and hence is blocked by this.

Scope-in

Scope and status of this PR (and package):

Define the migration's API
Create high-level API to migrate forward&backward migrations
Create CLI to run the migrations
Write documentation to README
- Document tests, how to write them, run them etc
Write test coverage

Scope-out

Thinks that are not in scope of this PR and will be scope of separate PR to js-ipfs that will integrate this package:

Disabling/enabling automatic migrations
Propagating information about migrations to JS-IPFS readme
Integrate to js-ipfs with posibility to automatically run the migrations

Blocked by

Currently, this depends on async/await refactor of

Problems

If the automatic migrations will be disabled in browser environment, how does the user can manually migrate its repo, when it does not have access to CLI?
If IPFSRepo() is used with custom options (e.g. using datastore-level for key storage backend, instead of default datastore-fs), this is not propagated to migration. ==> Should repo.options be propagated through migrate()/revert() to each migration?
6dc649b added support for passing options to the migrations
Since now it is possible to pass custom options to migrations, what should be done with the CLI? How to define the options for the datastores? They are not stored in config, if I understand it correctly...

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

…astore instance. License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

AuHau · 2019-04-28T16:32:53Z

So the PR is ready for review. I would also ask for the guidance of what next steps should be (ofc after reviews)? I assume something like:

Move repo under IPFS org
License file move to Protocol Labs?
Integrate CI
Release

As a community member, I can't do much of these, but I would love to work with you, so you could just hire me and then I could do them! 😉

Blocked by release of async/await versions of

interface-datastore
js-datastore-fs
js-datastore-level

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

AuHau · 2019-07-23T16:55:22Z

I have also moved the repo under IPFS org. What kind of teams should be assigned to it? I have added JS Team, but not sure if some others should have access also.

Also what about "Lead Maintainer"? I can do it, or if somebody else should be it (because PL guidelines or something), then I don't mind that either.

run.md

src/commands.js

Co-Authored-By: dirkmc <dirkmdev@gmail.com>

src/cli.js

src/index.js

dirkmc · 2019-07-24T23:28:32Z

src/repo/init.js

+
+    return true
+  } catch (e) {
+    log('While checking if repo is initialized error was thrown: ' + e.message)


Please use explain-error here

src/repo/version.js

Co-Authored-By: dirkmc <dirkmdev@gmail.com>

dirkmc · 2019-07-26T18:05:11Z

@AuHau thank you again for all your hard work. This is looking really good 👍

@alanshaw I've done a couple of passes over the code and suggested some changes, all of which @AuHau has addressed. I think this is ready for you to review 🚀

daviddias

There is a ton of kickass work here!! thank you so much @AuHau

Let's get it out of the door. I've made some comments on the README. some are nitpicks others are org structure.

daviddias · 2019-10-10T09:38:16Z

.editorconfig

+indent_size = 2
+trim_trailing_whitespace = true
+insert_final_newline = true
+end_of_line = lf


Mind adding this file to .gitignore?

Well I can, but I think it is a good thing to have around as it unities the code style. I know that for that there is aegir lint but this integrates directly with IDEs. You don't use it in other projects? I feel like I took it from some other JS package...
For example here: https://github.com/ipfs/js-ipfs/blob/master/.editorconfig

README.md

daviddias · 2019-10-10T09:47:44Z

README.md

+ * `options.ignoreLock` (bool, optional) - if true will not lock the repo when applying migrations. Use with caution.
+ * `options.repoOptions` (object, optional) - options that are passed to migrations, that use them to construct the datastore. (options are the same as for IPFSRepo).
+ * `options.onProgress` (function, optional) - callback that is called after finishing execution of each migration to report progress.


Is this for progress bars?

README.md

daviddias

2 more :)

README.md

Co-Authored-By: David Dias <daviddias.p@gmail.com>

README.md

src/index.js

src/repo/lock.js

AuHau added 13 commits March 5, 2019 13:53

Basic setup

e214b8f

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Basic migrator functionality

e1f8d93

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Strict mode

04b8b4d

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Linting

b386cb6

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Working CLI

48525dc

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Improving CLI usablity

1748537

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Dropping migration class interface

fd07c3e

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Improving error handling and other tweaks of migrations

5091b72

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Refactoring to drop ipfs-repo in favor of datastore

9eb9418

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Refactor migrator to pass only repo's path into migration and not dat…

3b18eba

…astore instance. License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

'add' command for CLI

d9311b0

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Linting

6a1a379

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Writing proper README

5a537ce

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

AuHau mentioned this pull request Apr 4, 2019

Create an utility for IPFS Repo migrations ipfs/js-ipfs#1115

Closed

AuHau added 5 commits April 8, 2019 15:23

Revert tests

5e6c4ba

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Migrate tests

b68f252

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

More tests

7e5992a

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Tests for lock.js and version.js

6df2da1

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Description of tests

1a15eea

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

AuHau changed the title ~~[WIP] Initial work~~ Initial work Apr 27, 2019

Update interface-datastore dependency

e7eb523

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

AuHau added 8 commits April 28, 2019 10:29

Fixing package-lock with correct interface-datastore version

348c8c4

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Update wording of CLI reporter

fc7166a

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Correct doctype for optional parameters

1446453

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Exposing repo.getVersion

966080f

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Improving verification of reversibility

b64412b

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Adding check if repo is initialized

c9d4724

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Renaming logging key

526d191

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

Adding parameter ignoreLock

11111b7

License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com>

dirkmc suggested changes Jul 23, 2019

View reviewed changes

AuHau and others added 6 commits July 24, 2019 07:34

Grammar fixes

405d259

Co-Authored-By: dirkmc <dirkmdev@gmail.com>

Messages tweak

4ab9c4d

Co-Authored-By: dirkmc <dirkmdev@gmail.com>

Default paramaters

77ca00e

Co-Authored-By: dirkmc <dirkmdev@gmail.com>

Tweaks

f6787a7

Merge branch 'dev' of github.com:AuHau/js-ipfs-repo-migrations into dev

94cdc7a

fix: update functions signature in templates

4fc8244

AuHau force-pushed the dev branch from f4fab25 to 4fc8244 Compare July 24, 2019 07:00

AuHau added 2 commits July 24, 2019 09:07

doc: CI/codecov/dependencies badges

188268a

style: lint fix

1a6d6c6

dirkmc suggested changes Jul 24, 2019

View reviewed changes

AuHau and others added 3 commits July 25, 2019 09:45

Message wordings

3e4415d

Co-Authored-By: dirkmc <dirkmdev@gmail.com>

fix: tweaks

a0ce795

style: package json lint fix

6996b6f

fix: update add command

edc7530

AuHau force-pushed the dev branch from 20d7e77 to edc7530 Compare July 29, 2019 15:22

AuHau added 3 commits September 16, 2019 15:25

feat: utils for properly getting Datastore and its options

724d02b

feat: when error occures save last succesfull migration version

c93c732

style: lint

813f2aa

AuHau mentioned this pull request Oct 9, 2019

Migration of Blockstore to use multihash instead of CID as key ipfs/js-ipfs#2522

Closed

9 tasks

daviddias suggested changes Oct 10, 2019

View reviewed changes

README.md Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

fix: apply review suggestions

a3d1712

Co-Authored-By: David Dias <daviddias.p@gmail.com>

AuHau force-pushed the dev branch from c3e183e to a3d1712 Compare October 10, 2019 15:36

AuHau merged commit aae9aec into master Oct 17, 2019

AuHau deleted the dev branch October 17, 2019 15:49

alanshaw reviewed Oct 18, 2019

View reviewed changes

AuHau mentioned this pull request Oct 24, 2019

feat: release metadata ipfs/aegir#457

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial work #1

Initial work #1

AuHau commented Apr 4, 2019 •

edited

Loading

AuHau commented Apr 28, 2019 •

edited

Loading

AuHau commented Jul 23, 2019

dirkmc Jul 24, 2019

dirkmc commented Jul 26, 2019

daviddias left a comment

daviddias Oct 10, 2019

AuHau Oct 10, 2019

daviddias Oct 10, 2019

AuHau Oct 10, 2019

daviddias left a comment

Initial work #1

Initial work #1

Conversation

AuHau commented Apr 4, 2019 • edited Loading

Scope-in

Scope-out

Blocked by

Problems

AuHau commented Apr 28, 2019 • edited Loading

Blocked by release of async/await versions of

AuHau commented Jul 23, 2019

dirkmc Jul 24, 2019

Choose a reason for hiding this comment

dirkmc commented Jul 26, 2019

daviddias left a comment

Choose a reason for hiding this comment

daviddias Oct 10, 2019

Choose a reason for hiding this comment

AuHau Oct 10, 2019

Choose a reason for hiding this comment

daviddias Oct 10, 2019

Choose a reason for hiding this comment

AuHau Oct 10, 2019

Choose a reason for hiding this comment

daviddias left a comment

Choose a reason for hiding this comment

AuHau commented Apr 4, 2019 •

edited

Loading

AuHau commented Apr 28, 2019 •

edited

Loading