CIDv1 link specifications do not work ( hardcoded mdag expectation somewhere )

[mib-kd743naq]

Version information:

go-ipfs version: 0.4.5-dev-bf9927f
Repo version: 4
System version: amd64/linux
Golang version: go1.7.1

Type:

Bug

Priority:

P4

Description:

Full description at https://ipfs.io/ipfs/QmXUmGtjEKnceju74LDStC24DDmJWxnxdH7aQPLYk9zJe5

=== Setup
String "abcdef\ngh" broken into 3 chunks on the (and including) newline

=== Problem
~$ ipfs cat zdj7WmC19DKCa9i2viQtsyqymhUFUgJzU8mZN4ytku3L75XAD | hexdump -C
Error: merkledag: not found
...

[whyrusleeping]

Do you mind pasting the commands you used to generate the objects in question?

[mib-kd743naq]

@whyrusleeping There is no command I can currently share.

As I said previously these are all experiments of mine trying to figure out how I can use ipfs for my own needs, entirely outside of go-ipfs, but remaining compatible with it (i.e. having http://ipfs.io/ipfs/<my hash> work).

I am essentially assembling all the bytes by hand, and then feeding them to http://127.0.0.1:5001/api/v0/block/put?format=raw or similar. The current "codebase" is a string of piped invocations of small converter utilities, with hand-added sizes etc, and most definitely not suitable for public consumption, sorry :/

[whyrusleeping]

okay, can you provide a set of commands like:

printf "foobar" | ipfs block put --format=raw

That recreate each object?

[mib-kd743naq]

Sure thing:

~$ for d in \
  Qmb6pVMgw4hMmsLU6Mz28ZV5uHzceGxR6pj7k4wKS2WLNZ \
  QmRnnQXyW5wNz6rvjw4fQQ8XWaLvyfkvJBztC3WGx1gmyg \
  QmaxVysa8dDZtk5g9AfdQouDVZdNZ3D2NFePZrME61X8GX
do \
  ipfs block get $d > $d.data
done

~$ for l in \
  zdj7WgpD1zodrFvjBSWMXL2y9dR7z7H1TJdBhrTLZcS4LGPH9 \
  zdj7WjDDokjEjue5N97KRJnNRvpdj5mUfNFsEGqpAmUEVxSXh \
  zdj7WmC19DKCa9i2viQtsyqymhUFUgJzU8mZN4ytku3L75XAD
do \
  ipfs block get $l > $l.link
done

~$ ls -l
total 24
-rw-r--r-- 1 ipfs ipfs    8 Nov 28 15:02 QmaxVysa8dDZtk5g9AfdQouDVZdNZ3D2NFePZrME61X8GX.data
-rw-r--r-- 1 ipfs ipfs   12 Nov 28 15:02 Qmb6pVMgw4hMmsLU6Mz28ZV5uHzceGxR6pj7k4wKS2WLNZ.data
-rw-r--r-- 1 ipfs ipfs    7 Nov 28 15:02 QmRnnQXyW5wNz6rvjw4fQQ8XWaLvyfkvJBztC3WGx1gmyg.data
-rw-r--r-- 1 ipfs ipfs   49 Nov 28 15:02 zdj7WgpD1zodrFvjBSWMXL2y9dR7z7H1TJdBhrTLZcS4LGPH9.link
-rw-r--r-- 1 ipfs ipfs   92 Nov 28 15:02 zdj7WjDDokjEjue5N97KRJnNRvpdj5mUfNFsEGqpAmUEVxSXh.link
-rw-r--r-- 1 ipfs ipfs   92 Nov 28 15:02 zdj7WmC19DKCa9i2viQtsyqymhUFUgJzU8mZN4ytku3L75XAD.link

~$ ipfs repo gc
removed zdj7WgpD1zodrFvjBSWMXL2y9dR7z7H1TJdBhrTLZcS4LGPH9
removed zdj7WjDDokjEjue5N97KRJnNRvpdj5mUfNFsEGqpAmUEVxSXh
removed zdj7WmC19DKCa9i2viQtsyqymhUFUgJzU8mZN4ytku3L75XAD
removed QmRnnQXyW5wNz6rvjw4fQQ8XWaLvyfkvJBztC3WGx1gmyg
removed Qmb6pVMgw4hMmsLU6Mz28ZV5uHzceGxR6pj7k4wKS2WLNZ
removed QmaxVysa8dDZtk5g9AfdQouDVZdNZ3D2NFePZrME61X8GX

Killing daemon, restarting with ipfs daemon --offline

~$ for d in \
  Qmb6pVMgw4hMmsLU6Mz28ZV5uHzceGxR6pj7k4wKS2WLNZ \
  QmRnnQXyW5wNz6rvjw4fQQ8XWaLvyfkvJBztC3WGx1gmyg \
  QmaxVysa8dDZtk5g9AfdQouDVZdNZ3D2NFePZrME61X8GX
do \
  ipfs block put < $d.data
done

Qmb6pVMgw4hMmsLU6Mz28ZV5uHzceGxR6pj7k4wKS2WLNZ
QmRnnQXyW5wNz6rvjw4fQQ8XWaLvyfkvJBztC3WGx1gmyg
QmaxVysa8dDZtk5g9AfdQouDVZdNZ3D2NFePZrME61X8GX

~$ for l in \
  zdj7WgpD1zodrFvjBSWMXL2y9dR7z7H1TJdBhrTLZcS4LGPH9 \
  zdj7WjDDokjEjue5N97KRJnNRvpdj5mUfNFsEGqpAmUEVxSXh \
  zdj7WmC19DKCa9i2viQtsyqymhUFUgJzU8mZN4ytku3L75XAD
do \
  ipfs block put -f protobuf < $l.link
done

zdj7WgpD1zodrFvjBSWMXL2y9dR7z7H1TJdBhrTLZcS4LGPH9
zdj7WjDDokjEjue5N97KRJnNRvpdj5mUfNFsEGqpAmUEVxSXh
zdj7WmC19DKCa9i2viQtsyqymhUFUgJzU8mZN4ytku3L75XAD

~$ for l in \
  zdj7WgpD1zodrFvjBSWMXL2y9dR7z7H1TJdBhrTLZcS4LGPH9 \
  zdj7WjDDokjEjue5N97KRJnNRvpdj5mUfNFsEGqpAmUEVxSXh \
  zdj7WmC19DKCa9i2viQtsyqymhUFUgJzU8mZN4ytku3L75XAD
do \
  echo -e "====\nCat-ing $l"
  ipfs cat $l | hexdump -C
done

====
Cat-ing zdj7WgpD1zodrFvjBSWMXL2y9dR7z7H1TJdBhrTLZcS4LGPH9
00000000  67 68                                             |gh|
00000002
====
Cat-ing zdj7WjDDokjEjue5N97KRJnNRvpdj5mUfNFsEGqpAmUEVxSXh
Error: merkledag: not found
====
Cat-ing zdj7WmC19DKCa9i2viQtsyqymhUFUgJzU8mZN4ytku3L75XAD
Error: merkledag: not found

~$ for b in \
  Qmb6pVMgw4hMmsLU6Mz28ZV5uHzceGxR6pj7k4wKS2WLNZ \
  QmRnnQXyW5wNz6rvjw4fQQ8XWaLvyfkvJBztC3WGx1gmyg \
  QmaxVysa8dDZtk5g9AfdQouDVZdNZ3D2NFePZrME61X8GX \
  zdj7WgpD1zodrFvjBSWMXL2y9dR7z7H1TJdBhrTLZcS4LGPH9 \
  zdj7WjDDokjEjue5N97KRJnNRvpdj5mUfNFsEGqpAmUEVxSXh \
  zdj7WmC19DKCa9i2viQtsyqymhUFUgJzU8mZN4ytku3L75XAD
do \
  echo -e "====\nDump of $b"
  ipfs block get $b | xxd -p -c 256
done

====
Dump of Qmb6pVMgw4hMmsLU6Mz28ZV5uHzceGxR6pj7k4wKS2WLNZ
0a0a08001206616263646566
====
Dump of QmRnnQXyW5wNz6rvjw4fQQ8XWaLvyfkvJBztC3WGx1gmyg
0a05080012010a
====
Dump of QmaxVysa8dDZtk5g9AfdQouDVZdNZ3D2NFePZrME61X8GX
0a06080012026768
====
Dump of zdj7WgpD1zodrFvjBSWMXL2y9dR7z7H1TJdBhrTLZcS4LGPH9
0a070802180222010212260a221220bb7b48844cfe1badbb5f09bb2188b5378b33ea0677ea27ffc6552399a19f30901802
====
Dump of zdj7WjDDokjEjue5N97KRJnNRvpdj5mUfNFsEGqpAmUEVxSXh
0a08080218032202010212260a2212203344dd6906640549056bacdaa1bd330528965652ff1e4c34baba2ae9605592bf180112280a2401701220a9279e1c0da57c0ec186af84479856f65f6d98bc05c2a862143823cf5a7c0fe81802
====
Dump of zdj7WmC19DKCa9i2viQtsyqymhUFUgJzU8mZN4ytku3L75XAD
0a08080218092202060312260a221220bd9cdb7208c3bf5848324922ce6f39cc7f1f239236b85b0fb50358d83ed63f46180612280a2401701220ccc43dfaf89b775d3606adea340eb203030708f26930f4a83d12a736d44242b01803

[mib-kd743naq]

@whyrusleeping in #3431 you said:

The protobuf nodes always default to generating a cidV0.

Surely this issue is the same then? Even though the block put -f protobuf returns a cidv1, internally it gets stored as a v0 somehow...?

[whyrusleeping]

@mib-kd743naq With block put, we never create a protobuf node, so at least here, we should be putting the right CID to disk, this issue is subtly different (but likely a very similar problem in any case)

[whyrusleeping]

@mib-kd743naq can you verify that this was fixed in latest master?

[mib-kd743naq]

Works as of 0c413de