RPC error: transport error: HTTP response too short: length 0, needed 12

[casey]

While indexing, I'm getting errors like " failed to fetch block 705096, retrying in 2s: JSON-RPC error: transport error: HTTP response too short: length 0, needed 12." exactly every 5000 blocks. Seems to continue though. Is this a known issue?

[dude513]

Exactly every 5000 blocks, me as well.
Started somewhere in a 200000 range, I can't recall.

[indexing blocks] ......... 333974/774477

ERROR ord::index::updater] failed to fetch block 334007, retrying in 2s: JSON-RPC error: transport error: HTTP response too short: length 0, needed 12.

[hashbender]

It's due to the commit that happens in updater.rs

Not sure exactly how rust works, but my guess would be the sync channel is blocked waiting for the loop to read off of it and the client ends up getting hung when attempting to write new block data

[dude513]

It's indexing slower now and with more frequent errors. @casey

60000 blocks in 12 hours.

Update: I killed the process and started again because it became deadly slow.
After ord index again it doesn't help at all, it is 1 block per 1 second now. 😴
I am using the latest code (ord 0.4.2) on Ubuntu 22.04.

[hashbender]

It's very write-heavy - using an NVME will speed things up a lot

[OrpingtonClose]

It's very write-heavy - using an NVME will speed things up a lot

used a ramdisk to do a ord index but it is similarily slow

[dude513]

@hashbender it has nothing to do with disks, I am on SSD.

I am confused why it is not the highest priority now because ord is not usable.

I tried to roll back to different versions and with 0.4.0 I got almost all blocks within a few minutes, however below this version it throws some weirdness when trying to ord index.

We know that there is a bug somewhere between 0.4.0 and 0.4.2.

Even though I got almost all blocks (10000 left) now it is stuck with any version.
It throws an error every single block.

[indexing blocks] █████░░░ 764050/774732[2023-02-02T15:45:07Z ERROR ord::index::updater] failed to fetch block 764083, retrying in 2s: JSON-RPC error: transport error: HTTP response too short: length 0, needed 12.
[indexing blocks] █████░░░ 764051/774732[2023-02-02T15:47:01Z ERROR ord::index::updater] failed to fetch block 764084, retrying in 2s: JSON-RPC error: transport error: HTTP response too short: length 0, needed 12.
[indexing blocks] █████░░░ 764052/774732[2023-02-02T15:48:46Z ERROR ord::index::updater] failed to fetch block 764085, retrying in 2s: JSON-RPC error: transport error: HTTP response too short: length 0, needed 12.

[dannydeezy]

is it safe for me to rollback to 0.4.0, index the chain, then upgrade back to 0.4.2?

[dude513]

is it safe for me to rollback to 0.4.0, index the chain, then upgrade back to 0.4.2?

Yes, it is safe to try it out.
If something went wrong you can remove .local/share/ord folder to start indexing blocks from the beginning.

[dannydeezy]

rolling back didn't work, my ord is stuck 😢

[dude513]

@hashbender it has nothing to do with disks, I am on SSD.

I am confused why it is not the highest priority now because ord is not usable.

I tried to roll back to different versions and with 0.4.0 I got almost all blocks within a few minutes, however below this version it throws some weirdness when trying to ord index.

We know that there is a bug somewhere between 0.4.0 and 0.4.2.

Even though I got almost all blocks (10000 left) now it is stuck with any version. It throws an error every single block.

[indexing blocks] █████░░░ 764050/774732[2023-02-02T15:45:07Z ERROR ord::index::updater] failed to fetch block 764083, retrying in 2s: JSON-RPC error: transport error: HTTP response too short: length 0, needed 12.
[indexing blocks] █████░░░ 764051/774732[2023-02-02T15:47:01Z ERROR ord::index::updater] failed to fetch block 764084, retrying in 2s: JSON-RPC error: transport error: HTTP response too short: length 0, needed 12.
[indexing blocks] █████░░░ 764052/774732[2023-02-02T15:48:46Z ERROR ord::index::updater] failed to fetch block 764085, retrying in 2s: JSON-RPC error: transport error: HTTP response too short: length 0, needed 12.

This time, I was able to index all blocks with 0.3.0, but could not create any inscriptions because it says the version is too old and I need to reindex with a newer version.

[dannydeezy]

i feel like this should be the highest priority issue, seems like most people are unable to sync

[dude513]

Any success? @dannydeezy @OrpingtonClose
I tried all possible versions dozens of time. Latest 0.4.2 might index up to 400000 block then it is 1 block per second with crashes. 0.4.0 and 0.4.1 are more stable and can index about 99% of the blocks, then constant crashes and 5 blocks per hour. I tried to sync 98% with above versions, stop ord and continue with 0.4.2, but it is dead slow and crashes at some point anyways, starting again it goes back where it initially started.

[jbdatko]

I was hitting this bug on 0.4.2. Mine was stuck several times, finally crashed at 772406. I essentially kept restarting with ord wallet inscriptions and it finally (after 2 days of manually restarting), was able to sync.

Perhaps a workaround for those experiencing this now is to run the indexing in a bash loop, on tmux or similar until it completes.

[dude513]

I was hitting this bug on 0.4.2. Mine was stuck several times, finally crashed at 772406. I essentially kept restarting with ord wallet inscriptions and it finally (after 2 days of manually restarting), was able to sync.

Perhaps a workaround for those experiencing this now is to run the indexing in a bash loop, on tmux or similar until it completes.

You were lucky, I get 5 blocks per hour after block ~764000.

[dude513]

@hashbender it has nothing to do with disks, I am on SSD.
I am confused why it is not the highest priority now because ord is not usable.
I tried to roll back to different versions and with 0.4.0 I got almost all blocks within a few minutes, however below this version it throws some weirdness when trying to ord index.
We know that there is a bug somewhere between 0.4.0 and 0.4.2.
Even though I got almost all blocks (10000 left) now it is stuck with any version. It throws an error every single block.

[indexing blocks] █████░░░ 764050/774732[2023-02-02T15:45:07Z ERROR ord::index::updater] failed to fetch block 764083, retrying in 2s: JSON-RPC error: transport error: HTTP response too short: length 0, needed 12.
[indexing blocks] █████░░░ 764051/774732[2023-02-02T15:47:01Z ERROR ord::index::updater] failed to fetch block 764084, retrying in 2s: JSON-RPC error: transport error: HTTP response too short: length 0, needed 12.
[indexing blocks] █████░░░ 764052/774732[2023-02-02T15:48:46Z ERROR ord::index::updater] failed to fetch block 764085, retrying in 2s: JSON-RPC error: transport error: HTTP response too short: length 0, needed 12.

This time, I was able to index all blocks with 0.3.0, but could not create any inscriptions because it says the version is too old and I need to reindex with a newer version.

More specifically this. @jbdatko

[brandonblack]

Seems like as the sat ranges get more fragmented later in the sync the volume of writes per 5k block flush gets really high. ~10 minutes to index, ~5 minutes to flush on my machine at block height 635000-640000.

Quick math tells me that it would take 45PiB or so to store a pointer from each individual sat to its TXO, so over time fragmenting of the sat space will become a real problem for ordinal sat tracking.

[brandonblack]

Watching iostat during the flush shows a pattern of ~14 seconds of reading ~20MB/s, then ~2.5 seconds of writing ~300MB/s on my NVME SSD.

[brandonblack]

Perhaps a variant of ord that only indexes sats of interest (inscribed or otherwise) and does a path-based lookup for any other sat (eg. when picking a sat to inscribe) would serve most users better than the current method of indexing all sats by range.

[diwakergupta]

Is there a workaround for this issue? I saw some mentions of using older versions but wasn't clear if those work reliably.

I'm on a beefy Mac M1, 32GB RAM, 1TB SSD. Syncing bitcoin node from genesis took < 8 hours. Indexing in ord has been running for > 24 hrs and still ongoing. Effectively unusable at this point.

[raphjaph]

Seems like as the sat ranges get more fragmented later in the sync the volume of writes per 5k block flush gets really high. ~10 minutes to index, ~5 minutes to flush on my machine at block height 635000-640000.

Quick math tells me that it would take 45PiB or so to store a pointer from each individual sat to its TXO, so over time fragmenting of the sat space will become a real problem for ordinal sat tracking.

Yes, fragmenting gets worse over time. At the moment you will need 64gb or more of RAM to have a chance at syncing the ord index with --index-sats.

[raphjaph]

Perhaps a variant of ord that only indexes sats of interest (inscribed or otherwise) and does a path-based lookup for any other sat (eg. when picking a sat to inscribe) would serve most users better than the current method of indexing all sats by range.

We we're thinking of doing exactly this. Building the database on-demand, but this would mean a complete redesign so we put that on hold. Maybe it's time to resurface that though.

[diwakergupta]

Is there a workaround for this issue? I saw some mentions of using older versions but wasn't clear if those work reliably.

I'm on a beefy Mac M1, 32GB RAM, 1TB SSD. Syncing bitcoin node from genesis took < 8 hours. Indexing in ord has been running for > 24 hrs and still ongoing. Effectively unusable at this point.

FYI @dude513 recommendation worked for me: 0.4.0 is significantly faster at indexing (two orders of magnitude) than 0.4.2 - so this very much seems like a regression in the latest release.

[alex27riva]

I downgraded to version 0.4.0 but the indexing of the last 10000 blocks is very slow, is this normal?
My hardware: Ryzen 5 2500U, 8GB RAM, Samsung 970 EVO Plus 2TB (NVMe)

[dude513]

I downgraded to version 0.4.0 but the indexing of the last 10000 blocks is very slow, is this normal? My hardware: Ryzen 5 2500U, 8GB RAM, Samsung 970 EVO Plus 2TB (NVMe)

You can't use ord currently. Consider downloading someone's index file such as here: https://github.com/casey/ord/discussions/1494#discussioncomment-4878035.

[MattSenter]

FWIW I too got by this by downgrading to 0.4.0. Sped right through indexing on 0.4.0 but was stalling on 0.4.2.

[MattSenter]

I spoke too soon.. stuck at last 10k blocks

[stephancill]

Managed to index after switching to 0.4.0. Last few thousand blocks took the bulk of the time to index.

[andrewtoth]

https://github.com/casey/ord/pull/1516 will fix these errors since blocks will no longer be requested through RPC.

[Prestonsr]

Tried the above and still got the error several times. What I did to resolve the issue just in time to miss 100k was I typed bitcoin-cli setnetworkactive false then watched disk usage on the activity monitor to hit about 32 GB. The index will look frozen until then. (about 20 minutes) Then type bitcoin-cli setnetworkactive true. Then ord wallet balance

[pavillag]

I get this error when indexing finishes:

Any suggestions? I don't see a fix here yet

[donnlee]

getting this issue on ord 0.5.0
progress bar starts at 775000/777096 every time i retry.

[andrewtoth]

@donnlee I think this fixed the issue https://github.com/casey/ord/pull/1637

[donnlee]

@andrewtoth , ok I will try branch andrewtoth:handle-block-count-error Thank you

[vicariousdrama]

I'm also having this issue and testing https://github.com/casey/ord/pull/1637 now

[donnlee]

@andrewtoth , I confirm that casey@50f2ad7 on ord 0.5.0 has fixed the issue of blocks re-indexing at 775000
This is awesome. I was able to finish the last 1% of indexing 🎉

[andrewtoth]

@donnlee great! Can you comment that in #1637 please?

[coconutpalms]

I got here from google because I am seeing the same error message when I run ord wallet create with an rpc endpoint provided by getblock.io. It's not fixed by downgrading to 0.4.0 so it's probably a different error.

[andrewtoth]

@coconutpalms not sure it's a good idea to use a third party bitcoind. ord uses the wallet so the private keys will be held by the third party.

[coconutpalms]

I'm concerned about downloading the blockchain because of US restrictions on illegal data, is there a way to run a node and inscribe ordinals without doing that?

[andrewtoth]

@coconutpalms if that's your concern I regret to inform you that ord downloads (but does not save to disk) the entire blockchain to create the index.

[raphjaph]

I'm going to close this, since this seems stale. Please reopen if not.