Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ValueError: cannot fit 'int' into an offset-sized integer (python3) - OverflowError: integer too large (pypy3) #88

Closed
schulmenueplaner-de opened this issue Apr 14, 2021 · 7 comments
Closed

ValueError: cannot fit 'int' into an offset-sized integer (python3) - OverflowError: integer too large (pypy3) #88

schulmenueplaner-de opened this issue Apr 14, 2021 · 7 comments

Comments

@schulmenueplaner-de
Copy link

Hi,

I have a raw 750GB image of a drive which used to have one ntfs partition of that size,
but now its showing only a 449gb ntf partition and windows can not mount the partition ("unformatted").

First run of RecuperaBit works fine, then it throws following error:
(64bit debian 10 (buster) pypy3 / python3)

pypy3:

...
ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
INFO:root:Parsing INDX records
INFO:root:Reading boot sectors
INFO:root:Repaired MFT entry #0 - $MFT in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #1 - $MFTMirr in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #2 - $LogFile in partition at offset 2048 from backup
DEBUG:root:Dropping bogus NTFS partition with MFT position 913644690 generated by MFT mirror of partition at offset 913639528
DEBUG:root:Dropping bogus NTFS partition with MFT position 913508696 generated by MFT mirror of partition at offset 913508680
DEBUG:root:Dropping bogus NTFS partition with MFT position 915820154 generated by MFT mirror of partition at offset 915814992
DEBUG:root:Dropping bogus NTFS partition with MFT position 915826346 generated by MFT mirror of partition at offset 915821184
DEBUG:root:Dropping bogus NTFS partition with MFT position 915832538 generated by MFT mirror of partition at offset 915827376
DEBUG:root:Dropping bogus NTFS partition with MFT position 915868434 generated by MFT mirror of partition at offset 915863272
DEBUG:root:Dropping bogus NTFS partition with MFT position 914665816 generated by MFT mirror of partition at offset 914665800
DEBUG:root:Dropping bogus NTFS partition with MFT position 977548198 generated by MFT mirror of partition at offset 977543036
DEBUG:root:Dropping bogus NTFS partition with MFT position 1357141790 generated by MFT mirror of partition at offset 1357136628
INFO:root:Finding partition geometry
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
Traceback (most recent call last):
  File "main.py", line 374, in <module>
    main()
  File "main.py", line 357, in main
    parts.update(scanner.get_partitions())
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 837, in get_partitions
    self.finalize_reconstruction(part)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 673, in finalize_reconstruction
    self.add_from_attribute_list(parsed, part, node.offset)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 628, in add_from_attribute_list
    _integrate_attribute_list(parsed, part, image)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 219, in _integrate_attribute_list
    dump = sectors(image, real_pos, length, 1)
  File "/root/RecuperaBit/recuperabit/utils.py", line 44, in sectors
    image.seek(offset * bsize)
OverflowError: integer too large

python3:

...
ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
INFO:root:Parsing INDX records
INFO:root:Reading boot sectors
DEBUG:root:Dropping bogus NTFS partition with MFT position 914665816 generated by MFT mirror of partition at offset 914665800
INFO:root:Repaired MFT entry #0 - $MFT in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #1 - $MFTMirr in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #2 - $LogFile in partition at offset 2048 from backup
DEBUG:root:Dropping bogus NTFS partition with MFT position 977548198 generated by MFT mirror of partition at offset 977543036
DEBUG:root:Dropping bogus NTFS partition with MFT position 1357141790 generated by MFT mirror of partition at offset 1357136628
DEBUG:root:Dropping bogus NTFS partition with MFT position 913644690 generated by MFT mirror of partition at offset 913639528
DEBUG:root:Dropping bogus NTFS partition with MFT position 915820154 generated by MFT mirror of partition at offset 915814992
DEBUG:root:Dropping bogus NTFS partition with MFT position 915826346 generated by MFT mirror of partition at offset 915821184
DEBUG:root:Dropping bogus NTFS partition with MFT position 915832538 generated by MFT mirror of partition at offset 915827376
DEBUG:root:Dropping bogus NTFS partition with MFT position 915868434 generated by MFT mirror of partition at offset 915863272
DEBUG:root:Dropping bogus NTFS partition with MFT position 913508696 generated by MFT mirror of partition at offset 913508680
INFO:root:Finding partition geometry
INFO:root:Finalizing MFT reconstruction of partition at offset 914665800
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
Traceback (most recent call last):
  File "main.py", line 374, in <module>
    main()
  File "main.py", line 357, in main
    parts.update(scanner.get_partitions())
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 837, in get_partitions
    self.finalize_reconstruction(part)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 673, in finalize_reconstruction
    self.add_from_attribute_list(parsed, part, node.offset)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 628, in add_from_attribute_list
    _integrate_attribute_list(parsed, part, image)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 219, in _integrate_attribute_list
    dump = sectors(image, real_pos, length, 1)
  File "/root/RecuperaBit/recuperabit/utils.py", line 44, in sectors
    image.seek(offset * bsize)
ValueError: cannot fit 'int' into an offset-sized integer

I can provide the image and logfile in a VM if needed.

Regards

Ralph
@Lazza Lazza closed this as completed in 0d29f62 Apr 18, 2021
@Lazza
Copy link
Owner

Lazza commented Apr 18, 2021

Could you please check the latest commit and try again with both?

Thank you.

@schulmenueplaner-de
Copy link
Author

schulmenueplaner-de commented Apr 20, 2021 •
edited
Loading

Hi Lazza, that worked, thank you !

I got into the CLI (pypy3), but unfortunately it only restores about 20 Gb from partition 29
and not much from partition 33 (which should be identical with 29)

> recoverable
Partition #29 -> Partition (NTFS, 698.63 GB, 10774 files, Recoverable, Offset: 2048, Offset (b): 1048576, Sec/Clus: 8, MFT offset: 6293504, MFT mirror offset: 2064)
Partition #33 -> Partition (NTFS, ??? b, 48962 files, Recoverable, Offset: 2048, Offset (b): 1048576, Sec/Clus: 8, MFT offset: 706320448, MFT mirror offset: None)
Partition #38 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 913639528, Offset (b): 467783438336, Sec/Clus: 1, MFT offset: 913641586, MFT mirror offset: 913644690)
Partition #41 -> Partition (NTFS, 450.00 MB, 37 files, Recoverable, Offset: 913508680, Offset (b): 467716444160, Sec/Clus: 8, MFT offset: 913815880, MFT mirror offset: 913508696)
Partition #42 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 915814992, Offset (b): 468897275904, Sec/Clus: 1, MFT offset: 915817050, MFT mirror offset: 915820154)
Partition #43 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 915821184, Offset (b): 468900446208, Sec/Clus: 1, MFT offset: 915823242, MFT mirror offset: 915826346)
Partition #44 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 915827376, Offset (b): 468903616512, Sec/Clus: 1, MFT offset: 915829434, MFT mirror offset: 915832538)
Partition #45 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 915863272, Offset (b): 468921995264, Sec/Clus: 1, MFT offset: 915865330, MFT mirror offset: 915868434)
Partition #118 -> Partition (NTFS, 19.45 GB, 128499 files, Recoverable, Offset: 914665800, Offset (b): 468308889600, Sec/Clus: 8, MFT offset: 920957256, MFT mirror offset: 914665816)
Partition #119 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 977543036, Offset (b): 500502034432, Sec/Clus: 1, MFT offset: 977545094, MFT mirror offset: 977548198)
Partition #120 -> Partition (NTFS, ??? b, 4555 files, Recoverable, Offset: 2048, Offset (b): 1048576, Sec/Clus: 8, MFT offset: 1339275104, MFT mirror offset: None)
Partition #121 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 1357136628, Offset (b): 694853953536, Sec/Clus: 1, MFT offset: 1357138686, MFT mirror offset: 1357141790)
Partition #122 -> Partition (NTFS, ??? b, 513 files, Recoverable, Offset: 2048, Offset (b): 1048576, Sec/Clus: 8, MFT offset: 1447197184, MFT mirror offset: None)
Partition #123 -> Partition (NTFS, ??? b, 4608 files, Recoverable, Offset: 2048, Offset (b): 1048576, Sec/Clus: 8, MFT offset: 1456092400, MFT mirror offset: None)

I do have a backup for most pictures but the pictures on this drive were sorted / renamed / adjusted ...

Thanks again for RecuperaBit and you quick reply.

@Lazza
Copy link
Owner

Lazza commented Apr 22, 2021

I would first try to export a CSV file of the interesting partitions (e.g. 29, 33 and 118 that do seem they have a lot of files). Then study them and verify their contents.

When you say it restores "not much" what do you mean? Did you simply try to restore from the Root node or did you check the Lost Files as well?

@Lazza
Copy link
Owner

Lazza commented Apr 22, 2021

PS:

I got into the CLI (pypy3)

Just to be sure, is the fix also working on Python 3 rather than Pypy3? Thank you!

@schulmenueplaner-de
Copy link
Author

Unfortunately your fix does not work with python3

ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
INFO:root:Parsing INDX records
INFO:root:Reading boot sectors
DEBUG:root:Dropping bogus NTFS partition with MFT position 914665816 generated by MFT mirror of partition at offset 914665800
INFO:root:Repaired MFT entry #0 - $MFT in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #1 - $MFTMirr in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #2 - $LogFile in partition at offset 2048 from backup
DEBUG:root:Dropping bogus NTFS partition with MFT position 977548198 generated by MFT mirror of partition at offset 977543036
DEBUG:root:Dropping bogus NTFS partition with MFT position 1357141790 generated by MFT mirror of partition at offset 1357136628
DEBUG:root:Dropping bogus NTFS partition with MFT position 913644690 generated by MFT mirror of partition at offset 913639528
DEBUG:root:Dropping bogus NTFS partition with MFT position 915820154 generated by MFT mirror of partition at offset 915814992
DEBUG:root:Dropping bogus NTFS partition with MFT position 915826346 generated by MFT mirror of partition at offset 915821184
DEBUG:root:Dropping bogus NTFS partition with MFT position 915832538 generated by MFT mirror of partition at offset 915827376
DEBUG:root:Dropping bogus NTFS partition with MFT position 915868434 generated by MFT mirror of partition at offset 915863272
DEBUG:root:Dropping bogus NTFS partition with MFT position 913508696 generated by MFT mirror of partition at offset 913508680
INFO:root:Finding partition geometry
INFO:root:Finalizing MFT reconstruction of partition at offset 914665800
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
Traceback (most recent call last):
 File "main.py", line 374, in <module>
   main()
 File "main.py", line 357, in main
   parts.update(scanner.get_partitions())
 File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 837, in get_partitions
   self.finalize_reconstruction(part)
 File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 673, in finalize_reconstruction
   self.add_from_attribute_list(parsed, part, node.offset)
 File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 628, in add_from_attribute_list
   _integrate_attribute_list(parsed, part, image)
 File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 219, in _integrate_attribute_list
   dump = sectors(image, real_pos, length, 1)
 File "/root/RecuperaBit/recuperabit/utils.py", line 44, in sectors
   image.seek(offset * bsize)
ValueError: cannot fit 'int' into an offset-sized integer

in comparison to pypy3

ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
INFO:root:Parsing INDX records
INFO:root:Reading boot sectors
INFO:root:Repaired MFT entry #0 - $MFT in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #1 - $MFTMirr in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #2 - $LogFile in partition at offset 2048 from backup
DEBUG:root:Dropping bogus NTFS partition with MFT position 913644690 generated by MFT mirror of partition at offset 913639528
DEBUG:root:Dropping bogus NTFS partition with MFT position 913508696 generated by MFT mirror of partition at offset 913508680
DEBUG:root:Dropping bogus NTFS partition with MFT position 915820154 generated by MFT mirror of partition at offset 915814992
DEBUG:root:Dropping bogus NTFS partition with MFT position 915826346 generated by MFT mirror of partition at offset 915821184
DEBUG:root:Dropping bogus NTFS partition with MFT position 915832538 generated by MFT mirror of partition at offset 915827376
DEBUG:root:Dropping bogus NTFS partition with MFT position 915868434 generated by MFT mirror of partition at offset 915863272
DEBUG:root:Dropping bogus NTFS partition with MFT position 914665816 generated by MFT mirror of partition at offset 914665800
DEBUG:root:Dropping bogus NTFS partition with MFT position 977548198 generated by MFT mirror of partition at offset 977543036
DEBUG:root:Dropping bogus NTFS partition with MFT position 1357141790 generated by MFT mirror of partition at offset 1357136628
INFO:root:Finding partition geometry
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
DEBUG:root:Found MATCH in positions {44512280} with weight 9 (100.0%)
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 913639528
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 913508680
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 915814992
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 915821184
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 915827376
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 915863272
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 914665800
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 977543036
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
DEBUG:root:Found MATCH in positions {661972816} with weight 142 (97.93103448275862%)
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 1357136628
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
DEBUG:root:Found MATCH in positions {722298656} with weight 14 (100.0%)
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
DEBUG:root:Found MATCH in positions {630871856} with weight 197 (99.4949494949495%)
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:MFT for partition at offset 913639528 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 913639512 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 913639528, Offset (b): 467783438336, Sec/Clus: 1, MFT offset: 913641586, MFT mirror offset: 913644690) (fragmented MFT)
INFO:root:MFT for partition at offset 915814992 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 915814976 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 915814992, Offset (b): 468897275904, Sec/Clus: 1, MFT offset: 915817050, MFT mirror offset: 915820154) (fragmented MFT)
INFO:root:MFT for partition at offset 915821184 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 915821168 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 915821184, Offset (b): 468900446208, Sec/Clus: 1, MFT offset: 915823242, MFT mirror offset: 915826346) (fragmented MFT)
INFO:root:MFT for partition at offset 915827376 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 915827360 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 915827376, Offset (b): 468903616512, Sec/Clus: 1, MFT offset: 915829434, MFT mirror offset: 915832538) (fragmented MFT)
INFO:root:MFT for partition at offset 915863272 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 915863256 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 915863272, Offset (b): 468921995264, Sec/Clus: 1, MFT offset: 915865330, MFT mirror offset: 915868434) (fragmented MFT)
INFO:root:MFT for partition at offset 977543036 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 977543020 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 977543036, Offset (b): 500502034432, Sec/Clus: 1, MFT offset: 977545094, MFT mirror offset: 977548198) (fragmented MFT)
INFO:root:MFT for partition at offset 1357136628 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 1357136612 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 1357136628, Offset (b): 694853953536, Sec/Clus: 1, MFT offset: 1357138686, MFT mirror offset: 1357141790) (fragmented MFT)
INFO:root:124 partitions found.

Write command ("help" for details):
>

118 is an old Windows Install do not need to recover that ...
Thank you for the tip with the CSV file, I ll try that.

Let me know if I can assist with python3 / tests.

Lazza added a commit that referenced this issue Apr 23, 2021
@Lazza
Fix #88 for Python 3 (cPython) as well
e1a5d1d
@Lazza
Copy link
Owner

Lazza commented Apr 23, 2021

Thank you for your feedback! I have pushed another commit that hopefully will help with cPython as well.

If you could try that one as well, it would be great.

@schulmenueplaner-de
Copy link
Author

It also works with python3 (tested: version 3.7.3) now. Thank you !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Lazza @schulmenueplaner-de